API: Make describe changes backwards compatible
State: closed

Adds the new behavior as a feature flag / deprecation.

Closes https://github.com/pandas-dev/pandas/issues/33903

(Do we have a list of issues for deprecations introduced in 1.x?)

Labels:
name: API Design
name: Deprecate
description: Functionality to remove in pandas

Comments:
cc @david-cortes.
> (Do we have a list of issues for deprecations introduced in 1.x?)

https://github.com/pandas-dev/pandas/issues/30228
BTW, some other remarks on the new behaviour: this should also be enabled in DataFrame.describe by default? 
Now (on master), a datetime is not included by default. Which made sense before, as it was regarded as a categorical and not numeric columns. But with `datetime_as_numeric`, it should be included by default?

Also, I would include "std" as well for datetimes, but with a NaN entry. That makes that the keys are always all the same for all numeric types, regardless of the exact dtype (and that also ensures the ordering is preserved when when doing describe on a dataframe with both numeric and datetime columns)
I was surprised to see that master didn't treat datetimes as numeric in `describe`. But I don't know how high-priority fixing that is (though a keyword calling `datetime_is_numeric` does bump the priority).
I think we should fix it, because if we add a keyword to enable the future behaviour, we should ensure we have the proper future behaviour we want (which is now not yet the case, IMO). Unless we disallow the keyword for dataframe .. 
(now, doesn't necessarily need to happen in this PR though)
Fixing that will get a bit messy, since it overlaps with the `include` and `exclude` keywords, and it's unclear how timedelta & period should behave (are they also numeric-like? Should the single keyword `datetime_is_numeric` control all of those? If so, should it be renamed?)
In the spirit of practicality over purity do we really need to do this? Outside of the Dask test do we expect end users would be relying on the old behavior?
> it's unclear how timedelta & period should behave (are they also numeric-like? Should the single keyword datetime_is_numeric control all of those? If so, should it be renamed?)

For timedelta that is clear, I think, since that also returns the numeric describe output, while period does not?


It seems timedelta is alraedy included by default, on released version:

```
In [7]: pd.DataFrame({'a': pd.timedelta_range("2012", periods=3), 'b': [1, 2, 3]}).describe()                                                                                                                      
Out[7]: 
                            a    b
count                       3  3.0
mean   1 days 00:00:00.000002  2.0
std           1 days 00:00:00  1.0
min    0 days 00:00:00.000002  1.0
25%    0 days 12:00:00.000002  1.5
50%    1 days 00:00:00.000002  2.0
75%    1 days 12:00:00.000002  2.5
max    2 days 00:00:00.000002  3.0
```

> In the spirit of practicality over purity do we really need to do this? Outside of the Dask test do we expect end users would be relying on the old behavior?

Hard to say. It's not hard to construct code that relies on the old behavior though.
My point is that this seems like a lot of churn for potentially negligible value add. I can't think of a pipeline where the old behavior is actually useful, so I don't think worth adding a keyword that we subsequently plan to deprecate for the sake of maintaining compat
There are a lot of things in pandas that I personally don't find particularly useful, but there are probably still a lot of people using those things. So I also find it hard to say whether it is important here. But it also doesn't seem difficult to actually do it with a deprecation.

But whether we go through a deprecation or not, we still need to agree on what we think the new / future behaviour should be. 
I would say that it should follow what we already do for timedelta, so include it by default for dataframe.

Split the tests and fixed the merge conflicts.

I'm happy with this as is since it restores the default 1.0 behavior. If we want additional changes then let's do them as followups.
> I'm happy with this as is since it restores the default 1.0 behavior. If we want additional changes then let's do them as followups.

So I suppose this means you didn't change the dataframe behaviour (which is certainly fine to leave for a follow-up). But should be then maybe raise a NotImplementedError when specifying `datetime_as_numeric=True` for DataFrame?
lgtm. @jorisvandenbossche comment here or in a followup ok
@jorisvandenbossche can you remind me what the dataframe issue is? That datetimes are not included in this output while timedeltas are?

```python
In [4]: pd.DataFrame({'a': pd.date_range("2012", periods=3), 'b': [1, 2, 3]}).describe()
   ...:
Out[4]:
         b
count  3.0
mean   2.0
std    1.0
min    1.0
25%    1.5
50%    2.0
75%    2.5
max    3.0

In [6]: pd.DataFrame({'a': pd.timedelta_range("2012", periods=3), 'b': [1, 2, 3]}).describe()
   ...:
Out[6]:
                               a    b
count                          3  3.0
mean   1 days 00:00:00.000002012  2.0
std              1 days 00:00:00  1.0
min    0 days 00:00:00.000002012  1.0
25%    0 days 12:00:00.000002012  1.5
50%    1 days 00:00:00.000002012  2.0
75%    1 days 12:00:00.000002012  2.5
max    2 days 00:00:00.000002012  3.0
```

Interestingly if you have datetime, numeric, and timedelta, then the timedelta is not included:

```python
In [5]: pd.DataFrame({'a': pd.date_range("2012", periods=3), 'b': [1, 2, 3], 'c': pd.period_range('2000', periods=3)}).describe()
   ...:
Out[5]:
         b
count  3.0
mean   2.0
std    1.0
min    1.0
25%    1.5
50%    2.0
75%    2.5
max    3.0
```

This all seems buggy, but I hope can be handled separately.
Yes, that needs to be cleaned up and can indeed be handled separately. 

But the one thing that might be relevant for this PR:

```
pd.DataFrame({'a': pd.date_range("2012", periods=3), 'b': [1, 2, 3]}).describe(datetime_is_numeric=True)
```

is not actually doing what the keyword is expected to be doing (I suppose, didn't try with this branch). 
And if we want to enable that in a follow-up PR, we should maybe not allow one to pass that now for the DataFrame case (because if we would then "fix" it in 1.2, it's kind of a breaking change, although nobody of course should rely on it). So we could raise a NotImplementedError that keyword is specified and `self` is a DataFrame.
@jorisvandenbossche gotcha. Added a test and implemented that.

```python
In [3]: pd.DataFrame({'a': pd.date_range("2012", periods=3), 'b': [1, 2, 3]}).describe(datetime_is_numeric=True)
Out[3]:
                         a    b
count                    3  3.0
mean   2012-01-02 00:00:00  2.0
min    2012-01-01 00:00:00  1.0
25%    2012-01-01 12:00:00  1.5
50%    2012-01-02 00:00:00  2.0
75%    2012-01-02 12:00:00  2.5
max    2012-01-03 00:00:00  3.0
std                    NaN  1.0
```
hopefully fixed the conflict correctly. merging on green.
Thanks. All green.
Err, wait, the whatsnew doesn't look right. Will fix.
cool