implement+test mean for datetimelike EA/Index/Series
State: closed

This takes over from #23890, implementing+testing reductions one at a time.

- [ ] closes #xxxx
- [x] tests added / passed
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ ] whatsnew entry


Labels:
name: Testing
description: pandas testing functions or related to the test suite
name: Datetime
description: Datetime data dtype
name: Algos
description: Non-arithmetic algos: value_counts, factorize, sorting, isin, clip, shift, diff
name: ExtensionArray
description: Extending pandas with custom dtypes or arrays.

Comments:
we are still treating datetimes as non-numeric yes? meaning that by default they will be exlcuded from say .mean() and you have to pass ``numeric_only=False``?

# [Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/24757?src=pr&el=h1) Report
> Merging [#24757](https://codecov.io/gh/pandas-dev/pandas/pull/24757?src=pr&el=desc) into [master](https://codecov.io/gh/pandas-dev/pandas/commit/453fa85a8b88ca22c7b878a3fcf97e068f11b6c4?src=pr&el=desc) will **increase** coverage by `<.01%`.
> The diff coverage is `100%`.

[![Impacted file tree graph](https://codecov.io/gh/pandas-dev/pandas/pull/24757/graphs/tree.svg?width=650&token=eZ4WkYLtcO&height=150&src=pr)](https://codecov.io/gh/pandas-dev/pandas/pull/24757?src=pr&el=tree)

```diff
@@            Coverage Diff             @@
##           master   #24757      +/-   ##
==========================================
+ Coverage   92.38%   92.38%   +<.01%     
==========================================
  Files         166      166              
  Lines       52363    52378      +15     
==========================================
+ Hits        48376    48391      +15     
  Misses       3987     3987
```

| Flag | Coverage Δ | |
|---|---|---|
| #multiple | `90.81% <100%> (ø)` | :arrow_up: |
| #single | `42.91% <20%> (-0.01%)` | :arrow_down: |

| [Impacted Files](https://codecov.io/gh/pandas-dev/pandas/pull/24757?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [pandas/core/arrays/datetimelike.py](https://codecov.io/gh/pandas-dev/pandas/pull/24757/diff?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvYXJyYXlzL2RhdGV0aW1lbGlrZS5weQ==) | `97.72% <100%> (+0.04%)` | :arrow_up: |
| [pandas/core/indexes/datetimelike.py](https://codecov.io/gh/pandas-dev/pandas/pull/24757/diff?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvaW5kZXhlcy9kYXRldGltZWxpa2UucHk=) | `98.52% <100%> (ø)` | :arrow_up: |
| [pandas/core/series.py](https://codecov.io/gh/pandas-dev/pandas/pull/24757/diff?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvc2VyaWVzLnB5) | `93.69% <100%> (+0.01%)` | :arrow_up: |

------

[Continue to review full report at Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/24757?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/24757?src=pr&el=footer). Last update [453fa85...30eeb64](https://codecov.io/gh/pandas-dev/pandas/pull/24757?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).

# [Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/24757?src=pr&el=h1) Report
> Merging [#24757](https://codecov.io/gh/pandas-dev/pandas/pull/24757?src=pr&el=desc) into [master](https://codecov.io/gh/pandas-dev/pandas/commit/8d124ea4c5200f218db7cea8e3ff504b0045a4e6?src=pr&el=desc) will **decrease** coverage by `<.01%`.
> The diff coverage is `100%`.

[![Impacted file tree graph](https://codecov.io/gh/pandas-dev/pandas/pull/24757/graphs/tree.svg?width=650&token=eZ4WkYLtcO&height=150&src=pr)](https://codecov.io/gh/pandas-dev/pandas/pull/24757?src=pr&el=tree)

```diff
@@            Coverage Diff             @@
##           master   #24757      +/-   ##
==========================================
- Coverage   91.88%   91.87%   -0.01%     
==========================================
  Files         174      174              
  Lines       50692    50708      +16     
==========================================
+ Hits        46576    46588      +12     
- Misses       4116     4120       +4
```

| Flag | Coverage Δ | |
|---|---|---|
| #multiple | `90.41% <100%> (ø)` | :arrow_up: |
| #single | `41.79% <17.64%> (-0.1%)` | :arrow_down: |

| [Impacted Files](https://codecov.io/gh/pandas-dev/pandas/pull/24757?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [pandas/core/indexes/datetimes.py](https://codecov.io/gh/pandas-dev/pandas/pull/24757/diff?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvaW5kZXhlcy9kYXRldGltZXMucHk=) | `96.36% <ø> (ø)` | :arrow_up: |
| [pandas/core/indexes/timedeltas.py](https://codecov.io/gh/pandas-dev/pandas/pull/24757/diff?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvaW5kZXhlcy90aW1lZGVsdGFzLnB5) | `90.96% <ø> (ø)` | :arrow_up: |
| [pandas/core/arrays/datetimelike.py](https://codecov.io/gh/pandas-dev/pandas/pull/24757/diff?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvYXJyYXlzL2RhdGV0aW1lbGlrZS5weQ==) | `97.93% <100%> (+0.04%)` | :arrow_up: |
| [pandas/core/indexes/datetimelike.py](https://codecov.io/gh/pandas-dev/pandas/pull/24757/diff?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvaW5kZXhlcy9kYXRldGltZWxpa2UucHk=) | `98.14% <100%> (ø)` | :arrow_up: |
| [pandas/core/series.py](https://codecov.io/gh/pandas-dev/pandas/pull/24757/diff?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvc2VyaWVzLnB5) | `93.62% <100%> (+0.01%)` | :arrow_up: |
| [pandas/io/gbq.py](https://codecov.io/gh/pandas-dev/pandas/pull/24757/diff?src=pr&el=tree#diff-cGFuZGFzL2lvL2dicS5weQ==) | `78.94% <0%> (-10.53%)` | :arrow_down: |
| [pandas/core/frame.py](https://codecov.io/gh/pandas-dev/pandas/pull/24757/diff?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvZnJhbWUucHk=) | `97% <0%> (-0.12%)` | :arrow_down: |

------

[Continue to review full report at Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/24757?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/24757?src=pr&el=footer). Last update [8d124ea...111c345](https://codecov.io/gh/pandas-dev/pandas/pull/24757?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).

> we are still treating datetimes as non-numeric yes? meaning that by default they will be exlcuded from say .mean() and you have to pass numeric_only=False?

I think that makes the most sense.
Does this close https://github.com/pandas-dev/pandas/issues/24752?
> Does this close #24752?

I don't think so, will need to double-check.
No, no it does not.
> I guess these have to be uniform periods so mean is just another period, though I think this could easily be a non-sensical return values, e.g. what is the mean of `pd.date_range('2012', freq='M', periods=4)`

Eyeballing it at about 2012-03-14.  Why would that be nonsensical?

it’s no longer a M freq Period. my point is the mean if a span is an odd thing 
> Can you explain that more? Maybe from an implementation point (viewing it as integers) it is clear what to do, but semantically for periods, I certainly don't find this obvious.

Period/PeriodArray/PeriodIndex is the closest thing we have to "lower-resolution Timestamp/DTA/DTI".  In this context, the mean of a bunch of Periods is analogous to a mean of a bunch of Timestamps.
> it’s no longer a M freq Period. my point is the mean if a span is an odd thing

@jreback in the example you gave, it isn't a Period at all.  It is a collection of Timestamps that happen to be a month apart.
I'm guessing @jreback's example was a typo and meant a `period_range`.

I'm still wrapping my head around the meaning of a mean for period data. The thing tripping me up is the necessary "rounding" involved in the mean. I think a good analogy is to integers.

```python
In [14]: np.arange(12).mean()
Out[14]: 5.5

In [15]: pd.period_range('2000', periods=12, freq='M').mean()
Out[15]: Period('2000-06', 'M')
```

so the `mean[int] -> float`.

It's not obvious to me that preserving the type and rounding is preferable to type casting (to a  Timestamp). But I don't think returning a Timestamp is feasible because then (IIUC) you have to worry about weird things like length of months, etc.

So the only real option is returning a Period that's rounded to the nearest period for our freq. But it took me a decent amount of thinking to get to that conclusion, and I'm still not sure it's obviously correct. For now, I'd recommend disabling mean for Period, unless we have a compelling reason to support it.
> Period/PeriodArray/PeriodIndex is the closest thing we have to "lower-resolution Timestamp/DTA/DTI". In this context, the mean of a bunch of Periods is analogous to a mean of a bunch of Timestamps.

It might be the closest thing, but it is not a lower-resolution timestamp. It represents a time span, and I don't think it is that clear what the mean of several time spans should be.

>  For now, I'd recommend disabling mean for Period, unless we have a compelling reason to support it.

\+1

@jbrockmendel do you have a use case in mind?
could provide a nice NotImplememtedError saying to convert to time stamps first 
@jreback can you confirm Tom's hypothesis that you meant period_range instead of date_range?  i.e. are we all OK with `DatetimeArray.mean`?

For PeriodArray I agree that for non-Tick frequencies it gets wonky.  Special-casing Tick vs non-Tick would be a hassle I'm OK with avoiding.  So will disable this for PeriodArray.
yes was a typo

ok on DTA / TDA but i think consensus is to not try to handle PA array currently 
> I don't see a problem delegating to TDI here if its a timedelta type. there must be something else wrong if you need this check.

The problem is that TDI/TDA doesn't have `std`, `sum`, etc yet, while ndarray[m8[ns]] does.

@TomAugspurger am I not communicating this clearly?
@jorisvandenbossche when convenient, can I get your help clearing up an apparent communication problem?
Sorry, catching up on emails...

Making sure I understand the PR correctly. This ensure that Series/Index .mean works for DatetimeArray/TimedeltaArray, while not breaking `.str` for DTA/DTA-backed Series/Index?.

If you didn't include the `and hasattr(TimedeltaIndex, name)`, then we'd end up calling `TimedeltaIndex(...).std()`, which fails, because TimedeltaIndex.std isn't a method?

IOW, this works on master

```python
In [2]: arr = pd.array([1, 2, 3], dtype='timedelta64[ns]')

In [3]: pd.Series(arr).std()
Out[3]: Timedelta('0 days 00:00:00.000000')
```

but would break without the `and hasattr(TimedeltaIndex, name)`?
> [...] but would break without the and hasattr(TimedeltaIndex, name)?

Right.
Thanks,  once all of the ops that we want are implemented, the `and hasattr(TimedeltaIndex, name)` can be removed?

Does that make sense @jreback? We can't exclude that check until TDA has all the methods, and we probably prefer smaller PRs?
why don't we add NotImplementedErr to those ops then? 
> why don't we add NotImplementedErr to those ops then?

For starters because that would also cause the relevant tests to fail, just slightly differently.

Second because after this PR goes through the next one will implement std, followed by median and whatever else is relevant.
ok can we do one of 2 things here:

- remove Timedelta change for now until we don't need to add this accessor check
- leave the change, but add a big TODO (and followup as soon as possible)?
added TODO comment
> Given the discussion we had on the other issue (where we disallowed M8 again for mean on Series), I am actually not sure that we should implement DatetimeArray.mean

Do you mean disallowed M8 for mean on DataFrame with `numeric_only=None`?  I recall that discussion, and that the decision was made mainly to avoid an API change that no one wanted to document at the last minute.

Is there a reason why `DatetimeArray.mean` would not be meaningful or have an ambiguous meaning?  If not, why wouldn't you want the method to exist?
> Is there a reason why DatetimeArray.mean would not be meaningful or have an ambiguous meaning? If not, why wouldn't you want the method to exist?

I am with @jbrockmendel here, DTA is well defined an meaningful, PeriodArray.mean is problematic which is the other discussion
> Is there a reason why DatetimeArray.mean would not be meaningful or have an ambiguous meaning? 

Don't you need floating-point to represent the mean? What's the expected value of

```python
np.mean([pd.Timestmap(1), pd.Timestamp(2)])
```

IIUC we would need to round that to `Timestamp(1)` or `Timestamp(2)`, which I would find a bit surprising. 
> Is there a reason why DatetimeArray.mean would not be meaningful or have an ambiguous meaning? If not, why wouldn't you want the method to exist?

Do you know if there is a reason why numpy choose to not support it?

I find the argument of Tom a convincing one: a mean of integers is a float, which can loose precision / give rounding. Altough, the same is true for timedelta of course .. (edit: but, timedelta supports sum and division with integer already, so supporting mean comes naturally then)
Ah I didn't realize that timedelta did implement it. Looking at the exception

```python
In [5]: np.mean(np.array([1, 2], dtype='M8[ns]'))
---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
<ipython-input-5-343ebecd9dd6> in <module>
----> 1 np.mean(np.array([1, 2], dtype='M8[ns]'))

~/Envs/pandas-dev/lib/python3.7/site-packages/numpy/core/fromnumeric.py in mean(a, axis, dtype, out, keepdims)
   3116
   3117     return _methods._mean(a, axis=axis, dtype=dtype,
-> 3118                           out=out, **kwargs)
   3119
   3120

~/Envs/pandas-dev/lib/python3.7/site-packages/numpy/core/_methods.py in _mean(a, axis, dtype, out, keepdims)
     73             is_float16_result = True
     74
---> 75     ret = umr_sum(arr, axis, dtype, out, keepdims)
     76     if isinstance(ret, mu.ndarray):
     77         ret = um.true_divide(

TypeError: ufunc add cannot use operands with types dtype('<M8[ns]') and dtype('<M8[ns]')
```

Not supporting datetimes is likely not a deliberate choice one way or the other, just a consequence of not having sum for datetimes.