REGR: fix TimedeltaIndex sum and datetime subtraction with NaT (#25282, #25317)
State: closed

closes #25282
closes #25317
- [x] tests added / passed
- [ ] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [x] whatsnew entry


Labels:
name: Datetime
description: Datetime data dtype
name: Regression
description: Functionality that used to work in a prior pandas version

Comments:
# [Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/25329?src=pr&el=h1) Report
> Merging [#25329](https://codecov.io/gh/pandas-dev/pandas/pull/25329?src=pr&el=desc) into [master](https://codecov.io/gh/pandas-dev/pandas/commit/4be995c13452766da48a82f3ecd0b5a24e48f526?src=pr&el=desc) will **increase** coverage by `<.01%`.
> The diff coverage is `100%`.

[![Impacted file tree graph](https://codecov.io/gh/pandas-dev/pandas/pull/25329/graphs/tree.svg?width=650&token=eZ4WkYLtcO&height=150&src=pr)](https://codecov.io/gh/pandas-dev/pandas/pull/25329?src=pr&el=tree)

```diff
@@            Coverage Diff             @@
##           master   #25329      +/-   ##
==========================================
+ Coverage   91.72%   91.72%   +<.01%     
==========================================
  Files         173      173              
  Lines       52831    52835       +4     
==========================================
+ Hits        48458    48462       +4     
  Misses       4373     4373
```

| Flag | Coverage Δ | |
|---|---|---|
| #multiple | `90.28% <100%> (ø)` | :arrow_up: |
| #single | `41.72% <50%> (ø)` | :arrow_up: |

| [Impacted Files](https://codecov.io/gh/pandas-dev/pandas/pull/25329?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [pandas/core/arrays/timedeltas.py](https://codecov.io/gh/pandas-dev/pandas/pull/25329/diff?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvYXJyYXlzL3RpbWVkZWx0YXMucHk=) | `88.39% <100%> (+0.09%)` | :arrow_up: |
| [pandas/core/arrays/datetimes.py](https://codecov.io/gh/pandas-dev/pandas/pull/25329/diff?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvYXJyYXlzL2RhdGV0aW1lcy5weQ==) | `97.79% <100%> (ø)` | :arrow_up: |

------

[Continue to review full report at Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/25329?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/25329?src=pr&el=footer). Last update [4be995c...3fa3e81](https://codecov.io/gh/pandas-dev/pandas/pull/25329?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).

# [Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/25329?src=pr&el=h1) Report
> Merging [#25329](https://codecov.io/gh/pandas-dev/pandas/pull/25329?src=pr&el=desc) into [master](https://codecov.io/gh/pandas-dev/pandas/commit/f4568fd76e864d8aee3d23f5a81302262d6e0dcb?src=pr&el=desc) will **decrease** coverage by `<.01%`.
> The diff coverage is `87.5%`.

[![Impacted file tree graph](https://codecov.io/gh/pandas-dev/pandas/pull/25329/graphs/tree.svg?width=650&token=eZ4WkYLtcO&height=150&src=pr)](https://codecov.io/gh/pandas-dev/pandas/pull/25329?src=pr&el=tree)

```diff
@@            Coverage Diff             @@
##           master   #25329      +/-   ##
==========================================
- Coverage   91.73%   91.73%   -0.01%     
==========================================
  Files         173      173              
  Lines       52832    52837       +5     
==========================================
+ Hits        48467    48471       +4     
- Misses       4365     4366       +1
```

| Flag | Coverage Δ | |
|---|---|---|
| #multiple | `90.3% <87.5%> (-0.01%)` | :arrow_down: |
| #single | `41.72% <37.5%> (ø)` | :arrow_up: |

| [Impacted Files](https://codecov.io/gh/pandas-dev/pandas/pull/25329?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [pandas/core/indexes/base.py](https://codecov.io/gh/pandas-dev/pandas/pull/25329/diff?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvaW5kZXhlcy9iYXNlLnB5) | `96.55% <100%> (ø)` | :arrow_up: |
| [pandas/core/arrays/datetimes.py](https://codecov.io/gh/pandas-dev/pandas/pull/25329/diff?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvYXJyYXlzL2RhdGV0aW1lcy5weQ==) | `97.79% <100%> (ø)` | :arrow_up: |
| [pandas/core/arrays/timedeltas.py](https://codecov.io/gh/pandas-dev/pandas/pull/25329/diff?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvYXJyYXlzL3RpbWVkZWx0YXMucHk=) | `88.2% <75%> (-0.11%)` | :arrow_down: |

------

[Continue to review full report at Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/25329?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/25329?src=pr&el=footer). Last update [f4568fd...67ab597](https://codecov.io/gh/pandas-dev/pandas/pull/25329?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).

@jbrockmendel in general, can you try to give PRs a bit more informative titles? (I don't know all the issue numbers by heart :-))
Side note: I've been getting this same [macos 3.5 buld failure](https://dev.azure.com/pandas-dev/pandas/_build/results?buildId=8360&view=logs&jobId=7c58e677-adbe-554c-f717-831a032b8982) in some of my PRs which _seem_ unrelated to the main changes. Looking into it but triaging just in case you find something first.
@mroeschke I'm pretty sure the failing builds are 32bit, so `np.arange(5)` returns a np.int32 array instead of the np.int64 I'm used to.  Pending the new CI run, I think changing this to `np.arange(5, dtype=np.int64)` should fix it.
I take it back, no idea what’s up with the Mac 3.5 errors
can you merge master, some failures
@jreback @jorisvandenbossche : This is green again.  I know you guys all approved @jbrockmendel PR pre-master-merging, but just wanted to ping in case there was anything else.
@jbrockmendel do you know why this first only failed on numpy dev? (or why the final change was only needed for numpy dev) Something changed in numpy / something we might need to report in numpy?
I don't have numpy dev installed locally, but would speculate that np.sum(ndarray[m8ns]) changed from returning a scalar to returning a zero-dim ndarray
I believe both returned a 0dim array

```python
In [7]: np.__version__
Out[7]: '1.17.0.dev0+8063fa6'

In [8]: type(np.array([1, 2], dtype='timedelta64[ns]').sum())
Out[8]: numpy.timedelta64
```

```python
In [8]: np.__version__
Out[8]: '1.16.0'

In [9]: type(np.array([1, 2], dtype='timedelta64[ns]').sum())
Out[9]: numpy.timedelta64
```
@jbrockmendel I believe this handles 0-dim scalars already and @TomAugspurger example is covered?
@jreback yes this is fine.  The zerodim example Tom posted is disproving my guess as to Joris's question about what changed in numpy-dev.
k merge away
I think it is still worth to investigate this further, to know if this was an intentional change in numpy or if we should report it.
@jorisvandenbossche I agree it would be nice to figure out what changed in numpy, but is that a blocker for this PR?  That sounds like a separate Issue.
Yeah, certainly no problem to already merge this (we should just be aware that we are possibly putting a workaround in place that might be needed in other places as well)

Opened and issue in numpy: https://github.com/numpy/numpy/issues/12997
@jbrockmendel thanks!