REGR: allow reindexing datetimelike with upcast / raise deprecation warning
State: closed

Closes https://github.com/pandas-dev/pandas/issues/42921

- Fixes inconsistency/regression in the deprecation of `date` coercion (https://github.com/pandas-dev/pandas/pull/39767) -> now this is also done for single-axis reindexing (xref https://github.com/pandas-dev/pandas/pull/44798#issuecomment-988264205)
- This also fixes the regression reported in https://github.com/pandas-dev/pandas/issues/42921(although it is not fully clear if we really want to "fix" this regression)

Labels:
name: Indexing
description: Related to indexing on series/frames, not to indexes themselves
name: Deprecate
description: Functionality to remove in pandas

Comments:
Thanks @jorisvandenbossche 

just to be clear...

```python
s.reindex(
    pd.date_range("2020-01-01", "2020-01-04", freq="1d"), fill_value=pd.Timedelta(0)
)
```

on 1.2.5

```
2020-01-01               None
2020-01-02               None
2020-01-03               None
2020-01-04    0 days 00:00:00
Freq: D, dtype: object
```

and with this PR
```
2020-01-01                NaT
2020-01-02                NaT
2020-01-03                NaT
2020-01-04    0 days 00:00:00
Freq: D, dtype: object
```

so this is not restoring 1.2.5 behaviour exactly.

However, the dtype is still `object`, whereas in the issue OP the expected behavior is for `timedelta64[ns]` (which is the result when dtype is explictly specfied, second part of https://github.com/pandas-dev/pandas/issues/42921#issuecomment-897568579)

so maybe should not yet close #42921 with this PR?
moving this to 1.4. we don't need to backport this
I moved this back to 1.3.5 milestone for now (let's discuss first before moving). 

It _does_ fix a regression from 1.2->1.3. Code that worked on 1.2, is raising an error on 1.3. 

@simonjayhawkins you're right that None vs NaT is not exactly the same (I didn't notice that yet, can look into why that is the case), but it's also a much smaller difference compared to raising an error in 1.3


> However, the dtype is still `object`, whereas in the issue OP the expected behavior is for `timedelta64[ns]` (which is the result when dtype is explictly specfied, second part of [#42921 (comment)](https://github.com/pandas-dev/pandas/issues/42921#issuecomment-897568579))

As you explained in that issue, it is expected that this doesn't give `timedelta64[ns]`, since the example starts with a datetime64 dtype.

> As you explained in that issue, it is expected that this doesn't give `timedelta64[ns]`, since the example starts with a datetime64 dtype.

That's fine. we probably need to repeat/confirm this in the issue (that the expected in the OP is incorrect) so that it can be closed off.
@jorisvandenbossche it may be a regression but this is delaying the long delayed release
it's simply too late
@jorisvandenbossche pls don't move this back
it is too late for this patch which hasn't been looked at and will not before release
will cases that go through ExtensionBlock.take_nd be affected/corrected?
> will cases that go through ExtensionBlock.take_nd be affected/corrected?

No, because they are already strict in preserving the dtype anyway
>> will cases that go through ExtensionBlock.take_nd be affected/corrected?

> No, because they are already strict in preserving the dtype anyway

It isn't clear to me that this strictness is correct.  i.e. in test_reindex_fill_value_datetimelike_upcast what happens if you add a couple of EA dtypes to the 'dtype' param?

> It isn't clear to me that this strictness is correct.

It has been like that for a long time (for ever?), and changing (or discussing) that is out of scope for this PR I think.
> that is out of scope for this PR I think.

Sure.

Looks like ArrayManager tests are failing.
@jorisvandenbossche needs rebase
@phofl @jbrockmendel if one of you could rebase this
rebased
@jbrockmendel ok here?
Nothing that should block this.  I'd like to get a response here https://github.com/pandas-dev/pandas/pull/44839/files#r774813668 eventually
The array_manager runs through take_1d which does not promote to object in case of incompatible fill values. Not familiar enough with the logic to decide on a fix here
> The array_manager runs through take_1d which does not promote to object in case of incompatible fill values. Not familiar enough with the logic to decide on a fix here

Yah, trying to make AM behavior consistent is a fool's errand.
@jbrockmendel or @phofl can you rebase this
@jbrockmendel or @phofl if you can rebase this
merged master
hmm a code check / pre-commit issue if you can fixup @jbrockmendel 
@meeseeksdev backport 1.4.x
Something went wrong ... Please have a look at my logs.