BUG: Fix issue with datetime[ns, tz] input in Block.setitem GH32395
State: closed

- [x] closes #32395 
- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [x] whatsnew entry


Labels:
name: Bug
name: Indexing
description: Related to indexing on series/frames, not to indexes themselves

Comments:
https://github.com/pandas-dev/pandas/pull/29334 might actually fix this (or if not is a bit cleaner)
https://github.com/pandas-dev/pandas/pull/29334 doesn't quite seem to fix this (some of the new tests here fail on that branch).
FYI, I'm currently planning to push this to the next release.
Any thoughts on - https://github.com/pandas-dev/pandas/pull/32479#discussion_r392056376?
> #29334 doesn't quite seem to fix this (some of the new tests here fail on that branch).

#29334 has been merged and did not fix.

if this has stalled, is reverting #29139 an option?

@h-vishal can you merge master to resolve conflicts and move whatsnew to 1.1.0
I've been working on Block.setitem, will see if I can find a nice option that doesnt entail reverting #29139.
@simonjayhawkins I have updated this, please look at alternate solutions discussed in https://github.com/pandas-dev/pandas/pull/32479#discussion_r392056376 and https://github.com/pandas-dev/pandas/pull/32479#discussion_r395987552 if one of those is preferred I will push another update.

@jbrockmendel let me know if there is an update on this.
@h-vishal the thing to do here is to add a clause into `Block.setitem` just after the `elif exact_match and is_categorical_dtype(arr_value.dtype)` check:

```
        elif exact_match and is_extension_array_dtype(arr_value.dtype):
            # GH#32395 if we're going to replace the values entirely, just
            #  substitute in the new array
            return self.make_block(arr_value)
```

Then update the tests to use tm.assert_series_equal and tm.assert_frame_equal as others have suggested.

caveat 1: i havent run the whole test suite on this yet
caveat 2: this works if we are setting `obj.loc[:] = other` but I havent looked at `obj.loc[:halfway] = other[:halfway]`
@jbrockmendel this will create a column with the corresponding `datetime64[ns, tz]` `dtype` when a full column assignment occurs so the current tests can be modified to use `assert_frame_equals`, partial column assignments to `datetime64[ns, tz]` data would create `object` `dtype` columns for which I can add additional tests.

Would it be better to use `is_datetime64tz_dtype(arr_value.dtype)` instead of `is_extension_array_dtype(arr_value.dtype)`?


> this will create a column with the corresponding datetime64[ns, tz] dtype when a full column assignment occurs so the current tests can be modified to use assert_frame_equals

I think thats the preferred behavior.  We can let others weigh in cc @jreback @simonjayhawkins  @TomAugspurger @WillAyd 

> partial column assignments to datetime64[ns, tz] data would create object dtype columns for which I can add additional tests.

Please do (or see if you can find existing tests for this behavior)

> Would it be better to use `is_datetime64tz_dtype(arr_value.dtype)` instead of `is_extension_array_dtype(arr_value.dtype)`?

I think we're going to want this behavior for any EA dtype, not just dt64tz.  But best not to enable without tests for more EA dtypes.  So this either needs more thorough tests, or you could do is_dt64_tz for now with a TODO comment to expand to other dtypes, and open an issue to that effect
While writing the new tests I came across a bug with a slice assignment of a sparse column - an `int32` sparse column on assignment gets converted to a `float64` dense column instead of an `int32` dense column as described in the documentation, this bug is also present in the 0.25 version. I didn't find an associated issue for this but it's possible that I missed it being mentioned somewhere. If this is not a known issue, I can create a new issue for this.

```
>>> df = pd.DataFrame(np.random.randn(100), columns=["data"])
>>> df.iloc[5:95] = np.nan
>>> df = df.astype(pd.SparseDtype("int", np.nan))
>>> df2 = pd.DataFrame(index=df.index)
>>> df2.loc[df.index[:10], "data"] = df["data"][:10]
>>> df.data
0     0
1    -1
2    -1
3     0
4     0
     ..
95    0
96    0
97    0
98    0
99    0
Name: data, Length: 100, dtype: Sparse[int32, nan]
>>> df2.data
0     0.0
1    -1.0
2    -1.0
3     0.0
4     0.0
     ...
95    NaN
96    NaN
97    NaN
98    NaN
99    NaN
Name: data, Length: 100, dtype: float64
```
> While writing the new tests I came across a bug with a slice assignment of a sparse column

does this PR fix the newly found bug?
> > While writing the new tests I came across a bug with a slice assignment of a sparse column
> 
> does this PR fix the newly found bug?

@jbrockmendel no, it's unrelated, confirmed in version 0.25 as well, better to create a new issue for it.
@h-vishal can you rebase and address comments
thanks @h-vishal 
Owee, I'm MrMeeseeks, Look at me.

There seem to be a conflict, please backport manually. Here are approximate instructions:

1. Checkout backport branch and update it.

```
$ git checkout 1.0.x
$ git pull
```

2. Cherry pick the first parent branch of the this PR on top of the older branch:
```
$ git cherry-pick -m1 0f2ca378a2b692edba7baab19ea138b887838cc0
```

3. You will likely have some merge/cherry-pick conflict here, fix them and commit:

```
$ git commit -am 'Backport PR #32479: BUG: Fix issue with datetime[ns, tz] input in Block.setitem GH32395'
```

4. Push to a named branch :

```
git push YOURFORK 1.0.x:auto-backport-of-pr-32479-on-1.0.x
```

5. Create a PR against branch 1.0.x, I would have named this PR:

> "Backport PR #32479 on branch 1.0.x"

And apply the correct labels and milestones.

Congratulation you did some good work ! Hopefully your backport PR will be tested by the continuous integration and merged soon!

If these instruction are inaccurate, feel free to [suggest an improvement](https://github.com/MeeseeksBox/MeeseeksDev).