Repository: pandas-dev/pandas
Pull request: fix: scalar timestamp assignment (#19843)
- [x] closes #19843
- [x] tests added / passed
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [x] whatsnew entry


Problem statement:
BUG: scalar assignment of a tz-aware is object dtype
[3] should be a ``datetime64[ns, UTC]``

```
In [1]: df = pd.DataFrame({'A': [0, 1]})

In [3]: df['now'] = pd.Timestamp('20130101', tz='UTC')

In [4]: df
Out[4]: 
   A                        now
0  0  2013-01-01 00:00:00+00:00
1  1  2013-01-01 00:00:00+00:00

In [5]: df.dtypes
Out[5]: 
A       int64
now    object
dtype: object

In [6]: df['now2'] = pd.DatetimeIndex([pd.Timestamp('20130101', tz='UTC')]).repeat(len(df))

In [7]: df.dtypes
Out[7]: 
A                     int64
now                  object
now2    datetime64[ns, UTC]
dtype: object
```

Hints from issue discussion:
I will try and fix this.
great!
Currently, ```infer_dtype_from_scalar``` (on datetimey/timestampy objects) returns a ```np.datetime64``` if no timezone is given, and defaults to ```np.object_``` on objects with timezones. Fixing this problem means returning something else, rather than ```np.object_```. 

Ideally return ```DatetimeTZDtypeType```. However, this crashes on ```np.empty(shape, dtype=dtype)``` in ```cast_scalar_to_array```. Seems like this should work, but it doesn't.

Quick fix is returning ```np.datetime64``` rather than ```np.object_```. You lose the timezone name, but numpy applies the correct offset before saving so the numbers are correct. This change doesn't break any tests, and results in the following behavior:

```
In [1]: df = pd.DataFrame({'A': [0, 1]})

In [3]: df['now'] = pd.Timestamp('20130101', tz='UTC')

In [5]: df.dtypes
Out[5]: 
A               int64
now    datetime64[ns]
dtype: object

In [6]: df['now2'] = pd.DatetimeIndex([pd.Timestamp('20130101', tz='UTC')]).repeat(len(df))

In [7]: df.dtypes
Out[7]: 
A                     int64
now          datetime64[ns]
now2    datetime64[ns, UTC]
dtype: object
```
Raises some inconsistencies, potentially problems with mixing in timezone-naive datetimes. Is the quick fix good enough?
@DylanDmitri you don't want to *ever* have numpy deal with timezones, they are completely wrong. ``infer_dtype_from_scalar`` has a ``pandas_dtype`` parameter that will make this work. We should actully just change this to do this by default (though this might break other things)
Been busy the last week, sorry.  Here's the problem code (from line 2874 of ```frame.py```)

```
# BEFORE
value = cast_scalar_to_array(len(self.index), value)
value = maybe_cast_to_datetime(value, value.dtype)
```

Main issue: ```cast_scalar_to_array``` defaults to dtype ```np.object_```, which is then ignored by ```maybe_cast_to_datetime```. Want to capture the real pandas dtype, and then pass that into ```maybe_cast_to_datetime```, which then works properly.

```
# AFTER
from pandas.core.dtypes.cast import infer_dtype_from_scalar
pandas_dtype, _ = infer_dtype_from_scalar(value, pandas_dtype=True)

value = cast_scalar_to_array(len(self.index), value)
value = maybe_cast_to_datetime(value, pandas_dtype)
```

This fixes the problem. Will check tests, and have a PR soon.

Commit message: fix: scalar timestamp assignment (#19843)

Patch to review:
diff --git a/pandas/core/frame.py b/pandas/core/frame.py
index ff4064b3f8c56..b2a508c8c3daa 100644
--- a/pandas/core/frame.py
+++ b/pandas/core/frame.py
@@ -2868,9 +2868,15 @@ def reindexer(value):
                 value = maybe_infer_to_datetimelike(value)
 
         else:
-            # upcast the scalar
+            # cast ignores pandas dtypes. so save the dtype first
+            from pandas.core.dtypes.cast import infer_dtype_from_scalar
+            pd_dtype, _ = infer_dtype_from_scalar(value, pandas_dtype=True)
+
+            # upcast
             value = cast_scalar_to_array(len(self.index), value)
-            value = maybe_cast_to_datetime(value, value.dtype)
+
+            # then add dtype back in
+            value = maybe_cast_to_datetime(value, pd_dtype)
 
         # return internal types directly
         if is_extension_type(value) or is_extension_array_dtype(value):
diff --git a/pandas/tests/scalar/timestamp/test_assignment_dtype.py b/pandas/tests/scalar/timestamp/test_assignment_dtype.py
new file mode 100644
index 0000000000000..f706b21ec7c43
--- /dev/null
+++ b/pandas/tests/scalar/timestamp/test_assignment_dtype.py
@@ -0,0 +1,23 @@
+import pandas as pd
+from pandas.core.dtypes.dtypes import DatetimeTZDtype
+
+
+# referencing #19843: scalar assignment of a tz-aware is object dtype
+
+class TestTimestampProperties(object):
+
+    def test_scalar_assignment(self):
+        df = pd.DataFrame(index=(0, 1, 2))
+
+        df['now'] = pd.Timestamp('20130101', tz='UTC')
+
+        assert isinstance(df.dtypes[0], DatetimeTZDtype)
+
+    def test_datetime_index_assignment(self):
+        df = pd.DataFrame(index=(0, 1, 2))
+
+        di = pd.DatetimeIndex(
+            [pd.Timestamp('20130101', tz='UTC')]).repeat(len(df))
+        df['now'] = di
+
+        assert isinstance(df.dtypes[0], DatetimeTZDtype)


Code review comments:
diff_hunk: @@ -0,0 +1,23 @@
+import pandas as pd
+from pandas.core.dtypes.dtypes import DatetimeTZDtype
+
+
+# referencing #19843: scalar assignment of a tz-aware is object dtype
path: pandas/tests/scalar/timestamp/test_assignment_dtype.py
text: @user2:
I think we can find a home for this in another test file (and don't need a new one).  In addition, the comment should be under the class declaration.

@user1 : I think `tests/frame/test_indexing.py` makes sense here.  What do you think?

@author:
Sounds good, should they go under ```TestDataFrameIndexingDatetimeWithTZ```, or in their own class?

@user2:
Same class sounds good for now.