BUG: DataFrame with scalar tzaware Timestamp
State: closed

- [x] closes #42505
- [x] tests added / passed
- [x] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit) for how to run them
- [ ] whatsnew entry

@simonjayhawkins where does the whatsnew for this go?

Labels:
name: Regression
description: Functionality that used to work in a prior pandas version
name: Timezones
description: Timezone data dtype
name: Constructors
description: Series/DataFrame/Index/pd.array Constructors

Comments:
> @simonjayhawkins where does the whatsnew for this go?

wdyt?

`sequence_to_dt64ns` is called from `sequence_to_datetimes` which is called from `maybe_infer_to_datetimelike` and `maybe_cast_to_datetime` in  `pandas/core/dtypes/cast.py` so this change is not narrowly targeted to the reported issue or reverting to previous behavior.

I have no doubt that this is the correct long term fix, option 2 in https://github.com/pandas-dev/pandas/issues/42505#issuecomment-972787122 but not sure if other interactions would expose other latent bugs.

This is fine for 1.4 IMO, but would a more narrowly targeted fix for 1.3.5 be safer, option 3 in https://github.com/pandas-dev/pandas/issues/42505#issuecomment-972787122

We must have little testing of scalars passed to `DatetimeArray._from_sequence`. Is there a risk that fixing this properly here change any existing behavior?
> wdyt?

I guess for now put it in the 1.4 file and discuss whether to backport?

> We must have little testing of scalars passed to DatetimeArray._from_sequence. Is there a risk that fixing this properly here change any existing behavior?

Only incorrect behavior AFAICT.

> but would a more narrowly targeted fix for 1.3.5 be safer, option 3 in #42505 (comment)

I'm not clear on how we can get more targeted than this.
this is fine, can you add a whatsnew for 1.3.5
Owee, I'm MrMeeseeks, Look at me.

There seem to be a conflict, please backport manually. Here are approximate instructions:

1. Checkout backport branch and update it.

```
git checkout 1.3.x
git pull
```

2. Cherry pick the first parent branch of the this PR on top of the older branch:
```
git cherry-pick -m1 a327ad18105a4efab9f7bc57fbf2dfd22130e933
```

3. You will likely have some merge/cherry-pick conflict here, fix them and commit:

```
git commit -am 'Backport PR #44518: BUG: DataFrame with scalar tzaware Timestamp'
```

4. Push to a named branch:

```
git push YOURFORK 1.3.x:auto-backport-of-pr-44518-on-1.3.x
```

5. Create a PR against branch 1.3.x, I would have named this PR:

> "Backport PR #44518 on branch 1.3.x (BUG: DataFrame with scalar tzaware Timestamp)"

And apply the correct labels and milestones.

Congratulations — you did some good work! Hopefully your backport PR will be tested by the continuous integration and merged soon!

Remember to remove the `Still Needs Manual Backport` label once the PR gets merged.

If these instructions are inaccurate, feel free to [suggest an improvement](https://github.com/MeeseeksBox/MeeseeksDev).
                
@meeseeksdev backport 1.3.x
Owee, I'm MrMeeseeks, Look at me.

There seem to be a conflict, please backport manually. Here are approximate instructions:

1. Checkout backport branch and update it.

```
git checkout 1.3.x
git pull
```

2. Cherry pick the first parent branch of the this PR on top of the older branch:
```
git cherry-pick -m1 a327ad18105a4efab9f7bc57fbf2dfd22130e933
```

3. You will likely have some merge/cherry-pick conflict here, fix them and commit:

```
git commit -am 'Backport PR #44518: BUG: DataFrame with scalar tzaware Timestamp'
```

4. Push to a named branch:

```
git push YOURFORK 1.3.x:auto-backport-of-pr-44518-on-1.3.x
```

5. Create a PR against branch 1.3.x, I would have named this PR:

> "Backport PR #44518 on branch 1.3.x (BUG: DataFrame with scalar tzaware Timestamp)"

And apply the correct labels and milestones.

Congratulations — you did some good work! Hopefully your backport PR will be tested by the continuous integration and merged soon!

Remember to remove the `Still Needs Manual Backport` label once the PR gets merged.

If these instructions are inaccurate, feel free to [suggest an improvement](https://github.com/MeeseeksBox/MeeseeksDev).
                
@jbrockmendel if you can push the backport :-<>
> > but would a more narrowly targeted fix for 1.3.5 be safer, option 3 in #42505 (comment)
> 
> I'm not clear on how we can get more targeted than this.

I was thinking special case in `construct_1d_arraylike_from_scalar`. The return type from `infer_dtype_from_scalar` is now a `Timestamp` whereas it was a `int` in pandas 1.2.5

So we then have the block... `if isinstance(dtype, ExtensionDtype)` where we pass the value onto `DataTimeArray._from_sequence`, so my thinking was to pass value.value if is instance of Timestamp to restore the 1.2.5 behavior explicitly without touching `DataTimeArray._from_sequence`.

so I was thinking something like...

```diff
diff --git a/pandas/core/dtypes/cast.py b/pandas/core/dtypes/cast.py
index 6d5162f3fe..a02e8e085c 100644
--- a/pandas/core/dtypes/cast.py
+++ b/pandas/core/dtypes/cast.py
@@ -1900,6 +1900,8 @@ def construct_1d_arraylike_from_scalar(
 
     """
 
+    from pandas.core.arrays import DatetimeArray
+
     if dtype is None:
         try:
             dtype, value = infer_dtype_from_scalar(value, pandas_dtype=True)
@@ -1908,6 +1910,8 @@ def construct_1d_arraylike_from_scalar(
 
     if isinstance(dtype, ExtensionDtype):
         cls = dtype.construct_array_type()
+        if issubclass(cls, DatetimeArray) and isinstance(value, Timestamp):
+            value = value.value
         subarr = cls._from_sequence([value] * length, dtype=dtype)
 
     else:
```

with an appropriate code comment about the hack.

The non-extension array block seems to include some special handling for `dtype.kind in ["M", "m"]` to unbox the values so this location for a fix, unboxing the values from a TimeStamp is maybe not so strange after all?

> > We must have little testing of scalars passed to DatetimeArray._from_sequence. Is there a risk that fixing this properly here change any existing behavior?
> 
> Only incorrect behavior AFAICT.

yes. any change in behavior from doing the proper fix for 1.3.5 can probably be put under the "bug fix" umbrella, so maybe this is fine.