BUG: `pd.concat` dataframes with different datetime64 resolutions
State: closed

- [x] Closes #53640
- [x] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests)
- [x] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit)
- [ ] *I assume that no changelog is needed since this bug does not exist in the latest version of pandas*


Labels:
name: Reshaping
description: Concat, Merge/Join, Stack/Unstack, Explode
name: Stale
name: Non-Nano
description: datetime64/timedelta64 with non-nanosecond resolution

Comments:
This pull request is stale because it has been open for thirty days with no activity. Please [update](https://pandas.pydata.org/pandas-docs/stable/development/contributing.html#updating-your-pull-request) and respond to this comment if you're still interested in working on this.
I can confirm this issue still exists in pandas 2.1.1. 

This bug together with the latest changes in pyarrow 13.0.0 https://github.com/apache/arrow/issues/38171 makes it difficult to use `pd.concat` and `pd.merge` https://github.com/pandas-dev/pandas/issues/55212.

```py
t1 = '2023-09-01'
t2 = '2023-09-01 01:00:00'
t3 = '2023-09-01 01:30:00'
ds1 = pd.date_range(t1, t1, freq='30T')
ds2 = pd.date_range(t2, t3, freq='30T')
df1 = pd.DataFrame({
    'ds': ds1.astype('datetime64[us]'),
    'y': list(range(len(ds1))),
})
df2 = pd.DataFrame({
    'ds': ds2.astype('datetime64[ns]'),
    'y': list(range(len(ds2))),
})
pd.concat([df1, df2], axis=0)
```

And the error
```
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/user/conda-envs/api/lib/python3.9/site-packages/pandas/core/reshape/concat.py", line 393, in concat
    return op.get_result()
  File "/home/user/conda-envs/api/lib/python3.9/site-packages/pandas/core/reshape/concat.py", line 680, in get_result
    new_data = concatenate_managers(
  File "/home/user/conda-envs/api/lib/python3.9/site-packages/pandas/core/internals/concat.py", line 189, in concatenate_managers
    values = _concatenate_join_units(join_units, copy=copy)
  File "/home/user/conda-envs/api/lib/python3.9/site-packages/pandas/core/internals/concat.py", line 486, in _concatenate_join_units
    concat_values = concat_compat(to_concat, axis=1)
  File "/home/user/conda-envs/api/lib/python3.9/site-packages/pandas/core/dtypes/concat.py", line 132, in concat_compat
    return cls._concat_same_type(to_concat_eas)
  File "/home/user/conda-envs/api/lib/python3.9/site-packages/pandas/core/arrays/datetimelike.py", line 2251, in _concat_same_type
    new_obj = super()._concat_same_type(to_concat, axis)
  File "/home/user/conda-envs/api/lib/python3.9/site-packages/pandas/core/arrays/_mixins.py", line 230, in _concat_same_type
    return super()._concat_same_type(to_concat, axis=axis)
  File "arrays.pyx", line 190, in pandas._libs.arrays.NDArrayBacked._concat_same_type
ValueError: all the input array dimensions except for the concatenation axis must match exactly, but along dimension 1, the array at index 0 has size 1 and the array at index 1 has size 2
```
Will this also close #55067, #55374?
> Will this also close #55067, #55374?

@jbrockmendel Yes, I can confirm that #55067 and #55374 raise no ValueError with this PR.

Also, I tried what you suggested, i.e., parametrize existing tests but find it too hard to realize. Currently I made a slightly more comprehensive test than the previous version and parametrized it with different datetime resolutions. Can you check if that suffices?
> Also, I tried what you suggested, i.e., parametrize existing tests but find it too hard to realize

I suggest defining these fixtures in tests.reshape.concat.test_datetimes
```
@pytest.fixture(params=["s", "ms", "us", "ns"])
def unit(request):
    return request.param


unit2 = unit
```

Then pass these fixtures to test_concat_tz_series3 and cast `first` to `f"M8[{unit}]"` and `second` to `f"M8[{unit2}]"`.  This test would then hit the bug that this PR fixes.

Updating other tests in this file to use the same pattern would be optional.
A few comments, otherwise looking good
Done, thanks for the review @jbrockmendel!
Thanks @Charlie-XIAO 
@mroeschke 
Looks like this is missing a whatsnew.

Also, would you be open to backporting this to 2.1.x?
It seems quite a few people are being hit by this.
Ah, yeah +1 to backport this to 2.1.x
@meeseeksdev backport 2.1.x.
Owee, I'm MrMeeseeks, Look at me.

There seem to be a conflict, please backport manually. Here are approximate instructions:

1. Checkout backport branch and update it.

```
git checkout 2.1.x
git pull
```

2. Cherry pick the first parent branch of the this PR on top of the older branch:
```
git cherry-pick -x -m1 f7bda1734cde43adb2e867304668e56e14c17a9f
```

3. You will likely have some merge/cherry-pick conflict here, fix them and commit:

```
git commit -am 'Backport PR #53641: BUG: `pd.concat` dataframes with different datetime64 resolutions'
```

4. Push to a named branch:

```
git push YOURFORK 2.1.x:auto-backport-of-pr-53641-on-2.1.x
```

5. Create a PR against branch 2.1.x, I would have named this PR:

> "Backport PR #53641 on branch 2.1.x (BUG: `pd.concat` dataframes with different datetime64 resolutions)"

And apply the correct labels and milestones.

Congratulations — you did some good work! Hopefully your backport PR will be tested by the continuous integration and merged soon!

Remember to remove the `Still Needs Manual Backport` label once the PR gets merged.

If these instructions are inaccurate, feel free to [suggest an improvement](https://github.com/MeeseeksBox/MeeseeksDev).
                
@lithomas1 - This issue still exists on 2.1.4, was the backport included in the release?
Unfortunately not, there seem to be other changes necessary in addition to this one to patch the bug for the 2.1 series.
(the added tests here fail when backported to 2.1.)

If you're curious, my attempt to do it was here https://github.com/pandas-dev/pandas/pull/56311.
This PR fixes the issue with concatenating dataframes that have different datetime64 resolutions.