REGR: DataFrame.agg with axis=1, EA dtype, and duplicate index
State: closed

- [x] closes #42380
- [x] tests added / passed
- [x] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing.html#code-standards) for how to run them
- [x] whatsnew entry

The issue in transpose was a bug that was introduced in 1.0.0, but induced a regression in 1.3.0 due to its use in `DataFrame.agg`.

Labels:
name: Reshaping
description: Concat, Merge/Join, Stack/Unstack, Explode
name: Regression
description: Functionality that used to work in a prior pandas version
name: Apply
description: Apply, Aggregate, Transform, Map

Comments:
thanks @rhshadrach 
@meeseeksdev backport 1.3.x
Something went wrong ... Please have  a look at my logs.
> The issue in transpose was a bug that was introduced in 1.0.0

in pandas 0.25.3 the returned DataFrame had object dtype.

```
>>> pd.__version__
'0.25.3'
>>> 
>>> df = pd.DataFrame(
...     {"a": list("abcde"), "b": list("abcde")}, index=list("aabbc"), dtype="category"
... )
>>> 
>>> df.T.dtypes
a    object
a    object
b    object
b    object
c    object
dtype: object
>>> 
```
on master, we now have value dependent behaviour

```
>>> pd.__version__
'1.4.0.dev0+193.g87d785599e'
>>> 
>>> df = pd.DataFrame(
...     {"a": list("abcde"), "b": list("abcde")}, index=list("aabbc"), dtype="category"
... )
>>> 
>>> df.T.dtypes
a    category
a    category
b    category
b    category
c    category
dtype: object
>>> 
>>> df = pd.DataFrame(
...     {"a": list("abcde"), "b": list("fghij")}, index=list("aabbc"), dtype="category"
... )
>>> 
>>> df.T.dtypes
a    object
a    object
b    object
b    object
c    object
dtype: object
>>> 
```
@simonjayhawkins I think the column dtypes, not values, are what determines the resulting dtype. As such, I would not describe this as value-dependent behavior. The result looks like the correct one to me, but can open an issue if you would like this discussed further.
> The result looks like the correct one to me

The result is now certainly consistent with the result for a unique index. (which also changed behavior between 0.25.3 and 1.0.0)

> but can open an issue if you would like this discussed further.

this has been backported for a patch release. and we could release anytime dependent on number and severity of reported regression.

My preference is to not having an open issue, but to keep the 1.3.x ready for release.

Any reason not just to revert #40428 instead?



> this has been backported for a patch release. and we could release anytime dependent on number and severity of reported regression.
> 
> My preference is to not having an open issue, but to keep the 1.3.x ready for release.

Are you saying we should not fix the transpose regression from 0.25.3 -> 1.0.0 in 1.3.1? If that's the case, then I would support reverting #40428, assuming it does fix the issue in `agg`. However, the code prior to #40428 used transpose as well, so I don't understand how it could fix it (but might be missing something).
> Are you saying we should not fix the transpose regression from 0.25.3 -> 1.0.0 in 1.3.1?

I am not saying that. I basically just wanted clarification of why #40428 was not reverted. I assume that because the root cause was determined as the change in #30091, that was fixed instead?

However, that change does not revert to the behavior in 0.25.3 because of the dtype change. (but that is also reasonable as the new behavior matches that of a unique index)

> If that's the case, then I would support reverting #40428, assuming it does fix the issue in `agg`. 

it is ok to fix prior regressions or bug fixes in patch releases, so that alone is not a reason to prefer the revert option.

> However, the code prior to #40428 used transpose as well, so I don't understand how it could fix it (but might be missing something).

that is odd.
Thanks @simonjayhawkins, your assessment is correct - the root cause of the agg regression was the transpose regression. Both are tested in this PR.

> However, that change does not revert to the behavior in 0.25.3 because of the dtype change. (but that is also reasonable as the new behavior matches that of a unique index)

The behavior in 0.25.3 is undesirable. When a DataFrame has a single EA dtype, 0.25.3's transpose would convert it to object whereas it should remain the EA dtype. #30091 fixed this, but at the expense of silently dropping data when the index has duplicates. This PR maintains the desired behavior of #30091, but in a way that does not drop data when the index has duplicates.