Repository: pandas-dev/pandas
Pull request: REG: DataFrame/Series.transform with list and non-list dict values
- [x] closes #40018
- [x] tests added / passed
- [x] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing.html#code-standards) for how to run them
- [x] whatsnew entry

Currently on master, `agg` does some preprocessing on dict-like arguments that `transform` does not, leading to incorrect behavior in `transform`. This PR moves `agg`'s preprocessing into `validate_dictlike_arg` which is used by both `agg` and `transform`, fixing the issue.

As identified [in #40018](https://github.com/pandas-dev/pandas/issues/40018#issuecomment-785678614), this is a regression from 1.1 -> 1.2 (almost certainly from one of my PRs, but I haven't checked which). Backporting this to 1.2 will probably require a bit of a different fix.

Problem statement:
BUG: DataFrame/Series.transform with list and non-list dict values
```
df = pd.DataFrame({'a': [1, 2], 'b': [1, 4], 'c': [1, 4]})
print(df.transform({'b': ['sqrt', 'abs'], 'c': 'sqrt'}))
print(df['a'].transform({'b': ['sqrt', 'abs'], 'c': 'sqrt'}))
```

produces

```
     b        c
  sqrt abs    c
0  1.0   1  1.0
1  2.0   4  2.0
          b             c
       sqrt abs         a
0  1.000000   1  1.000000
1  1.414214   2  1.414214
```

In each case the last column label on the 2nd level should be "sqrt" and not "a" or "c".

Hints from issue discussion:
In which version are you getting this? I am running them on `pandas==1.1.5` and the first case is producing the correct output. 
The second case errors out as 

`---------------------------------------------------------------------------
SpecificationError                        Traceback (most recent call last)
<ipython-input-2-e206dee29221> in <module>
----> 1 print(df['a'].transform({'b': ['sqrt', 'abs'], 'c': 'sqrt'}))

/opt/conda/lib/python3.8/site-packages/pandas/core/series.py in transform(self, func, axis, *args, **kwargs)
   4080         # Validate the axis parameter
   4081         self._get_axis_number(axis)
-> 4082         return super().transform(func, *args, **kwargs)
   4083 
   4084     def apply(self, func, convert_dtype=True, args=(), **kwds):

/opt/conda/lib/python3.8/site-packages/pandas/core/generic.py in transform(self, func, *args, **kwargs)
  10674         2  1.414214   7.389056
  10675         """
> 10676         result = self.agg(func, *args, **kwargs)
  10677         if is_scalar(result) or len(result) != len(self):
  10678             raise ValueError("transforms cannot produce aggregated results")

/opt/conda/lib/python3.8/site-packages/pandas/core/series.py in aggregate(self, func, axis, *args, **kwargs)
   4047             func = dict(kwargs.items())
   4048 
-> 4049         result, how = self._aggregate(func, *args, **kwargs)
   4050         if result is None:
   4051 

/opt/conda/lib/python3.8/site-packages/pandas/core/base.py in _aggregate(self, arg, *args, **kwargs)
    338                         raise SpecificationError("nested renamer is not supported")
    339                     elif isinstance(obj, ABCSeries):
--> 340                         raise SpecificationError("nested renamer is not supported")
    341                     elif isinstance(obj, ABCDataFrame) and k not in obj.columns:
    342                         raise KeyError(f"Column '{k}' does not exist!")

SpecificationError: nested renamer is not supported
`
Thanks @smohr003! I've verified your reported behavior on 1.1.x and that the code matches my reported behavior on 1.2.x (and master). The first output I would classify as a regression, but the 2nd behavior (using on a Series) is the correct one. That is, passing a dictionary to Series.transform should not raise.

Commit message: BUG: DataFrame/Series.transform with list and non-list dict values

Patch to review:
diff --git a/doc/source/whatsnew/v1.3.0.rst b/doc/source/whatsnew/v1.3.0.rst
index 32a2514b3b6a3..54f5ea720f673 100644
--- a/doc/source/whatsnew/v1.3.0.rst
+++ b/doc/source/whatsnew/v1.3.0.rst
@@ -327,6 +327,7 @@ Numeric
 - Bug in :func:`select_dtypes` different behavior between Windows and Linux with ``include="int"`` (:issue:`36569`)
 - Bug in :meth:`DataFrame.apply` and :meth:`DataFrame.agg` when passed argument ``func="size"`` would operate on the entire ``DataFrame`` instead of rows or columns (:issue:`39934`)
 - Bug in :meth:`DataFrame.transform` would raise ``SpecificationError`` when passed a dictionary and columns were missing; will now raise a ``KeyError`` instead (:issue:`40004`)
+- Bug in :meth:`DataFrame.transform` and :meth:`Series.transform` would have incorrect column labels when passed a dictionary with a mix of list and non-list values (:issue:`40018`)
 -
 
 Conversion
diff --git a/pandas/core/apply.py b/pandas/core/apply.py
index db4203e5158ef..3ea07e4425a51 100644
--- a/pandas/core/apply.py
+++ b/pandas/core/apply.py
@@ -280,7 +280,7 @@ def transform_dict_like(self, func):
         if len(func) == 0:
             raise ValueError("No transform functions were provided")
 
-        self.validate_dictlike_arg("transform", obj, func)
+        func = self.validate_dictlike_arg("transform", obj, func)
 
         results: Dict[Hashable, FrameOrSeriesUnion] = {}
         for name, how in func.items():
@@ -421,32 +421,17 @@ def agg_dict_like(self, _axis: int) -> FrameOrSeriesUnion:
         -------
         Result of aggregation.
         """
+        from pandas.core.reshape.concat import concat
+
         obj = self.obj
         arg = cast(AggFuncTypeDict, self.f)
 
-        is_aggregator = lambda x: isinstance(x, (list, tuple, dict))
-
         if _axis != 0:  # pragma: no cover
             raise ValueError("Can only pass dict with axis=0")
 
         selected_obj = obj._selected_obj
 
-        self.validate_dictlike_arg("agg", selected_obj, arg)
-
-        # if we have a dict of any non-scalars
-        # eg. {'A' : ['mean']}, normalize all to
-        # be list-likes
-        # Cannot use arg.values() because arg may be a Series
-        if any(is_aggregator(x) for _, x in arg.items()):
-            new_arg: AggFuncTypeDict = {}
-            for k, v in arg.items():
-                if not isinstance(v, (tuple, list, dict)):
-                    new_arg[k] = [v]
-                else:
-                    new_arg[k] = v
-            arg = new_arg
-
-        from pandas.core.reshape.concat import concat
+        arg = self.validate_dictlike_arg("agg", selected_obj, arg)
 
         if selected_obj.ndim == 1:
             # key only used for output
@@ -542,12 +527,13 @@ def maybe_apply_multiple(self) -> Optional[FrameOrSeriesUnion]:
 
     def validate_dictlike_arg(
         self, how: str, obj: FrameOrSeriesUnion, func: AggFuncTypeDict
-    ) -> None:
+    ) -> AggFuncTypeDict:
         """
-        Raise if dict-like argument is invalid.
+        Handler for dict-like argument.
 
         Ensures that necessary columns exist if obj is a DataFrame, and
-        that a nested renamer is not passed.
+        that a nested renamer is not passed. Also normalizes to all lists
+        when values consists of a mix of list and non-lists.
         """
         assert how in ("apply", "agg", "transform")
 
@@ -567,6 +553,23 @@ def validate_dictlike_arg(
                 cols_sorted = list(safe_sort(list(cols)))
                 raise KeyError(f"Column(s) {cols_sorted} do not exist")
 
+        is_aggregator = lambda x: isinstance(x, (list, tuple, dict))
+
+        # if we have a dict of any non-scalars
+        # eg. {'A' : ['mean']}, normalize all to
+        # be list-likes
+        # Cannot use func.values() because arg may be a Series
+        if any(is_aggregator(x) for _, x in func.items()):
+            new_func: AggFuncTypeDict = {}
+            for k, v in func.items():
+                if not is_aggregator(v):
+                    # mypy can't realize v is not a list here
+                    new_func[k] = [v]  # type:ignore[list-item]
+                else:
+                    new_func[k] = v
+            func = new_func
+        return func
+
 
 class FrameApply(Apply):
     obj: DataFrame
diff --git a/pandas/tests/apply/test_frame_transform.py b/pandas/tests/apply/test_frame_transform.py
index 1888ddd8ec4aa..47bc69656a597 100644
--- a/pandas/tests/apply/test_frame_transform.py
+++ b/pandas/tests/apply/test_frame_transform.py
@@ -103,6 +103,17 @@ def test_transform_dictlike(axis, float_frame, box):
     tm.assert_frame_equal(result, expected)
 
 
+def test_transform_dictlike_mixed():
+    # GH 40018 - mix of lists and non-lists in values of a dictionary
+    df = DataFrame({"a": [1, 2], "b": [1, 4], "c": [1, 4]})
+    result = df.transform({"b": ["sqrt", "abs"], "c": "sqrt"})
+    expected = DataFrame(
+        [[1.0, 1, 1.0], [2.0, 4, 2.0]],
+        columns=MultiIndex([("b", "c"), ("sqrt", "abs")], [(0, 0, 1), (0, 1, 0)]),
+    )
+    tm.assert_frame_equal(result, expected)
+
+
 @pytest.mark.parametrize(
     "ops",
     [
diff --git a/pandas/tests/apply/test_series_transform.py b/pandas/tests/apply/test_series_transform.py
index e67ea4f14e4ac..24d619cb2bbb1 100644
--- a/pandas/tests/apply/test_series_transform.py
+++ b/pandas/tests/apply/test_series_transform.py
@@ -2,6 +2,8 @@
 import pytest
 
 from pandas import (
+    DataFrame,
+    MultiIndex,
     Series,
     concat,
 )
@@ -55,6 +57,17 @@ def test_transform_dictlike(string_series, box):
     tm.assert_frame_equal(result, expected)
 
 
+def test_transform_dictlike_mixed():
+    # GH 40018 - mix of lists and non-lists in values of a dictionary
+    df = Series([1, 4])
+    result = df.transform({"b": ["sqrt", "abs"], "c": "sqrt"})
+    expected = DataFrame(
+        [[1.0, 1, 1.0], [2.0, 4, 2.0]],
+        columns=MultiIndex([("b", "c"), ("sqrt", "abs")], [(0, 0, 1), (0, 1, 0)]),
+    )
+    tm.assert_frame_equal(result, expected)
+
+
 def test_transform_wont_agg(string_series):
     # GH 35964
     # we are trying to transform with an aggregator


Code review comments:
diff_hunk: @@ -327,6 +327,7 @@ Numeric
 - Bug in :func:`select_dtypes` different behavior between Windows and Linux with ``include="int"`` (:issue:`36569`)
 - Bug in :meth:`DataFrame.apply` and :meth:`DataFrame.agg` when passed argument ``func="size"`` would operate on the entire ``DataFrame`` instead of rows or columns (:issue:`39934`)
 - Bug in :meth:`DataFrame.transform` would raise ``SpecificationError`` when passed a dictionary and columns were missing; will now raise a ``KeyError`` instead (:issue:`40004`)
+- Bug in :meth:`DataFrame.transform` and :meth:`Series.transform` would have incorrect column labels when passed a dictionary with a mix of list and non-list values (:issue:`40018`)
path: doc/source/whatsnew/v1.3.0.rst
text: @user1:
can you put in 1.2.3