Repository: pandas-dev/pandas
Pull request: DEPR: Series/DataFrame.transform partial failure except TypeError
- [x] closes #40211
- [x] tests added / passed
- [x] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing.html#code-standards) for how to run them
- [x] whatsnew entry

Partial revert of #40238. I did not realize how `TypeError` was handled differently than other errors in other UDF functions (https://github.com/pandas-dev/pandas/issues/40211#issuecomment-792030597). The deprecation is now only for partial failure when something other than TypeError is raised.

In order to make this work, transform needed to passthrough TypeErrors when they occur so we can correctly decide how to act with a list- or dict-like.

Problem statement:
API: DataFrame.agg has no partial failure
Synthetic example:

```
def foo(x):
    if x.name == "a":
        raise ValueError
    return x

def bar(x):
    if x.name == "a":
        raise ValueError
    return x.sum()

df = pd.DataFrame(
    {'a': [1, 2, 3], 'b': [1, 2, 3]}
)
print(df.transform([foo]))
print(df.agg([bar]))
```

The transform call results in

```
    b
  foo
0   1
1   2
2   3
```

whereas the agg call fails outright due to the `raise ValueError`. Series/DataFrame `apply` just calls `agg`. I also tried but couldn't find an example similar to transform's behavior of partial failure using `DataFrame.groupby` with apply/agg/transform, but wasn't able to (the code paths are a bit complex here, so I've resorted to blackbox testing).

My thinking here is having `transform` fail outright in the example above is the good way to go. It would be simpler from a code perspective and avoids silent failure, although perhaps it would make a user drop nuisance columns. I'll also mention that one of the things I'd like to work toward is having e.g. 

    df.transform(['mean'])

have the same performance as

    df.transform('mean')

for which having transform allow partial failure like this means there would need to be a fallback.

cc @jorisvandenbossche @jreback @jbrockmendel

Hints from issue discussion:
> My thinking here is having transform fail outright in the example above is the good way to go. It would be simpler from a code perspective and avoids silent failure

Agreed.

> DataFrame.groupby

This becomes a giant PITA bc we have a bunch of places where we pin the 'name' attribute to an object before passing it to a UDF.  I'm not aware of any documentation for this.  But it seems like the kind of thing that would mess with your examples.
Reopening because I missed the behavior difference when raising a TypeError instead of a ValueError (see below). We deprecated any partial-failure in #40238 for Series/DataFrame.transform, but we could have instead only deprecated when the exception raised is not a TypeError.

While I personally prefer not allowing any partial failure (i.e. as a user, I think it is my responsibility to select the columns that I want an operation to work on), I think we should partially revert #40238 so that only TypeErrors can give rise to partial failure in Series/DataFrame.transform (any other exception type would instead raise in a future version). This would then be consistent with most of the groupby behavior. If we want to take it a step further and also not allow TypeErrors, that can be done later.

Summary with TypeError (code below):

DataFrame.agg with callable: raised TypeError()
DataFrame.agg with list: partial failure allowed
DataFrame.transform with callable: raised AssertionError()
DataFrame.transform with list: partial failure allowed (but is deprecated in 1.3)
DataFrame.groupby(...).agg with callable: partial failure allowed
DataFrame.groupby(...).agg with list: partial failure allowed
DataFrame.groupby(...).transform with callable: partial failure allowed
DataFrame.groupby(...).transform with list: raised TypeError("unhashable type: 'list'")

Note: `df.transform(func)` where `func` is a callable will fallback to `func(df)` which is why the assertion error is raised.

Code:

```
def foo(x):
    if np.sum(np.sum(x)) < 10:
        raise TypeError
    return x

def bar(x):
    if np.sum(np.sum(x)) < 10:
        raise TypeError
    return x.sum()

df = pd.DataFrame(
    {'a': [1, 2, 3], 'b': [400, 500, 600]}
)

for use_gb in (False, True):
    for method in ('agg', 'transform'):
        for is_list in (False, True):
            obj = df
            if use_gb:
                obj = obj.groupby(level=0)
            func = foo if method == 'transform' else bar
            if is_list:
                func = [func]
            try:
                res = getattr(obj, method)(func)
                assert res.shape[1] == 1
                result = 'partial failure allowed'
            except Exception as err:
                result = f'raised {repr(err)}'
            print(
                f'DataFrame{".groupby(...)" if use_gb else ""}.{method} '
                f'{"with list" if is_list else "with callable"}: {result}'
            )
```

Commit message: Merge branch 'master' of https://github.com/pandas-dev/pandas into transform_partial_failure

 Conflicts:
	doc/source/whatsnew/v1.3.0.rst

Patch to review:
diff --git a/doc/source/whatsnew/v1.3.0.rst b/doc/source/whatsnew/v1.3.0.rst
index fad734a0e39ad..f1f4e7acc3e96 100644
--- a/doc/source/whatsnew/v1.3.0.rst
+++ b/doc/source/whatsnew/v1.3.0.rst
@@ -368,7 +368,7 @@ Deprecations
 - Using ``.astype`` to convert between ``datetime64[ns]`` dtype and :class:`DatetimeTZDtype` is deprecated and will raise in a future version, use ``obj.tz_localize`` or ``obj.dt.tz_localize`` instead (:issue:`38622`)
 - Deprecated casting ``datetime.date`` objects to ``datetime64`` when used as ``fill_value`` in :meth:`DataFrame.unstack`, :meth:`DataFrame.shift`, :meth:`Series.shift`, and :meth:`DataFrame.reindex`, pass ``pd.Timestamp(dateobj)`` instead (:issue:`39767`)
 - Deprecated :meth:`.Styler.set_na_rep` and :meth:`.Styler.set_precision` in favour of :meth:`.Styler.format` with ``na_rep`` and ``precision`` as existing and new input arguments respectively (:issue:`40134`, :issue:`40425`)
-- Deprecated allowing partial failure in :meth:`Series.transform` and :meth:`DataFrame.transform` when ``func`` is list-like or dict-like; will raise if any function fails on a column in a future version (:issue:`40211`)
+- Deprecated allowing partial failure in :meth:`Series.transform` and :meth:`DataFrame.transform` when ``func`` is list-like or dict-like and raises anything but ``TypeError``; ``func`` raising anything but a ``TypeError`` will raise in a future version (:issue:`40211`)
 - Deprecated support for ``np.ma.mrecords.MaskedRecords`` in the :class:`DataFrame` constructor, pass ``{name: data[name] for name in data.dtype.names}`` instead (:issue:`40363`)
 
 .. ---------------------------------------------------------------------------
diff --git a/pandas/core/apply.py b/pandas/core/apply.py
index 3a2c2d7124963..7577c7089af67 100644
--- a/pandas/core/apply.py
+++ b/pandas/core/apply.py
@@ -227,8 +227,11 @@ def transform(self) -> FrameOrSeriesUnion:
         func = cast(AggFuncTypeBase, func)
         try:
             result = self.transform_str_or_callable(func)
-        except Exception:
-            raise ValueError("Transform function failed")
+        except Exception as err:
+            if isinstance(err, TypeError):
+                raise err
+            else:
+                raise ValueError("Transform function failed")
 
         # Functions that transform may return empty Series/DataFrame
         # when the dtype is not appropriate
@@ -265,6 +268,7 @@ def transform_dict_like(self, func):
 
         results: Dict[Hashable, FrameOrSeriesUnion] = {}
         failed_names = []
+        all_type_errors = True
         for name, how in func.items():
             colg = obj._gotitem(name, ndim=1)
             try:
@@ -275,16 +279,18 @@ def transform_dict_like(self, func):
                     "No transform functions were provided",
                 }:
                     raise err
-                else:
+                elif not isinstance(err, TypeError):
+                    all_type_errors = False
                     failed_names.append(name)
         # combine results
         if not results:
-            raise ValueError("Transform function failed")
+            klass = TypeError if all_type_errors else ValueError
+            raise klass("Transform function failed")
         if len(failed_names) > 0:
             warnings.warn(
-                f"{failed_names} did not transform successfully. "
-                f"Allowing for partial failure is deprecated, this will raise "
-                f"a ValueError in a future version of pandas."
+                f"{failed_names} did not transform successfully and did not raise "
+                f"a TypeError. If any error is raised except for TypeError, "
+                f"this will raise in a future version of pandas. "
                 f"Drop these columns/ops to avoid this warning.",
                 FutureWarning,
                 stacklevel=4,
diff --git a/pandas/tests/apply/test_frame_transform.py b/pandas/tests/apply/test_frame_transform.py
index 212a54b78dead..5b3878f5fa690 100644
--- a/pandas/tests/apply/test_frame_transform.py
+++ b/pandas/tests/apply/test_frame_transform.py
@@ -158,35 +158,68 @@ def test_transform_method_name(method):
 # mypy doesn't allow adding lists of different types
 # https://github.com/python/mypy/issues/5492
 @pytest.mark.parametrize("op", [*frame_kernels_raise, lambda x: x + 1])
-def test_transform_bad_dtype(op, frame_or_series):
+def test_transform_bad_dtype(op, frame_or_series, request):
     # GH 35964
+    if op == "rank":
+        request.node.add_marker(
+            pytest.mark.xfail(reason="GH 40418: rank does not raise a TypeError")
+        )
+
     obj = DataFrame({"A": 3 * [object]})  # DataFrame that will fail on most transforms
     if frame_or_series is not DataFrame:
         obj = obj["A"]
 
-    msg = "Transform function failed"
-
     # tshift is deprecated
     warn = None if op != "tshift" else FutureWarning
     with tm.assert_produces_warning(warn):
-        with pytest.raises(ValueError, match=msg):
+        with pytest.raises(TypeError, match="unsupported operand|not supported"):
             obj.transform(op)
-        with pytest.raises(ValueError, match=msg):
+        with pytest.raises(TypeError, match="Transform function failed"):
             obj.transform([op])
-        with pytest.raises(ValueError, match=msg):
+        with pytest.raises(TypeError, match="Transform function failed"):
             obj.transform({"A": op})
-        with pytest.raises(ValueError, match=msg):
+        with pytest.raises(TypeError, match="Transform function failed"):
             obj.transform({"A": [op]})
 
 
 @pytest.mark.parametrize("op", frame_kernels_raise)
-def test_transform_partial_failure(op):
-    # GH 35964 & GH 40211
-    match = "Allowing for partial failure is deprecated"
+def test_transform_partial_failure_typeerror(op):
+    # GH 35964
+
+    if op == "rank":
+        pytest.skip("GH 40418: rank does not raise a TypeError")
 
     # Using object makes most transform kernels fail
     df = DataFrame({"A": 3 * [object], "B": [1, 2, 3]})
 
+    expected = df[["B"]].transform([op])
+    result = df.transform([op])
+    tm.assert_equal(result, expected)
+
+    expected = df[["B"]].transform({"B": op})
+    result = df.transform({"A": op, "B": op})
+    tm.assert_equal(result, expected)
+
+    expected = df[["B"]].transform({"B": [op]})
+    result = df.transform({"A": [op], "B": [op]})
+    tm.assert_equal(result, expected)
+
+    expected = df.transform({"A": ["shift"], "B": [op]})
+    result = df.transform({"A": [op, "shift"], "B": [op]})
+    tm.assert_equal(result, expected)
+
+
+def test_transform_partial_failure_valueerror():
+    # GH 40211
+    match = ".*did not transform successfully and did not raise a TypeError"
+
+    def op(x):
+        if np.sum(np.sum(x)) < 10:
+            raise ValueError
+        return x
+
+    df = DataFrame({"A": [1, 2, 3], "B": [400, 500, 600]})
+
     expected = df[["B"]].transform([op])
     with tm.assert_produces_warning(FutureWarning, match=match):
         result = df.transform([op])
@@ -202,6 +235,11 @@ def test_transform_partial_failure(op):
         result = df.transform({"A": [op], "B": [op]})
     tm.assert_equal(result, expected)
 
+    expected = df.transform({"A": ["shift"], "B": [op]})
+    with tm.assert_produces_warning(FutureWarning, match=match, check_stacklevel=False):
+        result = df.transform({"A": [op, "shift"], "B": [op]})
+    tm.assert_equal(result, expected)
+
 
 @pytest.mark.parametrize("use_apply", [True, False])
 def test_transform_passes_args(use_apply, frame_or_series):
diff --git a/pandas/tests/apply/test_invalid_arg.py b/pandas/tests/apply/test_invalid_arg.py
index 690d6bed0cb9b..5a8f238cd5605 100644
--- a/pandas/tests/apply/test_invalid_arg.py
+++ b/pandas/tests/apply/test_invalid_arg.py
@@ -276,7 +276,7 @@ def test_transform_none_to_type():
     # GH#34377
     df = DataFrame({"a": [None]})
     msg = "Transform function failed"
-    with pytest.raises(ValueError, match=msg):
+    with pytest.raises(TypeError, match=msg):
         df.transform({"a": int})
 
 
diff --git a/pandas/tests/apply/test_series_apply.py b/pandas/tests/apply/test_series_apply.py
index dcb5de29da320..6722fc43aa75e 100644
--- a/pandas/tests/apply/test_series_apply.py
+++ b/pandas/tests/apply/test_series_apply.py
@@ -259,29 +259,64 @@ def test_transform(string_series):
 
 @pytest.mark.parametrize("op", series_transform_kernels)
 def test_transform_partial_failure(op, request):
-    # GH 35964 & GH 40211
+    # GH 35964
     if op in ("ffill", "bfill", "pad", "backfill", "shift"):
         request.node.add_marker(
             pytest.mark.xfail(reason=f"{op} is successful on any dtype")
         )
-    match = "Allowing for partial failure is deprecated"
+    if op in ("rank", "fillna"):
+        pytest.skip(f"{op} doesn't raise TypeError on object")
 
     # Using object makes most transform kernels fail
     ser = Series(3 * [object])
 
     expected = ser.transform(["shift"])
-    with tm.assert_produces_warning(FutureWarning, match=match):
-        result = ser.transform([op, "shift"])
+    result = ser.transform([op, "shift"])
     tm.assert_equal(result, expected)
 
     expected = ser.transform({"B": "shift"})
-    with tm.assert_produces_warning(FutureWarning, match=match):
-        result = ser.transform({"A": op, "B": "shift"})
+    result = ser.transform({"A": op, "B": "shift"})
     tm.assert_equal(result, expected)
 
     expected = ser.transform({"B": ["shift"]})
+    result = ser.transform({"A": [op], "B": ["shift"]})
+    tm.assert_equal(result, expected)
+
+    expected = ser.transform({"A": ["shift"], "B": [op]})
+    result = ser.transform({"A": [op, "shift"], "B": [op]})
+    tm.assert_equal(result, expected)
+
+
+def test_transform_partial_failure_valueerror():
+    # GH 40211
+    match = ".*did not transform successfully and did not raise a TypeError"
+
+    def noop(x):
+        return x
+
+    def raising_op(_):
+        raise ValueError
+
+    ser = Series(3 * [object])
+
+    expected = ser.transform([noop])
+    with tm.assert_produces_warning(FutureWarning, match=match):
+        result = ser.transform([noop, raising_op])
+    tm.assert_equal(result, expected)
+
+    expected = ser.transform({"B": noop})
     with tm.assert_produces_warning(FutureWarning, match=match):
-        result = ser.transform({"A": [op], "B": ["shift"]})
+        result = ser.transform({"A": raising_op, "B": noop})
+    tm.assert_equal(result, expected)
+
+    expected = ser.transform({"B": [noop]})
+    with tm.assert_produces_warning(FutureWarning, match=match):
+        result = ser.transform({"A": [raising_op], "B": [noop]})
+    tm.assert_equal(result, expected)
+
+    expected = ser.transform({"A": [noop], "B": [noop]})
+    with tm.assert_produces_warning(FutureWarning, match=match, check_stacklevel=False):
+        result = ser.transform({"A": [noop, raising_op], "B": [noop]})
     tm.assert_equal(result, expected)
 
 


Code review comments:
diff_hunk: @@ -227,8 +227,11 @@ def transform(self) -> FrameOrSeriesUnion:
         func = cast(AggFuncTypeBase, func)
         try:
             result = self.transform_str_or_callable(func)
-        except Exception:
-            raise ValueError("Transform function failed")
+        except Exception as err:
+            if isinstance(err, TypeError):
path: pandas/core/apply.py
text: @user1:
i think more idiomatic to do as 

```
except TypeError:
    raise
except Exception as err:
    raise ValueError("Transform function failed") from err
```

@author:
Thanks - agreed. Change made.