Repository: pandas-dev/pandas
Pull request: BUG: Fix agg ingore arg/kwargs when given list like func
- [x] closes #50624  (Replace xxxx with the GitHub issue number)
- [x] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [x] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [x] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.

Previous [Pull Request #50725](https://github.com/pandas-dev/pandas/pull/50725), for multiple elements-func list this time.

I would appreciate any feedback or suggestions on this draft pull request to help improve it. Thank you in advance for your help and cooperation.

Problem statement:
BUG: agg ignores args/kwargs when giving a list-like argument
I think the expected value of `(a, b)` in the function below is `(5, 6)`.

```
def foo(x, a=1, b=2):
    print(a, b)
    return x

df = pd.DataFrame({'x': [1, 2, 3]})
# 0 is for the axis argument; 5 is the first (and only) in args
print(df.agg([foo], 0, 5, b=6))
# 1 2
# 1 2
# 1 2
```

This issue may also persist with apply and transform, in DataFrame, Series, DataFrameGroupBy, SeriesGroupBy, Rolling, and Resampler, and with other types of arguments (single and/or dict-likes). This should be fully investigated to determine where args / kwargs are ignored.

Hints from issue discussion:
take
I have just resolved the issue for the given single-element function list. However, addressing the issue for a list containing multiple functions may require further considerations, such as potential naming conflicts and additional requirements. It may be necessary to have further discussions and potentially make adjustments to the original API design.

Commit message: Fix test failure

Patch to review:
diff --git a/pandas/core/apply.py b/pandas/core/apply.py
index c28da1bc758cd..6eee935389bd7 100644
--- a/pandas/core/apply.py
+++ b/pandas/core/apply.py
@@ -5,6 +5,7 @@
 from contextlib import nullcontext
 from functools import partial
 import inspect
+from types import FunctionType
 from typing import (
     TYPE_CHECKING,
     Any,
@@ -330,9 +331,18 @@ def agg_list_like(self) -> DataFrame | Series:
             # degenerate case
             if selected_obj.ndim == 1:
 
+                args_remain, kwargs_remain = self.args, self.kwargs
                 for a in arg:
                     colg = obj._gotitem(selected_obj.name, ndim=1, subset=selected_obj)
-                    new_res = colg.aggregate(a)
+                    if not isinstance(a, (np.ufunc, str)) and isinstance(
+                        a, FunctionType
+                    ):
+                        args_pass, kwargs_pass, args_remain, kwargs_remain = _map_args(
+                            a, args_remain, kwargs_remain
+                        )
+                        new_res = colg.aggregate(a, *args_pass, **kwargs_pass)
+                    else:
+                        new_res = colg.aggregate(a)
                     results.append(new_res)
 
                     # make sure we find a good name
@@ -344,7 +354,7 @@ def agg_list_like(self) -> DataFrame | Series:
                 indices = []
                 for index, col in enumerate(selected_obj):
                     colg = obj._gotitem(col, ndim=1, subset=selected_obj.iloc[:, index])
-                    new_res = colg.aggregate(arg)
+                    new_res = colg.aggregate(arg, *self.args, **self.kwargs)
                     results.append(new_res)
                     indices.append(index)
                 keys = selected_obj.columns.take(indices)
@@ -1506,3 +1516,54 @@ def validate_func_kwargs(
         no_arg_message = "Must provide 'func' or named aggregation **kwargs."
         raise TypeError(no_arg_message)
     return columns, func
+
+
+def _map_args(
+    func: AggFuncType, args: tuple, kwargs: dict
+) -> tuple[tuple, dict, tuple, dict]:
+    # GH 50624
+    """
+    Map arguments to function.
+    But for some cases with unnamed arguments, it will cause error.
+
+    Parameters
+    ----------
+    func : function
+    args : tuple
+    kwargs : dict
+
+    Returns
+    -------
+    args_pass : tuple
+        Args should be passed to func
+    kwargs_pass : dict
+        Kwargs should be passed to func
+    args_remain : tuple
+        Args should be passed to other functions
+    kwargs_remain : dict
+        Kwargs should be passed to other functions
+
+    Examples
+    --------
+    >>> def f(a=1, b=2):
+    ...     return a, b
+    >>> _map_args(f, (1,), {'b': 2, 'c': 3})
+    ((1,), {'b': 2}, (), {'c': 3})
+    >>> _map_args(f, (1, 2, 3), {'b': 4}) # maybe some unexpected results
+    ((1, 2), {}, (3,), {'b':4})
+    """
+    argspec = inspect.getfullargspec(func)
+    args_names = argspec.args + argspec.kwonlyargs
+
+    if len(args) >= len(args_names):
+        args_pass = args[: len(args_names)]
+        args_remain = args[len(args_names) :]
+        kwargs_pass = {}
+        kwargs_remain = kwargs
+    else:
+        args_pass = args
+        args_remain = ()
+        kwargs_pass = {k: v for k, v in kwargs.items() if k in args_names}
+        kwargs_remain = {k: v for k, v in kwargs.items() if k not in args_names}
+
+    return args_pass, kwargs_pass, args_remain, kwargs_remain
diff --git a/pandas/tests/apply/test_frame_apply.py b/pandas/tests/apply/test_frame_apply.py
index 7aaad4d2ad081..df84fadc643ec 100644
--- a/pandas/tests/apply/test_frame_apply.py
+++ b/pandas/tests/apply/test_frame_apply.py
@@ -1623,3 +1623,22 @@ def test_any_apply_keyword_non_zero_axis_regression():
 
     result = df.apply("any", 1)
     tm.assert_series_equal(result, expected)
+
+
+def test_agg_list_like_arg():
+    # GH 50624
+    df = DataFrame({"x": [1, 2, 3]})
+
+    def foo1(x, a=1, b=2):
+        return x + a + b
+
+    def foo2(x, c=3, d=4):
+        return x + c + d
+
+    result = df.agg([foo1, foo2], 0, 5, b=6, c=7, d=8)
+    expected = DataFrame(
+        [[12, 16], [13, 17], [14, 18]],
+        columns=MultiIndex.from_tuples([("x", "foo1"), ("x", "foo2")]),
+        index=[0, 1, 2],
+    )
+    tm.assert_frame_equal(result, expected)


Code review comments:
diff_hunk: @@ -330,9 +331,18 @@ def agg_list_like(self) -> DataFrame | Series:
             # degenerate case
             if selected_obj.ndim == 1:
 
+                args_remain, kwargs_remain = self.args, self.kwargs
                 for a in arg:
                     colg = obj._gotitem(selected_obj.name, ndim=1, subset=selected_obj)
-                    new_res = colg.aggregate(a)
+                    if not isinstance(a, (np.ufunc, str)) and isinstance(
+                        a, FunctionType
+                    ):
+                        args_pass, kwargs_pass, args_remain, kwargs_remain = _map_args(
+                            a, args_remain, kwargs_remain
+                        )
+                        new_res = colg.aggregate(a, *args_pass, **kwargs_pass)
+                    else:
+                        new_res = colg.aggregate(a)
path: pandas/core/apply.py
text: @user1:
I think we should always pass args, kwargs to the function, regardless of type and regardless of whether or not the function accepts them. If the user is passing things the function doesn't accept, then we should raise. Does this break something?

@author:
When we pass `df.agg([foo1, foo2], 0, 5, b=6, c=7, d=8)`, `5, b=6` for `foo1` and `c=7, d=8` for `foo2`, the `self.args = (5,)` and `self.kwags={'b': 6, 'c': 7, 'd': 8}`. 

I think the key is how do we split args(`self.arg, self.kwargs`) for the list-like functions(`arg`), and pass them to the corresponding function(`a`).

The func `_map_args` is aim to spilt `arg_pass = (5,) kwargs_pass = {'b': 6}` for `foo1`, and remainder could pass to `foo2`.  

This is the botched method I found, and it is true that it will only ignore incorrect args instead raise error.

@user1:
> I think the key is how do we split args(`self.arg, self.kwargs`) for the list-like functions(`arg`), and pass them to the corresponding function(`a`).

I don't believe we should split args between functions. When a list is passed, I believe the args and kwargs should be passed to all functions in that list. E.g. `df.agg([foo1, foo2], 0, 5, b=6, c=7, d=8)` should pass `args=(5,)` and `kwargs={'b': 6, 'c': 7, 'd': 8}` to both `foo1` and `foo2`.

@author:
I'm sorry I still don't fully understand what you're saying. 

When we pass `kwargs={'b': 6, 'c': 7, 'd': 8}` to `foo1` which `def foo1(a, b)`, how could we deal with the `TypeError` caused by unexpected keyword argument?  Raise? It doesn't seem quite right, as they are for the next `foo2`.

@user1:
> how could we deal with the `TypeError` caused by unexpected keyword argument? Raise? It doesn't seem quite right, as they are for the next `foo2`.

Yes - I think it should raise. If foo takes kwargs, then we can't tell if they whether or not they are for `foo`. Or perhaps the user has a typo in the kwargs that were provided. We'd have to guess.

I think `.agg([foo1, foo2], *args, **kwargs)` should take the results of `.agg(foo1, *args, **kwargs)` and `.agg(foo2, *args, **kwargs)` and combine the results. If the user provides functions which don't take the arguments provided, then it should raise. This way there is no guessing and no odd edge cases. It is easier for us to implement and easier for the user to reason about.

@author:
When user call `df.agg([foo1, foo2], 0, 5, b=6, c=7, d=8)` for `def foo1(a, b), def foo2(c, d)`, raise an error cased by unexpected argument is what we want? 

In other words, when user call `.agg([foo1, foo2], *args, **kwargs)`, only if `args, kwargs` can be passed and uesd by both `foo1, foo2` at the same time, we let it work, otherwise an error should be raised.

The example below is what we want: 
```python
def foo1(x, a=1, c=0):
    print(f"foo1: {a}, {c}")
    return x

def foo2(x, b=2, c=0):
    print(f"foo2: {b}, {c}")
    return x

df = pd.DataFrame({'x': [1]})
df.agg([foo1, foo2], 0, 3, c=4)
# foo1: 3, 4
# foo2: 3, 4
```

Do I understand this properly?

@user1:
Yes - that is correct. Note that in order to accomplish this, we can just always pass *args and **kwargs to the function; no introspection of the functions and their arguments is necessary.