DEPR: non-keyword arguments in any
State: closed

precursor to #44802 

**What does this implement/fix? Explain your changes.**
I used `@overload` definitions to specify that `DataFrame.any` should return type `Series` and `Series.any` should have return type `bool`.

**Any other comments?**
I am currently getting errors on checking with `mypy`, `The erased type of self "pandas.core.frame.DataFrame" is not a supertype of its class "pandas.core.generic.NDFrame"`. 

So I am currently trying to figure out how to resolve this error and whether using `@overload` is the correct direction to do this. Mentioning this pull request as **WIP**.

Labels:
name: Deprecate
description: Functionality to remove in pandas

Comments:
> **Any other comments?** I am currently getting errors on checking with `mypy`, `The erased type of self "pandas.core.frame.DataFrame" is not a supertype of its class "pandas.core.generic.NDFrame"`.

If I understand the error correctly: any is a method of NDArray, but your overloads do now only cover DataFrame and Series - NDFrame is not covered by the overloads (since neither of them is a super-class for NDFrame).

I think you will need a third overload that is compatible with the return type of the Series and DataFrame overload. Not sure whether this works:
```py
    @overload
    def any(
        self: NDFrame,
        axis: Axis = ...,
        bool_only: bool_t | None = ...,
        skipna: bool_t = ...,
        level: Level | None = ...,
        **kwargs,
    ) -> Series | bool_t:
        ...


```
@twoertwein Thanks for the suggestion. 
But the error still persists after I make the suggested changes. 
Why not define `any` in both `pandas/core/frame.py` and `pandas/core/series.py`, in which they each have the correct return type and call `super().any`?
> Why not define `any` in both `pandas/core/frame.py` and `pandas/core/series.py`, in which they each have the correct return type and call `super().any`?

Thanks @MarcoGorelli 
I was working on this approach exactly, once the `@overload` approach didn't seem to work. 
> Why not define `any` in both `pandas/core/frame.py` and `pandas/core/series.py`, in which they each have the correct return type and call `super().any`?

If the mypy error is a false-positive, I would prefer adding two ignore comments compared to "implementing" `any` in the sub-classes. This has the advantage that as soon as mypy is updated, we can remove the ignore comments. We don't get an automatic reminder if we implement `any` in the sub-classes.

At least in the toy example of https://github.com/python/mypy/issues/11759, the return types are correctly inferred even though mypy complains.
@twoertwein true, but there's the same construct for `replace`, where both `Series.replace` and `DataFrame.replace` just call `super().replace`, if I recall correctly

> This has the advantage that as soon as mypy is updated, we can remove the ignore comments.

Sure, but even then, what's the advantage of doing this via overloads rather than by overriding `any` in `DataFrame` and `Series`?

Even without the `mypy` false-positive, using overloads requires adding 3 overloads, while overriding in the child classes only requires 2 extra function definitions.

In terms of lines of code which need adding, overriding `any` adds fewer
Thank you @MarcoGorelli , I agree `super().any(...)` is the better solution. I was hoping that typing should ideally not change the implementation.
Nice, thanks! Could you also post screenshots of the docs for `Series.any` and `DataFrame.any`?

You can use `--single` for this, see https://pandas.pydata.org/docs/development/contributing_documentation.html#building-the-documentation

> 
> ```
> # compile the reference docs for a single function
> python make.py clean
> python make.py --single pandas.DataFrame.join
> ```
`DataFrame.any` 
![image](https://user-images.githubusercontent.com/40866128/146383422-48aad20b-a7fb-4cf3-87da-4ca825e2caed.png)

`Series.any`
![image](https://user-images.githubusercontent.com/40866128/146383799-86262169-22ea-410d-a223-c5aa9e2f1579.png)

There is a mismatch between the doc-string and the type annotations: The type annotations say that DataFrame.any returns a Series (doc: Series or DataFrame) and Seriesn.any returns a bool (doc: Series or scalar).
> There is a mismatch between the doc-string and the type annotations: The type annotations say that DataFrame.any returns a Series (doc: Series or DataFrame) and Seriesn.any returns a bool (doc: Series or scalar).

Yes, you're right!

So, the this needs some overloads too, to indicate that the return type depends on whether `level` was specified
`Overloaded function signatures 1 and 2 overlap with incompatible return types`
What could be a way to define these overload definitions without overlapping signatures as per our requirement?
ah, it's because `level` takes default values, and can be passed both positionally and as keyword, and it has default arguments which come before it. This makes overloading complicated, check https://medium.com/analytics-vidhya/making-sense-of-typing-overload-437e6deecade 

So, I'd suggest:
1. deprecate non-keyword arguments in `any`, see https://github.com/pandas-dev/pandas/pull/41523/files as an example
2. for now, the return types can be `DataFrame | Series` and `Series | bool`
3. in a follow-up PR (or, in the next version of pandas, once non-keyword arguments in `any` have been deprecated), overload the return types depending on the value of `level`

This has been a bit more challenging that I originally thought, but I hope you find it worthwhile and learn a few new things!
`Level` is defined as `Union[Hashable, int]` but `None` is also hashable, so they actually overlap.
@MarcoGorelli 
Do I also need to add a what's new note and test?
This pull request is stale because it has been open for thirty days with no activity. Please [update](https://pandas.pydata.org/pandas-docs/stable/development/contributing.html#updating-your-pull-request) and respond to this comment if you're still interested in working on this.
> This pull request is stale because it has been open for thirty days with no activity. Please [update](https://pandas.pydata.org/pandas-docs/stable/development/contributing.html#updating-your-pull-request) and respond to this comment if you're still interested in working on this.

Still working, busy with some academic work.
Currently the tests are not passing due to "Warning not set with correct stack level".

Also the warning is triggered even for `df.any()`.
> Also the warning is triggered even for `df.any()`.

ooh, you're right. What if the decorator is added to

https://github.com/pandas-dev/pandas/blob/0141d9212bfafde28d83c6e74605635952cde3e8/pandas/core/generic.py#L10945-L10946

? Maybe `deprecate_non_keyword_arguments` needs an extra `name` argument then, because if we put the decorator there then the warning will show something funny
> ? Maybe `deprecate_non_keyword_arguments` needs an extra `name` argument then, because if we put the decorator there then the warning will show something funny

Yes, it mentions `NDFrame._add_numeric_operations.<locals>.any` instead of `NDFrame.any` in the warning now. 😅 


😂 

I think a possible solution would be:
- add a `name` argument to `deprecate_nonkeyword_arguments`, which is `None` by default. If it's not `None`, then in `deprecate_nonkeyword_arguments`, then the warning message will use that instead of `func.__qualname__`
- in the `deprecate_nonkeyword_arguments` decorator for this function, pass `name="DataFrame.any and Series.any"`

Reckon this'd work fine?
> Nice, looks good to me pending green!
> 
> Once this is in, if you wanted to make a separate PR to sort out the return types, that'd be welcome - I promise the review will be faster this time 😄
> 

Yes, I will soon create a new PR for the return types.

Thanks for the review and sharing the lovely cat 🐱 GIF.
Thanks @MarcoGorelli and @twoertwein for the reviews 😄 
What needs to be changed for the failing checks?
> What needs to be changed for the failing checks?

You'll need to update those tests such that they don't throw a warning

```
=================================== FAILURES ===================================
__________________ TestDataFrameAnalytics.test_any_all_extra ___________________
[gw1] linux -- Python 3.10.4 /usr/share/miniconda/envs/pandas-dev/bin/python

self = <pandas.tests.frame.test_reductions.TestDataFrameAnalytics object at 0x7feb4634aa40>

    def test_any_all_extra(self):
        df = DataFrame(
            {
                "A": [True, False, False],
                "B": [True, True, False],
                "C": [True, True, True],
            },
            index=["a", "b", "c"],
        )
>       result = df[["A", "B"]].any(1)

pandas/tests/frame/test_reductions.py:1058: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

args = (       A      B
a   True   True
b  False   True
c  False  False, 1)
kwargs = {}, arguments = ''

    @wraps(func)
    def wrapper(*args, **kwargs):
        arguments = _format_argument_list(allow_args)
        if len(args) > num_allow_args:
>           warnings.warn(
                msg.format(arguments=arguments),
                FutureWarning,
                stacklevel=stacklevel,
            )
E           FutureWarning: In a future version of pandas all arguments of DataFrame.any and Series.any will be keyword-only.

pandas/util/_decorators.py:[31](https://github.com/pandas-dev/pandas/runs/5830982510?check_suite_focus=true#step:11:31)2: FutureWarning
```

So, change `result = df[["A", "B"]].any(1)` to `result = df[["A", "B"]].any(axis=1)`, and likewise for the others
What is the tool used in the tests for docstring validation?
> What is the tool used in the tests for docstring validation?

numpydoc

There was a failure on the main branch but it's been fixed now (see https://github.com/pandas-dev/pandas/pull/46690)
I don't know how to solve the current failing checks.
Some of them show `FileNotFoundError`. Could the given paths be wrong 🤔 ? (in `doc/source/user_guide/io.rst`)
They're unrelated, don't worry about them
lgm @MarcoGorelli over to you