Repository: pandas-dev/pandas

Issue:
API: What is the rationale for numeric_only of Categorical reductions?
Consider an ordered Categorical with missing values:

```
In [32]: cat = pd.Categorical(['a', np.nan, 'b', 'a'], ordered=True)

In [33]: cat.min()
Out[33]: nan

In [34]: cat.max()
Out[34]: 'b'

In [35]: cat.min(numeric_only=True)
Out[35]: 'a'

In [36]: cat.max(numeric_only=True)
Out[36]: 'b'

In [37]: cat.min(numeric_only=False)
Out[37]: nan

In [38]: cat.max(numeric_only=False)
Out[38]: 'b'
```

So from the observation above (and from the code: https://github.com/pandas-dev/pandas/blob/a89e19d59e0bff2d02e4647af1904e2c9701dd5f/pandas/core/arrays/categorical.py#L2199), it seems that `numeric_only` means that only the actual categories should be considered, and not the missing values (so codes that are not -1).

This struck me as strange, for the following reasons:

* The fact that -1 is used as the code for missing data is rather an implementation detail, but now actually determines min/max behaviour (missing value is always the minimum, but never the maximum, unless there are only missing values)

* This behaviour is different than the default for other data types in pandas, which is skipping missing values by default:

    ```
    In [1]: s = pd.Series([1, np.nan, 2, 1])  

    In [2]: s.min()
    Out[2]: 1.0

    In [3]: s.astype(pd.CategoricalDtype(ordered=True)).min()
    Out[3]: nan

    In [5]: s.min(skipna=False)
    Out[5]: nan
    ```

* The keyword in pandas to determine whether NaNs should be skipped or not for reductions is `skipna=True/False`, not `numeric_only` (this also means the `skipna` keyword for categorical series is broken / has no effect). 
  Apart from that, the name "numeric_only" is also strange to me to mean this (and is also not documented).

* The `numeric_only` keyword in reductions methods of DataFrame actually means something entirely different: should full columns be excluded from the result based on their dtype. 
  
    ```
    In [63]: cat = pd.Categorical(['a', np.nan, 'b', 'a'], ordered=True)

    In [64]: pd.Series(cat).min(numeric_only=True)
    Out[64]: 'a'

    In [65]: pd.DataFrame({'cat': cat}).min(numeric_only=True)
    Out[65]: Series([], dtype: float64)
    ```

From the above list, I don't see a good reason for having `numeric_only=False` as 1) the default behaviour and 2) altogether as an option (instead of skipna). But it seems this was implemented rather from the beginning that Categoricals were introduced.

*Am I missing something?* 
*Is there a reason we don't skip NaNs by default for Categorical?* 

Would it be an idea to deprecate `numeric_only` in favor of `skipna` and deprecate the default?

cc @jreback @jankatins 





Discussion:
To be honest I've never fully understood what `numeric_only` is supposed to represent even in other parts of the codebase (ex: groupby ops). That is, conceptually I understand it but not sure all the impacts and nuances to it are well communicated.

I haven't run into the issue described personally so not sure how much weight my opinion should carry here, but I would be OK with the deprecation you mention
All valid points!

- I think the difference in behavior of `min` and `max` when nans are present is arbitrary. I would expect both `min` and `max` to return nan, not only `min`. I don't know what the behavior is for other Series / Dataframes, so let's check that first and keep it consistent
- My thinking was, since numeric_only seems to be broken in `0.24.0`, why not move already to `skipna` without fixing and deprecating `numeric_only`
- Another option would be to support them both temporarily (and make one have precedence over the other), and deprecate `numric_only`, however the defaults behavior is different (`skipna` defaults to `True`, while `numeric_only` defaults to `None`)
>  I would expect both `min` and `max` to return nan, not only `min`

To be consistent with the rest of pandas, I think they both should *skip* NaNs by default, but if you say `skipna=False`, both to return NaN.

I think the the practical deprecation path could look like:

- if somebody specifies `numeric_only` (for the categorical case), raise a deprecation warning to say they need to specify `skipna` instead.
- in case of `min` ànd if there are NaNs present (the case that would change default behaviour), raise a warning that this will change and say that they can specify `skipna` to silence the warning and already have the future behaviour.

So that would indeed mean supporting both keywords temporarily.


I've updated the PR. 

This will keep the old behavior if `numeric_only` is specified, but it will change the default behavior if it is not specified. This is a bit harder to fix, since the default for `skipna` is `True`, so it's impossible to distinguish if it was set or not. See https://github.com/pandas-dev/pandas/blob/master/pandas/core/series.py#L3664 (it's already converted from `None` to `True` here https://github.com/pandas-dev/pandas/blob/master/pandas/core/generic.py#L10950)
> This is a bit harder to fix, since the default for `skipna` is `True`, so it's impossible to distinguish if it was set or not

Ah, yes, that's a complication. We might be able to push the conversion from None to True a bit down into `_reduce`, where we could special case the categorical one (as you are already doing in the PR to deal with the numeric/only/skipna)
This is probably ok to remove / deprecate this for *only* ``Categorical``; the PR #25304 is not acceptable regardless of this change.
> ok to remove / deprecate this for _only_ `Categorical`

It is *only* implemented for Categorical, no other array has this keyword, so no problem to only deprecate it there. The `numeric_only` keyword for DataFrame reductions is basically unrelated (except for the fact that it has the same name).

Resolving patch:
diff --git a/pandas/core/arrays/categorical.py b/pandas/core/arrays/categorical.py
--- a/pandas/core/arrays/categorical.py
+++ b/pandas/core/arrays/categorical.py
@@ -2157,7 +2157,8 @@ def _reduce(self, name, axis=0, **kwargs):
             raise TypeError(f"Categorical cannot perform the operation {name}")
         return func(**kwargs)
 
-    def min(self, numeric_only=None, **kwargs):
+    @deprecate_kwarg(old_arg_name="numeric_only", new_arg_name="skipna")
+    def min(self, skipna=True):
         """
         The minimum value of the object.
 
@@ -2173,17 +2174,18 @@ def min(self, numeric_only=None, **kwargs):
         min : the minimum of this `Categorical`
         """
         self.check_for_ordered("min")
-        if numeric_only:
-            good = self._codes != -1
-            pointer = self._codes[good].min(**kwargs)
-        else:
-            pointer = self._codes.min(**kwargs)
-        if pointer == -1:
-            return np.nan
+        good = self._codes != -1
+        if not good.all():
+            if skipna:
+                pointer = self._codes[good].min()
+            else:
+                return np.nan
         else:
-            return self.categories[pointer]
+            pointer = self._codes.min()
+        return self.categories[pointer]
 
-    def max(self, numeric_only=None, **kwargs):
+    @deprecate_kwarg(old_arg_name="numeric_only", new_arg_name="skipna")
+    def max(self, skipna=True):
         """
         The maximum value of the object.
 
@@ -2199,15 +2201,15 @@ def max(self, numeric_only=None, **kwargs):
         max : the maximum of this `Categorical`
         """
         self.check_for_ordered("max")
-        if numeric_only:
-            good = self._codes != -1
-            pointer = self._codes[good].max(**kwargs)
-        else:
-            pointer = self._codes.max(**kwargs)
-        if pointer == -1:
-            return np.nan
+        good = self._codes != -1
+        if not good.all():
+            if skipna:
+                pointer = self._codes[good].max()
+            else:
+                return np.nan
         else:
-            return self.categories[pointer]
+            pointer = self._codes.max()
+        return self.categories[pointer]
 
     def mode(self, dropna=True):
         """
diff --git a/pandas/core/series.py b/pandas/core/series.py
--- a/pandas/core/series.py
+++ b/pandas/core/series.py
@@ -3869,9 +3869,7 @@ def _reduce(
             self._get_axis_number(axis)
 
         if isinstance(delegate, Categorical):
-            # TODO deprecate numeric_only argument for Categorical and use
-            # skipna as well, see GH25303
-            return delegate._reduce(name, numeric_only=numeric_only, **kwds)
+            return delegate._reduce(name, skipna=skipna, **kwds)
         elif isinstance(delegate, ExtensionArray):
             # dispatch to ExtensionArray interface
             return delegate._reduce(name, skipna=skipna, **kwds)


Test patch:
diff --git a/pandas/tests/arrays/categorical/test_analytics.py b/pandas/tests/arrays/categorical/test_analytics.py
--- a/pandas/tests/arrays/categorical/test_analytics.py
+++ b/pandas/tests/arrays/categorical/test_analytics.py
@@ -35,31 +35,43 @@ def test_min_max(self):
         assert _min == "d"
         assert _max == "a"
 
+    @pytest.mark.parametrize("skipna", [True, False])
+    def test_min_max_with_nan(self, skipna):
+        # GH 25303
         cat = Categorical(
             [np.nan, "b", "c", np.nan], categories=["d", "c", "b", "a"], ordered=True
         )
-        _min = cat.min()
-        _max = cat.max()
-        assert np.isnan(_min)
-        assert _max == "b"
+        _min = cat.min(skipna=skipna)
+        _max = cat.max(skipna=skipna)
 
-        _min = cat.min(numeric_only=True)
-        assert _min == "c"
-        _max = cat.max(numeric_only=True)
-        assert _max == "b"
+        if skipna is False:
+            assert np.isnan(_min)
+            assert np.isnan(_max)
+        else:
+            assert _min == "c"
+            assert _max == "b"
 
         cat = Categorical(
             [np.nan, 1, 2, np.nan], categories=[5, 4, 3, 2, 1], ordered=True
         )
-        _min = cat.min()
-        _max = cat.max()
-        assert np.isnan(_min)
-        assert _max == 1
-
-        _min = cat.min(numeric_only=True)
-        assert _min == 2
-        _max = cat.max(numeric_only=True)
-        assert _max == 1
+        _min = cat.min(skipna=skipna)
+        _max = cat.max(skipna=skipna)
+
+        if skipna is False:
+            assert np.isnan(_min)
+            assert np.isnan(_max)
+        else:
+            assert _min == 2
+            assert _max == 1
+
+    @pytest.mark.parametrize("method", ["min", "max"])
+    def test_deprecate_numeric_only_min_max(self, method):
+        # GH 25303
+        cat = Categorical(
+            [np.nan, 1, 2, np.nan], categories=[5, 4, 3, 2, 1], ordered=True
+        )
+        with tm.assert_produces_warning(expected_warning=FutureWarning):
+            getattr(cat, method)(numeric_only=True)
 
     @pytest.mark.parametrize(
         "values,categories,exp_mode",
diff --git a/pandas/tests/reductions/test_reductions.py b/pandas/tests/reductions/test_reductions.py
--- a/pandas/tests/reductions/test_reductions.py
+++ b/pandas/tests/reductions/test_reductions.py
@@ -1043,7 +1043,7 @@ def test_min_max(self):
         )
         _min = cat.min()
         _max = cat.max()
-        assert np.isnan(_min)
+        assert _min == "c"
         assert _max == "b"
 
         cat = Series(
@@ -1053,30 +1053,24 @@ def test_min_max(self):
         )
         _min = cat.min()
         _max = cat.max()
-        assert np.isnan(_min)
+        assert _min == 2
         assert _max == 1
 
-    def test_min_max_numeric_only(self):
-        # TODO deprecate numeric_only argument for Categorical and use
-        # skipna as well, see GH25303
+    @pytest.mark.parametrize("skipna", [True, False])
+    def test_min_max_skipna(self, skipna):
+        # GH 25303
         cat = Series(
             Categorical(["a", "b", np.nan, "a"], categories=["b", "a"], ordered=True)
         )
+        _min = cat.min(skipna=skipna)
+        _max = cat.max(skipna=skipna)
 
-        _min = cat.min()
-        _max = cat.max()
-        assert np.isnan(_min)
-        assert _max == "a"
-
-        _min = cat.min(numeric_only=True)
-        _max = cat.max(numeric_only=True)
-        assert _min == "b"
-        assert _max == "a"
-
-        _min = cat.min(numeric_only=False)
-        _max = cat.max(numeric_only=False)
-        assert np.isnan(_min)
-        assert _max == "a"
+        if skipna is True:
+            assert _min == "b"
+            assert _max == "a"
+        else:
+            assert np.isnan(_min)
+            assert np.isnan(_max)
 
 
 class TestSeriesMode: