Repository: pandas-dev/pandas
Pull request #59414 (2024-08-05 09:43:33)

API/TST: expand tests for string any/all reduction + fix pyarrow-based implementation
While working on adding any/all support for the object-dtype version of the StringDtype in https://github.com/pandas-dev/pandas/pull/58451, I bumped into some issues with the current testing and implementation.

It seemed that our current testing is a bit limited, so I expanded the existing test to cover more cases and to be parametrized over all string dtype variants (including plain object dtype).

But that uncovered some issues with the current implementation of the pyarrow-backed version:

- `any` could return `pd.NA` in case of `skipna=False`, while for this version of the string dtype (not using Kleene logic), the result should always be True or False. This is an easy fix (we currently were only filling missing values in case of `all`, also do this for `any`)
- **How to treat missing values?** As truthy or falsey? The current pyarrow-based implementation explicitly filled the missing values with False, but the [documentation](https://pandas.pydata.org/docs/reference/api/pandas.Series.any.html) says _"If skipna is False, then NA are treated as True, because these are not equal to zero."_ 
  I find that a bit strange (my first expectation would be to treat missing values as False), but it _is_ explicitly documented, and follows the behaviour of current object dtype (eg  `pd.Series(["a". np.nan], dtype=object).all(skipna=False)` gives True, not False.

So for now this PR updated the pyarrow implementation to follow the documented behaviour.

- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


mroeschke: In 3.0, adjacently `.idxmax/min` will raise a `ValueError` if `skipna=False` and a NA value is encountered. There _could_ be an API behavior argument to align with that behavior in the future.

Besides that point, I would also agree with your initial expectation that NA values are falsey.
jorisvandenbossche: > In 3.0, adjacently `.idxmax/min` will raise a `ValueError` if `skipna=False` and a NA value is encountered. There _could_ be an API behavior argument to align with that behavior in the future.

For `any`/`all`, I think in the future we don't have to raise about missing values, but we can use Kleene logic and return pd.NA in some cases (although that might then just give an error downstream in your code ..).

That's the behaviour that is currently implemented for the nullable BooleanDtype, and if we plan to stick to that behaviour, I would say it's not worth to change the current NaN-based dtypes to start raise an error for NaNs (and we can preserve the strange "NaN is considered as True because it is not equal to 0" for now).



Patch:
diff --git a/ci/code_checks.sh b/ci/code_checks.sh
index c69b47ae1d..c53265e126 100755
--- a/ci/code_checks.sh
+++ b/ci/code_checks.sh
@@ -164,7 +164,6 @@ if [[ -z "$CHECK" || "$CHECK" == "docstrings" ]]; then
         -i "pandas.Series.str.center RT03,SA01" \
         -i "pandas.Series.str.decode PR07,RT03,SA01" \
         -i "pandas.Series.str.encode PR07,RT03,SA01" \
-        -i "pandas.Series.str.fullmatch RT03" \
         -i "pandas.Series.str.index RT03" \
         -i "pandas.Series.str.ljust RT03,SA01" \
         -i "pandas.Series.str.lower RT03" \
diff --git a/pandas/core/arrays/string_arrow.py b/pandas/core/arrays/string_arrow.py
index 869cc34d5f..fd43ddfdf5 100644
--- a/pandas/core/arrays/string_arrow.py
+++ b/pandas/core/arrays/string_arrow.py
@@ -697,9 +697,9 @@ class ArrowStringArrayNumpySemantics(ArrowStringArray):
         self, name: str, *, skipna: bool = True, keepdims: bool = False, **kwargs
     ):
         if name in ["any", "all"]:
-            if not skipna and name == "all":
-                nas = pc.invert(pc.is_null(self._pa_array))
-                arr = pc.and_kleene(nas, pc.not_equal(self._pa_array, ""))
+            if not skipna:
+                nas = pc.is_null(self._pa_array)
+                arr = pc.or_kleene(nas, pc.not_equal(self._pa_array, ""))
             else:
                 arr = pc.not_equal(self._pa_array, "")
             return ArrowExtensionArray(arr)._reduce(
diff --git a/pandas/core/groupby/groupby.py b/pandas/core/groupby/groupby.py
index c07bc56377..1a3a32375b 100644
--- a/pandas/core/groupby/groupby.py
+++ b/pandas/core/groupby/groupby.py
@@ -2437,7 +2437,8 @@ class GroupBy(BaseGroupBy[NDFrameT]):
         Parameters
         ----------
         ddof : int, default 1
-            Degrees of freedom.
+            Delta Degrees of Freedom. The divisor used in calculations is ``N - ddof``,
+            where ``N`` represents the number of elements.
 
         engine : str, default None
             * ``'cython'`` : Runs the operation through C-extensions from cython.
diff --git a/pandas/core/strings/accessor.py b/pandas/core/strings/accessor.py
index b37e22c9a9..8e6183c434 100644
--- a/pandas/core/strings/accessor.py
+++ b/pandas/core/strings/accessor.py
@@ -1418,6 +1418,9 @@ class StringMethods(NoNewAttributesMixin):
         Returns
         -------
         Series/Index/array of boolean values
+            The function returns a Series, Index, or array of boolean values,
+            where True indicates that the entire string matches the regular
+            expression pattern and False indicates that it does not.
 
         See Also
         --------


Test patch:
diff --git a/pandas/tests/reductions/test_reductions.py b/pandas/tests/reductions/test_reductions.py
index 63e9e89cab..66799732be 100644
--- a/pandas/tests/reductions/test_reductions.py
+++ b/pandas/tests/reductions/test_reductions.py
@@ -1062,25 +1062,62 @@ class TestSeriesReductions:
         assert df.any().all()
         assert not df.all().any()
 
-    def test_any_all_pyarrow_string(self):
+    def test_any_all_string_dtype(self, any_string_dtype):
         # GH#54591
-        pytest.importorskip("pyarrow")
-        ser = Series(["", "a"], dtype="string[pyarrow_numpy]")
+        if (
+            isinstance(any_string_dtype, pd.StringDtype)
+            and any_string_dtype.na_value is pd.NA
+        ):
+            # the nullable string dtype currently still raise an error
+            # https://github.com/pandas-dev/pandas/issues/51939
+            ser = Series(["a", "b"], dtype=any_string_dtype)
+            with pytest.raises(TypeError):
+                ser.any()
+            with pytest.raises(TypeError):
+                ser.all()
+            return
+
+        ser = Series(["", "a"], dtype=any_string_dtype)
         assert ser.any()
         assert not ser.all()
+        assert ser.any(skipna=False)
+        assert not ser.all(skipna=False)
 
-        ser = Series([None, "a"], dtype="string[pyarrow_numpy]")
+        ser = Series([np.nan, "a"], dtype=any_string_dtype)
         assert ser.any()
         assert ser.all()
-        assert not ser.all(skipna=False)
+        assert ser.any(skipna=False)
+        assert ser.all(skipna=False)  # NaN is considered truthy
 
-        ser = Series([None, ""], dtype="string[pyarrow_numpy]")
+        ser = Series([np.nan, ""], dtype=any_string_dtype)
         assert not ser.any()
         assert not ser.all()
+        assert ser.any(skipna=False)  # NaN is considered truthy
+        assert not ser.all(skipna=False)
 
-        ser = Series(["a", "b"], dtype="string[pyarrow_numpy]")
+        ser = Series(["a", "b"], dtype=any_string_dtype)
         assert ser.any()
         assert ser.all()
+        assert ser.any(skipna=False)
+        assert ser.all(skipna=False)
+
+        ser = Series([], dtype=any_string_dtype)
+        assert not ser.any()
+        assert ser.all()
+        assert not ser.any(skipna=False)
+        assert ser.all(skipna=False)
+
+        ser = Series([""], dtype=any_string_dtype)
+        assert not ser.any()
+        assert not ser.all()
+        assert not ser.any(skipna=False)
+        assert not ser.all(skipna=False)
+
+        ser = Series([np.nan], dtype=any_string_dtype)
+        assert not ser.any()
+        assert ser.all()
+        assert ser.any(skipna=False)  # NaN is considered truthy
+        assert ser.all(skipna=False)  # NaN is considered truthy
 
     def test_timedelta64_analytics(self):
         # index min/max