Repository: pandas-dev/pandas
Pull request #59782 (2024-09-11 20:46:51)

BUG: groupby().any() returns true for groups with timedelta all NaT
PR Summary:

This PR addresses an issue where groupby().any() returns True for groups where all Timedelta values are NaT, which is inconsistent with other dtypes (like integers and strings). For non-Timedelta dtypes, groupby().any() correctly returns False when all values in a group are null.
Reproduction:

python

import pandas as pd

# Current behavior: incorrect output for all NaT groups
df = pd.DataFrame([pd.Timedelta(1), pd.NaT]).groupby([0, 1]).any()

For other dtypes (integers and strings):

python

df = pd.DataFrame([1, None]).groupby([0, 1]).any()  # Returns False
df = pd.DataFrame(["a", None]).groupby([0, 1]).any()  # Returns False

Expected behavior for Timedelta values: groupby().any() should return False for groups where all values are NaT.
Fix Details:

    Modified the _call_cython_op function in the groupby aggregation to correctly handle Timedelta types. Now, when all values in a group are NaT, the any() method will return False.
    This ensures consistent behavior across all dtypes, including Timedelta.

Tests Added:

    A new test has been added to check that groupby().any() returns False for groups with only NaT Timedelta values and True for groups with at least one non-NaT Timedelta.

Closing:

This PR fixes the inconsistent behavior of groupby().any() for Timedelta values and improves the overall correctness of the method. Let me know if there are any additional changes needed.

Resolves: #59712

Note: I am a complete novice to open source and this is my first ever PR. I'm taking an open source systems class is college and we're suppposed to attempt to contribute to a existing project we use and I chose pandas as it is a tool that has helped me alot. So i ask for any guidance anyone can give me about any potential mistakes or improvements i can make here. Appreciate it!


mroeschke: Thanks for the pull request but this is already being worked on in https://github.com/pandas-dev/pandas/pull/59750. Closing to let that contributor finish the effort, but we can reopen this if it goes stale 
Petroncini: Hi @mroeschke , could this be reopened?
Petroncini: 
Hi @rhshadrach, how's this? I was having some issues with the linux 32 bit tests but it seems resolved now.

Petroncini: Hi @rhshadrach, how's this?

Patch:
diff --git a/doc/source/whatsnew/v3.0.0.rst b/doc/source/whatsnew/v3.0.0.rst
index 41ba80989a..6ebb51cd3e 100644
--- a/doc/source/whatsnew/v3.0.0.rst
+++ b/doc/source/whatsnew/v3.0.0.rst
@@ -652,6 +652,7 @@ Plotting
 Groupby/resample/rolling
 ^^^^^^^^^^^^^^^^^^^^^^^^
 - Bug in :meth:`.DataFrameGroupBy.__len__` and :meth:`.SeriesGroupBy.__len__` would raise when the grouping contained NA values and ``dropna=False`` (:issue:`58644`)
+- Bug in :meth:`.DataFrameGroupBy.any` that returned True for groups where all Timedelta values are NaT. (:issue:`59712`)
 - Bug in :meth:`.DataFrameGroupBy.groups` and :meth:`.SeriesGroupby.groups` that would not respect groupby argument ``dropna`` (:issue:`55919`)
 - Bug in :meth:`.DataFrameGroupBy.median` where nat values gave an incorrect result. (:issue:`57926`)
 - Bug in :meth:`.DataFrameGroupBy.quantile` when ``interpolation="nearest"`` is inconsistent with :meth:`DataFrame.quantile` (:issue:`47942`)
diff --git a/pandas/core/groupby/ops.py b/pandas/core/groupby/ops.py
index da80969b61..0e99178642 100644
--- a/pandas/core/groupby/ops.py
+++ b/pandas/core/groupby/ops.py
@@ -371,6 +371,10 @@ class WrappedCythonOp:
 
         is_datetimelike = dtype.kind in "mM"
 
+        if self.how in ["any", "all"]:
+            if mask is None:
+                mask = isna(values)
+
         if is_datetimelike:
             values = values.view("int64")
             is_numeric = True
@@ -380,12 +384,10 @@ class WrappedCythonOp:
             values = values.astype(np.float32)
 
         if self.how in ["any", "all"]:
-            if mask is None:
-                mask = isna(values)
             if dtype == object:
                 if kwargs["skipna"]:
                     # GH#37501: don't raise on pd.NA when skipna=True
-                    if mask.any():
+                    if mask is not None and mask.any():
                         # mask on original values computed separately
                         values = values.copy()
                         values[mask] = True


Test patch:
diff --git a/pandas/tests/groupby/test_grouping.py b/pandas/tests/groupby/test_grouping.py
index fc2a8a9700..6bb2eaf89b 100644
--- a/pandas/tests/groupby/test_grouping.py
+++ b/pandas/tests/groupby/test_grouping.py
@@ -1180,3 +1180,15 @@ def test_grouping_by_key_is_in_axis():
     result = gb.sum()
     expected = DataFrame({"a": [1, 2], "b": [1, 2], "c": [7, 5]})
     tm.assert_frame_equal(result, expected)
+
+
+def test_groupby_any_with_timedelta():
+    # GH#59712
+    df = DataFrame({"value": [pd.Timedelta(1), pd.NaT]})
+
+    result = df.groupby(np.array([0, 1], dtype=np.int64))["value"].any()
+
+    expected = Series({0: True, 1: False}, name="value", dtype=bool)
+    expected.index = expected.index.astype(np.int64)
+
+    tm.assert_series_equal(result, expected)