Repository: pandas-dev/pandas
Pull request: REGR: fix non-reduction apply with tz-aware objects
Closes https://github.com/pandas-dev/pandas/issues/31505


Problem statement:
REGR: `copy()` within `apply()` raises ValueError: cannot create a DatetimeTZBlock without a tz, as of 1.0.0
#### Code Sample, a copy-pastable example if possible

Minimal example:
```python
import pandas as pd

df = pd.DataFrame({"foo": [pd.Timestamp("2020", tz="UTC")]}, dtype="object")
df.apply(lambda col: col.copy())  # raises exception below
```

Real-life usage:
```python
def filter_dataframe_by_dict(df, filters):
    """
    Filter the specified dataframe to only those rows which match the specified filters

    Parameters
    ----------
    df : pd.DataFrame
    
    filters : Mapping
        dict, keyed by a subset of `df.columns`

    Returns
    -------
    pd.DataFrame
       Same columns as `df`, including only those rows which match `filters` on all specified values.
    """
    filters = pd.Series(filters, dtype="object")
    mask = df[filters.index].apply(
        # astype("object") calls `copy()` internally, and is necessary to ensure dtype-agnostic 
        # comparisons.
        lambda row: row.astype("object").equals(filters), axis="columns"
    )
    return df[mask]

records = pd.DataFrame(columns = ["foo", "bar", "baz"])

records.loc[0] = {"foo": pd.Timestamp("2019", tz="UTC"), "bar": 1, "baz": 6.283}
records.loc[1] = {"foo": pd.Timestamp("2020", tz="UTC"), "bar": 2, "baz": 6.283}

filters = {"foo": pd.Timestamp("2020", tz="UTC")}

filter_dataframe_by_dict(records, filters)  # raises below exception
```

Exception:
```
Traceback (most recent call last):
  File "pandas_bug.py", line 29, in <module>
    df.apply(lambda col: col.copy())
  File ".venv/lib/python3.6/site-packages/pandas/core/frame.py", line 6875, in apply
    return op.get_result()
  File ".venv/lib/python3.6/site-packages/pandas/core/apply.py", line 186, in get_result
    return self.apply_standard()
  File ".venv/lib/python3.6/site-packages/pandas/core/apply.py", line 296, in apply_standard
    values, self.f, axis=self.axis, dummy=dummy, labels=labels
  File "pandas/_libs/reduction.pyx", line 617, in pandas._libs.reduction.compute_reduction
  File "pandas/_libs/reduction.pyx", line 127, in pandas._libs.reduction.Reducer.get_result
  File "pandas_bug.py", line 29, in <lambda>
    df.apply(lambda row: row.copy(), axis="columns")
  File ".venv/lib/python3.6/site-packages/pandas/core/generic.py", line 5810, in copy
    data = self._data.copy(deep=deep)
  File ".venv/lib/python3.6/site-packages/pandas/core/internals/managers.py", line 794, in copy
    res = self.apply("copy", deep=deep)
  File ".venv/lib/python3.6/site-packages/pandas/core/internals/managers.py", line 442, in apply
    applied = getattr(b, f)(**kwargs)
  File ".venv/lib/python3.6/site-packages/pandas/core/internals/blocks.py", line 696, in copy
    return self.make_block_same_class(values, ndim=self.ndim)
  File ".venv/lib/python3.6/site-packages/pandas/core/internals/blocks.py", line 281, in make_block_same_class
    return make_block(values, placement=placement, ndim=ndim, klass=type(self))
  File ".venv/lib/python3.6/site-packages/pandas/core/internals/blocks.py", line 3028, in make_block
    return klass(values, ndim=ndim, placement=placement)
  File ".venv/lib/python3.6/site-packages/pandas/core/internals/blocks.py", line 1723, in __init__
    values = self._maybe_coerce_values(values)
  File ".venv/lib/python3.6/site-packages/pandas/core/internals/blocks.py", line 2306, in _maybe_coerce_values
    raise ValueError("cannot create a DatetimeTZBlock without a tz")
ValueError: cannot create a DatetimeTZBlock without a tz
```
#### Problem description

Essentially, the problem arises when `.copy()` is called from with an `.apply` call, on a row/column of a DataFrame which:
* has dtype `object`
* consists solely of tz-aware `Timestamp` objects

While this seems like a fairly artificial set of conditions, as the "real world example" above is intended to demonstrate, it can indeed occur "organically". Appending rows to an initially empty `DataFrame` results in the dtype defaulting to `object` for all columns, so if the `filters` passed to `filter_dataframe_by_dict` happen to only consist of timestamp-valued columns, the error conditions are met.

Note that the error *only* seems to arise in `.apply` calls; making the same calls on the rows return using `iterrrows()` or `iloc` works just fine.

This problem only occurs as of the latest pandas 1.0.0 release.

#### Output of ``pd.show_versions()``

<details>

```
INSTALLED VERSIONS
------------------
commit           : None
python           : 3.6.8.final.0
python-bits      : 64
OS               : Linux
OS-release       : 4.15.0-74-generic
machine          : x86_64
processor        : x86_64
byteorder        : little
LC_ALL           : None
LANG             : en_GB.UTF-8
LOCALE           : en_GB.UTF-8

pandas           : 1.0.0
numpy            : 1.17.4
pytz             : 2019.3
dateutil         : 2.8.1
pip              : 19.3.1
setuptools       : 41.6.0
Cython           : None
pytest           : 5.3.0
hypothesis       : None
sphinx           : None
blosc            : None
feather          : None
xlsxwriter       : None
lxml.etree       : None
html5lib         : None
pymysql          : None
psycopg2         : None
jinja2           : None
IPython          : None
pandas_datareader: None
bs4              : None
bottleneck       : None
fastparquet      : None
gcsfs            : None
lxml.etree       : None
matplotlib       : None
numexpr          : None
odfpy            : None
openpyxl         : None
pandas_gbq       : None
pyarrow          : None
pytables         : None
pytest           : 5.3.0
pyxlsb           : None
s3fs             : None
scipy            : None
sqlalchemy       : 1.3.11
tables           : None
tabulate         : None
xarray           : None
xlrd             : None
xlwt             : None
xlsxwriter       : None
numba            : None
```
</details>

Hints from issue discussion:

Commit message: REGR: fix non-reduction apply with tz-aware objects

Patch to review:
diff --git a/pandas/core/apply.py b/pandas/core/apply.py
index 81e1d84880f60..2a50fce5ae68e 100644
--- a/pandas/core/apply.py
+++ b/pandas/core/apply.py
@@ -299,10 +299,12 @@ def apply_standard(self):
                 result = libreduction.compute_reduction(
                     values, self.f, axis=self.axis, dummy=dummy, labels=labels
                 )
-            except ValueError as err:
-                if "Function does not reduce" not in str(err):
-                    # catch only ValueError raised intentionally in libreduction
-                    raise
+            except ValueError:
+                # TODO there are still cases that give another error (eg GH-31505)
+                # if "Function does not reduce" not in str(err):
+                #     # catch only ValueError raised intentionally in libreduction
+                #     raise
+                pass
             except TypeError:
                 # e.g. test_apply_ignore_failures we just ignore
                 if not self.ignore_failures:
diff --git a/pandas/tests/frame/test_apply.py b/pandas/tests/frame/test_apply.py
index e98f74e133ea9..ad4f86b57d435 100644
--- a/pandas/tests/frame/test_apply.py
+++ b/pandas/tests/frame/test_apply.py
@@ -703,6 +703,12 @@ def apply_list(row):
         )
         tm.assert_series_equal(result, expected)
 
+    def test_apply_noreduction_tzaware_object(self):
+        # https://github.com/pandas-dev/pandas/issues/31505
+        df = pd.DataFrame({"foo": [pd.Timestamp("2020", tz="UTC")]}, dtype="object")
+        result = df.apply(lambda col: col.copy())
+        tm.assert_frame_equal(result, df)
+
 
 class TestInferOutputShape:
     # the user has supplied an opaque UDF where


Code review comments:
diff_hunk: @@ -299,10 +299,12 @@ def apply_standard(self):
                 result = libreduction.compute_reduction(
                     values, self.f, axis=self.axis, dummy=dummy, labels=labels
                 )
-            except ValueError as err:
-                if "Function does not reduce" not in str(err):
-                    # catch only ValueError raised intentionally in libreduction
-                    raise
+            except ValueError:
+                # TODO there are still cases that give another error (eg GH-31505)
+                # if "Function does not reduce" not in str(err):
+                #     # catch only ValueError raised intentionally in libreduction
+                #     raise
+                pass
path: pandas/core/apply.py
text: @author:
@user1 I suppose you don't like this "fix", but the reality seems to be that there are other ways that the libreduction can raise an error, that should still be catched. I can try to find a way to avoid calling libreduction (add a case to the `if` check above), but I have the feeling that this would just be a check for the specifc case reported, and there will then just be other cases that raise other errors.

@author:
So the reason it is now getting here, is because it is object dtype (we already check for extension dtypes and datetimelike dtypes to not get here). Maybe in general object dtype should not go through the cython code?

@user1:
stepping back for a second, it isn't clear to me why it's trying to create a DatetimeTZBlock instead of an ObjectBlock

@author:
Yep. I tried to step into with the debugger, but didn't directly find the cause. It seems there is some inference at some step, which infers that it is datetimetz dtype, but then the values are not converted and still objects, which then causes an error in creating the block.

@author:
At some point, it created a DatetimeTZBlock with `array([None], dtype=object)` or `array([Timestamp('2020-01-01 00:00:00+0000', tz='UTC')], dtype=object)`as its values. It is then copying that block that fails (through `make_block_same_class`), but the question is how it created such a block in the first place

@user1:
> but the question is how it created such a block in the first place

It looks like in Reducer.get_result self.typ is Series, so we're calling the Series constructor at some point and getting type inference there.  If I add in libreduction L117 `dtype=arr.dtype` tot he `self.typ(...)` call the tests this adds passes (haven't run the rest of the test suite with this edit)

@user2:
Is it safe to assume that reductions are instances of the dtype there, or do we need inference? Naively, it seems like that would fail of `.mean()` on integer dtypes, since that would return a float.

@user1:
> Is it safe to assume that reductions are instances of the dtype there, or do we need inference?

I _think_ this is creating a Series that is effectively a view on some portion of our original Series, so it should be OK.  Definitely merits more eyeballs.

Do we have an idea how big the gains from this are?  this piece of code is disproportionately difficult to maintain.

@user2:
> I think this is creating a Series that is effectively a view on some portion of our original Series, so it should be OK. Definitely merits more eyeballs.

Yep, I think you're correct.

@user2:
@author can you try the suggestion in https://github.com/pandas-dev/pandas/pull/31614/files#r374228865 to pass `dtype=self.dtype` to that constructor? 

@author:
It's indeed only creating a subset of the original data, so the dtype should always be correct, and so avoids dtype inference (the calculation of the function only happens afterwards, and thus also a potential dtype change like int->float for mean)