Repository: pandas-dev/pandas
Pull request: API: make min/max on empty datetime df consistent with datetime serie…
…s (#33704)

- [x] closes #33704
- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [x] whatsnew entry

Fixes the following issue:
```python
import pandas as pd
df = pd.DataFrame(dict(x=pd.to_datetime([])))
df.max()
```
throws a `ValueError` but 
```python
pd.Series(pd.to_datetime([])).max()
```
results in `NaT`.
With this change, calling an DataFrame on en empty `pd.to_datetime([])`, results in
```
In[1]: df = pd.DataFrame(dict(x=pd.to_datetime([])))
In[2]: df.max()
Out[5]: 
x   NaT
dtype: datetime64[ns]
```


Problem statement:
BUG: min/max of empty datetime dataframe raises
- [x] I have checked that this issue has not already been reported.

- [x] I have confirmed this bug exists on the latest version of pandas.

- [ ] (optional) I have confirmed this bug exists on the master branch of pandas.

#### Code Sample, a copy-pastable example

```python
import pandas as pd
df = pd.DataFrame(dict(x=pd.to_datetime([])))
df.max()
```

```python-traceback
Traceback (most recent call last):
  File "<ipython-input-17-be9940feb663>", line 1, in <module>
    df.max()
  File "pandas/core/generic.py", line 11215, in stat_func
    f, name, axis=axis, skipna=skipna, numeric_only=numeric_only
  File "pandas/core/frame.py", line 7907, in _reduce
    result = f(values)
  File "pandas/core/frame.py", line 7865, in f
    return op(x, axis=axis, skipna=skipna, **kwds)
  File "pandas/core/nanops.py", line 109, in f
    return _na_for_min_count(values, axis)
  File "pandas/core/nanops.py", line 392, in _na_for_min_count
    result.fill(fill_value)
ValueError: cannot convert float NaN to integer
```

#### Problem description

When taking the min/max of an empty datetime dataframe, a ValueError is raised. This is surprising, and inconsistent with the case of an empty datetime series, where min/max return NaT.

#### Expected Output

```python
x   NaT
dtype: datetime64[ns]
```

#### Output of ``pd.show_versions()``

<details>

INSTALLED VERSIONS
------------------
commit           : None
python           : 3.7.7.final.0
python-bits      : 64
OS               : Linux
OS-release       : 4.20.11-100.fc28.x86_64
machine          : x86_64
processor        : x86_64
byteorder        : little
LC_ALL           : None
LANG             : en_GB.UTF-8
LOCALE           : en_GB.UTF-8

pandas           : 1.0.3
numpy            : 1.18.1
pytz             : 2019.3
dateutil         : 2.8.1
pip              : 20.0.2
setuptools       : 46.1.3.post20200330
Cython           : 0.29.15
pytest           : 5.4.1
hypothesis       : None
sphinx           : None
blosc            : None
feather          : None
xlsxwriter       : None
lxml.etree       : 4.5.0
html5lib         : None
pymysql          : None
psycopg2         : None
jinja2           : 2.11.1
IPython          : 7.13.0
pandas_datareader: None
bs4              : 4.9.0
bottleneck       : 1.3.2
fastparquet      : None
gcsfs            : None
lxml.etree       : 4.5.0
matplotlib       : 3.1.3
numexpr          : 2.7.1
odfpy            : None
openpyxl         : None
pandas_gbq       : None
pyarrow          : 0.15.1
pytables         : None
pytest           : 5.4.1
pyxlsb           : None
s3fs             : None
scipy            : 1.2.1
sqlalchemy       : 1.3.16
tables           : 3.6.1
tabulate         : None
xarray           : None
xlrd             : None
xlwt             : None
xlsxwriter       : None
numba            : None

</details>

Hints from issue discussion:
Note the difference here:

```python
In [1]: import pandas as pd                                                                                                                      
# initialize a series from pd.to_datetime
In [2]: srs = pd.Series(pd.to_datetime([]))                                                                                                      

In [3]: srs                                                                                                                                      
Out[3]: Series([], dtype: datetime64[ns])

In [4]: srs.max()                                                                                                                                
Out[4]: NaT

In [5]: df1 = pd.DataFrame(dict(x=pd.to_datetime([])))                                                                                           

In [6]: df1                                                                                                                                      
Out[6]: 
Empty DataFrame
Columns: [x]
Index: []
...
In [8]: df1.max()                                                                                                                                
---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
<ipython-input-8-ffa8d8284b6e> in <module>
----> 1 df1.max()

~/programming/python/pandas_tests/pandas/pandas/core/generic.py in stat_func(self, axis, skipna, level, numeric_only, **kwargs)
  11172             return self._agg_by_level(name, axis=axis, level=level, skipna=skipna)
  11173         #import pdb; pdb.set_trace()()
> 11174         return self._reduce(
  11175             func, name=name, axis=axis, skipna=skipna, numeric_only=numeric_only
  11176         )

~/programming/python/pandas_tests/pandas/pandas/core/frame.py in _reduce(self, op, name, axis, skipna, numeric_only, filter_type, **kwds)
   8377 
   8378             try:
-> 8379                 result = f(values)
   8380 
   8381             except TypeError:

~/programming/python/pandas_tests/pandas/pandas/core/frame.py in f(x)
   8296 
   8297         def f(x):
-> 8298             return op(x, axis=axis, skipna=skipna, **kwds)
   8299 
   8300         def _get_data(axis_matters):

~/programming/python/pandas_tests/pandas/pandas/core/nanops.py in f(values, axis, skipna, **kwds)
    111                 # correctly handle empty inputs and remove this check.
    112                 # It *may* just be `var`
--> 113                 return _na_for_min_count(values, axis)
    114 
    115             if _USE_BOTTLENECK and skipna and _bn_ok_dtype(values.dtype, bn_name):

~/programming/python/pandas_tests/pandas/pandas/core/nanops.py in _na_for_min_count(values, axis)
    386         result_shape = values.shape[:axis] + values.shape[axis + 1 :]
    387         result = np.empty(result_shape, dtype=values.dtype)
--> 388         result.fill(fill_value)
    389         return result
    390 

ValueError: cannot convert float NaN to integer

# initializing dataframe with an empty list of pandas datetimes
In [9]: df2 = pd.DataFrame([pd.to_datetime([])])                                                                                                 

In [10]: df2                                                                                                                                     
Out[10]: 
Empty DataFrame
Columns: []
Index: [0]

In [11]: df2.max()                                                                                                                               
Out[11]: Series([], dtype: float64)

# when only column x is taken
In [13]: df1.x.max()                                                                                                                             
Out[13]: NaT

```
This totally seems unexpected, especially `[13]`.

Commit message: API: make min/max on empty datetime df consistent with datetime series (#33704)

Patch to review:
diff --git a/pandas/core/nanops.py b/pandas/core/nanops.py
index 32b05872ded3f..8a90ccf0622b7 100644
--- a/pandas/core/nanops.py
+++ b/pandas/core/nanops.py
@@ -384,8 +384,7 @@ def _na_for_min_count(
     else:
         assert axis is not None  # assertion to make mypy happy
         result_shape = values.shape[:axis] + values.shape[axis + 1 :]
-        result = np.empty(result_shape, dtype=values.dtype)
-        result.fill(fill_value)
+        result = np.full(result_shape, fill_value)
         return result
 
 
diff --git a/pandas/tests/arithmetic/test_datetime64.py b/pandas/tests/arithmetic/test_datetime64.py
index 8c480faa4ee81..b131a2294780f 100644
--- a/pandas/tests/arithmetic/test_datetime64.py
+++ b/pandas/tests/arithmetic/test_datetime64.py
@@ -2471,3 +2471,18 @@ def test_dt64arr_addsub_object_dtype_2d():
     assert result2.shape == (4, 1)
     assert result2.freq is None
     assert (result2.asi8 == 0).all()
+
+
+def test_sum_empty_df_series():
+    # Calling the following defined sum function returned an error for dataframes but
+    # returned NaT for series. # Check that the API is consistent in this sense when
+    # operating on empty Series/DataFrames. See GH:33704 for more information
+    df = pd.DataFrame(dict(x=pd.to_datetime([])))
+    series = pd.Series(pd.to_datetime([]))
+    assert (df.min().x is NaT) == (series.min() is NaT)
+    assert (df.max().x is NaT) == (series.max() is NaT)
+
+    df = pd.DataFrame(dict(x=[np.nan]))
+    series = pd.Series([np.nan])
+    assert np.isnan(df.min().x) == np.isnan(series.min())
+    assert np.isnan(df.max().x) == np.isnan(series.max())


Code review comments:
diff_hunk: @@ -384,8 +384,7 @@ def _na_for_min_count(
     else:
         assert axis is not None  # assertion to make mypy happy
         result_shape = values.shape[:axis] + values.shape[axis + 1 :]
-        result = np.empty(result_shape, dtype=values.dtype)
-        result.fill(fill_value)
+        result = np.full(result_shape, fill_value)
path: pandas/core/nanops.py
text: @user1:
does this not have a dtype parameter? does it matter?

@author:
The result parameter here is what is making all the trouble. We set the dtype to datetime but our `NaT` is treated as `NaN` which numpy internally tries to cast into an integer which yields the error: 
`ValueError: cannot convert float NaN to integer`.
The conversion happens somewhere inside the c-part of numpy, which I'm not familiar with.

But to come back to your question: for our result, specifying the dtype is not necessary. The given outputs are all corresponding to the type of the input. But good point, I should probably add this check to my tests.