Repository: pandas-dev/pandas
Pull request: Fix the output of df.describe on an empty categorical / object column 
- [x] closes #26397
- [x] tests added / passed
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [x] whatsnew entry


Problem statement:
DataFrame.describe excludes top and freq for empty DataFrame
Right now, the output shape and dtype of DataFrame.describe for object columns depends on whether the DataFrame is empty.

#### Code Sample, a copy-pastable example if possible

```python
In [75]: x = pd.DataFrame({"A": ['a', np.nan, np.nan]})

In [76]: x.describe()
Out[76]:
        A
count   1
unique  1
top     a
freq    1

In [77]: x.iloc[:0].describe()
Out[77]:
        A
count   0
unique  0
```
#### Problem description

This leads to instability in the output dtypes and shape.

Would people prefer that we use `np.NaN` or `None` for the top and freq in this case? I believe there's no ambiguity, since we drop missing values before computing.

While the output consistency would be nice, it's not clear to me what's actually best for users here.

Hints from issue discussion:
Makes sense to me
Seems reasonable to me - would probably also need a note in the docs explaining what None means in this context. Mind if I attempt a fix?
That would be great!

On Sat, May 18, 2019 at 11:10 AM enisnazif <notifications@github.com> wrote:

> Seems reasonable to me - would probably also need a note in the docs
> explaining what None means in this context. Mind if I attempt a fix?
>
> —
> You are receiving this because you authored the thread.
> Reply to this email directly, view it on GitHub
> <https://github.com/pandas-dev/pandas/issues/26397?email_source=notifications&email_token=AAKAOIWTQX3YM2K4JJCG26LPWAS7XA5CNFSM4HM5TRIKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODVWRLSY#issuecomment-493688267>,
> or mute the thread
> <https://github.com/notifications/unsubscribe-auth/AAKAOIXHHEB2LAIQB6E764DPWAS7XANCNFSM4HM5TRIA>
> .
>

Thanks - opened a PR at https://github.com/pandas-dev/pandas/pull/26474

Commit message: Add entry to api_breaking section for 0.25.0

Patch to review:
diff --git a/doc/source/whatsnew/v0.25.0.rst b/doc/source/whatsnew/v0.25.0.rst
index dc87ae8f48b8a..1adcd406d5f8d 100644
--- a/doc/source/whatsnew/v0.25.0.rst
+++ b/doc/source/whatsnew/v0.25.0.rst
@@ -184,6 +184,40 @@ are returned. (:issue:`21521`)
 
     df.groupby("a").ffill()
 
+``DataFrame`` describe on an empty categorical / object column will return top and freq
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+When calling :func:`describe`` on a :class:`DataFrame` with an empty categorical / object
+column, the 'top' and 'freq' columns were previously omitted, which was inconsistent with
+the output for non-empty columns. Now, the 'top' and 'freq' for the column will be shown
+as np.NaN in the case of an empty :class:`DataFrame` (:issue:`26397`)
+
+.. ipython:: python
+
+   df = pd.DataFrame({"empty_col": pd.Categorical([])})
+
+*Previous Behavior*:
+
+.. code-block:: python
+
+   In [3]: df.describe()
+   Out[3]:
+           empty_col
+   count           0
+   unique          0
+
+*New Behavior*:
+
+.. code-block:: python
+
+    In [4]: df.describe()
+    Out[4]:
+            empty_col
+    count         0.0
+    unique        0.0
+    top           NaN
+    freq          NaN
+
 
 .. _whatsnew_0250.api_breaking.deps:
 
diff --git a/pandas/core/arrays/categorical.py b/pandas/core/arrays/categorical.py
index d25ccd1b158be..8ca4ba182b545 100644
--- a/pandas/core/arrays/categorical.py
+++ b/pandas/core/arrays/categorical.py
@@ -1477,7 +1477,7 @@ def value_counts(self, dropna=True):
 
         if dropna or clean:
             obs = code if clean else code[mask]
-            count = bincount(obs, minlength=ncat or None)
+            count = bincount(obs, minlength=ncat or 0)
         else:
             count = bincount(np.where(mask, code, ncat))
             ix = np.append(ix, -1)
diff --git a/pandas/core/generic.py b/pandas/core/generic.py
index 76910f425836e..080bb6d0b4368 100644
--- a/pandas/core/generic.py
+++ b/pandas/core/generic.py
@@ -9920,6 +9920,12 @@ def describe_categorical_1d(data):
                     names += ['top', 'freq']
                     result += [top, freq]
 
+            # If the DataFrame is empty, set 'top' and 'freq' to None
+            # to maintain output shape consistency
+            else:
+                names += ['top', 'freq']
+                result += [None, None]
+
             return pd.Series(result, index=names, name=data.name)
 
         def describe_1d(data):
diff --git a/pandas/tests/frame/test_analytics.py b/pandas/tests/frame/test_analytics.py
index effe7eb47323d..487ff7932ec5f 100644
--- a/pandas/tests/frame/test_analytics.py
+++ b/pandas/tests/frame/test_analytics.py
@@ -588,6 +588,16 @@ def test_describe_categorical(self):
         result = df3.describe()
         tm.assert_numpy_array_equal(result["cat"].values, result["s"].values)
 
+    def test_describe_empty_categorical_column(self):
+        # GH 26397
+        # Ensure the index of an an empty categoric DataFrame column
+        # also contains (count, unique, top, freq)
+        df = pd.DataFrame({"empty_col": Categorical([])})
+        result = df.describe()
+        expected = DataFrame({'empty_col': [0, 0, None, None]},
+                             index=['count', 'unique', 'top', 'freq'])
+        tm.assert_frame_equal(result, expected)
+
     def test_describe_categorical_columns(self):
         # GH 11558
         columns = pd.CategoricalIndex(['int1', 'int2', 'obj'],
@@ -608,6 +618,7 @@ def test_describe_categorical_columns(self):
                              index=['count', 'mean', 'std', 'min', '25%',
                                     '50%', '75%', 'max'],
                              columns=exp_columns)
+
         tm.assert_frame_equal(result, expected)
         tm.assert_categorical_equal(result.columns.values,
                                     expected.columns.values)


Code review comments:
diff_hunk: @@ -184,6 +184,40 @@ are returned. (:issue:`21521`)
 
     df.groupby("a").ffill()
 
+``DataFrame`` describe on an empty categorical / object column will return top and freq
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+When calling :func:`describe`` on a :class:`DataFrame` with an empty categorical / object
+column, the 'top' and 'freq' columns were previously omitted, which was inconsistent with
+the output for non-empty columns. Now, the 'top' and 'freq' for the column will be shown
+as np.NaN in the case of an empty :class:`DataFrame` (:issue:`26397`)
+
+.. ipython:: python
+
+   df = pd.DataFrame({"empty_col": pd.Categorical([])})
+
path: doc/source/whatsnew/v0.25.0.rst
text: @user1:
can you add 'df' here (which shows the frame)

diff_hunk: @@ -184,6 +184,40 @@ are returned. (:issue:`21521`)
 
     df.groupby("a").ffill()
 
+``DataFrame`` describe on an empty categorical / object column will return top and freq
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+When calling :func:`describe`` on a :class:`DataFrame` with an empty categorical / object
+column, the 'top' and 'freq' columns were previously omitted, which was inconsistent with
+the output for non-empty columns. Now, the 'top' and 'freq' for the column will be shown
+as np.NaN in the case of an empty :class:`DataFrame` (:issue:`26397`)
+
+.. ipython:: python
+
+   df = pd.DataFrame({"empty_col": pd.Categorical([])})
+
+*Previous Behavior*:
+
+.. code-block:: python
+
+   In [3]: df.describe()
+   Out[3]:
+           empty_col
+   count           0
+   unique          0
+
+*New Behavior*:
+
+.. code-block:: python
+
+    In [4]: df.describe()
path: doc/source/whatsnew/v0.25.0.rst
text: @user1:
this one should be an ipython:: python block (and just df.describe()) as it will execute (previous behavior section is good)