Repository: pandas-dev/pandas

Issue:
DataFrame.nunique and Series.nunique not consistent when Empty
#### Code Sample, a copy-pastable example if possible

Example A:
```python
>>> import pandas as pd
>>> df = pd.DataFrame({"a": [1, 2], "b": [3, 4], "c": [5, 6]})
>>> assert df.nunique().tolist() == [df[col].nunique() for col in df.columns]
# Both equal [2, 2, 2]
```

Example B:
```python
>>> df = pd.DataFrame(columns=['a', 'b', 'c'])
>>> df.nunique()
# Empty DataFrame
# Columns: [a, b, c]
# Index: []

>>> [df[col].nunique() for col in df.columns]
# [0, 0, 0]
```

#### Problem description

In Example A, when a DataFrame isn't empty, getting nunique is consistent between the DataFrame and Series approaches; however, when a DataFrame is empty (Example B), DataFrame.nunique returns itself, while the Series approach returns 0. 

#### Expected Output

I would expect df.nunique to return 0 for each column, consistent with how a Series behaves. An empty object, by definition, has 0 unique elements in my mind.

#### Output of ``pd.show_versions()``

<details>

[paste the output of ``pd.show_versions()`` here below this line]
INSTALLED VERSIONS
------------------
commit           : None
python           : 3.6.5.final.0
python-bits      : 64
OS               : Darwin
OS-release       : 18.7.0
machine          : x86_64
processor        : i386
byteorder        : little
LC_ALL           : None
LANG             : en_US.UTF-8
LOCALE           : en_US.UTF-8

pandas           : 0.25.0
numpy            : 1.17.0
pytz             : 2019.2
dateutil         : 2.8.0
pip              : 18.1
setuptools       : 41.1.0
Cython           : None
pytest           : None
hypothesis       : None
sphinx           : None
blosc            : None
feather          : None
xlsxwriter       : None
lxml.etree       : None
html5lib         : None
pymysql          : None
psycopg2         : None
jinja2           : None
IPython          : 7.7.0
pandas_datareader: None
bs4              : None
bottleneck       : None
fastparquet      : None
gcsfs            : None
lxml.etree       : None
matplotlib       : None
numexpr          : None
odfpy            : None
openpyxl         : None
pandas_gbq       : None
pyarrow          : None
pytables         : None
s3fs             : None
scipy            : None
sqlalchemy       : None
tables           : None
xarray           : None
xlrd             : None
xlwt             : None
xlsxwriter       : None

</details>

Thank you!
DataFrame.nunique is incorrect for DataFrame with no columns
(edit by @TomAugspurger)

Current output:

```python
In [33]: pd.DataFrame(index=[0, 1]).nunique()
Out[33]:
Empty DataFrame
Columns: []
Index: [0, 1]
```

Expected Output is an empty series:

```python
Out[34]: Series([], dtype: float64)
```

Not sure what the expected dtype of that Series should be... probably object.

original post below:

---


#### Code Sample, a copy-pastable example if possible

With Pandas 0.20.3
```python
# create a DataFrame with 3 rows
df = pd.DataFrame({'a': ['A','B','C']})

# lookup unique values for each column, excluding 'a'
unique = df.loc[:, (df.columns != 'a')].nunique()
# this results in an empty Series, the index is also empty
unique.index.tolist()
>>> []
# and
unique[unique == 1].index.tolist()
>>> []
```

With pandas 0.23.3
```python
# create a DataFrame with 3 rows
df = pd.DataFrame({'a': ['A','B','C']})

# lookup unique values for each column, excluding 'a'
unique = df.loc[:, (df.columns != 'a')].nunique()
# this results in an empty Series, but the index is not empty
unique.index.tolist()
>>> [1,2,3]
also:
unique[unique == 1].index.tolist()
>>> [1,2,3]
```

Note:
```python
# if we have don't have an empty df, the behavior of nunique() seems fine:
df = pd.DataFrame({'a': ['A','B','C'], 'b': [1,1,1]})
unique = df.loc[:, (df.columns != 'a')].nunique()

unique[unique == 1]
>>> b    1
>>> dtype: int64
# and
unique[unique == 1].index.tolist()
>>> ['b']
```
#### Problem description

The change of behavior is a bit disturbing, and seems like it is a bug:
`nunique()` ends up creating a Series, and it should be a Series of the df columns, but that doesn't seem to be the case here, instead it is picking up the index of the df.

This is likely related to:

https://github.com/pandas-dev/pandas/issues/21932
https://github.com/pandas-dev/pandas/issues/21255

I am posting this because in my use case I use the list to drop the columns, but i end up with column names that do not exist in the df

<details>
INSTALLED VERSIONS
------------------
commit: None
python: 2.7.10.final.0
python-bits: 64
OS: Darwin
OS-release: 17.6.0
machine: x86_64
processor: i386
byteorder: little
LC_ALL: None
LANG: None
LOCALE: None.None
pandas: 0.23.3
pytest: None
pip: 10.0.1
setuptools: 39.2.0
Cython: None
numpy: 1.13.3
scipy: 0.19.1
pyarrow: None
xarray: None
IPython: None
sphinx: 1.5.5
patsy: 0.5.0
dateutil: 2.6.1
pytz: 2015.7
blosc: None
bottleneck: None
tables: None
numexpr: None
feather: None
matplotlib: None
openpyxl: None
xlrd: 0.9.4
xlwt: None
xlsxwriter: None
lxml: None
bs4: None
html5lib: None
sqlalchemy: None
pymysql: None
psycopg2: 2.7.3.2 (dt dec pq3 ext lo64)
jinja2: 2.10
s3fs: None
fastparquet: None
pandas_gbq: None
pandas_datareader: None

</details>



Discussion:
I also tried this in 0.25.1 and the behavior is the same.
Agreed, the expected output should be

```python
In [56]: pd.Series(0, index=df.columns)
Out[56]:
a    0
b    0
c    0
dtype: int64
```
Looks like something funny happening with `apply` (https://github.com/pandas-dev/pandas/blob/master/pandas/core/frame.py#L7966):
```python
[ins] In [35]: df.apply(Series.nunique, axis=0, dropna=True)                                           
Out[35]: 
Empty DataFrame
Columns: [a, b, c]
Index: []

[ins] In [36]: df.apply(Series.nunique, axis=0, dropna=False)                                          
Out[36]: 
Empty DataFrame
Columns: [a, b, c]
Index: []

[ins] In [37]: df.apply(Series.nunique, axis=0)                                                        
Out[37]: 
a   NaN
b   NaN
c   NaN
dtype: float64
```
You get similar behavior if you try taking a sum for instance:
```python
[ins] In [51]: df.apply(np.sum, axis=0, dropna=True)                                                   
Out[51]: 
Empty DataFrame
Columns: [a, b, c]
Index: []

[ins] In [52]: np.sum(df["a"])                                                                         
Out[52]: 0
```
@streamnsight : That indeed bothers me too.  Patch is welcome!

What is the latest version where this example fails (i.e. does it break after `0.20.3`)?
@gfyoung  it seems to work until 0.22
Ah, so this broke in `0.23.0`.  Marking for `0.23.4` then.
@gfyoung @TomAugspurger  
As I pointed out in the original post, I believe this is wider than just a problem with `nunique` as when I was looking for similar issues, I found a few posts about issues with empty Series / DataFrames. So IMO it is an issue with the way index is managed for an empty Series / DataFrame.
We'll use this issue specifically for nunique so that we can close it when
it's fixed and tested.

If there aren't already open issues for any other bugs you noticed, please
open a new issue.

On Thu, Jul 19, 2018 at 11:24 AM Emmanuel <notifications@github.com> wrote:

> @gfyoung <https://github.com/gfyoung> @TomAugspurger
> <https://github.com/TomAugspurger>
> As I pointed out in the original post, I believe this is wider than just a
> problem with nunique as when I was looking for similar issues, I found a
> few posts about issues with empty Series / DataFrames. So IMO it is an
> issue with the way index is managed for an empty Series / DataFrame.
>
> —
> You are receiving this because you were mentioned.
> Reply to this email directly, view it on GitHub
> <https://github.com/pandas-dev/pandas/issues/21959#issuecomment-406335722>,
> or mute the thread
> <https://github.com/notifications/unsubscribe-auth/ABQHIs3OH92s2QnQQ8GWQlIvjn0MBTwBks5uILK1gaJpZM4VT0Od>
> .
>

git bisect leads me to 76b35c6662136302d212448906c1c8bf8225fffd, but it's quite hard to tell. The underlying issue talks about indexing

> This fixes apply to work correctly when the returned shape mismatches the original. It will try to set the indices if it possible.

Might be a false lead, but it's just what git bisect gave me when running

```
import pandas as pd
assert len(pd.DataFrame(index=[0, 1]).nunique().index) == 0
```

Resolving patch:
diff --git a/pandas/core/apply.py b/pandas/core/apply.py
--- a/pandas/core/apply.py
+++ b/pandas/core/apply.py
@@ -204,17 +204,20 @@ def apply_empty_result(self):
         from pandas import Series
 
         if not should_reduce:
-
-            EMPTY_SERIES = Series([])
             try:
-                r = self.f(EMPTY_SERIES, *self.args, **self.kwds)
+                r = self.f(Series([]))
             except Exception:
                 pass
             else:
                 should_reduce = not isinstance(r, Series)
 
         if should_reduce:
-            return self.obj._constructor_sliced(np.nan, index=self.agg_axis)
+            if len(self.agg_axis):
+                r = self.f(Series([]))
+            else:
+                r = np.nan
+
+            return self.obj._constructor_sliced(r, index=self.agg_axis)
         else:
             return self.obj.copy()
 


Test patch:
diff --git a/pandas/tests/frame/test_apply.py b/pandas/tests/frame/test_apply.py
--- a/pandas/tests/frame/test_apply.py
+++ b/pandas/tests/frame/test_apply.py
@@ -116,6 +116,27 @@ def test_apply_with_reduce_empty(self):
         # Ensure that x.append hasn't been called
         assert x == []
 
+    @pytest.mark.parametrize("func", ["sum", "prod", "any", "all"])
+    def test_apply_funcs_over_empty(self, func):
+        # GH 28213
+        df = DataFrame(columns=["a", "b", "c"])
+
+        result = df.apply(getattr(np, func))
+        expected = getattr(df, func)()
+        assert_series_equal(result, expected)
+
+    def test_nunique_empty(self):
+        # GH 28213
+        df = DataFrame(columns=["a", "b", "c"])
+
+        result = df.nunique()
+        expected = Series(0, index=df.columns)
+        assert_series_equal(result, expected)
+
+        result = df.T.nunique()
+        expected = Series([], index=pd.Index([]))
+        assert_series_equal(result, expected)
+
     def test_apply_deprecate_reduce(self):
         empty_frame = DataFrame()