Repository: pandas-dev/pandas
Pull request: Groupby agg works with pd.Series.nunique, but groupby nunique fails with axis=1
- [x] closes #30253
- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [x] whatsnew entry


Problem statement:
Groupby agg works with pd.Series.nunique, but groupby nunique fails with axis=1
#### Code Sample, a copy-pastable example if possible

```python
df = pd.DataFrame({
    ('A', 'B'): [1, 2],
    ('A', 'C'): [1, 3],
    ('D', 'B'): [0, 0]
}, index=pd.bdate_range('20191212', '20191213'))

# DOESNT CRASH
df.groupby(axis=1, level=0).agg(pd.Series.nunique)

# CRASH
df.groupby(axis=1, level=0).nunique()

```
#### Problem description

The groupby nunique crashes with error "ValueError: all keys need to be the same shape", whereas the groupby agg with pd.Series.nunique does not crash

#### Expected Output

pd.DataFrame({
    'A': [1, 2],
    'D': [1, 1]
}, index=pd.bdate_range('20191212', '20191213'))


#### Output of ``pd.show_versions()``

<details>

INSTALLED VERSIONS
------------------
commit: None
python: 3.6.8.final.0
python-bits: 64
OS: Linux
OS-release: 2.6.32-642.6.2.el6.x86_64
machine: x86_64
processor: x86_64
byteorder: little
LC_ALL: None
LANG: en_US.UTF-8
LOCALE: en_US.UTF-8

pandas: 0.21.1
pytest: 4.4.1
pip: 18.1
setuptools: 40.8.0.post20190628
Cython: 0.27.3
numpy: 1.13.3
scipy: 0.19.1
pyarrow: None
xarray: 0.8.2
IPython: 5.1.0
sphinx: 1.8.5
patsy: 0.4.1
dateutil: 2.6.1
pytz: 2017.2
blosc: None
bottleneck: None
tables: 3.4.4
numexpr: 2.6.5
feather: None
matplotlib: 2.1.2
openpyxl: None
xlrd: None
xlwt: None
xlsxwriter: None
lxml: None
bs4: None
html5lib: 0.9999999
sqlalchemy: None
pymysql: None
psycopg2: None
jinja2: 2.8
s3fs: None
fastparquet: None
pandas_gbq: None
pandas_datareader: None

</details>

Hints from issue discussion:
this is. very old version of pandas pls try on latest and on master
df.groupby(axis=1, level=0).agg(pd.Series.nunique)

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/nlepleux/git/pandas-nlepleux/pandas/core/groupby/generic.py", line 942, in aggregate
    result = self._aggregate_frame(func)
  File "/home/nlepleux/git/pandas-nlepleux/pandas/core/groupby/generic.py", line 1123, in _aggregate_frame
    fres = func(data, *args, **kwargs)
  File "/home/nlepleux/git/pandas-nlepleux/pandas/core/base.py", line 1320, in nunique
    uniqs = self.unique()
  File "/home/nlepleux/git/pandas-nlepleux/pandas/core/generic.py", line 5201, in __getattr__
    return object.__getattribute__(self, name)
AttributeError: 'DataFrame' object has no attribute 'unique'
I think an issue with `axis=1` - this would work if transposing:

```python
>>> df.T.groupby(level=0).nunique().T
            A  D
2019-12-12  1  1
2019-12-13  2  1
```

Interested in trying to debug @nlepleux ?
I can surely try

Commit message: Merge remote-tracking branch 'upstream/master'

Patch to review:
diff --git a/doc/source/whatsnew/v1.0.0.rst b/doc/source/whatsnew/v1.0.0.rst
index 6fd758abb1f33..2826029d902db 100755
--- a/doc/source/whatsnew/v1.0.0.rst
+++ b/doc/source/whatsnew/v1.0.0.rst
@@ -833,6 +833,7 @@ Groupby/resample/rolling
 - Bug in :meth:`DataFrame.groupby` where ``any``, ``all``, ``nunique`` and transform functions would incorrectly handle duplicate column labels (:issue:`21668`)
 - Bug in :meth:`DataFrameGroupBy.agg` with timezone-aware datetime64 column incorrectly casting results to the original dtype (:issue:`29641`)
 - Bug in :meth:`DataFrame.groupby` when using axis=1 and having a single level columns index (:issue:`30208`)
+- Bug in :meth:`DataFrame.groupby` when using nunique on axis=1 (:issue:`30253`)
 
 Reshaping
 ^^^^^^^^^
diff --git a/pandas/core/groupby/generic.py b/pandas/core/groupby/generic.py
index bfe3c628169e9..19283883021d5 100644
--- a/pandas/core/groupby/generic.py
+++ b/pandas/core/groupby/generic.py
@@ -1813,10 +1813,17 @@ def groupby_series(obj, col=None):
             # Try to consolidate with normal wrapping functions
             from pandas.core.reshape.concat import concat
 
+            axis_number = obj._get_axis_number(self.axis)
+            if axis_number:
+                obj = obj.T
+
             results = [groupby_series(content, label) for label, content in obj.items()]
             results = concat(results, axis=1)
             results.columns.names = obj.columns.names
 
+            if axis_number:
+                results = results.T
+
         if not self.as_index:
             results.index = ibase.default_index(len(results))
         return results
diff --git a/pandas/tests/groupby/test_groupby.py b/pandas/tests/groupby/test_groupby.py
index 93d4dc6046735..a753fe204de2e 100644
--- a/pandas/tests/groupby/test_groupby.py
+++ b/pandas/tests/groupby/test_groupby.py
@@ -1994,3 +1994,22 @@ def test_dup_labels_output_shape(groupby_func, idx):
 
     assert result.shape == (1, 2)
     tm.assert_index_equal(result.columns, idx)
+
+
+def test_groupby_crash_on_nunique(axis):
+    # Fix following 30253
+    df = pd.DataFrame(
+        {("A", "B"): [1, 2], ("A", "C"): [1, 3], ("D", "B"): [0, 0]},
+        index=pd.bdate_range("20191212", "20191213"),
+    )
+    df.columns.names = ["COL1", "COL2"]
+
+    axis_number = df._get_axis_number(axis)
+    if not axis_number:
+        df = df.T
+
+    nunique_t = df.T.groupby(level=0, axis=axis).nunique().T
+
+    nunique = df.groupby(axis=int(not axis_number), level=0).nunique()
+
+    pd.testing.assert_frame_equal(nunique_t, nunique)


Code review comments:
diff_hunk: @@ -1994,3 +1994,22 @@ def test_dup_labels_output_shape(groupby_func, idx):
 
     assert result.shape == (1, 2)
     tm.assert_index_equal(result.columns, idx)
+
+
+def test_groupby_crash_on_nunique(axis):
+    # Fix following 30253
+    df = pd.DataFrame(
+        {("A", "B"): [1, 2], ("A", "C"): [1, 3], ("D", "B"): [0, 0]},
+        index=pd.bdate_range("20191212", "20191213"),
path: pandas/tests/groupby/test_groupby.py
text: @user1:
Unless these need to be date values (I don't think so) probably easier to just leave unspecified; otherwise it distracts a bit from what the test is aiming to fix

@author:
OK, will do

diff_hunk: @@ -1994,3 +1994,22 @@ def test_dup_labels_output_shape(groupby_func, idx):
 
     assert result.shape == (1, 2)
     tm.assert_index_equal(result.columns, idx)
+
+
+def test_groupby_crash_on_nunique(axis):
+    # Fix following 30253
+    df = pd.DataFrame(
+        {("A", "B"): [1, 2], ("A", "C"): [1, 3], ("D", "B"): [0, 0]},
+        index=pd.bdate_range("20191212", "20191213"),
+    )
+    df.columns.names = ["COL1", "COL2"]
+
+    axis_number = df._get_axis_number(axis)
+    if not axis_number:
+        df = df.T
+
+    nunique_t = df.T.groupby(level=0, axis=axis).nunique().T
+
+    nunique = df.groupby(axis=int(not axis_number), level=0).nunique()
+
+    pd.testing.assert_frame_equal(nunique_t, nunique)
path: pandas/tests/groupby/test_groupby.py
text: @user1:
```suggestion
    tm.assert_frame_equal(result, expected)
```

Note will have to update variable names as well

@author:
Sure, will do, thanks

diff_hunk: @@ -1813,10 +1813,17 @@ def groupby_series(obj, col=None):
             # Try to consolidate with normal wrapping functions
             from pandas.core.reshape.concat import concat
 
+            axis_number = obj._get_axis_number(self.axis)
+            if axis_number:
+                obj = obj.T
path: pandas/core/groupby/generic.py
text: @user1:
Rather than transpose I think should just specify the iterator, so something like:

```
if axis_number == 0:
    iter_func = obj.items
else:
    iter_func = obj.iterrows
```

Then change subsequent line to 

```python
results = [groupby_series(content, label) for label, content in iter_func()]
```


@author:
I'm trying that, thanks

diff_hunk: @@ -1994,3 +1994,22 @@ def test_dup_labels_output_shape(groupby_func, idx):
 
     assert result.shape == (1, 2)
     tm.assert_index_equal(result.columns, idx)
+
+
+def test_groupby_crash_on_nunique(axis):
+    # Fix following 30253
+    df = pd.DataFrame(
+        {("A", "B"): [1, 2], ("A", "C"): [1, 3], ("D", "B"): [0, 0]},
+        index=pd.bdate_range("20191212", "20191213"),
+    )
+    df.columns.names = ["COL1", "COL2"]
path: pandas/tests/groupby/test_groupby.py
text: @user1:
Can just specify `columns=` in the DataFrame constructor