Repository: pandas-dev/pandas
Pull request: Add test for unique aggregation returning different dtypes
- [ ] closes #22558 (Replace xxxx with the Github issue number)
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Problem statement:
unique aggregation unexpectedly returning different type
#### Code Sample

```python
import pandas as pd

x1 = pd.DataFrame({'a': [1, 2, 3], 'b': [1, 1, 1]})
x2 = pd.DataFrame({'a': [2, 2, 2], 'b': [1, 1, 1]})
aggregation = {'a': 'unique', 'b': 'unique'}

agg1 = x1.agg(aggregation)
agg2 = x2.agg(aggregation)

print("First aggregation:", type(agg1))
print(agg1)

print("Second aggregation:", type(agg2))
print(agg2)


```
#### Output
```
First aggregation: <class 'pandas.core.series.Series'>
a    [1, 2, 3]
b          [1]
dtype: object
Second aggregation: <class 'pandas.core.frame.DataFrame'>
   a  b
0  2  1
```

#### Problem description

When performing 'unique' aggregations on a dataframe, the results can be returned as different types in an unexpected manner.  

Generally, when performing a 'unique' aggregation on several columns of a dataframe as done above, a `pandas.Series` of numpy arrays is returned, with one element per aggregation column.  This, I think, is the expected behavior, and is demonstrated in the first aggregation above.

However, there is a special case.  When all aggregation columns have exactly 1 unique element, a `pandas.DataFrame` with one row is returned instead.   I'm pretty sure this is unintended behavior, and it requires special case handling when doing such aggregations.

#### Expected Output
```
First aggregation: <class 'pandas.core.series.Series'>
a    [1, 2, 3]
b          [1]
dtype: object
Second aggregation: <class 'pandas.core.series.Series'>
a          [2]
b          [1]
dtype: object
```


#### Output of ``pd.show_versions()``

<details>

INSTALLED VERSIONS
------------------
commit: None
python: 3.6.4.final.0
python-bits: 64
OS: Darwin
OS-release: 17.7.0
machine: x86_64
processor: i386
byteorder: little
LC_ALL: en_US.UTF-8
LANG: en_US.UTF-8
LOCALE: en_US.UTF-8

pandas: 0.23.1
pytest: None
pip: 9.0.3
setuptools: 39.0.1
Cython: None
numpy: 1.14.5
scipy: 1.1.0
pyarrow: None
xarray: None
IPython: 6.2.1
sphinx: 1.7.5
patsy: 0.5.0
dateutil: 2.7.0
pytz: 2018.3
blosc: None
bottleneck: None
tables: 3.4.2
numexpr: 2.6.4
feather: None
matplotlib: 2.2.0
openpyxl: None
xlrd: None
xlwt: None
xlsxwriter: None
lxml: None
bs4: None
html5lib: 1.0.1
sqlalchemy: 1.2.5
pymysql: None
psycopg2: None
jinja2: 2.10
s3fs: None
fastparquet: None
pandas_gbq: None
pandas_datareader: None


</details>

Hints from issue discussion:
How weird!  I would have expected a `Series` of lists for both as you said.  Investigation and PR are welcome!
After doing some digging I was able to find the cause for the mismatch in behavior.
https://github.com/pandas-dev/pandas/blob/a5fe9cf56c6552e34e3d3a933dcdb0ee4e485bf7/pandas/core/base.py#L538-L544
After getting the `result` of the 'unique' aggregation as an `OrderedDict`,
`OrderedDict([('a', array([1, 2, 3])), ('b', array([1]))]` and 
`OrderedDict([('a', array([2])), ('b', array([1]))]`
the code I'm linking tries to cast it to a `Dataframe`, and return a `Series` of lists if the casting fails. In other words, when the lengths of the lists in the values of `result` are equal a `DataFrame` is returned, and a `Series` otherwise.

I've confirmed that we can get the desired behavior if we don't try to cast to a `DataFrame` at all.

Thoughts?
I said give it a try, run tests, and see what happens!
These look correct on master now. Could use a test

```
In [24]: import pandas as pd
    ...:
    ...: x1 = pd.DataFrame({'a': [1, 2, 3], 'b': [1, 1, 1]})
    ...: x2 = pd.DataFrame({'a': [2, 2, 2], 'b': [1, 1, 1]})
    ...: aggregation = {'a': 'unique', 'b': 'unique'}
    ...:
    ...: agg1 = x1.agg(aggregation)
    ...: agg2 = x2.agg(aggregation)
    ...:
    ...: print("First aggregation:", type(agg1))
    ...: print(agg1)
    ...:
    ...: print("Second aggregation:", type(agg2))
    ...: print(agg2)
First aggregation: <class 'pandas.core.series.Series'>
a    [1, 2, 3]
b          [1]
dtype: object
Second aggregation: <class 'pandas.core.series.Series'>
a    [2]
b    [1]
dtype: object
```
take

Commit message: TST: Added test for consistent type with unique agg #22558

Patch to review:
diff --git a/pandas/tests/groupby/aggregate/test_aggregate.py b/pandas/tests/groupby/aggregate/test_aggregate.py
index 54ee32502bbc9..d9dbd32974d3c 100644
--- a/pandas/tests/groupby/aggregate/test_aggregate.py
+++ b/pandas/tests/groupby/aggregate/test_aggregate.py
@@ -1438,3 +1438,15 @@ def test_agg_of_mode_list(test, constant):
     expected = expected.set_index(0)
 
     tm.assert_frame_equal(result, expected)
+
+
+def test_unique_agg_type():
+    # GH#22558
+    df1 = DataFrame({"a": [1, 2, 3], "b": [1, 1, 1]})
+    df2 = DataFrame({"a": [2, 2, 2], "b": [1, 1, 1]})
+    aggregation = {"a": "unique", "b": "unique"}
+
+    agg1 = df1.agg(aggregation)
+    agg2 = df2.agg(aggregation)
+
+    tm.assert_class_equal(agg1, agg2)


Code review comments:
diff_hunk: @@ -1438,3 +1438,15 @@ def test_agg_of_mode_list(test, constant):
     expected = expected.set_index(0)
 
     tm.assert_frame_equal(result, expected)
+
+
+def test_unique_agg_type():
+    # GH#22558
+    df1 = DataFrame({"a": [1, 2, 3], "b": [1, 1, 1]})
+    df2 = DataFrame({"a": [2, 2, 2], "b": [1, 1, 1]})
+    aggregation = {"a": "unique", "b": "unique"}
+
+    agg1 = df1.agg(aggregation)
+    agg2 = df2.agg(aggregation)
+
+    tm.assert_class_equal(agg1, agg2)
path: pandas/tests/groupby/aggregate/test_aggregate.py
text: @user1:
Please define expected as a DataFrame and then use tm.assert_frame_equal. Also, this is not a groupby test, right?

@author:
You're right, this shouldn't be in groupby.  I've made the changes and moved it to a more appropriate file.