Repository: pandas-dev/pandas
Pull request: BUG: value_counts can handle the case even with empty groups (#28479)
* If applying rep to recons_labels go fail, use ids which has no
  consecutive duplicates instead. 

- [x] closes #28479 
- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [x] whatsnew entry

xuancong84 found that value_counts() crashes if `groupby` object contains empty groups.
However, even though I made the construction of DataFrame don't skip empty rows, it still crashed.
Till then, I already tried in many ways though, in this time I tried to correct the callee `self.grouper.recons_labels`. After several tests, I found that If freq of `Grouper` is too long so that it has empty groups in some periods then it crashes. And also have found that this is solved by using `ids` which has no consecutive duplicates instead of `self.grouper.recons_labels`.

Problem statement:
value_counts() crashes if groupby object contains empty groups
When you group some statistical counts for every day, it is possible that on some day there is no counts at all. This will result in empty groups in the groupby object. Performing value_counts() on such groupby objects causes crash.

The following example illustrates the problem:

```
import pandas as pd
df = pd.DataFrame({'Timestamp':[1565083561, 1565083561+86400, 1565083561+86500, 1565083561+86400*2, 1565083561+86400*3, 1565083561+86500*3, 1565083561+86400*4],
                   'Food':['apple', 'apple', 'banana', 'banana', 'orange', 'orange', 'pear']})
df['Datetime'] = pd.to_datetime(df['Timestamp'].apply(lambda t:str(t)), unit='s')
display(df)
dfg = df.groupby(pd.Grouper(freq='1D', key='Datetime'))
[print(g) for g in dfg]
display(dfg['Food'].value_counts())

df = df.drop([3])
display(df)
dfg = df.groupby(pd.Grouper(freq='1D', key='Datetime'))
[print(g) for g in dfg]
display(dfg['Food'].value_counts())
```

**This table does not contain days with empty data, value_counts() does not crash:**

Timestamp | Food | Datetime
-- | -- | --
1565083561 | apple | 2019-08-06 09:26:01
1565169961 | apple | 2019-08-07 09:26:01
1565170061 | banana | 2019-08-07 09:27:41
1565256361 | banana | 2019-08-08 09:26:01
1565342761 | orange | 2019-08-09 09:26:01
1565343061 | orange | 2019-08-09 09:31:01
1565429161 | pear | 2019-08-10 09:26:01

**After groupby each day:**
(Timestamp('2019-08-06 00:00:00', freq='D'),     Timestamp   Food            Datetime
0  1565083561  apple 2019-08-06 09:26:01)
(Timestamp('2019-08-07 00:00:00', freq='D'),     Timestamp    Food            Datetime
1  1565169961   apple 2019-08-07 09:26:01
2  1565170061  banana 2019-08-07 09:27:41)
(Timestamp('2019-08-08 00:00:00', freq='D'),     Timestamp    Food            Datetime
3  1565256361  banana 2019-08-08 09:26:01)
(Timestamp('2019-08-09 00:00:00', freq='D'),     Timestamp    Food            Datetime
4  1565342761  orange 2019-08-09 09:26:01
5  1565343061  orange 2019-08-09 09:31:01)
(Timestamp('2019-08-10 00:00:00', freq='D'),     Timestamp  Food            Datetime
6  1565429161  pear 2019-08-10 09:26:01)

**Result of value_counts():**
Datetime    Food  
2019-08-06  apple     1
2019-08-07  apple     1
`                 ` banana    1
2019-08-08  banana    1
2019-08-09  orange    2
2019-08-10  pear      1
Name: Food, dtype: int64


**This table contains a day with empty data (2019-08-08), value_counts() will crash:**

Timestamp | Food | Datetime
-- | -- | --
1565083561 | apple | 2019-08-06 09:26:01
1565169961 | apple | 2019-08-07 09:26:01
1565170061 | banana | 2019-08-07 09:27:41
1565342761 | orange | 2019-08-09 09:26:01
1565343061 | orange | 2019-08-09 09:31:01
1565429161 | pear | 2019-08-10 09:26:01

**After groupby each day (note the empty group on 2019-08-08):**
(Timestamp('2019-08-06 00:00:00', freq='D'),     Timestamp   Food            Datetime
0  1565083561  apple 2019-08-06 09:26:01)
(Timestamp('2019-08-07 00:00:00', freq='D'),     Timestamp    Food            Datetime
1  1565169961   apple 2019-08-07 09:26:01
2  1565170061  banana 2019-08-07 09:27:41)
_(Timestamp('2019-08-08 00:00:00', freq='D'), Empty DataFrame
Columns: [Timestamp, Food, Datetime]
Index: [])_
(Timestamp('2019-08-09 00:00:00', freq='D'),     Timestamp    Food            Datetime
4  1565342761  orange 2019-08-09 09:26:01
5  1565343061  orange 2019-08-09 09:31:01)
(Timestamp('2019-08-10 00:00:00', freq='D'),     Timestamp  Food            Datetime
6  1565429161  pear 2019-08-10 09:26:01)

**value_counts() crashes:**
```
---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
<ipython-input-543-5efc1c882109> in <module>
     14 [print(g) for g in dfg]
     15 print('This will cause crash:')
---> 16 display(dfg['Food'].value_counts())

~/anaconda3/lib/python3.7/site-packages/pandas/core/groupby/generic.py in value_counts(self, normalize, sort, ascending, bins, dropna)
   1137 
   1138         # multi-index components
-> 1139         labels = list(map(rep, self.grouper.recons_labels)) + [llab(lab, inc)]
   1140         levels = [ping.group_index for ping in self.grouper.groupings] + [lev]
   1141         names = self.grouper.names + [self._selection_name]

~/anaconda3/lib/python3.7/site-packages/numpy/core/fromnumeric.py in repeat(a, repeats, axis)
    469 
    470     """
--> 471     return _wrapfunc(a, 'repeat', repeats, axis=axis)
    472 
    473 

~/anaconda3/lib/python3.7/site-packages/numpy/core/fromnumeric.py in _wrapfunc(obj, method, *args, **kwds)
     54 def _wrapfunc(obj, method, *args, **kwds):
     55     try:
---> 56         return getattr(obj, method)(*args, **kwds)
     57 
     58     # An AttributeError occurs if the object does not have

ValueError: operands could not be broadcast together with shape (5,) (4,)
```



It turns out that this might result from a design flaw in DataFrame construction that it skips empty rows:
`pd.DataFrame.from_dict(data={'row1':{'a':1, 'b':2}, 'row2': {'a':3, 'b':4, 'c':5}, 'row3':{}}, orient='index').fillna(0)`


  | a | b | c
-- | -- | -- | --
row1 | 1 | 2 | 0
row2 | 3 | 4 | 5.0

Take note that row3 is not constructed at all. The correct behavior should output:

  | a | b | c
-- | -- | -- | --
row1 | 1 | 2 | 0.0
row2 | 3 | 4 | 5.0
row3 | 0 | 0 | 0.0

Hints from issue discussion:
Simpler repro; happens on Series as well, not just DataFrame. 

```python
import pandas as pd
ser = pd.Series([1, 2], index=pd.DatetimeIndex(['2019-09-01', '2019-09-03']))
ser.groupby(pd.Grouper(freq='D')).value_counts()
```

```python-traceback
---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
<ipython-input-3-0a3140bd4ce6> in <module>
----> 1 ser.groupby(pd.Grouper(freq='D')).value_counts()

C:\Miniconda3\envs\py37\lib\site-packages\pandas\core\groupby\generic.py in value_counts(self, normalize, sort, ascending, bins, dropna)
   1242
   1243         # multi-index components
-> 1244         labels = list(map(rep, self.grouper.recons_labels)) + [llab(lab, inc)]
   1245         levels = [ping.group_index for ping in self.grouper.groupings] + [lev]
   1246         names = self.grouper.names + [self._selection_name]

C:\Miniconda3\envs\py37\lib\site-packages\numpy\core\fromnumeric.py in repeat(a, repeats, axis)
    469
    470     """
--> 471     return _wrapfunc(a, 'repeat', repeats, axis=axis)
    472
    473

C:\Miniconda3\envs\py37\lib\site-packages\numpy\core\fromnumeric.py in _wrapfunc(obj, method, *args, **kwds)
     54 def _wrapfunc(obj, method, *args, **kwds):
     55     try:
---> 56         return getattr(obj, method)(*args, **kwds)
     57
     58     # An AttributeError occurs if the object does not have

ValueError: operands could not be broadcast together with shape (3,) (2,)
```


I wonder how can such a big issue never been reported in the past. Grouping according to days is a very common use right?
@xuancong84 I can reproduce on 0.19.2 (my earliest install), so this bug has been around a long time. My offhand guess is that `groupby(...).value_counts()` is a less common call, with users tending toward `groupby(['a', 'b']).count()` or some such rather than `groupby('a')['b'].value_counts()`.
Thanks for the report. Anybody interested in working on this?
Hi, I'm interested in this! May I take this?
I have found a cause of this issue by _pdb_ and somewhat _monkeypatch_.
the direct cause is that the prototype of `rep` func is `functools.partial(<function repeat at 0x7fd2c1cbfdd0>, repeats=array([1, 1]))` but `self.grouper.recons_labels` is `[array([0, 1, 2])]`

https://github.com/pandas-dev/pandas/blob/f08a1e62e31fc11e7e5bd7bec72b7e6d86473423/pandas/core/groupby/generic.py#L1258-L1264

So the number of params doesn't match at `map(rep, self.grouper.recons_labels)` in line 1262.

To solve this, I managed to find an indirect cause and they were `ids` and `val` because the `rep` func is made from them. with real code and psuedo-pdb:

```
s = pd.Series([42, 99], index=pd.DatetimeIndex(['2019-09-10', '2019-09-12'])
sg = s.groupby(pd.Grouper('1D'))
sg.value_counts()
    -> ...
    -> ids, val = ids[mask], val[mask]
    -> p ids, val
    [0, 2], [42, 99]
    -> ...
    -> *FAIL*
```

We need to supplement them when there are empty groups like this issue. In other words, we should
make them dense like as follows:
```
[0, 2], [42, 99] -> [0, 1, 2], [42, None, 99]
[0, 0, 3, 3, 3], [111, 111, 222, 222, 333] -> [0, 0, 1, 2, 3, 3, 3], [111, 111, None, None, 222, 222, 333]
```
the former is `ids` and the latter is `val`.

Now I have written the code block for making dense `ids` and `val`. and I'm writing some tests for this and finding where I should modify else like _doc/source/whatsnew_. Because this is my first contributing, I might miss something. So please let me know if I made a mistake or didn't follow the rule :D
However, this treatment only works when the index of SeriesGroupby is DatetimeIndex.
So I had it check that before enter the code block I wrote. It's like pretty ad-hoc though.

I think there are more fundamental solutions for this, but I'm just pandas newbie yet and what I able to do was just investigating local code block. I'll upload the rest tomorrow because I'm totally exhausted now D:...
I just tried it like this:

```
        # groupby removes null keys from groupings
        mask = ids != -1
        ids, val = ids[mask], val[mask]

        # when the index is sparse datetime, we should make it dense
        # otherwise, param of rep func below won't match with the labels
        if isinstance(self.obj.index, DatetimeIndex):
            dense_ids = np.arange(0, ids.max() + 1)

            if not np.array_equal(ids, dense_ids):
                unique_ids, count_ids = np.unique(ids, return_counts=True)
                additional_ids = unique_ids.repeat(count_ids - 1)
                dense_ids = np.sort(np.concatenate([dense_ids, additional_ids]))

                translate_idx = np.where(np.isin(dense_ids, ids), 1, None).nonzero()[0]

                dense_val = np.array([None]).repeat(dense_ids.shape)
                for i, v in zip(translate_idx, val):
                    dense_val[i] = v

                # [0, 2], [42, 77] --> [0, 1, 2], [42, None, 77]
                ids, val = dense_ids, dense_val

        if bins is None:
            lab, lev = algorithms.factorize(val, sort=True)
            llab = lambda lab, inc: lab[inc]
```

It solves this issue and passes the tests though, some of the benchmarks got significantly worse D:

```
      before           after         ratio
     [f08a1e62]       [c6cedf0b]
                      <fix-value_counts>
+         783±3μs          278±5ms   354.92  groupby.GroupByMethods.time_dtype_as_field('datetime', 'value_counts', 'transformation')
+        792±10μs          279±3ms   352.98  groupby.GroupByMethods.time_dtype_as_field('datetime', 'value_counts', 'direct')
+        959±10μs       3.40±0.1ms     3.54  groupby.GroupByMethods.time_dtype_as_field('float', 'value_counts', 'transformation')
+         941±2μs      3.25±0.02ms     3.45  groupby.GroupByMethods.time_dtype_as_field('float', 'value_counts', 'direct')
+         831±9μs       2.69±0.1ms     3.24  groupby.GroupByMethods.time_dtype_as_field('int', 'value_counts', 'transformation')
+        850±20μs      2.55±0.01ms     3.00  groupby.GroupByMethods.time_dtype_as_field('int', 'value_counts', 'direct')
+         841±3μs      2.51±0.02ms     2.99  groupby.GroupByMethods.time_dtype_as_group('float', 'value_counts', 'direct')
+        832±20μs       2.40±0.1ms     2.88  groupby.GroupByMethods.time_dtype_as_group('int', 'value_counts', 'transformation')
+         694±5μs         1.99±0ms     2.86  groupby.GroupByMethods.time_dtype_as_group('object', 'value_counts', 'transformation')
+        813±10μs      2.32±0.02ms     2.85  groupby.GroupByMethods.time_dtype_as_group('int', 'value_counts', 'direct')
+        852±10μs       2.42±0.1ms     2.84  groupby.GroupByMethods.time_dtype_as_group('float', 'value_counts', 'transformation')
+         699±5μs      1.97±0.01ms     2.82  groupby.GroupByMethods.time_dtype_as_group('object', 'value_counts', 'direct')
+       655±600μs         1.52±1ms     2.32  index_cached_properties.IndexCache.time_engine('MultiIndex')
+       51.5±20ms          107±1ms     2.07  multiindex_object.GetLoc.time_large_get_loc
+        67.1±1μs          134±8μs     2.00  indexing.NumericSeriesIndexing.time_loc_scalar(<class 'pandas.core.indexes.numeric.Int64Index'>, 'nonunique_monotonic_inc')
...
SOME BENCHMARKS HAVE CHANGED SIGNIFICANTLY.
PERFORMANCE DECREASED.
```

so I think that I should make `self.grouper.recons_labels` compact rather than make `idx` and `val` dense. But I'm not sure whether it'll work.
@0xF4D3C0D3 Thank you for your attempt!-:) Though, the benchmark result looks terrible. I do not think this is the right way to fix. In fact, you need to read my entire post especially the last part which causes missing rows. In general, the first step in value_counts() should not ignore empty groups, and the solution must be generic and must apply to any groupby objects, not just for DatetimeIndex.

Maybe you can add an additional option in the parent function, controlling whether to ignore empty rows in constructing the data frame to be passed to the 2nd part of value_counts() function (see the stack-trace).
Oh, I just overlooked the last part of your post. I only attempt about that time series. thanks to your advice, I'll be able to try in another way that more suitable. :D

But I don't understand what is `the data frame to be passed to the 2nd part of value_counts() function`.
Series.value_counts(self, normalize=False, sort=True, ascending=False, bins=None, dropna=True)
and
SeriesGroupBy.value_counts(self, normalize=False, sort=True, ascending=False, bins=None, dropna=True)

what's the meaning of `2nd part`?

Commit message: checkout sorting.py to upstream/master not origin/master

Patch to review:
diff --git a/doc/source/whatsnew/v1.0.0.rst b/doc/source/whatsnew/v1.0.0.rst
index 48c1173a372a7..d13c815f0b829 100644
--- a/doc/source/whatsnew/v1.0.0.rst
+++ b/doc/source/whatsnew/v1.0.0.rst
@@ -408,6 +408,7 @@ Other
 - :meth:`DataFrame.to_csv` and :meth:`Series.to_csv` now support dicts as ``compression`` argument with key ``'method'`` being the compression method and others as additional compression options when the compression method is ``'zip'``. (:issue:`26023`)
 - Bug in :meth:`Series.diff` where a boolean series would incorrectly raise a ``TypeError`` (:issue:`17294`)
 - :meth:`Series.append` will no longer raise a ``TypeError`` when passed a tuple of ``Series`` (:issue:`28410`)
+- :meth:`SeriesGroupBy.value_counts` will be able to handle the case even when the :class:`Grouper` makes empty groups (:issue: 28479)
 - Fix corrupted error message when calling ``pandas.libs._json.encode()`` on a 0d array (:issue:`18878`)
 
 .. _whatsnew_1000.contributors:
diff --git a/pandas/core/groupby/ops.py b/pandas/core/groupby/ops.py
index e380cf5930f97..e4335d39929b3 100644
--- a/pandas/core/groupby/ops.py
+++ b/pandas/core/groupby/ops.py
@@ -833,6 +833,14 @@ def levels(self):
     def names(self):
         return [self.binlabels.name]
 
+    @property
+    def recons_labels(self):
+        comp_ids, obs_ids, _ = self.group_info
+        if len(self.binlabels) != len(self.indices):
+            return [np.unique(comp_ids)]
+        labels = (ping.labels for ping in self.groupings)
+        return decons_obs_group_ids(comp_ids, obs_ids, self.shape, labels, xnull=True)
+
     @property
     def groupings(self):
         from pandas.core.groupby.grouper import Grouping
diff --git a/pandas/tests/groupby/test_value_counts.py b/pandas/tests/groupby/test_value_counts.py
index f8bd8843ab7e3..70d6b3db1923e 100644
--- a/pandas/tests/groupby/test_value_counts.py
+++ b/pandas/tests/groupby/test_value_counts.py
@@ -9,7 +9,7 @@
 import numpy as np
 import pytest
 
-from pandas import DataFrame, MultiIndex, Series, date_range
+from pandas import DataFrame, Grouper, MultiIndex, Series, date_range
 from pandas.util import testing as tm
 
 
@@ -79,3 +79,28 @@ def rebuild_index(df):
     # have to sort on index because of unstable sort on values
     left, right = map(rebuild_index, (left, right))  # xref GH9212
     tm.assert_series_equal(left.sort_index(), right.sort_index())
+
+
+@pytest.mark.parametrize("freq", ["1D", "2D", "1W", "1Y"])
+@pytest.mark.parametrize("size", [100, 1000])
+@pytest.mark.parametrize("frac", [0.1, 0.5, 1])
+def test_series_groupby_value_counts_with_grouper(freq, size, frac):
+    np.random.seed(42)
+
+    df = DataFrame.from_dict(
+        {
+            "date": date_range("2019-09-25", periods=size),
+            "name": np.random.choice(list("abcd"), size),
+        }
+    ).sample(frac=frac)
+
+    gr = df.groupby(Grouper(key="date", freq=freq))["name"]
+
+    # have to sort on index because of unstable sort on values xref GH9212
+    result = gr.value_counts().sort_index()
+    expected = gr.apply(Series.value_counts).sort_index()
+    expected.index.names = (
+        result.index.names
+    )  # .apply(Series.value_counts) can't create all names
+
+    tm.assert_series_equal(result, expected)


Code review comments:
diff_hunk: @@ -79,3 +79,28 @@ def rebuild_index(df):
     # have to sort on index because of unstable sort on values
     left, right = map(rebuild_index, (left, right))  # xref GH9212
     tm.assert_series_equal(left.sort_index(), right.sort_index())
+
+
+@pytest.mark.parametrize("freq", ["1D", "2D", "1W", "1Y"])
+@pytest.mark.parametrize("size", [100, 1000])
+@pytest.mark.parametrize("frac", [0.1, 0.5, 1])
+def test_series_groupby_value_counts_with_grouper(freq, size, frac):
+    np.random.seed(42)
path: pandas/tests/groupby/test_value_counts.py
text: @user1:
Why the seed? Ideally this passes for any seed; if not that's a problem.

@author:
For the reproducibility, I set the seed arbitrary value. Well, It's no problem to get rid of that line. And I just found several tests are also using `np.random.seed`: https://github.com/pandas-dev/pandas/search?q=np.random.seed&unscoped_q=np.random.seed

@user1:
Let's remove it.

@author:
yes, no problem!

diff_hunk: @@ -79,3 +79,28 @@ def rebuild_index(df):
     # have to sort on index because of unstable sort on values
     left, right = map(rebuild_index, (left, right))  # xref GH9212
     tm.assert_series_equal(left.sort_index(), right.sort_index())
+
+
+@pytest.mark.parametrize("freq", ["1D", "2D", "1W", "1Y"])
+@pytest.mark.parametrize("size", [100, 1000])
+@pytest.mark.parametrize("frac", [0.1, 0.5, 1])
+def test_series_groupby_value_counts_with_grouper(freq, size, frac):
+    np.random.seed(42)
+
+    df = DataFrame.from_dict(
+        {
+            "date": date_range("2019-09-25", periods=size),
+            "name": np.random.choice(list("abcd"), size),
+        }
+    ).sample(frac=frac)
+
+    gr = df.groupby(Grouper(key="date", freq=freq))["name"]
+
+    # have to sort on index because of unstable sort on values xref GH9212
+    result = gr.value_counts().sort_index()
+    expected = gr.apply(Series.value_counts).sort_index()
+    expected.index.names = (
+        result.index.names
+    )  # .apply(Series.value_counts) can't create all names
path: pandas/tests/groupby/test_value_counts.py
text: @user1:
I don't understand the comment here. Can it be removed?

@author:
Oh no! thx, it's just my mistake... I'll remove it

diff_hunk: @@ -833,6 +833,14 @@ def levels(self):
     def names(self):
         return [self.binlabels.name]
 
+    @property
+    def recons_labels(self):
+        comp_ids, obs_ids, _ = self.group_info
+        if len(self.binlabels) != len(self.indices):
path: pandas/core/groupby/ops.py
text: @user1:
Is this check not appropriate for the BaseGrouper class (disclaimer: I'm not familiar with groupby)? I'm a bit concerned about the amount of code duplication here.

@author:
This issue occurs when only `groupby` with `freq` for now. and then its grouper is `BinGrouper`. I was also able to use just the LBYL way to check whether this is `BaseGrouper` or `BinGrouper` at `BaseGrouper`'s `recons_labels`. Because I have a little bit worried about overhead to check instance type, thus I took overriding way. However, code duplication more matters, I'll move this logic into BaseGrouper. I don't know whose priority is higher, so let me know it, please.

@user1:
@user2 do you have thoughts here?

@user2:
So IIUC the issue here is that the length of the grouper exceeds the length of the object being grouped? Seems a little strange to just return right here but some context would help clarify

@author:
> So IIUC the issue here is that the length of the grouper exceeds the length of the object being grouped? Seems a little strange to just return right here but some context would help clarify

Yes, exactly, the issue occurs when each length of, the label list and the indices, (Its key has grouped index and value has values in each group) are different. However, I'm not sure where this logic should be placed. I always appreciate any suggestions for this.

@user3:
this type of change should be in decons_obs_group_ids; having this here is duplicative. This is pretty tricky code and the last thing we want is more of it.

@author:
@user3 I moved the logic from `recons_labels` to `decons_obs_group_ids`. And I modified it to work in `decons_obs_group_ids` by checking `shape` and the length of unique `comp_ids` and `labels`. I tried not to add new parameters that check the instance type instead, I make the most of the variables in that context.

@author:
@user3 I think a modification should be under the method of `BinGrouper` because this kind of issue occurs only with `groupby` having `freq`. And if it has `freq`, then it would be `BinGrouper`. I could somewhat manage to do it in `decons_obs_group_ids` though it's quite tricky as you see.  In the first place, if the type of grouper is BinGrouper, then it doesn't need both `labels` and `decons_obs_group_ids` and It's ok to just return unique comp_ids(checked by `pytest pandas` but not absolutely sure).

However why unique comp_ids? that's because `rep`, a partial function which is used at making result index expects unique ids that would be result index. `rep` seems to do its job well. And we know we should change something about `self.recons_labels`.

If you don't like current PR(tricky length checking), consider the overriding again, please. Thank you

@author:
My suggestion is this:
```diff
diff --git a/pandas/core/groupby/ops.py b/pandas/core/groupby/ops.py
@@ -792,6 +792,11 @@ class BinGrouper(BaseGrouper):
+    @user4
+    def recons_labels(self):
+        comp_ids, _, _ = self.group_info
+        return [np.unique(comp_ids)]
+
diff --git a/pandas/core/sorting.py b/pandas/core/sorting.py
@@ -162,10 +162,6 @@ def decons_obs_group_ids(comp_ids, obs_ids, shape, labels, xnull):
-    labels = list(labels)
-    unique_comp_ids = np.unique(comp_ids)
-    if (shape[0] != len(unique_comp_ids)) and (shape[0] == len(labels[0])):
-        return [unique_comp_ids]
```

I confirmed it passes all tests. If you don't mind, I'll push it.

@author:
@user3 any comment, please?