PERF: faster grouping
State: closed

closes #14293


Labels:
name: Groupby
name: Performance
description: Memory or execution speed performance

Comments:
cc @mrocklin
@wesm 

Does the performance regression pointed out in https://github.com/pydata/pandas/issues/14293#issuecomment-249365759 still hold here?

so that was a bug, now fixed.

this is size=2**21
large is 10k groups, small is 100

```
· Running 8 total benchmarks (2 commits * 1 environments * 4 benchmarks)
[  0.00%] · For pandas commit hash 45b79968:
[  0.00%] ·· Building for conda-py2.7-Cython-matplotlib-numexpr-numpy-openpyxl-pytables-scipy-sqlalchemy-xlrd-xlsxwriter-xlwt...
[  0.00%] ·· Benchmarking conda-py2.7-Cython-matplotlib-numexpr-numpy-openpyxl-pytables-scipy-sqlalchemy-xlrd-xlsxwriter-xlwt
[ 12.50%] ··· Running groupby.groupby_groups.time_groupby_groups_int64_large                                                                                                                       279.51ms
[ 25.00%] ··· Running groupby.groupby_groups.time_groupby_groups_int64_small                                                                                                                       104.08ms
[ 37.50%] ··· Running groupby.groupby_groups.time_groupby_groups_object_large                                                                                                                      374.57ms
[ 50.00%] ··· Running groupby.groupby_groups.time_groupby_groups_object_small                                                                                                                      151.01ms
[ 50.00%] · For pandas commit hash d9e51fe7:
[ 50.00%] ·· Building for conda-py2.7-Cython-matplotlib-numexpr-numpy-openpyxl-pytables-scipy-sqlalchemy-xlrd-xlsxwriter-xlwt...
[ 50.00%] ·· Benchmarking conda-py2.7-Cython-matplotlib-numexpr-numpy-openpyxl-pytables-scipy-sqlalchemy-xlrd-xlsxwriter-xlwt
[ 62.50%] ··· Running groupby.groupby_groups.time_groupby_groups_int64_large                                                                                                                       856.54ms
[ 75.00%] ··· Running groupby.groupby_groups.time_groupby_groups_int64_small                                                                                                                       674.41ms
[ 87.50%] ··· Running groupby.groupby_groups.time_groupby_groups_object_large                                                                                                                      392.38ms
[100.00%] ··· Running groupby.groupby_groups.time_groupby_groups_object_small                                                                                                                      262.49ms    

   before     after       ratio
  [d9e51fe7] [45b79968]
-  856.54ms   279.51ms      0.33  groupby.groupby_groups.time_groupby_groups_int64_large
-  674.41ms   104.08ms      0.15  groupby.groupby_groups.time_groupby_groups_int64_small
SOME BENCHMARKS HAVE CHANGED SIGNIFICANTLY.
```

These _are_ faster for `object` dtypes (and about 2x for _small), though the margin isn't as big as it should be I think.

## [Current coverage](https://codecov.io/gh/pydata/pandas/pull/14294?src=pr) is 85.26% (diff: 100%)

> Merging [#14294](https://codecov.io/gh/pydata/pandas/pull/14294?src=pr) into [master](https://codecov.io/gh/pydata/pandas/branch/master?src=pr) will decrease coverage by **<.01%**

``` diff
@@             master     #14294   diff @@
==========================================
  Files           140        140          
  Lines         50593      50599     +6   
  Methods           0          0          
  Messages          0          0          
  Branches          0          0          
==========================================
+ Hits          43137      43142     +5   
- Misses         7456       7457     +1   
  Partials          0          0          
```

> Powered by [Codecov](https://codecov.io?src=pr). Last update [b81d444...82d19dd](https://codecov.io/gh/pydata/pandas/compare/b81d444fb324eb637546a8e806554172d182fd52...82d19dd50755b66c6e7cefd57b0a693e61619a4d?src=pr)

these are 2*22 with 100, 10000 for small/large

```
   before     after       ratio
  [d9e51fe7] [87db6a4f]
-  136.59ms   107.11ms      0.78  groupby.groupby_indices.time_groupby_indices
-  996.17ms   699.80ms      0.70  groupby.groupby_groups.time_groupby_groups_object_large
-  606.62ms   319.25ms      0.53  groupby.groupby_groups.time_groupby_groups_object_small
-     1.80s   540.05ms      0.30  groupby.groupby_groups.time_groupby_groups_int64_large
-     1.49s   207.64ms      0.14  groupby.groupby_groups.time_groupby_groups_int64_small
```

@wesm 
@jorisvandenbossche 

any comments

roughly the same perf benefits here. with https://github.com/pydata/pandas/pull/14294/commits/580237924022eb74575420ad4433952c8de318dd

```
  [7dedbed8] [a731e2dd]
-  554.34ms   269.39ms      0.49  groupby.groupby_groups.time_groupby_groups_object_small
-     1.78s   430.09ms      0.24  groupby.groupby_groups.time_groupby_groups_int64_large
-     1.32s   180.49ms      0.14  groupby.groupby_groups.time_groupby_groups_int64_small
SOME BENCHMARKS HAVE CHANGED SIGNIFICANTLY.
```

lgtm