PERF: Fix performance regression with Series statistical ops (#25952)
State: closed

- [x] closes #25952
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [x] whatsnew entry

#25743 introduced a performance regression in `Series` `MultiIndex` statistical operations.

This happened due to the additional required check for Series groupby operations that already existed for `DataFrame`.  By ensuring that the check needs to be run, this regression can be avoided also potentially slightly speeding up similar `DataFrame` operations.

Labels:
name: Performance
description: Memory or execution speed performance

Comments:
Can you post ASV results for this? Also does this impact performance of DataFrame ops?
Here is the affected ASV tests:

```
       before           after         ratio
     [00c119c5]       [98abd413]
     <master>         <fix-groupby-performance>
-      20.8±0.2ms       18.2±0.5ms     0.88  stat_ops.SeriesMultiIndexOps.time_op(0, 'mad')
-      12.1±0.2ms       9.71±0.2ms     0.80  stat_ops.SeriesMultiIndexOps.time_op(0, 'skew')
-      12.1±0.2ms       9.61±0.2ms     0.79  stat_ops.SeriesMultiIndexOps.time_op(0, 'kurt')
-      8.58±0.2ms      5.96±0.03ms     0.69  stat_ops.SeriesMultiIndexOps.time_op(1, 'sem')
-      8.73±0.1ms       5.79±0.2ms     0.66  stat_ops.SeriesMultiIndexOps.time_op(0, 'sem')
-     8.92±0.08ms      5.92±0.09ms     0.66  stat_ops.SeriesMultiIndexOps.time_op(1, 'median')
-      8.67±0.1ms       5.71±0.1ms     0.66  stat_ops.SeriesMultiIndexOps.time_op(0, 'median')
-      7.23±0.2ms       4.33±0.4ms     0.60  stat_ops.SeriesMultiIndexOps.time_op(1, 'prod')
-      7.67±0.2ms       4.48±0.3ms     0.58  stat_ops.SeriesMultiIndexOps.time_op(0, 'std')
-      7.70±0.2ms      4.39±0.06ms     0.57  stat_ops.SeriesMultiIndexOps.time_op(1, 'var')
-      7.13±0.1ms      3.92±0.03ms     0.55  stat_ops.SeriesMultiIndexOps.time_op(0, 'sum')
-      7.35±0.2ms       4.01±0.1ms     0.55  stat_ops.SeriesMultiIndexOps.time_op(1, 'sum')
-      7.24±0.1ms       3.94±0.2ms     0.54  stat_ops.SeriesMultiIndexOps.time_op(1, 'mean')
-      7.18±0.1ms      3.90±0.05ms     0.54  stat_ops.SeriesMultiIndexOps.time_op(0, 'mean')
-     7.62±0.06ms      4.12±0.04ms     0.54  stat_ops.SeriesMultiIndexOps.time_op(0, 'var')
-      7.15±0.1ms      3.84±0.08ms     0.54  stat_ops.SeriesMultiIndexOps.time_op(0, 'prod')
```

Most `MultiIndex` `DataFrame` operations are improved as well, but only slightly.  Not enough to meet the 10% threshold.

I'll run a full ASV tonight to see if there are any other major affected areas.
# [Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/25953?src=pr&el=h1) Report
> Merging [#25953](https://codecov.io/gh/pandas-dev/pandas/pull/25953?src=pr&el=desc) into [master](https://codecov.io/gh/pandas-dev/pandas/commit/00c119c50728bee88a5e5d1538cf3a405e36997c?src=pr&el=desc) will **decrease** coverage by `0.01%`.
> The diff coverage is `80%`.

[![Impacted file tree graph](https://codecov.io/gh/pandas-dev/pandas/pull/25953/graphs/tree.svg?width=650&token=eZ4WkYLtcO&height=150&src=pr)](https://codecov.io/gh/pandas-dev/pandas/pull/25953?src=pr&el=tree)

```diff
@@            Coverage Diff             @@
##           master   #25953      +/-   ##
==========================================
- Coverage   91.82%    91.8%   -0.02%     
==========================================
  Files         175      175              
  Lines       52581    52582       +1     
==========================================
- Hits        48280    48271       -9     
- Misses       4301     4311      +10
```

| Flag | Coverage Δ | |
|---|---|---|
| #multiple | `90.35% <80%> (-0.02%)` | :arrow_down: |
| #single | `41.89% <0%> (-0.08%)` | :arrow_down: |

| [Impacted Files](https://codecov.io/gh/pandas-dev/pandas/pull/25953?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [pandas/core/groupby/grouper.py](https://codecov.io/gh/pandas-dev/pandas/pull/25953/diff?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvZ3JvdXBieS9ncm91cGVyLnB5) | `97.44% <80%> (-0.73%)` | :arrow_down: |
| [pandas/io/gbq.py](https://codecov.io/gh/pandas-dev/pandas/pull/25953/diff?src=pr&el=tree#diff-cGFuZGFzL2lvL2dicS5weQ==) | `75% <0%> (-12.5%)` | :arrow_down: |
| [pandas/core/indexes/base.py](https://codecov.io/gh/pandas-dev/pandas/pull/25953/diff?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvaW5kZXhlcy9iYXNlLnB5) | `96.57% <0%> (-0.22%)` | :arrow_down: |
| [pandas/core/frame.py](https://codecov.io/gh/pandas-dev/pandas/pull/25953/diff?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvZnJhbWUucHk=) | `96.79% <0%> (-0.12%)` | :arrow_down: |

------

[Continue to review full report at Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/25953?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/25953?src=pr&el=footer). Last update [00c119c...98abd41](https://codecov.io/gh/pandas-dev/pandas/pull/25953?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).

# [Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/25953?src=pr&el=h1) Report
> Merging [#25953](https://codecov.io/gh/pandas-dev/pandas/pull/25953?src=pr&el=desc) into [master](https://codecov.io/gh/pandas-dev/pandas/commit/48ea04fe5d3ba9b618152ae83aef703dbbb2c3f4?src=pr&el=desc) will **decrease** coverage by `0.01%`.
> The diff coverage is `100%`.

[![Impacted file tree graph](https://codecov.io/gh/pandas-dev/pandas/pull/25953/graphs/tree.svg?width=650&token=eZ4WkYLtcO&height=150&src=pr)](https://codecov.io/gh/pandas-dev/pandas/pull/25953?src=pr&el=tree)

```diff
@@            Coverage Diff             @@
##           master   #25953      +/-   ##
==========================================
- Coverage   91.98%   91.96%   -0.02%     
==========================================
  Files         175      175              
  Lines       52372    52369       -3     
==========================================
- Hits        48172    48161      -11     
- Misses       4200     4208       +8
```

| Flag | Coverage Δ | |
|---|---|---|
| #multiple | `90.52% <100%> (-0.01%)` | :arrow_down: |
| #single | `40.7% <0%> (-0.14%)` | :arrow_down: |

| [Impacted Files](https://codecov.io/gh/pandas-dev/pandas/pull/25953?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [pandas/core/groupby/grouper.py](https://codecov.io/gh/pandas-dev/pandas/pull/25953/diff?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvZ3JvdXBieS9ncm91cGVyLnB5) | `98.52% <100%> (+0.34%)` | :arrow_up: |
| [pandas/io/gbq.py](https://codecov.io/gh/pandas-dev/pandas/pull/25953/diff?src=pr&el=tree#diff-cGFuZGFzL2lvL2dicS5weQ==) | `78.94% <0%> (-10.53%)` | :arrow_down: |
| [pandas/core/indexes/base.py](https://codecov.io/gh/pandas-dev/pandas/pull/25953/diff?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvaW5kZXhlcy9iYXNlLnB5) | `96.72% <0%> (-0.22%)` | :arrow_down: |
| [pandas/core/frame.py](https://codecov.io/gh/pandas-dev/pandas/pull/25953/diff?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvZnJhbWUucHk=) | `96.9% <0%> (-0.12%)` | :arrow_down: |
| [pandas/util/testing.py](https://codecov.io/gh/pandas-dev/pandas/pull/25953/diff?src=pr&el=tree#diff-cGFuZGFzL3V0aWwvdGVzdGluZy5weQ==) | `90.61% <0%> (-0.11%)` | :arrow_down: |

------

[Continue to review full report at Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/25953?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/25953?src=pr&el=footer). Last update [48ea04f...92858c1](https://codecov.io/gh/pandas-dev/pandas/pull/25953?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).

whatsnew line has been removed.

Here are the 'full' ASV results.  I re-ran all the deviant tests and removed those that weren't consistent.  Here are the final results:

```
       before           after         ratio
     [00c119c5]       [98abd413]
     <master>         <fix-groupby-performance>
-      20.5±0.2ms       18.5±0.3ms     0.90  stat_ops.SeriesMultiIndexOps.time_op(0, 'mad')
-     4.49±0.04ms       3.77±0.1ms     0.84  groupby.Size.time_category_size
-      13.9±0.2ms       9.63±0.1ms     0.69  stat_ops.SeriesMultiIndexOps.time_op(0, 'kurt')
-     8.78±0.06ms       5.84±0.1ms     0.67  stat_ops.SeriesMultiIndexOps.time_op(1, 'sem')
-      8.82±0.3ms       5.80±0.1ms     0.66  stat_ops.SeriesMultiIndexOps.time_op(1, 'median')
-      8.94±0.4ms       5.82±0.1ms     0.65  stat_ops.SeriesMultiIndexOps.time_op(0, 'sem')
-      8.70±0.2ms       5.60±0.1ms     0.64  stat_ops.SeriesMultiIndexOps.time_op(0, 'median')
-      7.74±0.1ms      4.75±0.07ms     0.61  stat_ops.SeriesMultiIndexOps.time_op(1, 'std')
-      6.57±0.1ms       3.82±0.1ms     0.58  stat_ops.SeriesMultiIndexOps.time_op(0, 'mean')
-      8.62±0.1ms       4.96±0.2ms     0.58  groupby.TransformBools.time_transform_mean
-      7.64±0.1ms      4.37±0.09ms     0.57  stat_ops.SeriesMultiIndexOps.time_op(0, 'std')
-      7.72±0.2ms      4.39±0.09ms     0.57  stat_ops.SeriesMultiIndexOps.time_op(1, 'var')
-      7.31±0.1ms       4.13±0.1ms     0.57  stat_ops.SeriesMultiIndexOps.time_op(0, 'sum')
-      7.41±0.2ms      4.10±0.08ms     0.55  stat_ops.SeriesMultiIndexOps.time_op(1, 'sum')
-     7.63±0.09ms      4.19±0.09ms     0.55  stat_ops.SeriesMultiIndexOps.time_op(0, 'var')
-      7.47±0.2ms       4.06±0.2ms     0.54  stat_ops.SeriesMultiIndexOps.time_op(1, 'mean')
-      7.35±0.2ms       3.98±0.1ms     0.54  stat_ops.SeriesMultiIndexOps.time_op(1, 'prod')
-      7.27±0.2ms      3.83±0.07ms     0.53  stat_ops.SeriesMultiIndexOps.time_op(0, 'prod')
```

Notably, `groupby.Size.time_category_size` and `groupby.TransformBools.time_transform_mean` are also consistently faster by more than 10%.
Hello @ArtificialQualia! Thanks for updating this PR. We checked the lines you've touched for [PEP 8](https://www.python.org/dev/peps/pep-0008) issues, and found:



There are currently no PEP 8 issues detected in this Pull Request. Cheers! :beers: 

##### Comment last updated at 2019-04-28 20:04:36 UTC
Anything more needed on this PR?
merge master as well
Master has been merged

Correct, the change this is fixing was merged in 0.25.  No need for a whatsnew. (see #25743)

Here are the ASV results from a previous comment:

> Here are the 'full' ASV results. I re-ran all the deviant tests and removed those that weren't consistent. Here are the final results:
> 
> ```
>        before           after         ratio
>      [00c119c5]       [98abd413]
>      <master>         <fix-groupby-performance>
> -      20.5±0.2ms       18.5±0.3ms     0.90  stat_ops.SeriesMultiIndexOps.time_op(0, 'mad')
> -     4.49±0.04ms       3.77±0.1ms     0.84  groupby.Size.time_category_size
> -      13.9±0.2ms       9.63±0.1ms     0.69  stat_ops.SeriesMultiIndexOps.time_op(0, 'kurt')
> -     8.78±0.06ms       5.84±0.1ms     0.67  stat_ops.SeriesMultiIndexOps.time_op(1, 'sem')
> -      8.82±0.3ms       5.80±0.1ms     0.66  stat_ops.SeriesMultiIndexOps.time_op(1, 'median')
> -      8.94±0.4ms       5.82±0.1ms     0.65  stat_ops.SeriesMultiIndexOps.time_op(0, 'sem')
> -      8.70±0.2ms       5.60±0.1ms     0.64  stat_ops.SeriesMultiIndexOps.time_op(0, 'median')
> -      7.74±0.1ms      4.75±0.07ms     0.61  stat_ops.SeriesMultiIndexOps.time_op(1, 'std')
> -      6.57±0.1ms       3.82±0.1ms     0.58  stat_ops.SeriesMultiIndexOps.time_op(0, 'mean')
> -      8.62±0.1ms       4.96±0.2ms     0.58  groupby.TransformBools.time_transform_mean
> -      7.64±0.1ms      4.37±0.09ms     0.57  stat_ops.SeriesMultiIndexOps.time_op(0, 'std')
> -      7.72±0.2ms      4.39±0.09ms     0.57  stat_ops.SeriesMultiIndexOps.time_op(1, 'var')
> -      7.31±0.1ms       4.13±0.1ms     0.57  stat_ops.SeriesMultiIndexOps.time_op(0, 'sum')
> -      7.41±0.2ms      4.10±0.08ms     0.55  stat_ops.SeriesMultiIndexOps.time_op(1, 'sum')
> -     7.63±0.09ms      4.19±0.09ms     0.55  stat_ops.SeriesMultiIndexOps.time_op(0, 'var')
> -      7.47±0.2ms       4.06±0.2ms     0.54  stat_ops.SeriesMultiIndexOps.time_op(1, 'mean')
> -      7.35±0.2ms       3.98±0.1ms     0.54  stat_ops.SeriesMultiIndexOps.time_op(1, 'prod')
> -      7.27±0.2ms      3.83±0.07ms     0.53  stat_ops.SeriesMultiIndexOps.time_op(0, 'prod')
> ```
> Notably, `groupby.Size.time_category_size` and `groupby.TransformBools.time_transform_mean` are also consistently faster by more than 10%.


thanks