PERF: optimized median func when bottleneck not present
State: closed

closes #16468
 
Uses np.nanmedian to compute median instead of internal algos.median when bottleneck is not present/turned off. Has order of magnitude performance benefits

`df = pd.DataFrame(np.random.randn(10000, 2), columns=list('AB'))`

Before
```
>>> pd.set_option('use_bottleneck', False)
>>> %timeit df.median(1)
318 ms ± 8.56 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)

>>> %timeit pd.Series(np.nanmedian(df.values, axis=1), index=df.index)
1.83 ms ± 123 µs per loop (mean ± std. dev. of 7 runs, 1000 loops each)

>>> pd.set_option('use_bottleneck', True)
>>> %timeit df.median(1)
239 µs ± 2.23 µs per loop (mean ± std. dev. of 7 runs, 1000 loops each)
```

After
```
>>> pd.set_option('use_bottleneck', False)
>>> %timeit df.median(1)
1.89 ms ± 53.2 µs per loop (mean ± std. dev. of 7 runs, 1000 loops each)

>>> pd.set_option('use_bottleneck', True)
>>> %timeit df.median(1)
227 µs ± 11 µs per loop (mean ± std. dev. of 7 runs, 1000 loops each)
```



Labels:
name: Performance
description: Memory or execution speed performance
name: Numeric Operations
description: Arithmetic, Comparison, and Logical operations

Comments:
needs an asv (run with and w/o bottleneck)
Do i need to run all the tests or can i get away with a subset?
Also do I have to manually turn bottleneck on and off for the tests?
> Also do I have to manually turn bottleneck on and off for the tests?

have 2 tests, 1 with it forced on, 1 with forced off

> Do i need to run all the tests or can i get away with a subset?

just the median tests :>
actually I don't think bottleneck is in our asv config. (separate issue is it should be)
> just the median tests :\>

does that just entail `test_nanops.py`?
@rohanp `asv continuous -f 1.1 upstream/master HEAD -b median` should work. See http://pandas.pydata.org/pandas-docs/stable/contributing.html#running-the-performance-test-suite
I don't understand why the tests are failing. How do I see the test cases that failed?
click on the red x
I'm not quite sure what you mean. I figured out how to see the details of the tests that failed; for example on circleci I can see

```
self = <pandas.tests.test_nanops.TestnanopsDataFrame object at 0x7f67d2d65c88>

    def test_nanmedian(self):
        with warnings.catch_warnings(record=True):
            self.check_funs(nanops.nanmedian, np.median, allow_complex=False,
                            allow_str=False, allow_date=False,
>                           allow_tdelta=True, allow_obj='convert')

pandas/tests/test_nanops.py:374: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
pandas/tests/test_nanops.py:245: in check_funs
    **kwargs)
pandas/tests/test_nanops.py:234: in check_fun
    targarnanval, **kwargs)
pandas/tests/test_nanops.py:193: in check_fun_data
    check_dtype=check_dtype)
pandas/tests/test_nanops.py:146: in check_results
    tm.assert_almost_equal(targ, res, check_dtype=check_dtype)
pandas/util/testing.py:234: in assert_almost_equal
    **kwargs)
pandas/_libs/testing.pyx:59: in pandas._libs.testing.assert_almost_equal (pandas/_libs/testing.c:4156)
    cpdef assert_almost_equal(a, b,
pandas/_libs/testing.pyx:173: in pandas._libs.testing.assert_almost_equal (pandas/_libs/testing.c:3274)
    raise_assert_detail(obj, msg, lobj, robj)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

obj = 'numpy array', message = 'numpy array values are different (20.0 %)'
left = '[[nan, nan, nan, nan, nan], [nan, nan, nan, nan, nan], [nan, nan, nan, nan, nan], [nan, nan, nan, nan, nan], [nan, nan, nan, nan, nan], [nan, nan, nan, nan, nan], [nan, nan, nan, nan, nan]]'
right = '[[0.0511871111488, -0.458724351947, -0.0208572588354, 0.00471620824214, -0.487450340243], [0.254176310551, -0.0309721..., -0.2722898607, 0.174928469595], [-0.431481894512, 0.245270844279, 0.140055994353, -0.0727601005076, 0.238682362073]]'
```

But what I do not understand is how to find the input that caused the test to fail. I checked the `test_nanops.py` file and while it has the tests it does not seem to have the test cases.
@rohanp these tests have several layers of indirection (to allow more code reuse). I would try `pytest pandsa/tests/test_nanops.py -k test_nanmedian --pdb` to drop into a debugger on failure so you can inspect what's going on. But the basic idea is `test_nanmedian` -> `check_funs` -> `check_fun` -> `check_fun_data` -> `check_results`, which is failing for one of the input array.

The actual arrays will be something like `self.arr_float` or `arr_float_nan` and are accessed with `getattr`.
can you rebase and update?
can you rebase and update
can you rebase / update according to comments
Hello @rohanp! Thanks for updating the PR.



Cheers ! There are no PEP8 issues in this Pull Request. :beers: 

##### Comment last updated on January 21, 2018 at 21:03 Hours UTC
Hi, sorry for the delay. I think I figured out the bug. Thanks @TomAugspurger for the debugging command. How do I rebase and update? 
@rohanp you can do the following (assuming that pandas-dev/pandas remote repo is called 'upstream')

```
git fetch upstream
git checkout rohanp
git merge upstream/master # after this you can fix merge conflicts, and do 'git commit' to continue
git push origin rohanp
```
Hi @jorisvandenbossche, my changes are causing the tests to fail. I'm still looking into why. I'll remove that once I get to the bottom of what the problem is.
> my changes are causing the tests to fail. I'm still looking into why

see my comment above, isnull was recently be renamed to isna in this file, that is the reason all tests are failing now (but, we also asked to remove the part that uses isnull (or explain why it is needed)) 
Hi, I had put that in because the tests were failing and I thought that would fix it (but apparently not). The tests are still failing without it. 
needs a rebase. ping on green.
@rohanp you have some failing tests
I don't understand—all the errors seem to be in tests unrelated to nanmedian.

```
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> traceback >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>

self = <pandas.tests.series.test_operators.TestSeriesOperators object at 0x113abe278>
test_input = 0    foo
1    bar
2    baz
dtype: object, error_type = <class 'TypeError'>

    @pytest.mark.parametrize(
        "test_input,error_type",
        [
            (pd.Series([]), ValueError),
    
            # For strings, or any Series with dtype 'O'
            (pd.Series(['foo', 'bar', 'baz']), TypeError),
            (pd.Series([(1,), (2,)]), TypeError),
    
            # For mixed data types
            (
                pd.Series(['foo', 'foo', 'bar', 'bar', None, np.nan, 'baz']),
                TypeError
            ),
        ]
    )
    def test_assert_argminmax_raises(self, test_input, error_type):
        """
            Cases where ``Series.argmax`` and related should raise an exception
            """
        with pytest.raises(error_type):
>           test_input.argmin()

pandas/tests/series/test_operators.py:1882: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
pandas/core/series.py:1310: in idxmin
    i = nanops.nanargmin(_values_from_object(self), skipna=skipna)
pandas/core/nanops.py:513: in nanargmin
    isfinite=True)
pandas/core/nanops.py:212: in _get_values
    mask = _isfinite(values)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

values = array(['foo', 'bar', 'baz'], dtype=object)

    def _isfinite(values):
        if is_datetime_or_timedelta_dtype(values):
            return isna(values)
        if (is_complex_dtype(values) or is_float_dtype(values) or
                is_integer_dtype(values) or is_bool_dtype(values)):
            return ~np.isfinite(values)
>       return ~np.isfinite(values.astype('float64'))
E       ValueError: could not convert string to float: 'baz'

pandas/core/nanops.py:255: ValueError
```
you need to rebase on master and build the extensions, this was a recent change.
The same tests seem to be failing :-/
@rohanp There went something wrong when merging upstream master (when solving the conflicts, you removed some code). See the diff now includes changes in the argmin/argmax methods. If you undo those changes, the tests will pass.
# [Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/16509?src=pr&el=h1) Report
> Merging [#16509](https://codecov.io/gh/pandas-dev/pandas/pull/16509?src=pr&el=desc) into [master](https://codecov.io/gh/pandas-dev/pandas/commit/34c4ffd7c454848311f71e6869b5cad4bc132449?src=pr&el=desc) will **decrease** coverage by `<.01%`.
> The diff coverage is `80%`.

[![Impacted file tree graph](https://codecov.io/gh/pandas-dev/pandas/pull/16509/graphs/tree.svg?width=650&token=eZ4WkYLtcO&height=150&src=pr)](https://codecov.io/gh/pandas-dev/pandas/pull/16509?src=pr&el=tree)

```diff
@@            Coverage Diff             @@
##           master   #16509      +/-   ##
==========================================
- Coverage      91%      91%   -0.01%     
==========================================
  Files         162      162              
  Lines       49547    49554       +7     
==========================================
+ Hits        45092    45096       +4     
- Misses       4455     4458       +3
```

| Flag | Coverage Δ | |
|---|---|---|
| #multiple | `88.78% <80%> (+0.01%)` | :arrow_up: |
| #single | `40.24% <10%> (-0.07%)` | :arrow_down: |

| [Impacted Files](https://codecov.io/gh/pandas-dev/pandas/pull/16509?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [pandas/core/nanops.py](https://codecov.io/gh/pandas-dev/pandas/pull/16509?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvbmFub3BzLnB5) | `96.52% <80%> (-1.13%)` | :arrow_down: |
| [pandas/io/gbq.py](https://codecov.io/gh/pandas-dev/pandas/pull/16509?src=pr&el=tree#diff-cGFuZGFzL2lvL2dicS5weQ==) | `25% <0%> (-58.34%)` | :arrow_down: |
| [pandas/core/frame.py](https://codecov.io/gh/pandas-dev/pandas/pull/16509?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvZnJhbWUucHk=) | `97.72% <0%> (-0.1%)` | :arrow_down: |
| [pandas/plotting/\_converter.py](https://codecov.io/gh/pandas-dev/pandas/pull/16509?src=pr&el=tree#diff-cGFuZGFzL3Bsb3R0aW5nL19jb252ZXJ0ZXIucHk=) | `65.05% <0%> (+1.81%)` | :arrow_up: |

------

[Continue to review full report at Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/16509?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/16509?src=pr&el=footer). Last update [34c4ffd...8b2fd36](https://codecov.io/gh/pandas-dev/pandas/pull/16509?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).

# [Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/16509?src=pr&el=h1) Report
> Merging [#16509](https://codecov.io/gh/pandas-dev/pandas/pull/16509?src=pr&el=desc) into [master](https://codecov.io/gh/pandas-dev/pandas/commit/6ba65ca8a69560d526762bdbded64b3139f66d7a?src=pr&el=desc) will **increase** coverage by `<.01%`.
> The diff coverage is `100%`.

[![Impacted file tree graph](https://codecov.io/gh/pandas-dev/pandas/pull/16509/graphs/tree.svg?width=650&token=eZ4WkYLtcO&height=150&src=pr)](https://codecov.io/gh/pandas-dev/pandas/pull/16509?src=pr&el=tree)

```diff
@@            Coverage Diff             @@
##           master   #16509      +/-   ##
==========================================
+ Coverage   91.57%   91.57%   +<.01%     
==========================================
  Files         150      150              
  Lines       48700    48702       +2     
==========================================
+ Hits        44595    44597       +2     
  Misses       4105     4105
```

| Flag | Coverage Δ | |
|---|---|---|
| #multiple | `89.94% <100%> (ø)` | :arrow_up: |
| #single | `41.71% <16.66%> (-0.01%)` | :arrow_down: |

| [Impacted Files](https://codecov.io/gh/pandas-dev/pandas/pull/16509?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [pandas/core/nanops.py](https://codecov.io/gh/pandas-dev/pandas/pull/16509/diff?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvbmFub3BzLnB5) | `96.69% <100%> (+0.01%)` | :arrow_up: |

------

[Continue to review full report at Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/16509?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/16509?src=pr&el=footer). Last update [6ba65ca...027a2b2](https://codecov.io/gh/pandas-dev/pandas/pull/16509?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).

@jorisvandenbossche thank you for that catch! I've now double checked that nothing in the diff has changed except for the contents of `nanmedian`, and the tests are passing on my local computer, but some of the same errors are popping up on the CI tests. 

pandas/util/testing.py:1105: AssertionError
___________________________ TestPanel4d.test_median ____________________________

self = <pandas.tests.test_panel4d.TestPanel4d object at 0x7f3358785e50>

    def test_median(self):
        def wrapper(x):
            if isna(x).any():
                return np.nan
            return np.median(x)
    
>       self._check_stat_op('median', wrapper)

pandas/tests/test_panel4d.py:54: 
It looks like the tests are passing on when I run `pytest pandas` locally as well as on AppVeyor, but failing on Travis CI and CircleCI.

Could someone please explain how I can replicate the environment for CircleCI on my local computer so that I can debug? I tried looking in the [ci folder](https://github.com/rohanp/pandas/tree/master/ci) but am not sure what I should be looking for (a conda environment.yml?)