Repository: pandas-dev/pandas
Pull request: PERF: optimized median func when bottleneck not present
closes #16468
 
Uses np.nanmedian to compute median instead of internal algos.median when bottleneck is not present/turned off. Has order of magnitude performance benefits

`df = pd.DataFrame(np.random.randn(10000, 2), columns=list('AB'))`

Before
```
>>> pd.set_option('use_bottleneck', False)
>>> %timeit df.median(1)
318 ms ± 8.56 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)

>>> %timeit pd.Series(np.nanmedian(df.values, axis=1), index=df.index)
1.83 ms ± 123 µs per loop (mean ± std. dev. of 7 runs, 1000 loops each)

>>> pd.set_option('use_bottleneck', True)
>>> %timeit df.median(1)
239 µs ± 2.23 µs per loop (mean ± std. dev. of 7 runs, 1000 loops each)
```

After
```
>>> pd.set_option('use_bottleneck', False)
>>> %timeit df.median(1)
1.89 ms ± 53.2 µs per loop (mean ± std. dev. of 7 runs, 1000 loops each)

>>> pd.set_option('use_bottleneck', True)
>>> %timeit df.median(1)
227 µs ± 11 µs per loop (mean ± std. dev. of 7 runs, 1000 loops each)
```



Problem statement:
PERF: .median(axis=1) perf issues
```
In [2]: df = pd.DataFrame(np.random.randn(10000, 2), columns=list('AB'))

In [3]: result1 = df.median(1)

In [4]: result2 = pd.Series(np.nanmedian(df.values, axis=1), index=df.index)

In [5]: result1.equals(result2)
Out[5]: True

In [6]: %timeit result1 = df.median(1)
241 µs ± 4.71 µs per loop (mean ± std. dev. of 7 runs, 1000 loops each)

In [7]: %timeit df.median(1)
250 µs ± 5.74 µs per loop (mean ± std. dev. of 7 runs, 1000 loops each)

In [8]: %timeit pd.Series(np.nanmedian(df.values, axis=1), index=df.index)
1.77 ms ± 32.7 µs per loop (mean ± std. dev. of 7 runs, 1000 loops each)

In [9]: pd.set_option('use_bottleneck', False)

In [10]: result3 = df.median(1)

In [11]: result1.equals(result3)
Out[11]: True

In [12]: %timeit df.median(1)
317 ms ± 9.66 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)
```

So, if *bottleneck* is installed, then ``df.median(1)`` is blazingly fast. However if its NOT installed (or not used), then we fallback to ``np.apply_along_axis(our_median_impl)``, so our median impl is pretty fast itself, but it only handles 1d, so this is a pythonic loop.

To fix we can use ``np.nanmedian`` soln if available (its in >= numpy 1.9, currently we support >= 1.7).

Hints from issue discussion:
I get similar results 

```
>>> df = pd.DataFrame(np.random.randn(10000, 2), columns=list('AB'))

>>> pd.set_option('use_bottleneck', False)
>>> %timeit df.median(1) 
327 ms ± 4.51 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)

>>> %timeit pd.Series(np.nanmedian(df.values, axis=1), index=df.index)
1.83 ms ± 123 µs per loop (mean ± std. dev. of 7 runs, 1000 loops each)

>>> pd.set_option('use_bottleneck', True)
>>> %timeit df.median(1)
239 µs ± 2.23 µs per loop (mean ± std. dev. of 7 runs, 1000 loops each)
```

Why isn't bottleneck a dependency of Pandas? I didn't even know I did not have it installed until now. Even when I set `pd.set_option('use_bottleneck', True)`, Pandas did not give me any warning that I did not have it installed.








see here: http://pandas.pydata.org/pandas-docs/stable/install.html#recommended-dependencies

these *could* be deps, but ``pip`` used to have trouble with these things and they didn't work on all platforms.

and https://github.com/pandas-dev/pandas/issues/9422, which bottleneck changed in 1.0 (breaking the previous, IMHO correct API).
in any event, this is easily fixed by using ``np.nanmedian`` as I said. (which again only recently came about in last 1-2 years).
okay, working on the fix
done: https://github.com/pandas-dev/pandas/pull/16509

Commit message: pep8

Patch to review:
diff --git a/doc/source/whatsnew/v0.21.0.txt b/doc/source/whatsnew/v0.21.0.txt
index 4f55c6388c728..8aac5c410d7e7 100644
--- a/doc/source/whatsnew/v0.21.0.txt
+++ b/doc/source/whatsnew/v0.21.0.txt
@@ -298,6 +298,7 @@ Performance Improvements
 ~~~~~~~~~~~~~~~~~~~~~~~~
 
 - Improved performance of instantiating :class:`SparseDataFrame` (:issue:`16773`)
+- Improved performance of  :func:`median()` when bottleneck is not installed but :func:`np.namnmedian` is present (:issue:`16468`)
 - :attr:`Series.dt` no longer performs frequency inference, yielding a large speedup when accessing the attribute (:issue:`17210`)
 
 
diff --git a/pandas/core/nanops.py b/pandas/core/nanops.py
index b2bbf1c75b7ea..aa4aaad01da49 100644
--- a/pandas/core/nanops.py
+++ b/pandas/core/nanops.py
@@ -5,7 +5,7 @@
 from distutils.version import LooseVersion
 
 import numpy as np
-from pandas import compat
+from pandas import compat, _np_version_under1p9
 from pandas._libs import tslib, algos, lib
 from pandas.core.dtypes.common import (
     _get_dtype,
@@ -344,12 +344,22 @@ def nanmedian(values, axis=None, skipna=True):
 
     values, mask, dtype, dtype_max = _get_values(values, skipna)
 
+    if not skipna:
+        return _wrap_results(np.median(values, axis=axis), dtype)
+
     def get_median(x):
         mask = notna(x)
         if not skipna and not mask.all():
             return np.nan
         return algos.median(_values_from_object(x[mask]))
 
+    if _np_version_under1p9:
+        agg1d = get_median
+        aggaxis = lambda values, axis: np.apply_along_axis(get_median, axis, values)
+    else:
+        agg1d = np.nanmedian
+        aggaxis = np.nanmedian
+
     if not is_float_dtype(values):
         values = values.astype('f8')
         values[mask] = np.nan
@@ -363,8 +373,7 @@ def get_median(x):
     if values.ndim > 1:
         # there's a non-empty array to apply over otherwise numpy raises
         if notempty:
-            return _wrap_results(
-                np.apply_along_axis(get_median, axis, values), dtype)
+            return _wrap_results(aggaxis(values, axis), dtype)
 
         # must return the correct shape, but median is not defined for the
         # empty set so return nans of shape "everything but the passed axis"
@@ -377,7 +386,7 @@ def get_median(x):
         return _wrap_results(ret, dtype)
 
     # otherwise return a scalar value
-    return _wrap_results(get_median(values) if notempty else np.nan, dtype)
+    return _wrap_results(agg1d(values) if notempty else np.nan, dtype)
 
 
 def _get_counts_nanvar(mask, axis, ddof, dtype=float):


Code review comments:
diff_hunk: @@ -344,12 +344,22 @@ def nanmedian(values, axis=None, skipna=True):
 
     values, mask, dtype, dtype_max = _get_values(values, skipna)
 
+    if not skipna:
path: pandas/core/nanops.py
text: @user1:
why are you adding this?

@author:
The tests compare nanmedian to numpy's median function. If this special case were not there, this function would return a non-null result for input with null values and skipna turned off. Here is a debug log demonstrating this: https://gist.github.com/rohanp/1082ad5a5e199f6b1cb8ade965c4519f

@user1:
so fix the test to do the correct comparison

@author:
Isn't the desired behavior for nanmedian to be like median when skipna is turned off though? If not, then what is the skipna flag supposed to do?

@user1:
yes, so use the ``skipna`` flag as a switch when generating which function to use.