Repository: pandas-dev/pandas

Issue:
ValueError when trying to compute Quantile 
``` python
In [7]: df = pd.DataFrame(np.random.randn(10, 2))

In [8]: df.iloc[1, 1] = np.nan

In [9]: df.quantile(.5)
---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
<ipython-input-9-66d518aa86c6> in <module>()
----> 1 df.quantile(.5)

/Users/tom.augspurger/Envs/py3/lib/python3.5/site-packages/pandas-0.19.0rc1+21.ge596cbf-py3.5-macosx-10.11-x86_64.egg/pandas/core/frame.py in quantile(self, q, axis, numeric_only, interpolation)
   5152                                      axis=1,
   5153                                      interpolation=interpolation,
-> 5154                                      transposed=is_transposed)
   5155
   5156         if result.ndim == 2:

/Users/tom.augspurger/Envs/py3/lib/python3.5/site-packages/pandas-0.19.0rc1+21.ge596cbf-py3.5-macosx-10.11-x86_64.egg/pandas/core/internals.py in quantile(self, **kwargs)
   3142
   3143     def quantile(self, **kwargs):
-> 3144         return self.reduction('quantile', **kwargs)
   3145
   3146     def setitem(self, **kwargs):

/Users/tom.augspurger/Envs/py3/lib/python3.5/site-packages/pandas-0.19.0rc1+21.ge596cbf-py3.5-macosx-10.11-x86_64.egg/pandas/core/internals.py in reduction(self, f, axis, consolidate, transposed, **kwargs)
   3071         for b in self.blocks:
   3072             kwargs['mgr'] = self
-> 3073             axe, block = getattr(b, f)(axis=axis, **kwargs)
   3074
   3075             axes.append(axe)

/Users/tom.augspurger/Envs/py3/lib/python3.5/site-packages/pandas-0.19.0rc1+21.ge596cbf-py3.5-macosx-10.11-x86_64.egg/pandas/core/internals.py in quantile(self, qs, interpolation, axis, mgr)
   1325             values = _block_shape(values[~mask], ndim=self.ndim)
   1326             if self.ndim > 1:
-> 1327                 values = values.reshape(result_shape)
   1328
   1329         from pandas import Float64Index

ValueError: total size of new array must be unchanged
```

original post follows

---

I have a simple dataframe that I created as follows:

`df[df['Week of'] == week]`

where `week` is a week name I'm filtering by

I have been taking the quartile values of this dataframe as follows:

`df[df['Week of'] == week].quantile(.25)`

However since the update to Pandas 0.19 I am receiving the error (this code worked fine before):

`values = values.reshape(result_shape)`
`ValueError: total size of new array must be unchanged`



Discussion:
Can you please make this a fully reproducible example with dummy data?

I have tracked this error down to there being NaN values in some, but not all, of the columns for a row (2 out of 10 in this case). I then tried to compute the quartile of that DF and pandas did not like this. My solution is to plug the NaN values with 0.

Edited in a reproducible example. Hard to say for sure, but maybe related to https://github.com/pydata/pandas/commit/4de83d25d751d8ca102867b2d46a5547c01d7248

It's definitely related to a (float) block having some cols with missing values:

``` python
In [11]: df = pd.DataFrame(np.random.randint(0, 10, size=(10, 2)))
In [13]: df.iloc[1, 1] = np.nan

In [14]: df.quantile(.5)
Out[14]:
0    4.5
1    7.0
Name: 0.5, dtype: float64
```

and 

``` python
In [15]: df = pd.DataFrame(np.random.randn(10, 2))

In [17]: df.iloc[0, :] = np.nan

In [18]: df.quantile(.5)
Out[18]:
0    0.347815
1    0.072105
Name: 0.5, dtype: float64
```

both work

```
In [10]: pd.__version__
Out[10]: '0.19.0'

In [11]: np.random.seed(1234)

In [12]: df = pd.DataFrame(np.random.randn(10, 2))
    ...:
    ...:
    ...: df.iloc[0, :] = np.nan
    ...:

In [13]: df
Out[13]:
          0         1
0       NaN       NaN
1  1.432707 -0.312652
2 -0.720589  0.887163
3  0.859588 -0.636524
4  0.015696 -2.242685
5  1.150036  0.991946
6  0.953324 -2.021255
7 -0.334077  0.002118
8  0.405453  0.289092
9  1.321158 -1.546906

In [14]: df.median()
Out[14]:
0    0.859588
1   -0.312652
dtype: float64

In [15]: df.quantile(0.5)
Out[15]:
0    0.859588
1   -0.312652
Name: 0.5, dtype: float64

In [16]: df = pd.DataFrame(np.random.randint(0, 10, size=(10, 2)))
    ...: df.iloc[1, 1] = np.nan
    ...:
    ...:

In [17]: df
Out[17]:
   0    1
0  0  3.0
1  2  NaN
2  1  3.0
3  1  3.0
4  7  1.0
5  7  4.0
6  0  5.0
7  1  5.0
8  9  9.0
9  4  0.0

In [18]: df.median()
Out[18]:
0    1.5
1    3.0
dtype: float64

In [19]: df.quantile(0.5)
Out[19]:
0    1.5
1    3.0
Name: 0.5, dtype: float64
```

@Rubyj you'll have to show a complete end-to-end reproducible example. This was a bug in 0.18.1 but is correct in 0.19.0.

@jreback the problem seems to be a DataFrame with a FloatBloack that has at least 1 col with _no_ missing values and at least 1 col with some missing values (see my edit at the top of the OP)

@jreback 

@TomAugspurger provided a reproducible example for me in my original post and added the labels that you removed. Not sure if you saw that. Thank you Tom :+1: 

@TomAugspurger your example works, I see that you changed the top of post. thanks.

so in this case, the individual dims needs to be iterated (corresponding with the columns). with the quantiling then combined, rather than doing this all at once. numpy doesn't handle the nans in the quantiling.


Resolving patch:
diff --git a/pandas/core/internals.py b/pandas/core/internals.py
--- a/pandas/core/internals.py
+++ b/pandas/core/internals.py
@@ -6,7 +6,6 @@
 from collections import defaultdict
 
 import numpy as np
-from numpy import percentile as _quantile
 
 from pandas.core.base import PandasObject
 
@@ -1315,16 +1314,38 @@ def quantile(self, qs, interpolation='linear', axis=0, mgr=None):
 
         values = self.get_values()
         values, _, _, _ = self._try_coerce_args(values, values)
-        mask = isnull(self.values)
-        if not lib.isscalar(mask) and mask.any():
 
-            # even though this could be a 2-d mask it appears
-            # as a 1-d result
-            mask = mask.reshape(values.shape)
-            result_shape = tuple([values.shape[0]] + [-1] * (self.ndim - 1))
-            values = _block_shape(values[~mask], ndim=self.ndim)
-            if self.ndim > 1:
-                values = values.reshape(result_shape)
+        def _nanpercentile1D(values, mask, q, **kw):
+            values = values[~mask]
+
+            if len(values) == 0:
+                if is_scalar(q):
+                    return self._na_value
+                else:
+                    return np.array([self._na_value] * len(q),
+                                    dtype=values.dtype)
+
+            return np.percentile(values, q, **kw)
+
+        def _nanpercentile(values, q, axis, **kw):
+
+            mask = isnull(self.values)
+            if not is_scalar(mask) and mask.any():
+                if self.ndim == 1:
+                    return _nanpercentile1D(values, mask, q, **kw)
+                else:
+                    # for nonconsolidatable blocks mask is 1D, but values 2D
+                    if mask.ndim < values.ndim:
+                        mask = mask.reshape(values.shape)
+                    if axis == 0:
+                        values = values.T
+                        mask = mask.T
+                    result = [_nanpercentile1D(val, m, q, **kw) for (val, m)
+                              in zip(list(values), list(mask))]
+                    result = np.array(result, dtype=values.dtype, copy=False).T
+                    return result
+            else:
+                return np.percentile(values, q, axis=axis, **kw)
 
         from pandas import Float64Index
         is_empty = values.shape[axis] == 0
@@ -1343,13 +1364,13 @@ def quantile(self, qs, interpolation='linear', axis=0, mgr=None):
             else:
 
                 try:
-                    result = _quantile(values, np.array(qs) * 100,
-                                       axis=axis, **kw)
+                    result = _nanpercentile(values, np.array(qs) * 100,
+                                            axis=axis, **kw)
                 except ValueError:
 
                     # older numpies don't handle an array for q
-                    result = [_quantile(values, q * 100,
-                                        axis=axis, **kw) for q in qs]
+                    result = [_nanpercentile(values, q * 100,
+                                             axis=axis, **kw) for q in qs]
 
                 result = np.array(result, copy=False)
                 if self.ndim > 1:
@@ -1368,7 +1389,7 @@ def quantile(self, qs, interpolation='linear', axis=0, mgr=None):
                 else:
                     result = np.array([self._na_value] * len(self))
             else:
-                result = _quantile(values, qs * 100, axis=axis, **kw)
+                result = _nanpercentile(values, qs * 100, axis=axis, **kw)
 
         ndim = getattr(result, 'ndim', None) or 0
         result = self._try_coerce_result(result)


Test patch:
diff --git a/pandas/tests/frame/test_quantile.py b/pandas/tests/frame/test_quantile.py
--- a/pandas/tests/frame/test_quantile.py
+++ b/pandas/tests/frame/test_quantile.py
@@ -262,6 +262,11 @@ def test_quantile_datetime(self):
                              index=[0.5], columns=[0, 1])
         assert_frame_equal(result, expected)
 
+        # empty when numeric_only=True
+        # FIXME (gives empty frame in 0.18.1, broken in 0.19.0)
+        # result = df[['a', 'c']].quantile(.5)
+        # result = df[['a', 'c']].quantile([.5])
+
     def test_quantile_invalid(self):
         msg = 'percentiles should all be in the interval \\[0, 1\\]'
         for invalid in [-1, 2, [0.5, -1], [0.5, 2]]:
@@ -340,3 +345,95 @@ def test_quantile_box(self):
                              pd.Timedelta('2 days')]],
                            index=[0.5], columns=list('AaBbCc'))
         tm.assert_frame_equal(res, exp)
+
+    def test_quantile_nan(self):
+
+        # GH 14357 - float block where some cols have missing values
+        df = DataFrame({'a': np.arange(1, 6.0), 'b': np.arange(1, 6.0)})
+        df.iloc[-1, 1] = np.nan
+
+        res = df.quantile(0.5)
+        exp = Series([3.0, 2.5], index=['a', 'b'], name=0.5)
+        tm.assert_series_equal(res, exp)
+
+        res = df.quantile([0.5, 0.75])
+        exp = DataFrame({'a': [3.0, 4.0], 'b': [2.5, 3.25]}, index=[0.5, 0.75])
+        tm.assert_frame_equal(res, exp)
+
+        res = df.quantile(0.5, axis=1)
+        exp = Series(np.arange(1.0, 6.0), name=0.5)
+        tm.assert_series_equal(res, exp)
+
+        res = df.quantile([0.5, 0.75], axis=1)
+        exp = DataFrame([np.arange(1.0, 6.0)] * 2, index=[0.5, 0.75])
+        tm.assert_frame_equal(res, exp)
+
+        # full-nan column
+        df['b'] = np.nan
+
+        res = df.quantile(0.5)
+        exp = Series([3.0, np.nan], index=['a', 'b'], name=0.5)
+        tm.assert_series_equal(res, exp)
+
+        res = df.quantile([0.5, 0.75])
+        exp = DataFrame({'a': [3.0, 4.0], 'b': [np.nan, np.nan]},
+                        index=[0.5, 0.75])
+        tm.assert_frame_equal(res, exp)
+
+    def test_quantile_nat(self):
+
+        # full NaT column
+        df = DataFrame({'a': [pd.NaT, pd.NaT, pd.NaT]})
+
+        res = df.quantile(0.5, numeric_only=False)
+        exp = Series([pd.NaT], index=['a'], name=0.5)
+        tm.assert_series_equal(res, exp)
+
+        res = df.quantile([0.5], numeric_only=False)
+        exp = DataFrame({'a': [pd.NaT]}, index=[0.5])
+        tm.assert_frame_equal(res, exp)
+
+        # mixed non-null / full null column
+        df = DataFrame({'a': [pd.Timestamp('2012-01-01'),
+                              pd.Timestamp('2012-01-02'),
+                              pd.Timestamp('2012-01-03')],
+                        'b': [pd.NaT, pd.NaT, pd.NaT]})
+
+        res = df.quantile(0.5, numeric_only=False)
+        exp = Series([pd.Timestamp('2012-01-02'), pd.NaT], index=['a', 'b'],
+                     name=0.5)
+        tm.assert_series_equal(res, exp)
+
+        res = df.quantile([0.5], numeric_only=False)
+        exp = DataFrame([[pd.Timestamp('2012-01-02'), pd.NaT]], index=[0.5],
+                        columns=['a', 'b'])
+        tm.assert_frame_equal(res, exp)
+
+    def test_quantile_empty(self):
+
+        # floats
+        df = DataFrame(columns=['a', 'b'], dtype='float64')
+
+        res = df.quantile(0.5)
+        exp = Series([np.nan, np.nan], index=['a', 'b'], name=0.5)
+        tm.assert_series_equal(res, exp)
+
+        res = df.quantile([0.5])
+        exp = DataFrame([[np.nan, np.nan]], columns=['a', 'b'], index=[0.5])
+        tm.assert_frame_equal(res, exp)
+
+        # FIXME (gives empty frame in 0.18.1, broken in 0.19.0)
+        # res = df.quantile(0.5, axis=1)
+        # res = df.quantile([0.5], axis=1)
+
+        # ints
+        df = DataFrame(columns=['a', 'b'], dtype='int64')
+
+        # FIXME (gives empty frame in 0.18.1, broken in 0.19.0)
+        # res = df.quantile(0.5)
+
+        # datetimes
+        df = DataFrame(columns=['a', 'b'], dtype='datetime64')
+
+        # FIXME (gives NaNs instead of NaT in 0.18.1 or 0.19.0)
+        # res = df.quantile(0.5, numeric_only=False)
diff --git a/pandas/tests/series/test_quantile.py b/pandas/tests/series/test_quantile.py
--- a/pandas/tests/series/test_quantile.py
+++ b/pandas/tests/series/test_quantile.py
@@ -184,3 +184,35 @@ def test_quantile_nat(self):
 
         res = Series([pd.NaT, pd.NaT]).quantile([0.5])
         tm.assert_series_equal(res, pd.Series([pd.NaT], index=[0.5]))
+
+    def test_quantile_empty(self):
+
+        # floats
+        s = Series([], dtype='float64')
+
+        res = s.quantile(0.5)
+        self.assertTrue(np.isnan(res))
+
+        res = s.quantile([0.5])
+        exp = Series([np.nan], index=[0.5])
+        tm.assert_series_equal(res, exp)
+
+        # int
+        s = Series([], dtype='int64')
+
+        res = s.quantile(0.5)
+        self.assertTrue(np.isnan(res))
+
+        res = s.quantile([0.5])
+        exp = Series([np.nan], index=[0.5])
+        tm.assert_series_equal(res, exp)
+
+        # datetime
+        s = Series([], dtype='datetime64[ns]')
+
+        res = s.quantile(0.5)
+        self.assertTrue(res is pd.NaT)
+
+        res = s.quantile([0.5])
+        exp = Series([pd.NaT], index=[0.5])
+        tm.assert_series_equal(res, exp)