Repository: pandas-dev/pandas
Pull request: BUG: Coercing bool types to int in qcut
- [x] closes #20303 
- [x] tests added / passed: `pytest pandas/tests/reshape/test_qcut.py pandas/tests/reshape/test_cut.py -v` 
- [x] passes `black pandas`
- [x] passes `git diff upstream/master --name-only -- "*.py" | xargs flake8`
- [x] whatsnew entry


Problem statement:
qcut raising TypeError for boolean Series
#### Code Sample, a copy-pastable example if possible

```python
import pandas as pd
pd.qcut(pd.Series([True, False, False, False, False, False, True]), 6, duplicates="drop", precision=2)
```
#### Problem description

Pandas throws a TypeError:
```
Traceback (most recent call last):
  File "/tmp/pandas/env/lib/python3.5/site-packages/numpy/core/fromnumeric.py", line 52, in _wrapfunc
    return getattr(obj, method)(*args, **kwds)
TypeError: Cannot cast ufunc multiply output from dtype('float64') to dtype('bool') with casting rule 'same_kind'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/tmp/pandas/env/src/pandas/pandas/core/reshape/tile.py", line 210, in qcut
    dtype=dtype, duplicates=duplicates)
  File "/tmp/pandas/env/src/pandas/pandas/core/reshape/tile.py", line 254, in _bins_to_cuts
    dtype=dtype)
  File "/tmp/pandas/env/src/pandas/pandas/core/reshape/tile.py", line 351, in _format_labels
    precision = _infer_precision(precision, bins)
  File "/tmp/pandas/env/src/pandas/pandas/core/reshape/tile.py", line 429, in _infer_precision
    levels = [_round_frac(b, precision) for b in bins]
  File "/tmp/pandas/env/src/pandas/pandas/core/reshape/tile.py", line 429, in <listcomp>
    levels = [_round_frac(b, precision) for b in bins]
  File "/tmp/pandas/env/src/pandas/pandas/core/reshape/tile.py", line 422, in _round_frac
    return np.around(x, digits)
  File "/tmp/pandas/env/lib/python3.5/site-packages/numpy/core/fromnumeric.py", line 2837, in around
    return _wrapfunc(a, 'round', decimals=decimals, out=out)
  File "/tmp/pandas/env/lib/python3.5/site-packages/numpy/core/fromnumeric.py", line 62, in _wrapfunc
    return _wrapit(obj, method, *args, **kwds)
  File "/tmp/pandas/env/lib/python3.5/site-packages/numpy/core/fromnumeric.py", line 42, in _wrapit
    result = getattr(asarray(obj), method)(*args, **kwds)
TypeError: Cannot cast ufunc multiply output from dtype('float64') to dtype('bool') with casting rule 'same_kind'
```

If the second parameter for qcut is changed from 6 to 7, a different TypeError is raised:

```
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/tmp/pandas/env/src/pandas/pandas/core/reshape/tile.py", line 207, in qcut
    bins = algos.quantile(x, quantiles)
  File "/tmp/pandas/env/src/pandas/pandas/core/algorithms.py", line 903, in quantile
    return algos.arrmap_float64(q, _get_score)
  File "pandas/_libs/algos_common_helper.pxi", line 416, in pandas._libs.algos.arrmap_float64
  File "/tmp/pandas/env/src/pandas/pandas/core/algorithms.py", line 888, in _get_score
    idx % 1)
  File "/tmp/pandas/env/src/pandas/pandas/core/algorithms.py", line 876, in _interpolate
    return a + (b - a) * fraction
TypeError: numpy boolean subtract, the `-` operator, is deprecated, use the bitwise_xor, the `^` operator, or the logical_xor function instead.
```

#### Expected Output

Something like

```
0      (0.29, 1.0]
1    (-0.01, 0.29]
2    (-0.01, 0.29]
3    (-0.01, 0.29]
4    (-0.01, 0.29]
5    (-0.01, 0.29]
6      (0.29, 1.0]
dtype: category
Categories (2, interval[float64]): [(-0.01, 0.29] < (0.29, 1.0]]
```

#### Output of ``pd.show_versions()``

Using Pandas 0.23.0.dev0+516.g74e6c78, also reproducable with 0.22.0. In Pandas 0.20.3, the first TypeError is also reproducable, but the second command (with 7 instead of 6) works.

<details>

INSTALLED VERSIONS
------------------
commit: None
python: 3.5.2.final.0
python-bits: 64
OS: Linux
OS-release: 4.13.0-36-generic
machine: x86_64
processor: x86_64
byteorder: little
LC_ALL: None
LANG: de_DE.UTF-8
LOCALE: de_DE.UTF-8

pandas: 0.23.0.dev0+516.g74e6c78
pytest: None
pip: 9.0.1
setuptools: 38.5.2
Cython: 0.27.3
numpy: 1.14.1
scipy: None
pyarrow: None
xarray: None
IPython: None
sphinx: None
patsy: None
dateutil: 2.7.0
pytz: 2018.3
blosc: None
bottleneck: None
tables: None
numexpr: None
feather: None
matplotlib: None
openpyxl: None
xlrd: None
xlwt: None
xlsxwriter: None
lxml: None
bs4: None
html5lib: None
sqlalchemy: None
pymysql: None
psycopg2: None
jinja2: None
s3fs: None
fastparquet: None
pandas_gbq: None
pandas_datareader: None

</details>

Hints from issue discussion:
I suppose could coerce bool types to int in the code. a PR to do this would be great.
```
In [12]: pd.qcut(pd.Series([True, False, False, False, False, False, True]).astype(int), 7, duplicates="drop", precision=2)
Out[12]: 
0      (0.29, 1.0]
1    (-0.01, 0.29]
2    (-0.01, 0.29]
3    (-0.01, 0.29]
4    (-0.01, 0.29]
5    (-0.01, 0.29]
6      (0.29, 1.0]
dtype: category
Categories (2, interval[float64]): [(-0.01, 0.29] < (0.29, 1.0]]
```
I'd be willing to tackle this issue as my first contribution if no one else is already working on it 

@jreback does this still require  a PR - if so, im happy to contribute 
sure #20400 was almost there but didn’t get merged

Commit message: switch np.where condition to check if non nan

Patch to review:
diff --git a/doc/source/whatsnew/v0.25.2.rst b/doc/source/whatsnew/v0.25.2.rst
index 9789c9fce3541..380f081e0650f 100644
--- a/doc/source/whatsnew/v0.25.2.rst
+++ b/doc/source/whatsnew/v0.25.2.rst
@@ -103,6 +103,7 @@ Other
 - Compatibility with Python 3.8 in :meth:`DataFrame.query` (:issue:`27261`)
 - Fix to ensure that tab-completion in an IPython console does not raise
   warnings for deprecated attributes (:issue:`27900`).
+- :func:`qcut` now handles bool ndarray/Series (:issue:`20303`)
 
 .. _whatsnew_0.252.contributors:
 
diff --git a/pandas/core/reshape/tile.py b/pandas/core/reshape/tile.py
index ab354a21a33df..98c95a4a2f6e6 100644
--- a/pandas/core/reshape/tile.py
+++ b/pandas/core/reshape/tile.py
@@ -11,6 +11,7 @@
 from pandas.core.dtypes.common import (
     _NS_DTYPE,
     ensure_int64,
+    is_bool_dtype,
     is_categorical_dtype,
     is_datetime64_dtype,
     is_datetime64tz_dtype,
@@ -423,8 +424,8 @@ def _bins_to_cuts(
 
 def _coerce_to_type(x):
     """
-    if the passed data is of datetime/timedelta type,
-    this method converts it to numeric so that cut method can
+    if the passed data is of datetime/timedelta or bool type,
+    this method converts it to numeric so that cut or qcut method can
     handle it
     """
     dtype = None
@@ -437,10 +438,15 @@ def _coerce_to_type(x):
     elif is_timedelta64_dtype(x):
         x = to_timedelta(x)
         dtype = np.dtype("timedelta64[ns]")
+    elif is_bool_dtype(x):
+        dtype = x.dtype
 
     if dtype is not None:
         # GH 19768: force NaT to NaN during integer conversion
-        x = np.where(x.notna(), x.view(np.int64), np.nan)
+        if is_bool_dtype(x):
+            x = np.where(~np.isnan(x), x.astype(int), np.nan)
+        else:
+            x = np.where(x.notna(), x.view(np.int64), np.nan)
 
     return x, dtype
 
diff --git a/pandas/tests/reshape/test_cut.py b/pandas/tests/reshape/test_cut.py
index a2ebf2359f55f..1a410c1a8fe32 100644
--- a/pandas/tests/reshape/test_cut.py
+++ b/pandas/tests/reshape/test_cut.py
@@ -585,3 +585,17 @@ def test_timedelta_cut_roundtrip():
         ["0 days 23:57:07.200000", "2 days 00:00:00", "3 days 00:00:00"]
     )
     tm.assert_index_equal(result_bins, expected_bins)
+
+
+@pytest.mark.parametrize("bins", [6, 7])
+@pytest.mark.parametrize(
+    "box, compare",
+    [(Series, tm.assert_series_equal), (np.array, tm.assert_categorical_equal)],
+)
+def test_cut_bool_coercion_to_int(bins, box, compare):
+    # issue 20303
+    x = box(np.random.randint(2, size=200))
+    expected = cut(x, bins, duplicates="drop")
+    data = x.astype(bool)
+    result = cut(data, bins, duplicates="drop")
+    compare(result, expected)
diff --git a/pandas/tests/reshape/test_qcut.py b/pandas/tests/reshape/test_qcut.py
index cb46918157e89..77e14932cb0a9 100644
--- a/pandas/tests/reshape/test_qcut.py
+++ b/pandas/tests/reshape/test_qcut.py
@@ -236,3 +236,17 @@ def test_date_like_qcut_bins(arg, expected_bins):
     ser = Series(arg)
     result, result_bins = qcut(ser, 2, retbins=True)
     tm.assert_index_equal(result_bins, expected_bins)
+
+
+@pytest.mark.parametrize("bins", [6, 7])
+@pytest.mark.parametrize(
+    "box, compare",
+    [(Series, tm.assert_series_equal), (np.array, tm.assert_categorical_equal)],
+)
+def test_qcut_bool_coercion_to_int(bins, box, compare):
+    # issue 20303
+    x = box(np.random.randint(2, size=200))
+    expected = qcut(x, bins, duplicates="drop")
+    data = x.astype(bool)
+    result = qcut(data, bins, duplicates="drop")
+    compare(result, expected)


Code review comments:
diff_hunk: @@ -444,7 +444,7 @@ def _coerce_to_type(x):
     if dtype is not None:
         # GH 19768: force NaT to NaN during integer conversion
         if is_bool_dtype(x):
path: pandas/core/reshape/tile.py
text: @author:
@user1 Not entirely sure where this should go - Adding `x.astype(int)` under  `elif is_bool_dtype(x)` higher up throws an error in tests with the existing `np.where(x.notna(), x.view(np.int64), np.nan)` statement if x is ndarray - it passes if x is Series though.
 `AttributeError: 'numpy.ndarray' object has no attribute 'isnan'`
    
It passes if adding i`s_bool_dtype(x)`  with new `np.where` condition using `~np.isnan(x)` like i've done here to account for x being an ndarray

@user1:
so don't use ``np.isnan``, change this to 

```x = np.where(notna(x), x.astype(np.int64, copy=False), np.nan)```

@author:
I've gone with doing the integer conversion in the `elif` block higher up and leaving `dtype` as `None` as @user2 suggested, rather than making any changes here.

diff_hunk: @@ -103,6 +103,7 @@ Other
 - Compatibility with Python 3.8 in :meth:`DataFrame.query` (:issue:`27261`)
 - Fix to ensure that tab-completion in an IPython console does not raise
   warnings for deprecated attributes (:issue:`27900`).
+- :func:`qcut` now handles bool ndarray/Series (:issue:`20303`)
path: doc/source/whatsnew/v0.25.2.rst
text: @user1:
Can you move this to 1.0?  Also, maybe a little more direct to say "boolean input" instead of specifying specific containers.

diff_hunk: @@ -585,3 +585,17 @@ def test_timedelta_cut_roundtrip():
         ["0 days 23:57:07.200000", "2 days 00:00:00", "3 days 00:00:00"]
     )
     tm.assert_index_equal(result_bins, expected_bins)
+
+
+@pytest.mark.parametrize("bins", [6, 7])
+@pytest.mark.parametrize(
+    "box, compare",
+    [(Series, tm.assert_series_equal), (np.array, tm.assert_categorical_equal)],
+)
+def test_cut_bool_coercion_to_int(bins, box, compare):
+    # issue 20303
+    x = box(np.random.randint(2, size=200))
path: pandas/tests/reshape/test_cut.py
text: @user1:
Can explicitly specify the values here?  If we're adding `list` might need to make things a little more verbose since `astype` won't work for `list`.  Maybe a setup like:
```python
data_expected = box([0, 1, 1, 0, 1] * 10)
data_result = box([False, True, True, False, True] * 10)
```
Likewise for the `qcut` test.

diff_hunk: @@ -437,10 +438,15 @@ def _coerce_to_type(x):
     elif is_timedelta64_dtype(x):
         x = to_timedelta(x)
         dtype = np.dtype("timedelta64[ns]")
+    elif is_bool_dtype(x):
+        dtype = x.dtype
path: pandas/core/reshape/tile.py
text: @user1:
I think it should be fine to just do the integer conversion here and leave `dtype` as `None`:
```python
elif is_bool_dtype(x):
   # GH 20303
   x = x.astype(np.int64)
```
It looks like `dtype` is only actually used for datetimelike  dtypes throughout the `cut`/`qcut` process.  This also means that you don't need to worry about what goes on in the `if dtype is not None:` block.

diff_hunk: @@ -585,3 +585,17 @@ def test_timedelta_cut_roundtrip():
         ["0 days 23:57:07.200000", "2 days 00:00:00", "3 days 00:00:00"]
     )
     tm.assert_index_equal(result_bins, expected_bins)
+
+
+@pytest.mark.parametrize("bins", [6, 7])
+@pytest.mark.parametrize(
+    "box, compare",
+    [(Series, tm.assert_series_equal), (np.array, tm.assert_categorical_equal)],
path: pandas/tests/reshape/test_cut.py
text: @user1:
Can you add `list` as a `box` since this should also work for something like `[True, True, False]`?