Repository: pandas-dev/pandas
Pull request: Tests for where() with categorical
- [X] closes #18888
- [X] tests added / passed
- [X] passes `black pandas`
- [X] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`


Hey everyone, 

I wrote some tests for issue #18888. I don't know exactly if the format is correct but if any change is needed I'm happy to make it right. 

Which 'whatsnew entry' should I report for both new tests?

Also, if you think this is a decent contribution and if there's no problem with doing that, it would be nice to tag this PR with hacktoberfest-accepted label :)

Problem statement:
Category type is discarded with `where` series method
#### Code Sample, a copy-pastable example if possible

```python
>>> s = pd.Series(["A","A","B","B","C"],dtype='category')
>>> s.dtype
CategoricalDtype(categories=['A', 'B', 'C'], ordered=False)
>>> s.where(s!="C", s)
0    A
1    A
2    B
3    B
4    C
dtype: object
>>> s2 = pd.Series(range(5))
>>> s2.dtype
dtype('int64')
>>> s2.where(s2<3,s2+3)
0    0
1    1
2    2
3    6
4    7
dtype: int64
```
#### Problem description

Categories are dropped when using the `pd.Series.where` method. This increases memory usage for categorical data by making large temporaries and increases the noisiness of code as the type must be reinforced after the statement.

#### Expected Output

```
>>> s.where(s!="C", s).dtype
CategoricalDtype(categories=['A', 'B', 'C'], ordered=False)
```

#### Output of ``pd.show_versions()``

Is the one in the ubuntu repos:

```
>>> pd.show_versions()

INSTALLED VERSIONS
------------------
commit: None
python: 3.5.2.final.0
python-bits: 64
OS: Linux
OS-release: 4.4.0-43-Microsoft
machine: x86_64
processor: x86_64
byteorder: little
LC_ALL: None
LANG: en_US.UTF-8
LOCALE: en_US.UTF-8

pandas: 0.21.0
pytest: None
pip: 9.0.1
setuptools: 20.7.0
Cython: None
numpy: 1.13.3
scipy: 1.0.0
pyarrow: None
xarray: None
IPython: 6.2.1
sphinx: None
patsy: None
dateutil: 2.6.1
pytz: 2017.3
blosc: None
bottleneck: None
tables: None
numexpr: None
feather: None
matplotlib: None
openpyxl: None
xlrd: 1.1.0
xlwt: None
xlsxwriter: None
lxml: None
bs4: None
html5lib: 1.0b10
sqlalchemy: None
pymysql: None
psycopg2: None
jinja2: 2.8
s3fs: None
fastparquet: None
pandas_gbq: None
pandas_datareader: None
```

Hints from issue discussion:
Thx for the report. In addition, categorical `DataFrame` raises `ValueError`.

```
s = pd.Series(["A","A","B","B","C"],dtype='category')
df = s.to_frame()
df.where(df != "C")
# ValueError: Wrong number of items passed 5, placement implies 1
```

```
# OK
s = pd.Series(["A","A","B","B","C"])
df = s.to_frame()
df.where(df != "C")
```
This is working on master.

```
>>> import pandas as pd
>>>
>>> pd.__version__
'1.1.0.dev0+1381.g0db2286c9'
>>>
>>> s = pd.Series(["A", "A", "B", "B", "C"], dtype="category")
>>> s.dtype
CategoricalDtype(categories=['A', 'B', 'C'], ordered=False)
>>>
>>>
>>> s.where(s != "C")
0      A
1      A
2      B
3      B
4    NaN
dtype: category
Categories (3, object): [A, B, C]
>>>
>>> df = s.to_frame()
>>> df.where(df != "C")
     0
0    A
1    A
2    B
3    B
4  NaN
>>>
>>> _.dtypes
0    category
dtype: object
>>>
```

Commit message: fix black pandas

Patch to review:
diff --git a/pandas/tests/series/indexing/test_where.py b/pandas/tests/series/indexing/test_where.py
index c4a2cb90f7090..c4edcbdc1f0cf 100644
--- a/pandas/tests/series/indexing/test_where.py
+++ b/pandas/tests/series/indexing/test_where.py
@@ -452,3 +452,18 @@ def test_where_empty_series_and_empty_cond_having_non_bool_dtypes():
     ser = Series([], dtype=float)
     result = ser.where([])
     tm.assert_series_equal(result, ser)
+
+
+def test_where_category_not_dropping():
+
+    s = pd.Series(["A", "A", "B", "B", "C"], dtype="category")
+    exp = s.dtype
+    s.where(s != "C", s)
+    res = s.dtype
+    assert exp == res
+
+
+def test_where_categorical_frame_ValueError():
+    s = pd.Series(["A", "A", "B", "B", "C"], dtype="category")
+    df = s.to_frame()
+    df.where(df != "C")


Code review comments:
diff_hunk: @@ -452,3 +452,18 @@ def test_where_empty_series_and_empty_cond_having_non_bool_dtypes():
     ser = Series([], dtype=float)
     result = ser.where([])
     tm.assert_series_equal(result, ser)
+
+
+def test_where_category_not_dropping():
+
+    s = pd.Series(["A", "A", "B", "B", "C"], dtype="category")
+    exp = s.dtype
+    s.where(s != "C", s)
+    res = s.dtype
+    assert exp == res
+
+
+def test_where_categorical_frame_ValueError():
+    s = pd.Series(["A", "A", "B", "B", "C"], dtype="category")
+    df = s.to_frame()
+    df.where(df != "C")
path: pandas/tests/series/indexing/test_where.py
text: @user1:
here also construct the expected frame and `assert_frame_equal` against that

@author:
But here, after we apply df.where(df != "C") the result frame would have a NaN category. How can I construct the expected then?

@user1:
You can use `np.nan` or `pd.NA` to enter the null values.

We have a few examples in the user guide, under `Object Creation`:
https://pandas.pydata.org/pandas-docs/stable/user_guide/categorical.html

The `pd.Categorical` API reference is another place to look:
https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.Categorical.html

@author:
Great, thanks!

diff_hunk: @@ -452,3 +452,18 @@ def test_where_empty_series_and_empty_cond_having_non_bool_dtypes():
     ser = Series([], dtype=float)
     result = ser.where([])
     tm.assert_series_equal(result, ser)
+
+
+def test_where_category_not_dropping():
+
+    s = pd.Series(["A", "A", "B", "B", "C"], dtype="category")
+    exp = s.dtype
+    s.where(s != "C", s)
+    res = s.dtype
+    assert exp == res
path: pandas/tests/series/indexing/test_where.py
text: @user1:
you want to construct the expected result and test for the equality of the series using `tm.assert_series_equal` - see neighboring tests for examples

@user1:
use 
``` python
res = ...
expected = ...
assert_..._equal(res, expected)
```
in both tests

diff_hunk: @@ -452,3 +452,18 @@ def test_where_empty_series_and_empty_cond_having_non_bool_dtypes():
     ser = Series([], dtype=float)
     result = ser.where([])
     tm.assert_series_equal(result, ser)
+
+
+def test_where_category_not_dropping():
+
+    s = pd.Series(["A", "A", "B", "B", "C"], dtype="category")
+    exp = s.dtype
+    s.where(s != "C", s)
+    res = s.dtype
+    assert exp == res
+
+
+def test_where_categorical_frame_ValueError():
+    s = pd.Series(["A", "A", "B", "B", "C"], dtype="category")
+    df = s.to_frame()
path: pandas/tests/series/indexing/test_where.py
text: @user1:
I think you can use the dataframe constructor directly here

diff_hunk: @@ -452,3 +452,18 @@ def test_where_empty_series_and_empty_cond_having_non_bool_dtypes():
     ser = Series([], dtype=float)
     result = ser.where([])
     tm.assert_series_equal(result, ser)
+
+
+def test_where_category_not_dropping():
+
+    s = pd.Series(["A", "A", "B", "B", "C"], dtype="category")
+    exp = s.dtype
+    s.where(s != "C", s)
+    res = s.dtype
+    assert exp == res
+
+
+def test_where_categorical_frame_ValueError():
path: pandas/tests/series/indexing/test_where.py
text: @user1:
call this `test_where_frame_with_categorical_column`

diff_hunk: @@ -452,3 +452,18 @@ def test_where_empty_series_and_empty_cond_having_non_bool_dtypes():
     ser = Series([], dtype=float)
     result = ser.where([])
     tm.assert_series_equal(result, ser)
+
+
+def test_where_category_not_dropping():
path: pandas/tests/series/indexing/test_where.py
text: @user1:
call this `test_where_categorical_series`