Repository: pandas-dev/pandas

Issue:
minor inconsistency in Categorical.remove_categories error message
mypy error: `pandas\core\arrays\categorical.py:1125:13: error: Incompatible types in assignment (expression has type "List[Any]", variable has type "Set[Any]")  [assignment]`


#### Code Sample, a copy-pastable example if possible

```python
Python 3.7.4 (default, Aug  9 2019, 18:34:13) [MSC v.1915 64 bit (AMD64)] :: Anaconda, Inc. on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> import numpy as np
>>> import pandas as pd
>>> pd.__version__
'0.26.0.dev0+537.gd2bc6a497.dirty'
>>>
>>> cat = pd.Categorical(["a", "b", "c", "a"], ordered=True)
>>>
>>> cat
[a, b, c, a]
Categories (3, object): [a < b < c]
>>>
>>> cat = cat.remove_categories(["c"])
>>> cat
[a, b, NaN, a]
Categories (2, object): [a < b]
>>>
>>> cat = cat.remove_categories(["c"])
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "C:\Users\simon\pandas-simonjayhawkins\pandas\core\arrays\categorical.py", line 1130, in remove_categories
    raise ValueError(msg.format(not_included=not_included))
ValueError: removals must all be in old categories: {'c'}
>>>
>>> cat = cat.remove_categories(["c", np.nan])
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "C:\Users\simon\pandas-simonjayhawkins\pandas\core\arrays\categorical.py", line 1130, in remove_categories
    raise ValueError(msg.format(not_included=not_included))
ValueError: removals must all be in old categories: ['c']
>>> cat
[a, b, NaN, a]
Categories (2, object): [a < b]
>>>

```
#### Problem description

The error message shows invalid removals as a set or a list depending on whether the removal list contains a null value.

#### Expected Output

consistent message. preferably as a list.

#### Output of ``pd.show_versions()``

<details>

INSTALLED VERSIONS
------------------
commit           : d2bc6a49778407a02f31c40408e8d62cb24e6b64
python           : 3.7.4.final.0
python-bits      : 64
OS               : Windows
OS-release       : 10
machine          : AMD64
processor        : Intel64 Family 6 Model 58 Stepping 9, GenuineIntel
byteorder        : little
LC_ALL           : None
LANG             : en_GB.UTF-8
LOCALE           : None.None

pandas           : 0.26.0.dev0+537.gd2bc6a497.dirty
numpy            : 1.16.4
pytz             : 2019.2
dateutil         : 2.8.0
pip              : 19.2.2
setuptools       : 41.0.1
Cython           : 0.29.13
pytest           : 5.0.1
hypothesis       : 4.32.2
sphinx           : 1.8.5
blosc            : None
feather          : None
xlsxwriter       : 1.1.8
lxml.etree       : 4.4.1
html5lib         : 1.0.1
pymysql          : None
psycopg2         : None
jinja2           : 2.10.1
IPython          : 7.8.0
pandas_datareader: None
bs4              : 4.7.1
bottleneck       : 1.2.1
fastparquet      : 0.3.0
gcsfs            : None
lxml.etree       : 4.4.1
matplotlib       : 3.1.1
numexpr          : 2.7.0
odfpy            : None
openpyxl         : 2.6.2
pandas_gbq       : None
pyarrow          : 0.13.0
pytables         : None
s3fs             : 0.2.1
scipy            : 1.3.1
sqlalchemy       : 1.3.7
tables           : 3.5.2
xarray           : 0.12.1
xlrd             : 1.2.0
xlwt             : 1.3.0
xlsxwriter       : 1.1.8
>>>

</details>

minor inconsistency in Categorical.remove_categories error message
mypy error: `pandas\core\arrays\categorical.py:1125:13: error: Incompatible types in assignment (expression has type "List[Any]", variable has type "Set[Any]")  [assignment]`


#### Code Sample, a copy-pastable example if possible

```python
Python 3.7.4 (default, Aug  9 2019, 18:34:13) [MSC v.1915 64 bit (AMD64)] :: Anaconda, Inc. on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> import numpy as np
>>> import pandas as pd
>>> pd.__version__
'0.26.0.dev0+537.gd2bc6a497.dirty'
>>>
>>> cat = pd.Categorical(["a", "b", "c", "a"], ordered=True)
>>>
>>> cat
[a, b, c, a]
Categories (3, object): [a < b < c]
>>>
>>> cat = cat.remove_categories(["c"])
>>> cat
[a, b, NaN, a]
Categories (2, object): [a < b]
>>>
>>> cat = cat.remove_categories(["c"])
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "C:\Users\simon\pandas-simonjayhawkins\pandas\core\arrays\categorical.py", line 1130, in remove_categories
    raise ValueError(msg.format(not_included=not_included))
ValueError: removals must all be in old categories: {'c'}
>>>
>>> cat = cat.remove_categories(["c", np.nan])
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "C:\Users\simon\pandas-simonjayhawkins\pandas\core\arrays\categorical.py", line 1130, in remove_categories
    raise ValueError(msg.format(not_included=not_included))
ValueError: removals must all be in old categories: ['c']
>>> cat
[a, b, NaN, a]
Categories (2, object): [a < b]
>>>

```
#### Problem description

The error message shows invalid removals as a set or a list depending on whether the removal list contains a null value.

#### Expected Output

consistent message. preferably as a list.

#### Output of ``pd.show_versions()``

<details>

INSTALLED VERSIONS
------------------
commit           : d2bc6a49778407a02f31c40408e8d62cb24e6b64
python           : 3.7.4.final.0
python-bits      : 64
OS               : Windows
OS-release       : 10
machine          : AMD64
processor        : Intel64 Family 6 Model 58 Stepping 9, GenuineIntel
byteorder        : little
LC_ALL           : None
LANG             : en_GB.UTF-8
LOCALE           : None.None

pandas           : 0.26.0.dev0+537.gd2bc6a497.dirty
numpy            : 1.16.4
pytz             : 2019.2
dateutil         : 2.8.0
pip              : 19.2.2
setuptools       : 41.0.1
Cython           : 0.29.13
pytest           : 5.0.1
hypothesis       : 4.32.2
sphinx           : 1.8.5
blosc            : None
feather          : None
xlsxwriter       : 1.1.8
lxml.etree       : 4.4.1
html5lib         : 1.0.1
pymysql          : None
psycopg2         : None
jinja2           : 2.10.1
IPython          : 7.8.0
pandas_datareader: None
bs4              : 4.7.1
bottleneck       : 1.2.1
fastparquet      : 0.3.0
gcsfs            : None
lxml.etree       : 4.4.1
matplotlib       : 3.1.1
numexpr          : 2.7.0
odfpy            : None
openpyxl         : 2.6.2
pandas_gbq       : None
pyarrow          : 0.13.0
pytables         : None
s3fs             : 0.2.1
scipy            : 1.3.1
sqlalchemy       : 1.3.7
tables           : 3.5.2
xarray           : 0.12.1
xlrd             : 1.2.0
xlwt             : 1.3.0
xlsxwriter       : 1.1.8
>>>

</details>



Discussion:
can I work on it?
the difference in those codes is caused here:

https://github.com/pandas-dev/pandas/blob/c4489cbfa5f93e69301486e1ef867caed9f4eb9e/pandas/core/arrays/categorical.py#L1079-L1130

so you should wrap line 1116 by `list` because 1116 returns `set` and 1122 returns `list`.
on second thoughts, may be better to always display the invalid removals as a set.

this would make more sense for scalar input as well as input with duplicates.

may also mean that the error message is unchanged for the majority of use cases. 
> can I work on it?

go for it.
> on second thoughts, may be better to always display the invalid removals as a set.
> 
> this would make more sense for scalar input as well as input with duplicates.
> 
> may also mean that the error message is unchanged for the majority of use cases.

well then how about make the line 1121 returns as a set. like below:
`not_included = {x for x in not_included if notna(x)}`

and the `not_included` doesn't spread and have any side effects so it's safe to do this.
Can I work on this?

`      inplace = validate_bool_kwarg(inplace, "inplace")
        if not is_list_like(removals):
            removals = [removals]

        removal_set = set(list(removals))
        not_included = removal_set - set(self.dtype.categories)
        new_categories = [c for c in self.dtype.categories if c not in removal_set]

        # GH 10156
        if any(isna(removals)):
            not_included = {x for x in not_included if notna(x)}
            new_categories = [x for x in new_categories if notna(x)]

        if len(not_included) != 0:
            msg = "removals must all be in old categories: {not_included!s}"
            raise ValueError(msg.format(not_included=not_included))

        return self.set_categories(
            new_categories, ordered=self.ordered, rename=False, inplace=inplace
        )
`

I've made the changes. Shall I commit? I am new to contributing to issues on GitHub.
> I've made the changes. Shall I commit?

you'll also need to update `test_remove_categories`  in `pandas/tests/arrays/categorical/test_api.py`  with a `match` parameter to `pytest.raises` to confirm the changes, and add a case for a null value.

may be worth adding cases for the scaler and with duplicates. 
> 
> you'll also need to update `test_remove_categories` in `pandas/tests/arrays/categorical/test_api.py` with a `match` parameter to `pytest.raises` to confirm the changes, and add a case for a null value.

Meaning to remove a null value (as in not present)? Also I didn't understand what the `match` parameter will do.


can I work on it?
the difference in those codes is caused here:

https://github.com/pandas-dev/pandas/blob/c4489cbfa5f93e69301486e1ef867caed9f4eb9e/pandas/core/arrays/categorical.py#L1079-L1130

so you should wrap line 1116 by `list` because 1116 returns `set` and 1122 returns `list`.
on second thoughts, may be better to always display the invalid removals as a set.

this would make more sense for scalar input as well as input with duplicates.

may also mean that the error message is unchanged for the majority of use cases. 
> can I work on it?

go for it.
> on second thoughts, may be better to always display the invalid removals as a set.
> 
> this would make more sense for scalar input as well as input with duplicates.
> 
> may also mean that the error message is unchanged for the majority of use cases.

well then how about make the line 1121 returns as a set. like below:
`not_included = {x for x in not_included if notna(x)}`

and the `not_included` doesn't spread and have any side effects so it's safe to do this.
Can I work on this?

`      inplace = validate_bool_kwarg(inplace, "inplace")
        if not is_list_like(removals):
            removals = [removals]

        removal_set = set(list(removals))
        not_included = removal_set - set(self.dtype.categories)
        new_categories = [c for c in self.dtype.categories if c not in removal_set]

        # GH 10156
        if any(isna(removals)):
            not_included = {x for x in not_included if notna(x)}
            new_categories = [x for x in new_categories if notna(x)]

        if len(not_included) != 0:
            msg = "removals must all be in old categories: {not_included!s}"
            raise ValueError(msg.format(not_included=not_included))

        return self.set_categories(
            new_categories, ordered=self.ordered, rename=False, inplace=inplace
        )
`

I've made the changes. Shall I commit? I am new to contributing to issues on GitHub.
> I've made the changes. Shall I commit?

you'll also need to update `test_remove_categories`  in `pandas/tests/arrays/categorical/test_api.py`  with a `match` parameter to `pytest.raises` to confirm the changes, and add a case for a null value.

may be worth adding cases for the scaler and with duplicates. 
> 
> you'll also need to update `test_remove_categories` in `pandas/tests/arrays/categorical/test_api.py` with a `match` parameter to `pytest.raises` to confirm the changes, and add a case for a null value.

Meaning to remove a null value (as in not present)? Also I didn't understand what the `match` parameter will do.



Resolving patch:
diff --git a/pandas/core/arrays/categorical.py b/pandas/core/arrays/categorical.py
--- a/pandas/core/arrays/categorical.py
+++ b/pandas/core/arrays/categorical.py
@@ -1120,7 +1120,7 @@ def remove_categories(self, removals, inplace=False):
 
         # GH 10156
         if any(isna(removals)):
-            not_included = [x for x in not_included if notna(x)]
+            not_included = {x for x in not_included if notna(x)}
             new_categories = [x for x in new_categories if notna(x)]
 
         if len(not_included) != 0:


Test patch:
diff --git a/pandas/tests/arrays/categorical/test_api.py b/pandas/tests/arrays/categorical/test_api.py
--- a/pandas/tests/arrays/categorical/test_api.py
+++ b/pandas/tests/arrays/categorical/test_api.py
@@ -1,3 +1,5 @@
+import re
+
 import numpy as np
 import pytest
 
@@ -339,9 +341,13 @@ def test_remove_categories(self):
         tm.assert_categorical_equal(cat, new)
         assert res is None
 
-        # removal is not in categories
-        with pytest.raises(ValueError):
-            cat.remove_categories(["c"])
+    @pytest.mark.parametrize("removals", [["c"], ["c", np.nan], "c", ["c", "c"]])
+    def test_remove_categories_raises(self, removals):
+        cat = Categorical(["a", "b", "a"])
+        message = re.escape("removals must all be in old categories: {'c'}")
+
+        with pytest.raises(ValueError, match=message):
+            cat.remove_categories(removals)
 
     def test_remove_unused_categories(self):
         c = Categorical(["a", "b", "c", "d", "a"], categories=["a", "b", "c", "d", "e"])