Repository: pandas-dev/pandas

Issue:
`fillna` using columns of dtype `category` also fills non-NaN values
#### Code Sample, a copy-pastable example if possible

```python
import pandas as pd
import numpy as np

dct = {
    'A': ['a', 'b', 'c', 'b'], 
    'B': ['d', 'e', np.nan, np.nan]
}
df = pd.DataFrame.from_dict(dct).astype('category')
df['C'] = df['B']
df['C'].cat.add_categories(df['A'].cat.categories, inplace=True)
df['C'] = df['C'].fillna(df['A'])
```

**output**


|   | A | B | C
| :---: | ---: | ---: | ---:
| **0** | a | d | a
| **1** | b | e | b
| **2** | c | NaN | c
| **3** | b | NaN | b


#### Problem description

I have two columns, A and B,  of dtype `category`. Column B contains `NaN` values.
Applying `fillna` to B using A (after adding categories in A to categories in B), results in ALL values of B being overwritten with values of A. The issue is that `fillna` also fills non-`NaN` values.

#### Expected Output

Non-`NaN` values should not be overwritten:

|   | A | B | C
| :---: | ---: | ---: | ---:
| **0** | a | d | d
| **1** | b | e | e
| **2** | c | NaN | c
| **3** | b | NaN | b

#### Output of ``pd.show_versions()``

<details>

INSTALLED VERSIONS
------------------
commit: None
python: 3.7.3.final.0
python-bits: 64
OS: Windows
OS-release: 10
machine: AMD64
processor: Intel64 Family 6 Model 94 Stepping 3, GenuineIntel
byteorder: little
LC_ALL: None
LANG: None
LOCALE: None.None

pandas: 0.24.2
pytest: 4.3.1
pip: 19.0.3
setuptools: 40.8.0
Cython: 0.29.6
numpy: 1.16.2
scipy: 1.2.1
pyarrow: None
xarray: None
IPython: 7.4.0
sphinx: 1.8.5
patsy: 0.5.1
dateutil: 2.8.0
pytz: 2018.9
blosc: None
bottleneck: 1.2.1
tables: 3.5.1
numexpr: 2.6.9
feather: None
matplotlib: 3.0.3
openpyxl: 2.6.1
xlrd: 1.2.0
xlwt: 1.3.0
xlsxwriter: 1.1.5
lxml.etree: 4.3.2
bs4: 4.7.1
html5lib: 1.0.1
sqlalchemy: 1.3.1
pymysql: None
psycopg2: None
jinja2: 2.10
s3fs: None
fastparquet: None
pandas_gbq: None
pandas_datareader: None
gcsfs: None

</details>



Discussion:
Makes sense - investigation and PRs would certainly be welcome
@WillAyd Can I try this? 
Confirmed this happening for me too. 
Slightly simpler example

```python
In [36]: s = pd.Series(pd.Categorical(['d', 'e', None, None], categories=['d', 'e', 'a', 'b', 'c']))

In [37]: s.fillna(pd.Series(['a', 'b', 'c', 'd']))
Out[37]:
0    a
1    b
2    c
3    d
dtype: category
Categories (5, object): [d, e, a, b, c]
```

@gamis I don't think anyone has opened a PR for this, if you're interested in working on it. Even diagnosing the issue would be valuable.
Even simpler :)

```
>>> import pandas as pd
>>> pd.Categorical(['d', 'e', None, None], categories=['d', 'e', 'a', 'b', 'c']).fillna(pd.Series(['a', 'b', 'c', 'd']))
[a, b, c, d]
Categories (5, object): [d, e, a, b, c]
```

Resolving patch:
diff --git a/pandas/core/arrays/categorical.py b/pandas/core/arrays/categorical.py
--- a/pandas/core/arrays/categorical.py
+++ b/pandas/core/arrays/categorical.py
@@ -1840,8 +1840,8 @@ def fillna(self, value=None, method=None, limit=None):
                     raise ValueError("fill value must be in categories")
 
                 values_codes = _get_codes_for_values(value, self.categories)
-                indexer = np.where(values_codes != -1)
-                codes[indexer] = values_codes[values_codes != -1]
+                indexer = np.where(codes == -1)
+                codes[indexer] = values_codes[indexer]
 
             # If value is not a dict or Series it should be a scalar
             elif is_hashable(value):


Test patch:
diff --git a/pandas/tests/series/test_missing.py b/pandas/tests/series/test_missing.py
--- a/pandas/tests/series/test_missing.py
+++ b/pandas/tests/series/test_missing.py
@@ -578,6 +578,28 @@ def test_fillna_categorical(self, fill_value, expected_output):
         exp = Series(Categorical(expected_output, categories=["a", "b"]))
         tm.assert_series_equal(s.fillna(fill_value), exp)
 
+    @pytest.mark.parametrize(
+        "fill_value, expected_output",
+        [
+            (Series(["a", "b", "c", "d", "e"]), ["a", "b", "b", "d", "e"]),
+            (Series(["b", "d", "a", "d", "a"]), ["a", "d", "b", "d", "a"]),
+            (
+                Series(
+                    Categorical(
+                        ["b", "d", "a", "d", "a"], categories=["b", "c", "d", "e", "a"]
+                    )
+                ),
+                ["a", "d", "b", "d", "a"],
+            ),
+        ],
+    )
+    def test_fillna_categorical_with_new_categories(self, fill_value, expected_output):
+        # GH 26215
+        data = ["a", np.nan, "b", np.nan, np.nan]
+        s = Series(Categorical(data, categories=["a", "b", "c", "d", "e"]))
+        exp = Series(Categorical(expected_output, categories=["a", "b", "c", "d", "e"]))
+        tm.assert_series_equal(s.fillna(fill_value), exp)
+
     def test_fillna_categorical_raise(self):
         data = ["a", np.nan, "b", np.nan, np.nan]
         s = Series(Categorical(data, categories=["a", "b"]))