Repository: pandas-dev/pandas
Pull request: BUG-19214 int categoricals are formatted as ints
- [X] closes #19214
- [X] tests added / passed
- [X] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [X] whatsnew entry

This PR alters the `ExtensionArrayFormatter` such that if it receives a categorical series with only integers and `NaN`, it will print the integers as integers instead of converting to floats.

```
>>> # previous behaviour
>>> pd.Series([1,2, np.nan], dtype="object").astype("category")
0    1.0
1    2.0
2    NaN
dtype: category
Categories (2, int64): [1, 2]
>>> # new behaviour
>>> pd.Series([1,2, np.nan], dtype="object").astype("category")
0      1
1      2
2    NaN
dtype: category
Categories (2, int64): [1, 2]
```

Problem statement:
categorical series with null converts ints to float
```python
In [4]: import pandas as pd, numpy as np

In [5]: sr = pd.Series([1,2, np.nan], dtype="O").astype("category")
   ...: sr
   ...: 
Out[5]: 
0    1.0
1    2.0
2    NaN
dtype: category
Categories (2, int64): [1, 2]

In [6]: sr[0]
Out[6]: 1.0
```
Shouldn't it keep them as ints? Interestingly, `pd.Series(["a", 1,2, np.nan], dtype="O").astype("category")` doesn't do the conversion.

Hints from issue discussion:
so ``.loc`` does this correctly, meaning we defer to the underlying data source for getting the values, rather than do the array conversion (which ``__getitem__``) does. So its buggy, changing this may not be easy though.

```
In [1]: sr = pd.Series([1,2, np.nan], dtype="O").astype("category")

In [2]: sr
Out[2]: 
0    1.0
1    2.0
2    NaN
dtype: category
Categories (2, int64): [1, 2]

In [3]: sr.loc[0]
Out[3]: 1
```
cc @toobaz 
cc @jschendel 

if you want to wade thru some indexing code!
> pd.Series(["a", 1,2, np.nan], dtype="O").astype("category")

this is as expected, the objs in the series are actually store (and not as a numpy array).

Commit message: BUG-19214 int categoricals are formatted as ints

Patch to review:
diff --git a/doc/source/whatsnew/v0.24.0.rst b/doc/source/whatsnew/v0.24.0.rst
index a84fd118061bc..c3813e22dc765 100644
--- a/doc/source/whatsnew/v0.24.0.rst
+++ b/doc/source/whatsnew/v0.24.0.rst
@@ -1321,6 +1321,7 @@ Categorical
 - Bug in many methods of the ``.str``-accessor, which always failed on calling the ``CategoricalIndex.str`` constructor (:issue:`23555`, :issue:`23556`)
 - Bug in :meth:`Series.where` losing the categorical dtype for categorical data (:issue:`24077`)
 - Bug in :meth:`Categorical.apply` where ``NaN`` values could be handled unpredictably. They now remain unchanged (:issue:`24241`)
+- Bug where an integer ``Categorical`` would be formatted as if it had floats if ``NaN`` values were present (:issue:`19214`)
 
 Datetimelike
 ^^^^^^^^^^^^
diff --git a/pandas/io/formats/format.py b/pandas/io/formats/format.py
index 9dc2692f276e3..005b59abf608e 100644
--- a/pandas/io/formats/format.py
+++ b/pandas/io/formats/format.py
@@ -1176,6 +1176,11 @@ def _format_strings(self):
                                   na_rep=self.na_rep, digits=self.digits,
                                   space=self.space, justify=self.justify,
                                   leading_space=self.leading_space)
+
+        if (is_categorical_dtype(values.dtype) and
+                is_integer_dtype(values.dtype.categories)):
+            # integers were coerced to float for array with NaN (GH 19214)
+            return [value.replace(".0", "") for value in fmt_values]
         return fmt_values
 
 
diff --git a/pandas/tests/arrays/categorical/test_repr.py b/pandas/tests/arrays/categorical/test_repr.py
index 227edf60951e6..8f000157864f9 100644
--- a/pandas/tests/arrays/categorical/test_repr.py
+++ b/pandas/tests/arrays/categorical/test_repr.py
@@ -240,6 +240,10 @@ def test_categorical_repr_datetime_ordered(self):
 
         assert repr(c) == exp
 
+    def test_categorical_repr_int_with_nan(self):
+        s = Series([1, 2, np.nan], dtype="object").astype("category")
+        assert ".0" not in repr(s)
+
     def test_categorical_repr_period(self):
         idx = period_range('2011-01-01 09:00', freq='H', periods=5)
         c = Categorical(idx)


Code review comments:
diff_hunk: @@ -1176,6 +1176,11 @@ def _format_strings(self):
                                   na_rep=self.na_rep, digits=self.digits,
                                   space=self.space, justify=self.justify,
                                   leading_space=self.leading_space)
+
path: pandas/io/formats/format.py
text: @user1:
we have a CategoricalFormatter already, mod should be made there. Should be caught when formatting, not as a post-action.