Categorical: preserve ints when NaN are present

Some fixups for Categoricals.

Fixes: #3678
done