ENH: Provide an errors parameter to fillna
State: closed

 - [ ] progress towards #15533
 - [ ] tests added / passed
 - [ ] passes ``git diff upstream/master | flake8 --diff``
 - [ ] whatsnew entry


Labels:
name: Missing-data
description: np.nan, pd.NaT, pd.NA, dropna, isnull, interpolate
name: Error Reporting
description: Incorrect or improved errors from pandas

Comments:
Try it out yourself:

```python
pd.DataFrame({'a': [1,2,np.nan], 'b': [np.nan, 2, 3]}).fillna({'a': 'foo', 'b': 'bar'}, errors='raise')
pd.DataFrame({'a': [1,2,np.nan], 'b': [np.nan, 2, 3]}).fillna({'a': 'foo', 'b': 'bar'}, errors='force')
pd.DataFrame({'a': [1,2,np.nan], 'b': [np.nan, 2, 3]}).fillna({'a': 'foo', 'b': 'bar'}, errors='ignore')
```

`types/missing.py:valid_fill_value` has a test suite in `test_missing.py`, but I'm confused as to where I should put the tests for the new `fillna` bits...
@jreback Two API questions:

1. Pretty sure it should be `coerce`, not `force`. See e.g. [here](http://pandas.pydata.org/pandas-docs/version/0.17.0/generated/pandas.to_numeric.html).
2. Should the default behavior change? You stated [in discussion](https://github.com/pandas-dev/pandas/issues/15533#issuecomment-285442956) that the default behavior should be `ignore`, not `force`. The latter is the default right now, however. This seems like a really big change, I think it'd be wise to have a deprecation cycle first...
This fails seemingly every `sparse` test in the test suite. 👀

`sparse` things obviously already have hardcoded `dtype` requirements so I should find a way of disabling this feature for all such types.
Yeah, I think `'coerce'` instead of `'force'`. Not sure about changing the default. I don't think `ignore` should be the default. I think `raise` is more consistent with `to_numeric`. But it's a big change, and maybe not worth it.
I'm investigating the five tests in the suite that the current code still doesn't pass, and I've hit something I can't explain originating in `TestDataFrameMissingData.test_fillna_datetime_columns`.

At a high level the bug is the fact that with the following bit:

```
>>> pd.DataFrame({'A': [-1, -2, np.nan],
              'B': [pd.Timestamp('2013-01-01'), pd.Timestamp('2013-01-02'), pd.NaT],
              'C': ['foo', 'bar', None],
              'D': ['foo2', 'bar2', None]},
              index=pd.date_range('20130110', periods=3)).fillna('?')
```

`0.19.2` gives:

```
            A                    B    C     D
2013-01-10 -1  2013-01-01 00:00:00  foo  foo2
2013-01-11 -2  2013-01-02 00:00:00  bar  bar2
2013-01-12  ?                    ?    ?     ?
```

While this PR gives:

```
            A                    B    C     D
2013-01-10 -1  2013-01-01 00:00:00  foo  foo2
2013-01-11 -2  2013-01-02 00:00:00  bar  bar2
2013-01-12  ?           	NaT   ?     ?
```

This difference seems to be emitting from line 268 in `frame.py`, where in `DataFrame.__init__` you have:

        if isinstance(data, BlockManager):
            mgr = self._init_mgr(data, axes=dict(index=index, columns=columns),
                                 dtype=dtype, copy=copy)

In `0.19.2` this emits:

```
BlockManager
Items: Index(['A', 'B', 'C', 'D'], dtype='object')
Axis 1: DatetimeIndex(['2013-01-10', '2013-01-11', '2013-01-12'], dtype='datetime64[ns]', freq='D')
ObjectBlock: slice(0, 4, 1), 4 x 3, dtype: object
```

In this PR:

```
BlockManager
Items: Index(['A', 'B', 'C', 'D'], dtype='object')
Axis 1: DatetimeIndex(['2013-01-10', '2013-01-11', '2013-01-12'], dtype='datetime64[ns]', freq='D')
DatetimeBlock: slice(1, 2, 1), 1 x 3, dtype: datetime64[ns]
ObjectBlock: [0, 2, 3], 3 x 3, dtype: object
```

But how is this possible? This PR only adds a bit of if-else logic to the `fillna` method; it doesn't touch any of these deep internals (which are mostly beyond me anyway). So it *shouldn't* cause any differences in how block managers operate.

And the current master seems to pass all tests, including this one...huh.

I wouldn't call this an edge case, the old behavior definitely needs to be preserved.

Maybe I'm missing something, going to have to circle back to this later.
@ResidentMario this is related to this bug: https://github.com/pandas-dev/pandas/issues/15613 I think

with the default ``errors='ignore'`` this should return as 0.19.2
Thanks for the clarification, this is seems to be the same as [your comment](https://github.com/pandas-dev/pandas/issues/15613#issuecomment-285034706). Going to need to fiddle with it to find a workaround.

A more minimal example:

```python
pd.Series([pd.Timestamp('2015-01-01'), pd.NaT]).fillna("?")
```

Fills with `?` on master and `NaT` on `fillna-errors`.
Should probably write the tests first, this got picked up because a test happened to depend on this behavior, maybe there are other cases also like this.
Looking at the tests, I see three dissatisfactory cases:

1. `Period` columns are always `dtype('O')`, and there's no easy way of checking whether or not a column is period-type (previously discussed).
2. date-type things auto-coerce to `NaT`, so their default behavior is equivalent to `errors="ignore"` for everything else (see above).
3. A column with a `bool` and `nan` is `dtype('O')`. So e.g. we can't prevent `Series([True, np.nan]).fillna('foobar', errors='ignore')` from producing `True, 'foobar'`.

The second one needs to be solved, the first and last are limitations of `pandas` nullity handling that probably need to be lived with.
Looks like it was simpler than I had thought.

The problem is that to keep `generic/fillna` working with `categorical` types, I pass `None` instead of `'coerce'` down as the default value to `internals/fillna`. Adding:

```
if not errors:
    errors = 'coerce'
```

To the `fillna` method body fixed this problem. For now, this is OK. But, I'm hoping I'll find a better solution later.
Does this perhaps also fix the problem reported in https://github.com/pandas-dev/pandas/issues/16112?
No, trying an error mode on a sparse matrix will raise an error for now.
`TestDataFrameOperators.test_timestamp_compare` fails because of something remarkably subtle.

Suppose we have:

```python
import pandas as pd
from io import StringIO

res = pd.read_csv(StringIO(""",dates1,dates2,floatcol,intcol,stringcol
0,1970-01-01 00:00:00.000000000,1970-01-01 00:00:00.000000000,,,
1,1970-01-01 00:00:00.000000000,,,,
2,1970-01-01 00:00:00.000000000,1970-01-01 00:00:00.000000000,,,
3,1970-01-01 00:00:00.000000000,,,,
4,1970-01-01 00:00:00.000000000,,,,
5,1970-01-01 00:00:00.000000000,1970-01-01 00:00:00.000000000,,,
6,1970-01-01 00:00:00.000000000,1970-01-01 00:00:00.000000000,,,
7,1970-01-01 00:00:00.000000000,,,,
8,1970-01-01 00:00:00.000000000,1970-01-01 00:00:00.000000001,,,
9,1970-01-01 00:00:00.000000001,1970-01-01 00:00:00.000000001,,,"""), index_col=0)
```

On master:

```
In [14]: res.fillna(True).dtypes
Out[14]: 
dates1        object
dates2        object
floatcol     float64
intcol       float64
stringcol    float64
dtype: object
```

On the feature branch:

```
In [14]: res.fillna(True).dtypes
Out[14]: 
dates1        datetime64[ns]
dates2        object
floatcol     float64
intcol       float64
stringcol    float64
dtype: object
```

On master even though the first column is already full and doesn't receive any boolean fills, the column `dtype` still gets upcast to `object`. I think that's because `fillna` internally runs the operation, and the upcast, on the 2-by-n "double datetimes" `block`.

Since I now force this operation to be done column-by-column, `fillna` will no longer do this&mdash;it will leave that first column's `dtype` untouched. That's not a bug; that's a feature!

But, this test relies on the old behavior, because it then attempts:

```python
res.fillna(True).astype(bool)
```

Which fails with:

```
*** TypeError: cannot astype a datetimelike from [datetime64[ns]] to [bool]
```

Because while `object` columns can be cast to `bool`, `datetime64` columns cannot.

I suspect the other remaining failures are similar.
On master:

```python
import pandas as pd
from io import StringIO

res = pd.read_csv(StringIO(""",dates1,dates2
0,1970-01-01 00:00:00.000000000,1970-01-01 00:00:00.000000000
1,1970-01-01 00:00:00.000000000,"""), index_col=0)

res['dates1'] = pd.to_datetime(res['dates1'])
res.astype(bool)
```

Raises:

```
TypeError: cannot astype a datetimelike from [datetime64[ns]] to [bool]
```
@ResidentMario your last is correct. if you need to do booleans with that you have to fillna first.

```
In [25]: pd.to_datetime(res['dates2'])
Out[25]: 
0   1970-01-01
1          NaT
Name: dates2, dtype: datetime64[ns]

In [26]: pd.to_datetime(res['dates2']).fillna(False).astype(bool)
Out[26]: 
0     True
1    False
Name: dates2, dtype: bool
```
@jreback Unsure what you mean above, can you clarify?
@ResidentMario I mean your example above is exactly as expected an not an error.
Returning to this after a long absence.

@jreback's solution works in the reproduction case, but can't be used in implementation. The problem is that it uses `fillna`, and this changeset is a feature patch for...`fillna`.

The brute-force way of resolving this, mentioned at #16155, is to upcast the `datetime` to `object` `dtype` beforehand. That doesn't seem right at all.

But I'm just not seeing any hooks I can use to get in there and fix just this corner case specifically, either.

I'm stumped. Welcoming suggestions from others.

This is the last remaining test failure in this PR (assuming the driver failures are benign). There's still work to do with checking the impact on performance and going over the tests again, but this is otherwise reasonably close to a finish.
closing as stale. this is a bit easier in current master FYI.