ENH: standardize fill_value behavior across the API
State: closed

 - [ ] closes #15533 
 - [ ] tests added / passed
 - [ ] passes ``git diff upstream/master | flake8 --diff``
 - [ ] whatsnew entry


Labels:
name: Missing-data
description: np.nan, pd.NaT, pd.NA, dropna, isnull, interpolate
name: Error Reporting
description: Incorrect or improved errors from pandas

Comments:
This is a starting point for #15533. Right now I've only added `_is_fillable_values` and `validate_fill_value` methods to the bottom of `common.py`.

There's way too much magic ATM. Some specific questions:
* Is there a method for detecting `pandas` (non-`numpy`) time dtypes: `Timestamp`, `Period`, `Timedelta`? AFAIK all of the `common.py` ops are w.r.t. `numpy` dtypes (`datetime64` etc.).
* `common.py` ops accept individual objects *or* arrays and look at the `dtype` thereof, so we have to catch `numpy` and `pandas` data structs separately. What's a good way of tackling this? Just import all of the names and test them all?
So with date stuff, we can catch `numpy`/stdlib `datetime`/`Timestamp` using `is_datetime64_any_dtype`. We can catch `Timedelta` using `is_timedelta64_dtype`. 

But how do we catch `Period`? When fed to `is_datetime64_any_dtype` it returns `False`. Additionally, the following evaluates to `False` as well:

    is_period_dtype(pd.Series([pd.Period('2015-01-01')]).dtype)

Is this supposed to happen? The `Period` `numpy` `dtype` is just `'O'`...
Periods are object type when in a Series ATM. They have a specific dtype only in an Index.
There is an ``is_period_arraylike`` if you really need inference on an array.
numpy has pretty much nothing to do with dtypes anymore in pandas (except for some basic types).
See the method in the new commit.

How it works right now:
* `list_like`, `dict_like`, and `callable` fill values will always raise a `TypeError`.
* `isnull` fill values will always pass.
* If the unified `dtype` (the `dtype` you get when you cast to a `numpy` array) of a `Series` or `DataFrame` is `Object`, any object excepting the ones in the first bullet point will be accepted.
* The above includes `Period` dtype columns. Theoretically `Period` dtype columns should only accept `Period` fill values. However, because of the way periods are implemented, with an `O` `dtype`, there doesn't seem to be an easy way of conforming to this behavior without changing the method signature somehow. Periods just fall through to the general object case right now.
* If the unified `dtype` is `datetime64`, only `datetime` types will work.
* If the unified `dtype` is `timedelta64`, only `timedelta` types will work.

Is this behavior OK?
# [Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/15587?src=pr&el=h1) Report
> Merging [#15587](https://codecov.io/gh/pandas-dev/pandas/pull/15587?src=pr&el=desc) into [master](https://codecov.io/gh/pandas-dev/pandas/commit/09360d80da730008a6a89f38f3780bb1d55f9e25?src=pr&el=desc) will **decrease** coverage by `-0.03%`.
> The diff coverage is `75%`.


```diff
@@            Coverage Diff             @@
##           master   #15587      +/-   ##
==========================================
- Coverage   91.06%   91.03%   -0.03%     
==========================================
  Files         137      137              
  Lines       49307    49330      +23     
==========================================
+ Hits        44899    44908       +9     
- Misses       4408     4422      +14
```


| [Impacted Files](https://codecov.io/gh/pandas-dev/pandas/pull/15587?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [pandas/core/reshape.py](https://codecov.io/gh/pandas-dev/pandas/compare/09360d80da730008a6a89f38f3780bb1d55f9e25...1efdc4aca5d21949bcd091a24ed4e2c0b0907653?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvcmVzaGFwZS5weQ==) | `99.28% <100%> (ø)` | :white_check_mark: |
| [pandas/core/missing.py](https://codecov.io/gh/pandas-dev/pandas/compare/09360d80da730008a6a89f38f3780bb1d55f9e25...1efdc4aca5d21949bcd091a24ed4e2c0b0907653?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvbWlzc2luZy5weQ==) | `84.38% <71.42%> (-0.57%)` | :x: |
| [pandas/io/gbq.py](https://codecov.io/gh/pandas-dev/pandas/compare/09360d80da730008a6a89f38f3780bb1d55f9e25...1efdc4aca5d21949bcd091a24ed4e2c0b0907653?src=pr&el=tree#diff-cGFuZGFzL2lvL2dicS5weQ==) | `25% <0%> (-58.34%)` | :x: |
| [pandas/tools/merge.py](https://codecov.io/gh/pandas-dev/pandas/compare/09360d80da730008a6a89f38f3780bb1d55f9e25...1efdc4aca5d21949bcd091a24ed4e2c0b0907653?src=pr&el=tree#diff-cGFuZGFzL3Rvb2xzL21lcmdlLnB5) | `91.78% <0%> (-0.35%)` | :x: |
| [pandas/core/frame.py](https://codecov.io/gh/pandas-dev/pandas/compare/09360d80da730008a6a89f38f3780bb1d55f9e25...1efdc4aca5d21949bcd091a24ed4e2c0b0907653?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvZnJhbWUucHk=) | `97.87% <0%> (-0.06%)` | :x: |
| [pandas/formats/format.py](https://codecov.io/gh/pandas-dev/pandas/compare/09360d80da730008a6a89f38f3780bb1d55f9e25...1efdc4aca5d21949bcd091a24ed4e2c0b0907653?src=pr&el=tree#diff-cGFuZGFzL2Zvcm1hdHMvZm9ybWF0LnB5) | `95.33% <0%> (-0.01%)` | :x: |
| [pandas/io/excel.py](https://codecov.io/gh/pandas-dev/pandas/compare/09360d80da730008a6a89f38f3780bb1d55f9e25...1efdc4aca5d21949bcd091a24ed4e2c0b0907653?src=pr&el=tree#diff-cGFuZGFzL2lvL2V4Y2VsLnB5) | `79.67% <0%> (+0.03%)` | :white_check_mark: |
| [pandas/tseries/base.py](https://codecov.io/gh/pandas-dev/pandas/compare/09360d80da730008a6a89f38f3780bb1d55f9e25...1efdc4aca5d21949bcd091a24ed4e2c0b0907653?src=pr&el=tree#diff-cGFuZGFzL3RzZXJpZXMvYmFzZS5weQ==) | `96.65% <0%> (+0.06%)` | :white_check_mark: |
| [pandas/core/common.py](https://codecov.io/gh/pandas-dev/pandas/compare/09360d80da730008a6a89f38f3780bb1d55f9e25...1efdc4aca5d21949bcd091a24ed4e2c0b0907653?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvY29tbW9uLnB5) | `91.36% <0%> (+0.33%)` | :white_check_mark: |

------

[Continue to review full report at Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/15587?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/15587?src=pr&el=footer). Last update [09360d8...9eaa0f2](https://codecov.io/gh/pandas-dev/pandas/compare/09360d80da730008a6a89f38f3780bb1d55f9e25...9eaa0f241388ff7e5110d3396fefc1d8c3c4bab4?el=footer&src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
that's not right

a fill value will be applied per individual dtype 
so best to simply validate at a lower level then

look in internals and put this check in the fillna method
How about iterating through the sub-series column-by-column? Do `isinstance(obj, ABCFrame)` and if`True` do a `[_validate(col, dtype) for col in obj.columns]`.

On the design side. Suppose I have a mixed dtype DataFrame, let's say with a `str` column and a `bool` column. After some operation I now have a null value in each. `fill_value` doesn't implement columnar `dict` input like `fillna` does, so there's no way of handling these columns separately.

If `fill_value` is supposed to be a quick substitute for `fillna`, then this should be OK. If on the other hand the idea is that we want it there strictly to cover cases when we don't upcast the column `dtype` (which was the original motivation), then validation should be by-column, yeah.
@ResidentMario you can also do columnar. Keep in mind though that we generally will simply skip a non-compat fill-value. 

```
In [1]: df = DataFrame({'A':[1,2,3],'B':pd.date_range('20130101',periods=3)})

In [2]: df
Out[2]: 
   A          B
0  1 2013-01-01
1  2 2013-01-02
2  3 2013-01-03

In [3]: df.iloc[1] = np.nan

In [4]: df
Out[4]: 
     A          B
0  1.0 2013-01-01
1  NaN        NaT
2  3.0 2013-01-03

In [5]: df.fillna(0)
Out[5]: 
     A          B
0  1.0 2013-01-01
1  0.0 1970-01-01
2  3.0 2013-01-03

In [6]: df.fillna(Timestamp('20130201'))
Out[6]: 
                     A          B
0                    1 2013-01-01
1  2013-02-01 00:00:00 2013-02-01
2                    3 2013-01-03
```

so one could argue that both [5] and [6] are wrong or right. We generally leave this up to the user when having mixed dtypes. 
Exactly, leave it to the user&mdash;that's what the current implementation would do. So, do you think `fill_value` should follow the [5] & [6] case, the (current) "lossy" implementation, or a stricter (suggested) check-each-column implementation? I can do whichever.
closing in favor of #15563 

normally pls don't create *new* PR's for the same issue, just push to the same one.
This is a different PR, though. The goal here is to implement error handling for `fill_value` parameters to various methods; the goal of PR#15563 is to implement error handing for `fillna`. Presumably I'd then rebase from there and work the feature out here.

That being said I cherry-picked some of the commits here for that PR. So a totally new PR might just be cleaner anyway.

Sorry, took a while to get to the bottom of this particular molehill.