API: Coerce None according to the dtype of the container
State: closed

This pull request fixes #7939 by ensuring that None coercion always obeys the following rules:
1. If the container is numeric, None should be coerced to NaN
2. If the container is for time data, None should be coerced to NaT
3. If the container contains heterogenous data (i.e. objects), None should be preserved.
## Old behaviour examples
### Assigning None in integer containers

None is sometimes preserved, sometimes coerced to NaN, and sometimes causes an error.

``` python
>>> s = Series([1, 2, 3])
>>> s[0] = None # error

>>> s = Series([1, 2, 3])
>>> s.loc[0] = None
>>> s
0    None
1       2
2       3
dtype: object

>>> s = Series([1, 2, 3])
>>> s[s == 1] = None
>>> s
0   NaN
1     2
2     3
dtype: float64

>>> s = Series([1, 2, 3])
>>> s.loc[s == 1] = None
0    None
1       2
2       3
dtype: object

```

Datetimes are similar.
### Assigning None in object containers

None is sometimes preserved, and sometimes coerced.

``` python
>>> s = Series(["a", "b", "c"])
>>> s[0] = None
0    None
1       b
2       c
dtype: object

>>> s = Series(["a", "b", "c"])
>>> s.loc[0] = None
0    None
1       b
2       c
dtype: object

>>> s = Series(["a", "b", "c"])
>>> s[s == 'a'] = None
0    NaN
1      b
2      c
dtype: object

>>> s = Series(["a", "b", "c"])
>>> s.loc[s == 'a'] = None
0    None
1       b
2       c
dtype: object
```
## New behaviour examples
### None is always coerced to NaN in integer containers

``` python
>>> s = Series([1, 2, 3])
>>> s[0] = None
0    NaN
1      2
2      3
dtype: float64

>>> s = Series([1, 2, 3])
>>> s.loc[0] = None # same

>>> s = Series([1, 2, 3])
>>> s[s == 1] = None # same

>>> s = Series([1, 2, 3])
>>> s.loc[s == 1] = None # same
```
### None is always preserved in object containers

``` python
>>> s = Series(["a", "b", "c"])
>>> s[0] = None
0    None
1       b
2       c
dtype: object

>>> s = Series(["a", "b", "c"])
>>> s.loc[0] = None # same

>>> s = Series(["a", "b", "c"])
>>> s[s == 'a'] = None

>>> s = Series(["a", "b", "c"])
>>> s.loc[s == 'a'] = None # same
```


Labels:
name: Missing-data
description: np.nan, pd.NaT, pd.NA, dropna, isnull, interpolate
name: API Design

Comments:
see my comments in #7939. this would need to be consistent across lots of use cases. and so cannot accept this as is. feel to work on this and ping if needed. This would need a bunch more tests/corner case fixes.

OK, I've expanded this. I think the right thing to do is:
- Coerce None to nan for containers with numeric types
- Coerce None to NaT for containers with datetimes
- Preserve None otherwise

I've added tests to ensure this behaviour is consistent across different container types, and consistent between `__setitem__` and `.loc`.

Interestingly, no existing tests when I made these changes, so imagine these changes are unlikely to break any code in the wild.

can you show (in the top of this PR). The before and after (for some common cases)?

lots of tests are nice!

I've updated the PR with a summary and some before/after examples. I've also expanded the tests.

can you show the dtype of s after the operation (at the top of the PR), also point 1) should be `nan` (and not None) for numeric.

Its nice to have a summary at the top of the PR for later reference.

Thanks, updated.

from new, this should be a float dtype. (since its numeric and are coercing). you never want to end up `object` UNLESS you start from `object`.

```
s = Series([1, 2, 3])
>>> s[0] = None
0    NaN
1      2
2      3
dtype: object
```

Woops, you're absolutely right. I've fixed the code and expanded the tests accordingly.

OK, I've changed the tests to iterate over a set of test data, added tests for mixed dtypes on DataFrames, and used assertion functions. Let me know if you're happy or if there are further changes you'd like me to make.

ok, looks fine (1 comment above).

need an example in v0.15.0.txt API laying out all of the changes.

Use a code-block to show the before and then ipython to show the changes.

make as short/concisie as possible

I've added a keyword argument to `array_equivalent`, and added a small example in v0.15.0.txt. Let me know if this looks good.

OK, I've updated the docs, and Travis is green. @jreback I think this is ready for merge?

yep, just a coulpe of minor doc comments

perfect, ping when green

@Wilfred thanks!

Wonderful! Thank you for all your help and patience :)

@Wilfred no thank you!

this was an excellent PR. (I know it took a bunch of iterations), but this was a very subtle change (and the reason for the many tests / doc changes). the code change was trivial, but that's how it is some times. (and of course the opposite where you have lots of code changes and just a 1-liner in the docs)