make null lowercase a missing value
State: closed

xref #16079
closes https://github.com/pandas-dev/pandas/issues/16471

Labels:
name: Missing-data
description: np.nan, pd.NaT, pd.NA, dropna, isnull, interpolate
name: IO CSV
description: read_csv, to_csv

Comments:
1) You also need to add it to `parsers.pyx`
2) There's a test (also in `na_values.py`) that checks whether each NaN value you specified gets interpreted as such.  You should add `null` to that test.
Thanks,
1. will add it to parsers.pyx - _NA_VALUES line 279
2. na_values.py - added to the test, method test_default_na_values()  , should be in the pull request.BTW was not able to run pytest, maybe because the test class name doesn't start with the word test;wrote my own simple test with the data that triggered this issue, it passed.
 

    On Monday, May 29, 2017 4:15 PM, gfyoung <notifications@github.com> wrote:
 

    
   - You need to add it to parsers.pyx
   - There's a test (also in na_values.py) that checks whether each NaN value you specified gets interpreted as such. You should add null to that test.
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub, or mute the thread.  

   
@OlegShteynbuk : Yeah, you need to run `pytest test_parsers.py` instead (and yeah, that was the test I was referring to.  Good catch :smile:)
run pytest twice, second time with the class name, some tests skipped, all others passed, should be ok, right?
 pytest pandas/tests/io/parser/test_parsers.
725 passed, 13 skipped in 76.04 seconds

 pytest pandas/tests/io/parser/test_parsers.py::TestPythonParser
235 passed, 5 skipped in 30.06 seconds 
 

    On Monday, May 29, 2017 4:29 PM, gfyoung <notifications@github.com> wrote:
 

 @OlegShteynbuk : Yeah, you need to run pytest test_parsers.py instead (and yeah, that was the test I was referring to. Good catch 😄 )—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub, or mute the thread.  

   
@OlegShteynbuk : Nope, not an issue.
added it to parsers.pyx thanks to gfyoung
# [Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/16534?src=pr&el=h1) Report
> Merging [#16534](https://codecov.io/gh/pandas-dev/pandas/pull/16534?src=pr&el=desc) into [master](https://codecov.io/gh/pandas-dev/pandas/commit/e60dc4cff2c6e8a2283fbb906faeb8cb01df37ff?src=pr&el=desc) will **not change** coverage.
> The diff coverage is `n/a`.

[![Impacted file tree graph](https://codecov.io/gh/pandas-dev/pandas/pull/16534/graphs/tree.svg?width=650&height=150&src=pr&token=eZ4WkYLtcO)](https://codecov.io/gh/pandas-dev/pandas/pull/16534?src=pr&el=tree)

```diff
@@           Coverage Diff           @@
##           master   #16534   +/-   ##
=======================================
  Coverage   90.79%   90.79%           
=======================================
  Files         161      161           
  Lines       51063    51063           
=======================================
  Hits        46365    46365           
  Misses       4698     4698
```

| Flag | Coverage Δ | |
|---|---|---|
| #multiple | `88.63% <ø> (ø)` | :arrow_up: |
| #single | `40.15% <ø> (ø)` | :arrow_up: |

| [Impacted Files](https://codecov.io/gh/pandas-dev/pandas/pull/16534?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [pandas/io/common.py](https://codecov.io/gh/pandas-dev/pandas/pull/16534?src=pr&el=tree#diff-cGFuZGFzL2lvL2NvbW1vbi5weQ==) | `69.91% <ø> (ø)` | :arrow_up: |

------

[Continue to review full report at Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/16534?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/16534?src=pr&el=footer). Last update [e60dc4c...4a91508](https://codecov.io/gh/pandas-dev/pandas/pull/16534?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).

# [Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/16534?src=pr&el=h1) Report
> Merging [#16534](https://codecov.io/gh/pandas-dev/pandas/pull/16534?src=pr&el=desc) into [master](https://codecov.io/gh/pandas-dev/pandas/commit/e60dc4cff2c6e8a2283fbb906faeb8cb01df37ff?src=pr&el=desc) will **not change** coverage.
> The diff coverage is `n/a`.

[![Impacted file tree graph](https://codecov.io/gh/pandas-dev/pandas/pull/16534/graphs/tree.svg?src=pr&token=eZ4WkYLtcO&width=650&height=150)](https://codecov.io/gh/pandas-dev/pandas/pull/16534?src=pr&el=tree)

```diff
@@           Coverage Diff           @@
##           master   #16534   +/-   ##
=======================================
  Coverage   90.79%   90.79%           
=======================================
  Files         161      161           
  Lines       51063    51063           
=======================================
  Hits        46365    46365           
  Misses       4698     4698
```

| Flag | Coverage Δ | |
|---|---|---|
| #multiple | `88.63% <ø> (ø)` | :arrow_up: |
| #single | `40.15% <ø> (ø)` | :arrow_up: |

| [Impacted Files](https://codecov.io/gh/pandas-dev/pandas/pull/16534?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [pandas/io/common.py](https://codecov.io/gh/pandas-dev/pandas/pull/16534?src=pr&el=tree#diff-cGFuZGFzL2lvL2NvbW1vbi5weQ==) | `69.91% <ø> (ø)` | :arrow_up: |

------

[Continue to review full report at Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/16534?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/16534?src=pr&el=footer). Last update [e60dc4c...4a91508](https://codecov.io/gh/pandas-dev/pandas/pull/16534?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).

# [Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/16534?src=pr&el=h1) Report
> Merging [#16534](https://codecov.io/gh/pandas-dev/pandas/pull/16534?src=pr&el=desc) into [master](https://codecov.io/gh/pandas-dev/pandas/commit/e437ad594048cc28873df13ccf50cd39a4e88dcb?src=pr&el=desc) will **increase** coverage by `<.01%`.
> The diff coverage is `n/a`.

[![Impacted file tree graph](https://codecov.io/gh/pandas-dev/pandas/pull/16534/graphs/tree.svg?width=650&height=150&src=pr&token=eZ4WkYLtcO)](https://codecov.io/gh/pandas-dev/pandas/pull/16534?src=pr&el=tree)

```diff
@@            Coverage Diff             @@
##           master   #16534      +/-   ##
==========================================
+ Coverage   90.74%   90.75%   +<.01%     
==========================================
  Files         161      161              
  Lines       51074    51073       -1     
==========================================
+ Hits        46349    46350       +1     
+ Misses       4725     4723       -2
```

| Flag | Coverage Δ | |
|---|---|---|
| #multiple | `88.59% <ø> (ø)` | :arrow_up: |
| #single | `40.16% <ø> (ø)` | :arrow_up: |

| [Impacted Files](https://codecov.io/gh/pandas-dev/pandas/pull/16534?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [pandas/io/common.py](https://codecov.io/gh/pandas-dev/pandas/pull/16534?src=pr&el=tree#diff-cGFuZGFzL2lvL2NvbW1vbi5weQ==) | `69.91% <ø> (ø)` | :arrow_up: |
| [pandas/core/generic.py](https://codecov.io/gh/pandas-dev/pandas/pull/16534?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvZ2VuZXJpYy5weQ==) | `92.31% <0%> (+0.04%)` | :arrow_up: |
| [pandas/core/indexes/datetimes.py](https://codecov.io/gh/pandas-dev/pandas/pull/16534?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvaW5kZXhlcy9kYXRldGltZXMucHk=) | `95.33% <0%> (+0.09%)` | :arrow_up: |

------

[Continue to review full report at Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/16534?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/16534?src=pr&el=footer). Last update [e437ad5...da74438](https://codecov.io/gh/pandas-dev/pandas/pull/16534?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).

@OlegShteynbuk : See the checklist in the PR you submitted?  Just run through those and make sure you can check all of them off.  Then it should be good to merge!
@gfyoung looks ok, not sure if the last test change to parsers.pyx  is included, or need a new pull request.
I'm not sure you fully understood what I was asking you to do.  The code changes are fine, but there are several other things you need to do (i.e whatsnew) before this will be merged.  That's why we have the checkboxes in the PR you submitted.
where are the checkboxes located, can see checks for tests they all fine, i can write whatsnew but where is it?

If you look at the web version of GitHub, the very first text box under the PR title has the checkboxes.  Also, the whatsnew can be found under `doc/source/whatsnew` (probably want 0.21.0 as this is technically an API change)
there are checkboxes on the top, first comment, there is whatsnew checkbox don't see any links, need to add whatsnew somewhere
> there are checkboxes on the top, first comment

Correct.  Make sure you've satisfied everything listed on those checkboxes.

> need to add whatsnew somewhere

Yes, I gave you the file-path that you should follow in your local repository: `pandas/doc/source/whatsnew`
probably need a new .txt file under pandas/doc/source/whatsnew
No, you shouldn't.  You should add it to the 0.21 version.
ok, will add 0.21
added to whatsnew 0.21, commited to my branch
all other checkboxes looks fine, will be away for about an hour, do i need to do anything  else before leaving?
- The ``na_values`` argument for :func:`read_csv` function now has 'null' (in lower case) as a default value that is interpreted as NaN: (:issue:`16471`)

probably need to modify na_values in io.rst file too as it goes into docs
added 'null' to io.rst line 226,  looks like the one that is going into read_csv docs, on my branch; 
there is also a heading NA Values  (line 1023) describing default values, not sure about this one, most likely need to be modified too.
thanks @OlegShteynbuk 
@gfyoung do we have an issue about consolidating the duplicate definitions of NA values (in parsers & IO), maybe just remove from parsers and leave in IO is best. if not, can you create one?
consolidated the duplicate definitions of NA values (in parsers & IO), removed from parsers and left in IO, assert removed too;  probably need to create a different issue.
https://github.com/pandas-dev/pandas/pull/16589
consolidated NA values in parsers.pyx