ENH: Json fill_value for missing fields
State: closed

- [x] closes #16918
- [x] tests added / passed
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [x] whatsnew entry

A new argument `fill_value` is added to allow users to supply default values for missing fields for each column. This enhancement also addresses the undesirable type conversion from `int` to `float` due to missing fields as raised in #16918

Labels:
name: Missing-data
description: np.nan, pd.NaT, pd.NA, dropna, isnull, interpolate
name: IO JSON
description: read_json, to_json, json_normalize

Comments:
# [Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/27073?src=pr&el=h1) Report
> Merging [#27073](https://codecov.io/gh/pandas-dev/pandas/pull/27073?src=pr&el=desc) into [master](https://codecov.io/gh/pandas-dev/pandas/commit/e9555159afd8caabc88b45ca6c06a16efe2f6afc?src=pr&el=desc) will **decrease** coverage by `<.01%`.
> The diff coverage is `87.5%`.

[![Impacted file tree graph](https://codecov.io/gh/pandas-dev/pandas/pull/27073/graphs/tree.svg?width=650&token=eZ4WkYLtcO&height=150&src=pr)](https://codecov.io/gh/pandas-dev/pandas/pull/27073?src=pr&el=tree)

```diff
@@            Coverage Diff             @@
##           master   #27073      +/-   ##
==========================================
- Coverage   92.04%   92.03%   -0.01%     
==========================================
  Files         180      180              
  Lines       50714    50716       +2     
==========================================
- Hits        46679    46676       -3     
- Misses       4035     4040       +5
```

| Flag | Coverage Δ | |
|---|---|---|
| #multiple | `90.67% <87.5%> (-0.01%)` | :arrow_down: |
| #single | `41.86% <50%> (-0.09%)` | :arrow_down: |

| [Impacted Files](https://codecov.io/gh/pandas-dev/pandas/pull/27073?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [pandas/core/frame.py](https://codecov.io/gh/pandas-dev/pandas/pull/27073/diff?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvZnJhbWUucHk=) | `96.89% <100%> (-0.12%)` | :arrow_down: |
| [pandas/core/internals/construction.py](https://codecov.io/gh/pandas-dev/pandas/pull/27073/diff?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvaW50ZXJuYWxzL2NvbnN0cnVjdGlvbi5weQ==) | `95.95% <100%> (ø)` | :arrow_up: |
| [pandas/io/json/normalize.py](https://codecov.io/gh/pandas-dev/pandas/pull/27073/diff?src=pr&el=tree#diff-cGFuZGFzL2lvL2pzb24vbm9ybWFsaXplLnB5) | `96% <75%> (-0.94%)` | :arrow_down: |
| [pandas/io/gbq.py](https://codecov.io/gh/pandas-dev/pandas/pull/27073/diff?src=pr&el=tree#diff-cGFuZGFzL2lvL2dicS5weQ==) | `88.88% <0%> (-11.12%)` | :arrow_down: |

------

[Continue to review full report at Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/27073?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/27073?src=pr&el=footer). Last update [e955515...7c2b71c](https://codecov.io/gh/pandas-dev/pandas/pull/27073?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).

# [Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/27073?src=pr&el=h1) Report
> Merging [#27073](https://codecov.io/gh/pandas-dev/pandas/pull/27073?src=pr&el=desc) into [master](https://codecov.io/gh/pandas-dev/pandas/commit/e9555159afd8caabc88b45ca6c06a16efe2f6afc?src=pr&el=desc) will **decrease** coverage by `<.01%`.
> The diff coverage is `87.5%`.

[![Impacted file tree graph](https://codecov.io/gh/pandas-dev/pandas/pull/27073/graphs/tree.svg?width=650&token=eZ4WkYLtcO&height=150&src=pr)](https://codecov.io/gh/pandas-dev/pandas/pull/27073?src=pr&el=tree)

```diff
@@            Coverage Diff             @@
##           master   #27073      +/-   ##
==========================================
- Coverage   92.04%   92.03%   -0.01%     
==========================================
  Files         180      180              
  Lines       50714    50716       +2     
==========================================
- Hits        46679    46676       -3     
- Misses       4035     4040       +5
```

| Flag | Coverage Δ | |
|---|---|---|
| #multiple | `90.67% <87.5%> (-0.01%)` | :arrow_down: |
| #single | `41.86% <50%> (-0.09%)` | :arrow_down: |

| [Impacted Files](https://codecov.io/gh/pandas-dev/pandas/pull/27073?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [pandas/core/frame.py](https://codecov.io/gh/pandas-dev/pandas/pull/27073/diff?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvZnJhbWUucHk=) | `96.89% <100%> (-0.12%)` | :arrow_down: |
| [pandas/core/internals/construction.py](https://codecov.io/gh/pandas-dev/pandas/pull/27073/diff?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvaW50ZXJuYWxzL2NvbnN0cnVjdGlvbi5weQ==) | `95.95% <100%> (ø)` | :arrow_up: |
| [pandas/io/json/normalize.py](https://codecov.io/gh/pandas-dev/pandas/pull/27073/diff?src=pr&el=tree#diff-cGFuZGFzL2lvL2pzb24vbm9ybWFsaXplLnB5) | `96% <75%> (-0.94%)` | :arrow_down: |
| [pandas/io/gbq.py](https://codecov.io/gh/pandas-dev/pandas/pull/27073/diff?src=pr&el=tree#diff-cGFuZGFzL2lvL2dicS5weQ==) | `88.88% <0%> (-11.12%)` | :arrow_down: |

------

[Continue to review full report at Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/27073?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/27073?src=pr&el=footer). Last update [e955515...41baaa3](https://codecov.io/gh/pandas-dev/pandas/pull/27073?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).

> or are you suggesting something that cannot already be done

@jreback : By the time `read_json` completes, the damage is already done *per se* (the precision has been lost).  This PR attempts to prevent that from happening via the `fill_value` parameter.  There are two ways we could address this:

* Don't force casting to numeric in cases like these (API change).  This is the option we have in `to_numeric` (e.g. we keep the column as `object` when we risk losing precision).

* Allow people to work around this via `fill_value`.  This is an interesting option, but if we do this, I would expect that we also apply this to the rest of the IO interface (`read_csv`, `read_excel`, etc.)
@jreback @gfyoung  So which direction shall I pursue to fix the precision issue due to conversion?
@jiangyue12392 : Sorry for the silence!  We're actually in the midst of pushing a release candidate for `0.25.0`.  I would merge / rebase `master` for the time being.

@WillAyd @jreback : If either of you have bandwidth to give feedback on the two directions I gave above, that would be great here.
Yea I don’t think adding this to the API is the best approach. Certainly more complicated but would rather see if there’s a way to return Int64 with missing data

Sent from my iPhone

> On Jul 3, 2019, at 11:31 PM, gfyoung <notifications@github.com> wrote:
> 
> @jiangyue12392 : Sorry for the silence! We're actually in the midst of pushing a release candidate for 0.25.0. I would merge / rebase master for the time being.
> 
> @WillAyd @jreback : If either of you have bandwidth to give feedback on the two directions I gave above, that would be great here.
> 
> —
> You are receiving this because you were mentioned.
> Reply to this email directly, view it on GitHub, or mute the thread.

@WillAyd Can you point me to more information about the nullable integer type? I may be able to come up with some other solutions with that.
@jiangyue12392 there is a dedicated section in the docs to it:

https://pandas.pydata.org/pandas-docs/stable/user_guide/integer_na.html

It's a relatively new feature still so anything you can do to improve integration with that would be awesome!
@WillAyd @gfyoung @jreback I made another attempt to solve this issue with IntegerArray at #27335. Please advise.
superseded by #27335