Robustness improvement for normalize.py
State: closed

- [x] closes #26284
- [x] tests added / passed
- [ ] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ ] whatsnew entry

Improves robustness of json_normalize for inconsistently formatted json.
Fixes all observations in #26284

Labels:
name: IO JSON
description: read_json, to_json, json_normalize

Comments:
Hello @Marky0! Thanks for updating this PR. We checked the lines you've touched for [PEP 8](https://www.python.org/dev/peps/pep-0008) issues, and found:



* In the file [`pandas/tests/io/json/test_normalize.py`](https://github.com/pandas-dev/pandas/blob/d0dd9820668ddd4a7648ff9fbd581e67298c77db/pandas/tests/io/json/test_normalize.py):

> [Line 63:14](https://github.com/pandas-dev/pandas/blob/d0dd9820668ddd4a7648ff9fbd581e67298c77db/pandas/tests/io/json/test_normalize.py#L63): [E127](https://duckduckgo.com/?q=pep8%20E127) continuation line over-indented for visual indent
> [Line 64:15](https://github.com/pandas-dev/pandas/blob/d0dd9820668ddd4a7648ff9fbd581e67298c77db/pandas/tests/io/json/test_normalize.py#L64): [E127](https://duckduckgo.com/?q=pep8%20E127) continuation line over-indented for visual indent
> [Line 308:17](https://github.com/pandas-dev/pandas/blob/d0dd9820668ddd4a7648ff9fbd581e67298c77db/pandas/tests/io/json/test_normalize.py#L308): [E126](https://duckduckgo.com/?q=pep8%20E126) continuation line over-indented for hanging indent



##### Comment last updated at 2019-05-11 16:28:34 UTC
# [Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/26328?src=pr&el=h1) Report
> Merging [#26328](https://codecov.io/gh/pandas-dev/pandas/pull/26328?src=pr&el=desc) into [master](https://codecov.io/gh/pandas-dev/pandas/commit/17247edf8439d290cd5f9410df477af8d399c839?src=pr&el=desc) will **decrease** coverage by `51.33%`.
> The diff coverage is `0%`.

[![Impacted file tree graph](https://codecov.io/gh/pandas-dev/pandas/pull/26328/graphs/tree.svg?width=650&token=eZ4WkYLtcO&height=150&src=pr)](https://codecov.io/gh/pandas-dev/pandas/pull/26328?src=pr&el=tree)

```diff
@@             Coverage Diff             @@
##           master   #26328       +/-   ##
===========================================
- Coverage   92.04%   40.71%   -51.34%     
===========================================
  Files         175      175               
  Lines       52297    52306        +9     
===========================================
- Hits        48138    21296    -26842     
- Misses       4159    31010    +26851
```

| Flag | Coverage Δ | |
|---|---|---|
| #multiple | `?` | |
| #single | `40.71% <0%> (-0.18%)` | :arrow_down: |

| [Impacted Files](https://codecov.io/gh/pandas-dev/pandas/pull/26328?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [pandas/io/json/normalize.py](https://codecov.io/gh/pandas-dev/pandas/pull/26328/diff?src=pr&el=tree#diff-cGFuZGFzL2lvL2pzb24vbm9ybWFsaXplLnB5) | `7.47% <0%> (-89.47%)` | :arrow_down: |
| [pandas/io/formats/latex.py](https://codecov.io/gh/pandas-dev/pandas/pull/26328/diff?src=pr&el=tree#diff-cGFuZGFzL2lvL2Zvcm1hdHMvbGF0ZXgucHk=) | `0% <0%> (-100%)` | :arrow_down: |
| [pandas/io/sas/sas\_constants.py](https://codecov.io/gh/pandas-dev/pandas/pull/26328/diff?src=pr&el=tree#diff-cGFuZGFzL2lvL3Nhcy9zYXNfY29uc3RhbnRzLnB5) | `0% <0%> (-100%)` | :arrow_down: |
| [pandas/core/groupby/categorical.py](https://codecov.io/gh/pandas-dev/pandas/pull/26328/diff?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvZ3JvdXBieS9jYXRlZ29yaWNhbC5weQ==) | `0% <0%> (-100%)` | :arrow_down: |
| [pandas/tseries/plotting.py](https://codecov.io/gh/pandas-dev/pandas/pull/26328/diff?src=pr&el=tree#diff-cGFuZGFzL3RzZXJpZXMvcGxvdHRpbmcucHk=) | `0% <0%> (-100%)` | :arrow_down: |
| [pandas/tseries/converter.py](https://codecov.io/gh/pandas-dev/pandas/pull/26328/diff?src=pr&el=tree#diff-cGFuZGFzL3RzZXJpZXMvY29udmVydGVyLnB5) | `0% <0%> (-100%)` | :arrow_down: |
| [pandas/io/formats/html.py](https://codecov.io/gh/pandas-dev/pandas/pull/26328/diff?src=pr&el=tree#diff-cGFuZGFzL2lvL2Zvcm1hdHMvaHRtbC5weQ==) | `0% <0%> (-99.37%)` | :arrow_down: |
| [pandas/io/sas/sas7bdat.py](https://codecov.io/gh/pandas-dev/pandas/pull/26328/diff?src=pr&el=tree#diff-cGFuZGFzL2lvL3Nhcy9zYXM3YmRhdC5weQ==) | `0% <0%> (-91.16%)` | :arrow_down: |
| [pandas/io/sas/sas\_xport.py](https://codecov.io/gh/pandas-dev/pandas/pull/26328/diff?src=pr&el=tree#diff-cGFuZGFzL2lvL3Nhcy9zYXNfeHBvcnQucHk=) | `0% <0%> (-90.1%)` | :arrow_down: |
| [pandas/core/tools/numeric.py](https://codecov.io/gh/pandas-dev/pandas/pull/26328/diff?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvdG9vbHMvbnVtZXJpYy5weQ==) | `10.44% <0%> (-89.56%)` | :arrow_down: |
| ... and [131 more](https://codecov.io/gh/pandas-dev/pandas/pull/26328/diff?src=pr&el=tree-more) | |

------

[Continue to review full report at Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/26328?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/26328?src=pr&el=footer). Last update [17247ed...7b64d1b](https://codecov.io/gh/pandas-dev/pandas/pull/26328?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).

# [Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/26328?src=pr&el=h1) Report
> Merging [#26328](https://codecov.io/gh/pandas-dev/pandas/pull/26328?src=pr&el=desc) into [master](https://codecov.io/gh/pandas-dev/pandas/commit/17247edf8439d290cd5f9410df477af8d399c839?src=pr&el=desc) will **decrease** coverage by `<.01%`.
> The diff coverage is `100%`.

[![Impacted file tree graph](https://codecov.io/gh/pandas-dev/pandas/pull/26328/graphs/tree.svg?width=650&token=eZ4WkYLtcO&height=150&src=pr)](https://codecov.io/gh/pandas-dev/pandas/pull/26328?src=pr&el=tree)

```diff
@@            Coverage Diff             @@
##           master   #26328      +/-   ##
==========================================
- Coverage   92.04%   92.04%   -0.01%     
==========================================
  Files         175      175              
  Lines       52297    52301       +4     
==========================================
  Hits        48138    48138              
- Misses       4159     4163       +4
```

| Flag | Coverage Δ | |
|---|---|---|
| #multiple | `90.59% <100%> (ø)` | :arrow_up: |
| #single | `40.72% <0%> (-0.16%)` | :arrow_down: |

| [Impacted Files](https://codecov.io/gh/pandas-dev/pandas/pull/26328?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [pandas/io/json/normalize.py](https://codecov.io/gh/pandas-dev/pandas/pull/26328/diff?src=pr&el=tree#diff-cGFuZGFzL2lvL2pzb24vbm9ybWFsaXplLnB5) | `97.19% <100%> (+0.25%)` | :arrow_up: |
| [pandas/io/gbq.py](https://codecov.io/gh/pandas-dev/pandas/pull/26328/diff?src=pr&el=tree#diff-cGFuZGFzL2lvL2dicS5weQ==) | `78.94% <0%> (-10.53%)` | :arrow_down: |
| [pandas/compat/\_\_init\_\_.py](https://codecov.io/gh/pandas-dev/pandas/pull/26328/diff?src=pr&el=tree#diff-cGFuZGFzL2NvbXBhdC9fX2luaXRfXy5weQ==) | `93.93% <0%> (-0.35%)` | :arrow_down: |
| [pandas/core/frame.py](https://codecov.io/gh/pandas-dev/pandas/pull/26328/diff?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvZnJhbWUucHk=) | `97.01% <0%> (-0.12%)` | :arrow_down: |
| [pandas/core/sparse/frame.py](https://codecov.io/gh/pandas-dev/pandas/pull/26328/diff?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvc3BhcnNlL2ZyYW1lLnB5) | `95.48% <0%> (-0.01%)` | :arrow_down: |
| [pandas/core/computation/expr.py](https://codecov.io/gh/pandas-dev/pandas/pull/26328/diff?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvY29tcHV0YXRpb24vZXhwci5weQ==) | `96.7% <0%> (-0.01%)` | :arrow_down: |
| [pandas/core/sparse/scipy\_sparse.py](https://codecov.io/gh/pandas-dev/pandas/pull/26328/diff?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvc3BhcnNlL3NjaXB5X3NwYXJzZS5weQ==) | `100% <0%> (ø)` | :arrow_up: |

------

[Continue to review full report at Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/26328?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/26328?src=pr&el=footer). Last update [17247ed...d0dd982](https://codecov.io/gh/pandas-dev/pandas/pull/26328?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).

> I think here you are attempting to both handle `record_path` pointing to JSON objects while also accounting for error handling with missing keys.

> If that's correct I think you should just focus on solving the first issue and we can come back to the latter in a subsequent PR. Otherwise the review and update process may take much longer

@WillAyd you are correct that I have addressed two issues here. The primary one being missing keys, the secondary allowing `record_path`  to parse a single object. The latter only needs two lines  of code 194-196 (to check if the parsed result is anything other than a list, and if not make it a list because it is then correctly parsed into the dataframe).

I can move this to a new PR, but it seems a lot of effort for two lines ? Let me know your preference. 
Can someone explain why **pandas-dev.pandas (Windows py37_np141)** test is failing ?

**Cython-generated file 'pandas\_libs/algos.c' not found.**
> The latter only needs two lines of code 194-196 (to check if the parsed result is anything other than a list, and if not make it a list because it is then correctly parsed into the dataframe).

The fix may be two lines of code but the main problem is that the tests are now ambiguously mixed between the two changes so would still be preferable to have those as separate PRs.
> Can someone explain why **pandas-dev.pandas (Windows py37_np141)** test is failing ?
> 
> **Cython-generated file 'pandas_libs/algos.c' not found.**

Have seen this on a few other PRs something must have changed on the CI side with Windows lately. Not caused by this PR
@Marky0 if you want to merge master and address the comments
closing as stale, if you want to continue, pls ping.