PERF: json_normalize, for basic use case
State: closed

- [X] closes #15621
- [X] tests passed
- [X] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing.html#code-standards) for how to run them

Proposed speed up for very simple use cases using the `pd.json_normalize` function. E.g. `pd.json_normalize(data)`

The speed up can be seen in this example:
```
import datetime
import pandas as pd

# example json data
data = {"hello": ["thisisatest", 999898, datetime.date.today()],
        "nest1": {"nest2": {"nest3": "nest3_value", "nest3_int": 3445}},
        "nest1_list": {"nest2": ["blah", 32423, 546456.876, 92030234]},
        "hello2": "string"}

hundred_thousand_rows = [data for i in range(100000)]

s = time()
pd.json_normalize(hundred_thousand_rows)
pandas_json_normalize_time_taken = time() - s
print(f"\npandas time taken for a 100,000 rows: {pandas_json_normalize_time_taken} seconds")
```

With output from Pandas 1.2.2: `pandas time taken for a 100,000 rows: 3.0518009662628174 seconds`
From this branch: `pandas time taken for a 100,000 rows: 0.632451057434082 seconds`

To show tests pass for the appropriate file, ran `pytest pandas/tests/io/json/test_normalize.py -v` [test_normalize.py_pytest.log](https://github.com/pandas-dev/pandas/files/6039361/test_normalize.py_pytest.log)

To show pre-commit passed on file, ran `pre-commit run --files pandas/io/json/_normalize.py` [_normalize.py_pre-commit.log](https://github.com/pandas-dev/pandas/files/6039363/_normalize.py_pre-commit.log)

There was one code check that was caught, running `./ci/code_checks.sh` [code_checks.log](https://github.com/pandas-dev/pandas/files/6039360/code_checks.log)

```
pandas/io/json/_normalize.py:208: error: Incompatible types in assignment (expression has type "List[Union[List[Dict[Any, Any]], Dict[Any, Any]]]", variable has type "Dict[Any, Any]")  [assignment]
```
As it a type hint issue, decided to still make pull request. Can you advice on how to fix, I'm fairly new to type hints?

Kind regards,
Sam

Labels:
name: Performance
description: Memory or execution speed performance
name: IO JSON
description: read_json, to_json, json_normalize

Comments:
Just remembered forgot to update the changelog, woopsy
@jreback 
> is it possible to simply dispatch to this if the basic case is selected?

If possible that would be best, but am not familiar enough with pandas codebase to know where to look. Have tried looking around the `pandas/io/json/` but can't find where dispatch configuration is setup. Any advise for where to look/how to get started would be appreciated.

> why is ordering not preserved?

Oh it is, I need to update that part of the comment.

> do we have sufficient asv's for this? e.g. pls add the cases you are measuring.

I do, I will add those in at the next opportunity and think of a few more cases.


What would you recommend to fix the type hint issue I have?
```
pandas/io/json/_normalize.py:208: error: Incompatible types in assignment (expression has type "List[Union[List[Dict[Any, Any]], Dict[Any, Any]]]", variable has type "Dict[Any, Any]")  [assignment]
```
Hello @smpurkis! Thanks for updating this PR. We checked the lines you've touched for [PEP 8](https://www.python.org/dev/peps/pep-0008) issues, and found:





There are currently no PEP 8 issues detected in this Pull Request. Cheers! :beers: 

##### Comment last updated at 2021-03-03 19:07:29 UTC
Added asv, as there were none for `json_normalize` function. Although can't run locally as machine isn't powerful enough.
Have also touched up a comment and condition statement
asv failing, I've not written any before and my laptop is too slow to run them
asv benchmark should be working now
Can you run the related JSON benchmarks and post the output of them here? 
Running
```
asv continuous -f 1.1 -E virtualenv upstream/master HEAD -b json
```
gave this log: 
[asv-json-benchmark.log](https://github.com/pandas-dev/pandas/files/6051608/asv-json-benchmark.log)
so net effect is this

```
     <issue-15621-improve-json-normalize-perf>       <master>  
         320±2ms        285±0.5ms     0.89  io.json.NormalizeJSON.time_normalize_json('values', 'df_int_floats')
        326±20ms          277±9ms     0.85  io.json.NormalizeJSON.time_normalize_json('records', 'df_int_floats')
```
?
this doesnt' seem to match your results.
> this doesnt' seem to match your results.

Can you please explain a bit more on how you are comparing them. The test is the same, but asv sets up its own environment which surely would change the times.
Unless asv runs the same benchmark on pandas master, not sure how the results are useful
if you added an asv then we could see

this doesn't thatch your timings from the top (the ratio not the absolute time)
Assuming asv is comparing my forked master to main master then my addition in asv might be wrong. 
Will have another look, and do some reading up on asv
Found the issue, my checking of the parameters was incorrect. Reran the benchmark.

```
       before           after         ratio
     [a241cfc6]       [5959eaab]
     <issue-15621-improve-json-normalize-perf>       <master>  
-         317±1ms         65.5±2ms     0.21  io.json.NormalizeJSON.time_normalize_json('values', 'df_date_idx')
-       317±0.5ms       65.4±0.3ms     0.21  io.json.NormalizeJSON.time_normalize_json('split', 'df_date_idx')
-         317±2ms       65.3±0.9ms     0.21  io.json.NormalizeJSON.time_normalize_json('values', 'df_td_int_ts')
-         316±2ms       65.2±0.5ms     0.21  io.json.NormalizeJSON.time_normalize_json('index', 'df_date_idx')
-         317±1ms       65.4±0.4ms     0.21  io.json.NormalizeJSON.time_normalize_json('index', 'df_int_floats')
-       316±0.9ms       65.1±0.3ms     0.21  io.json.NormalizeJSON.time_normalize_json('values', 'df')
-       315±0.8ms       64.9±0.1ms     0.21  io.json.NormalizeJSON.time_normalize_json('columns', 'df_td_int_ts')
-       316±0.4ms       65.0±0.2ms     0.21  io.json.NormalizeJSON.time_normalize_json('index', 'df')
-       316±0.6ms       64.9±0.4ms     0.21  io.json.NormalizeJSON.time_normalize_json('split', 'df_td_int_ts')
-         316±2ms       64.9±0.3ms     0.21  io.json.NormalizeJSON.time_normalize_json('split', 'df_int_floats')
-       316±0.6ms       64.8±0.2ms     0.21  io.json.NormalizeJSON.time_normalize_json('records', 'df')
-         317±1ms       65.0±0.2ms     0.21  io.json.NormalizeJSON.time_normalize_json('records', 'df_date_idx')
-         317±1ms       65.0±0.5ms     0.21  io.json.NormalizeJSON.time_normalize_json('index', 'df_int_float_str')
-       317±0.4ms       65.1±0.2ms     0.20  io.json.NormalizeJSON.time_normalize_json('records', 'df_int_floats')
-       316±0.7ms       64.8±0.2ms     0.20  io.json.NormalizeJSON.time_normalize_json('values', 'df_int_floats')
-       316±0.2ms       64.8±0.2ms     0.20  io.json.NormalizeJSON.time_normalize_json('columns', 'df_int_floats')
-         315±1ms       64.5±0.3ms     0.20  io.json.NormalizeJSON.time_normalize_json('split', 'df_int_float_str')
-         318±1ms       64.9±0.1ms     0.20  io.json.NormalizeJSON.time_normalize_json('split', 'df')
-       317±0.9ms       64.8±0.2ms     0.20  io.json.NormalizeJSON.time_normalize_json('columns', 'df_date_idx')
-         317±1ms       64.8±0.2ms     0.20  io.json.NormalizeJSON.time_normalize_json('values', 'df_int_float_str')
-         318±1ms       64.8±0.3ms     0.20  io.json.NormalizeJSON.time_normalize_json('records', 'df_td_int_ts')
-       318±0.8ms       64.8±0.4ms     0.20  io.json.NormalizeJSON.time_normalize_json('records', 'df_int_float_str')
-         319±5ms       65.1±0.5ms     0.20  io.json.NormalizeJSON.time_normalize_json('columns', 'df_int_float_str')
-         317±2ms       64.6±0.2ms     0.20  io.json.NormalizeJSON.time_normalize_json('columns', 'df')
-         319±2ms       64.9±0.4ms     0.20  io.json.NormalizeJSON.time_normalize_json('index', 'df_td_int_ts')

SOME BENCHMARKS HAVE CHANGED SIGNIFICANTLY.
PERFORMANCE INCREASED.

```
Have made the whatsnew note and the changes you advised.
There is still the type hint issue originally picked up, haven't found a way to correct it
thanks @smpurkis very nice!