BUG: read_json does not respect chunksize
State: closed

- [X] closes #36791
- [X] closes #34548
- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [x] whatsnew entry


Labels:
name: Bug
name: IO JSON
description: read_json, to_json, json_normalize

Comments:
Thanks. Mind adding a unit test and whatsnew entry for version 1.2
I've added a test and addressed the comments.
can you also post the benchmarks for line read json (should show that this improves vs master)
I don't expect it to improve raw performance, it's about memory usage for large files. 
> I don't expect it to improve raw performance, it's about memory usage for large files.

we have memory benchmarks to measure this
@robertwb if you can post the memory benchmarks
the output is `BENCHMARKS NOT SIGNIFICANTLY CHANGED.` but IIUC `io.json.ReadJSONLines.peakmem_read_json_lines_concat` is reduced from 206M to 188M

maybe we need to change the benchmark to see a bigger effect.

<details>

```
(pandas-dev) simon@WIN-7LJRFMBC2ME:~/pandas/asv_bench$ asv continuous -f 1.1 master HEAD -b peakmem_read_json
· Creating environments
· Discovering benchmarks
·· Uninstalling from conda-py3.8-Cython0.29.21-jinja2-matplotlib-numba-numexpr-numpy-odfpy-openpyxl-pytables-pytest-scipy-sqlalchemy-xlrd-xlsxwriter-xlwt.
·· Building 4b856ee8 <json-chunksize> for conda-py3.8-Cython0.29.21-jinja2-matplotlib-numba-numexpr-numpy-odfpy-openpyxl-pytables-pytest-scipy-sqlalchemy-xlrd-xlsxwriter-xlwt.........................................
·· Installing 4b856ee8 <json-chunksize> into conda-py3.8-Cython0.29.21-jinja2-matplotlib-numba-numexpr-numpy-odfpy-openpyxl-pytables-pytest-scipy-sqlalchemy-xlrd-xlsxwriter-xlwt.
· Running 6 total benchmarks (2 commits * 1 environments * 3 benchmarks)
[  0.00%] · For pandas commit 4b856ee8 <json-chunksize> (round 1/1):
[  0.00%] ·· Benchmarking conda-py3.8-Cython0.29.21-jinja2-matplotlib-numba-numexpr-numpy-odfpy-openpyxl-pytables-pytest-scipy-sqlalchemy-xlrd-xlsxwriter-xlwt
[ 16.67%] ··· io.json.ReadJSONLines.peakmem_read_json_lines                                                           ok
[ 16.67%] ··· ========== ======
                index
              ---------- ------
                 int      250M
               datetime   250M
              ========== ======

[ 33.33%] ··· io.json.ReadJSONLines.peakmem_read_json_lines_concat                                                    ok
[ 33.33%] ··· ========== ======
                index
              ---------- ------
                 int      188M
               datetime   188M
              ========== ======

[ 50.00%] ··· io.json.ReadJSONLines.peakmem_read_json_lines_nrows                                                     ok
[ 50.00%] ··· ========== ======
                index
              ---------- ------
                 int      188M
               datetime   188M
              ========== ======

[ 50.00%] · For pandas commit 2733a109 <master> (round 1/1):
[ 50.00%] ·· Building for conda-py3.8-Cython0.29.21-jinja2-matplotlib-numba-numexpr-numpy-odfpy-openpyxl-pytables-pytest-scipy-sqlalchemy-xlrd-xlsxwriter-xlwt......................................
[ 50.00%] ·· Benchmarking conda-py3.8-Cython0.29.21-jinja2-matplotlib-numba-numexpr-numpy-odfpy-openpyxl-pytables-pytest-scipy-sqlalchemy-xlrd-xlsxwriter-xlwt
[ 66.67%] ··· io.json.ReadJSONLines.peakmem_read_json_lines                                                           ok
[ 66.67%] ··· ========== ======
                index
              ---------- ------
                 int      248M
               datetime   248M
              ========== ======

[ 83.33%] ··· io.json.ReadJSONLines.peakmem_read_json_lines_concat                                                    ok
[ 83.33%] ··· ========== ======
                index
              ---------- ------
                 int      206M
               datetime   206M
              ========== ======

[100.00%] ··· io.json.ReadJSONLines.peakmem_read_json_lines_nrows                                                     ok
[100.00%] ··· ========== ======
                index
              ---------- ------
                 int      195M
               datetime   195M
              ========== ======


BENCHMARKS NOT SIGNIFICANTLY CHANGED.
```

</details>

> the output is `BENCHMARKS NOT SIGNIFICANTLY CHANGED.` but IIUC `io.json.ReadJSONLines.peakmem_read_json_lines_concat` is reduced from 206M to 188M
> 
> maybe we need to change the benchmark to see a bigger effect.

so how do we know that this is working (and the previous was not), sure this is a difference, but wouldn't you expect a HUGE diffrence here? (e.g. maybe the benchmark should be reading with chunksize=1 or something) vs no chunksize.
> so how do we know that this is working (and the previous was not), sure this is a difference, but wouldn't you expect a HUGE diffrence here? (e.g. maybe the benchmark should be reading with chunksize=1 or something) vs no chunksize.

we do a concat of the chunks in the benchmark. maybe could just iterate through the chunks without the concat.

am currently running with chunksize=1 to compare.

> > so how do we know that this is working (and the previous was not), sure this is a difference, but wouldn't you expect a HUGE diffrence here? (e.g. maybe the benchmark should be reading with chunksize=1 or something) vs no chunksize.
> 
> we do a concat of the chunks in the benchmark. maybe could just iterate through the chunks without the concat.
> 
> am currently running with chunksize=1 to compare.

oh yeah the concat will increase memory :-> don't compare that
changing the chunksize to 10 (1 fails) make a bit more difference (to the memory used on master!)

```
       before           after         ratio
     [2733a109]       [d731dc87]
     <master>         <json-chunksize>
-            224M             189M     0.84  io.json.ReadJSONLines.peakmem_read_json_lines_concat('int')
-            224M             188M     0.84  io.json.ReadJSONLines.peakmem_read_json_lines_concat('datetime')

SOME BENCHMARKS HAVE CHANGED SIGNIFICANTLY.
```
> oh yeah the concat will increase memory :-> don't compare that

presumably we do the concat to compare the results with peakmem_read_json_lines

```
    def peakmem_read_json_lines(self, index):
        read_json(self.fname, orient="records", lines=True)
```
ok fair. @simonjayhawkins wouldn't object to changing the benchmarks (but this PR is fine).
thanks @robertwb 
@meeseeksdev backport 1.2.x