ENH: Adding json line parsing to pd.read_json #9180
State: closed

- [x] closes #9180
- [x] closed #13356 
- [x] tests added / passed
- [x] passes `git diff upstream/master | flake8 --diff`
- [x] whatsnew entry


Labels:
name: Enhancement
name: IO JSON
description: read_json, to_json, json_normalize

Comments:
lgtm, pls add a whatsnew note.

cc @Komnomnomnom 

how does this look?

@jreback I think I added all your suggestions. Thanks for the review!

## [Current coverage](https://codecov.io/gh/pydata/pandas/pull/13351?src=pr) is **84.31%**

> Merging [#13351](https://codecov.io/gh/pydata/pandas/pull/13351?src=pr) into [master](https://codecov.io/gh/pydata/pandas/branch/master?src=pr) will decrease coverage by **0.22%**

``` diff
@@             master     #13351   diff @@
==========================================
  Files           141        138     -3   
  Lines         51185      51177     -8   
  Methods           0          0          
  Messages          0          0          
  Branches          0          0          
==========================================
- Hits          43275      43151   -124   
- Misses         7910       8026   +116   
  Partials          0          0          
```

> Powered by [Codecov](https://codecov.io?src=pr). Last updated by [506520b...c56a6a8](https://codecov.io/gh/pydata/pandas/compare/506520bd35331aa82db50686c07d96594cac0c10...c56a6a854d6f521c6c0be8a7a960984c32a74dee?src=pr)

Dask.dataframe users have started asking about this: https://github.com/dask/dask/issues/1236

I think need to pass thru the encoding option first, then this PR can work.

Yeah I'm looking through the code a bit more. The problem is that the current code works for json without an encoding. So adding this feature would make the encoding required which is not exactly a good experience for the user. Additionally, it seems there are several different standards. I'm still grokking the differences and how the parser works and will update the PR when I figure out the right answer (or if someone else has a better patch)

@aterrel no, you just need to _do_ something if `encoding` is not `None` which would be the default.

This is _already_ handled by `io/common/get_filepath_or_buffer`, literally this just needs to be passed thru (with some tests :)

Not sure if this is the right place to have this commentary, so please let me know if there is a better place.

This code got a little messier than I like. The `get_filepath_or_buffer` only decodes for url's (which I guess I should add a test for) and if a file is given it basically returns the filepath back and none for the encoding.

The `_get_handler` adds files correctly but I have to keep around the encoding.

It would be nice to have a `gimme_unicode` function that given any file, url, or buffer would produce the unicode stream handler. `UnicodeReader` looks like a good candidate but it is specialized to csv.  

Anywho, please let me know if I understand the situation incorrectly. As is it might be good to accept the patch as is and have a new issue that unifies the unicode readers of csv and json (and other formats that are unicode).

@aterrel yeah its a little bit convoluted now, similar to a cleanup with compression, need to do same for unicode. I will make another issue for that.

xref https://github.com/pydata/pandas/issues/13401

this closes #13356 fully as well?

lgtm otherwise. ping on green.

@aterrel minor comments. ping when pushed and green.

Okay I'll try to get a few of these taken care of tonight. I should also make it be able to output to json lines as well.

BTW, there are now conflicts with this PR. Is the usual thing to rebase?

yes pls rebase

@jreback fixed it and added the ability to write to line delimited json. I ended up adding more than one line to the docs, because you know you need Hadoop in there for proper SEO =P

@aterrel Added minor doc comments.

Should there be some tests for the new `encoding` keyword? Maybe that is already tested through the use of `_get_handle` function, but there should at least be a test that checks that `read_json` accepts this keyword argument I think.

It is but I can put another test in. Looks like the coverage is only missing for the case when `to_json` recieves a `lines` parameter and not `orient=records`

@aterrel Github indicates that this cannot be merged (has conflicts). Did you rebase on upstream/master?

I did and it was green for a bit but I think the whatsnew changed again.

I'll rebase again when I get those tests put together.

can you rebase

@jreback @jorisvandenbossche I've rebased and added an encoding test.

fixed the doc issues, will add encodings for `to_json` later.

lgtm. @jorisvandenbossche ?

I understand that docs for `encoding` will be added in a follow-up PR? (as this is a new keyword as well?)

yes IIRC can add `encoding` in `.to_json` in a future issue; @aterrel can you create an issue for that as well.

For the rest, merge away!

thanks @aterrel 

nice PR!

give a check in a few hours (or prob tomorrow) in the built dev-docs and see that everything looks ok for the changes.

sweet. I'll follow along in the other issues to keep up all the issues.