read_json engine keyword and pyarrow integration
State: closed

- [ ] closes #48893
- [x] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [x] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [x] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:
name: IO JSON
description: read_json, to_json, json_normalize
name: Arrow
description: pyarrow functionality

Comments:
Thanks, I appreciate it 🙏🏻 Took me 11 days but finally starting to figure it out. 
So I pushed my recent changes but after I successfully synced my fork, I started to have some problems when running the tests:

1. Independent of the test I run I kept getting the following error:
```
ImportError while loading conftest '/Users/ardkosar1/Documents/personal_projects/pandas/pandas/conftest.py'.
pandas/conftest.py:595: in <module>
    "timedelta": tm.makeTimedeltaIndex(100),
pandas/_testing/__init__.py:401: in makeTimedeltaIndex
    return pd.timedelta_range(start="1 day", periods=k, freq=freq, name=name, **kwargs)
pandas/core/indexes/timedeltas.py:278: in timedelta_range
    tdarr = TimedeltaArray._generate_range(start, end, periods, freq, closed=closed)
pandas/core/arrays/timedeltas.py:271: in _generate_range
    start = Timedelta(start).as_unit("ns")
E   AttributeError: 'Timedelta' object has no attribute 'as_unit'
```
2. I tried commenting out line 595 in `pandas/conftest.py` to see if I could run the tests like that and I started getting the following error:
```
ImportError while loading conftest '/Users/ardkosar1/Documents/personal_projects/pandas/pandas/conftest.py'.
pandas/conftest.py:628: in <module>
    idx = Index(pd.array(tm.makeStringIndex(100), dtype="string[pyarrow]"))
pandas/core/construction.py:322: in array
    dtype = registry.find(dtype) or dtype
pandas/core/dtypes/base.py:521: in find
    return dtype_type.construct_from_string(dtype)
pandas/core/arrays/string_.py:157: in construct_from_string
    return cls(storage="pyarrow")
pandas/core/arrays/string_.py:111: in __init__
    raise ImportError(
E   ImportError: pyarrow>=6.0.0 is required for PyArrow backed StringArray.
```
I think the pyarrow version in the `pandas-dev` environment is `2.0.0` so I tried upgrading pyarrow and then I was able to run my tests; however, I am not sure if I got reliable results. But I wanted to push my updates anyways just to get feedback and fix any issues.

@WillAyd I successfully added engine fixture to all the tests; however, since pyarrows `read_json` doesn't support any of the `pandas.read_json`s arguments I had to add an `xfail` decorator to all the tests so if `engine="pyarrow"` it doesn't cause any errors when running the test suite. 
> Independent of the test I run I kept getting the following error:

Generally you will need to rebuild your cython code each time you pull in code (`python setup.py build_ext -j 4`)

Hi @abkosar Just following up to see if you need any further assistance on this PR. I would be interested in following up to this PR with https://github.com/pandas-dev/pandas/issues/48957 so let us know if you need any help
Hey @mroeschke, Sorry for no updates. Work has been really busy the past two weeks, I haven't had the time to sit and concentrate on the issue. I will start working on it tonight. First I will address your latest comments and we can take it from there. Appreciate the support! Thanks!
And also I don't know why this says there are conflicts because I don't what it shows in my fork.
> And also I don't know why this says there are conflicts because I don't what it shows in my fork.

Did you happen to follow these update instructions? https://pandas.pydata.org/pandas-docs/stable/development/contributing.html#updating-your-pull-request
Yep I checked it out but for me, it didn't work with `git merge upstream/main`, it worked with `rebase`. 
I am trying to run the tests, but I keep getting an error, making it hard for me to run tests and update something. After each rebase I also do `python setup.py build_ext -j 4` as per @mroeschke 's suggestion; however, I keep getting the following error:
```
ERROR: usage: pytest [options] [file_or_dir] [file_or_dir] [...]
pytest: error: unrecognized arguments: --strict-data-files
  inifile: <path>/pandas/pyproject.toml
  rootdir: <path>/pandas
```
I also tried to recreate the development environment, but it didn't work.
How are you running the tests? And to confirm you created your development environment according to https://pandas.pydata.org/pandas-docs/stable/development/contributing_environment.html?
Hey @abkosar mind if I push some commits to your PR? I would be very interested to see this feature in the next major release tentatively scheduled for next month
Hey @mroeschke! Please go ahead, but I was going to do some changes so should I hold off?
Go ahead and still make changes @abkosar! I was planning on pushing this over the finish line closer to the end of the month just in case you didn't have the bandwidth  
Sounds good, thanks! I will work on it this weekend definitely. I will first try to fix the testing errors I was getting and will let you know.
@mroeschke Just pushed the updates @WillAyd asked for.


@mroeschke I have a question regarding the whole rebasing and force pushing. 

To keep the PR clean, I usually squash all the commits I do for each follow-up and then give the squashed commit the same name, which is `ENH: Add engine keyword to read_json to enable reading from pyarrow #48893`. 

However, this gives me a headache each time because when I rebase upstream, it doesn't automatically put my most recent commit to the top (since I squash it, it involves all the changes I have done so far). When I squash and try to push, it tries to take all the commits starting from the first date I made a change. So every time I create a new branch, hard reset the branch to the first date I made a change and first update main and then cherry pick my commit to the top. 

Is there an easier way to do this?
> To keep the PR clean, I usually squash all the commits I do for each follow-up and then give the squashed commit the same name

No need to squash the commits. When merging to main we will squash the commits anyways. Generally we discourage force pushing since it can lose the history and can be hard to know what changed between updates
@mroeschke I did the updates and trying resolve the `This branch has conflicts that must be resolved` at the bottom. I fixed the merge conflicts and pushed but it still says there are conflicts. Can you please advise? Thanks!
It probably means you didn't fully resolve all the conflicts when merging main into your branch https://pandas.pydata.org/pandas-docs/stable/development/contributing.html#updating-your-pull-request
`Code Checks / Docstring validation, typing, and other manual pre-commit hooks (pull_request)` fails but it doesn't fail when I run `pre-commit` on `pandas/io/json/_json.py` on my local because I don't think pre-commit includes `mypy`. 
Here are the checks that need fixing

```
pandas/io/json/_json.py:875: error: Incompatible types in assignment (expression has type "Union[IOHandles[str], IOHandles[bytes]]", variable has type "Optional[IOHandles[str]]")  [assignment]
pandas/io/json/_json.py:884: error: Item "None" of "Optional[IOHandles[str]]" has no attribute "handle"  [union-attr]
pandas/io/json/_json.py:887: error: Argument 1 to "ArrowJsonParserWrapper" has incompatible type "Union[Any, IO[str]]"; expected "ReadBuffer[bytes]"  [arg-type]
```

```
Error: /home/runner/work/pandas/pandas/pandas/io/json/_json.py:486:PR03:pandas.read_json:Wrong parameters order. Actual: ('path_or_buf', 'orient', 'typ', 'dtype', 'convert_axes', 'convert_dates', 'keep_default_dates', 'precise_float', 'date_unit', 'encoding', 'encoding_errors', 'lines', 'chunksize', 'compression', 'nrows', 'storage_options', 'use_nullable_dtypes', 'engine'). Documented: ('path_or_buf', 'orient', 'typ', 'dtype', 'convert_axes', 'convert_dates', 'keep_default_dates', 'precise_float', 'date_unit', 'encoding', 'encoding_errors', 'engine', 'lines', 'chunksize', 'compression', 'nrows', 'storage_options', 'use_nullable_dtypes')
```
```
Error: /home/runner/work/pandas/pandas/pandas/io/json/_json.py:486:PR03:pandas.read_json:Wrong parameters order. Actual: ('path_or_buf', 'orient', 'typ', 'dtype', 'convert_axes', 'convert_dates', 'keep_default_dates', 'precise_float', 'date_unit', 'encoding', 'encoding_errors', 'lines', 'chunksize', 'compression', 'nrows', 'storage_options', 'use_nullable_dtypes', 'engine'). Documented: ('path_or_buf', 'orient', 'typ', 'dtype', 'convert_axes', 'convert_dates', 'keep_default_dates', 'precise_float', 'date_unit', 'encoding', 'encoding_errors', 'engine', 'lines', 'chunksize', 'compression', 'nrows', 'storage_options', 'use_nullable_dtypes')
```
> ```
> Error: /home/runner/work/pandas/pandas/pandas/io/json/_json.py:486:PR03:pandas.read_json:Wrong parameters order. Actual: ('path_or_buf', 'orient', 'typ', 'dtype', 'convert_axes', 'convert_dates', 'keep_default_dates', 'precise_float', 'date_unit', 'encoding', 'encoding_errors', 'lines', 'chunksize', 'compression', 'nrows', 'storage_options', 'use_nullable_dtypes', 'engine'). Documented: ('path_or_buf', 'orient', 'typ', 'dtype', 'convert_axes', 'convert_dates', 'keep_default_dates', 'precise_float', 'date_unit', 'encoding', 'encoding_errors', 'engine', 'lines', 'chunksize', 'compression', 'nrows', 'storage_options', 'use_nullable_dtypes')
> ```

This is fixed too.
I pushed a few commits addressing my comments. I think this should be okay for 2.0. Mind taking a look @phofl 
We are also documenting all key-words in the user guide I think, right at the top of the section.

Otherwise lgtm
thx @mroeschke  and thx @abkosar 
Thanks @mroeschke for all your patience! Happy to be involved!