load_datasets("json", ...) don't read local .json.gz properly
State: open | Author: junwang-wish (NONE)
Labels: name: bug

## Describe the bug
I have a local file `*.json.gz` and it can be read by `pandas.read_json(lines=True)`, but cannot be read by `load_datasets("json")` (resulting in 0 lines)

## Steps to reproduce the bug
```python
fpath = '/data/junwang/.cache/general/57b6f2314cbe0bc45dda5b78f0871df2/test.json.gz'
ds_panda = DatasetDict(
    test=Dataset.from_pandas(
        pd.read_json(fpath, lines=True)
    )
)
ds_direct = load_dataset(
    'json', data_files={
        'test': fpath
    }, features=Features(
        text_input=Value(dtype="string", id=None),
        text_output=Value(dtype="string", id=None)
    )
)
len(ds_panda['test']), len(ds_direct['test'])
```

## Expected results
Lines of `ds_panda['test']` and `ds_direct['test']` should match.

## Actual results
```
Using custom data configuration default-c0ef2598760968aa
Downloading and preparing dataset json/default to /data/junwang/.cache/huggingface/datasets/json/default-c0ef2598760968aa/0.0.0/e6070c77f18f01a5ad4551a8b7edfba20b8438b7cad4d94e6ad9378022ce4aab...
Dataset json downloaded and prepared to /data/junwang/.cache/huggingface/datasets/json/default-c0ef2598760968aa/0.0.0/e6070c77f18f01a5ad4551a8b7edfba20b8438b7cad4d94e6ad9378022ce4aab. Subsequent calls will reuse this data.

(62087, 0)
```

## Environment info
<!-- You can run the command `datasets-cli env` and copy-and-paste its output below. -->
- `datasets` version:
- Platform: Ubuntu 18.04.4 LTS 
- Python version: 3.8.13
- PyArrow version: 9.0.0


Comments: Hi @junwang-wish, thanks for reporting.

Unfortunately, I'm not able to reproduce the bug. Which version of `datasets` are you using? Does the problem persist if you update `datasets`?
```shell
pip install -U datasets
``` 

Thanks @albertvillanova I updated `datasets` from `2.5.1` to `2.5.2` and tested copying the `json.gz` to a different directory and my mind was blown:

```python
fpath = '/data/junwang/.cache/general/57b6f2314cbe0bc45dda5b78f0871df2/test.json.gz'
ds_panda = DatasetDict(
    test=Dataset.from_pandas(
        pd.read_json(fpath, lines=True)
    )
)
ds_direct = load_dataset(
    'json', data_files={
        'test': fpath
    }, features=Features(
        text_input=Value(dtype="string", id=None),
        text_output=Value(dtype="string", id=None)
    )
)
len(ds_panda['test']), len(ds_direct['test'])
```
produces 
```python
Using custom data configuration default-0e6cf24134163e8b
Found cached dataset json (/data/junwang/.cache/huggingface/datasets/json/default-0e6cf24134163e8b/0.0.0/e6070c77f18f01a5ad4551a8b7edfba20b8438b7cad4d94e6ad9378022ce4aab)
(1, 0)
```
but then I ran below command to see if the same file in a different directory leads to same discrepancy
```shell
cp /data/junwang/.cache/general/57b6f2314cbe0bc45dda5b78f0871df2/test.json.gz tmp_test.json.gz
```
and so I ran
```python
fpath = 'tmp_test.json.gz'
ds_panda = DatasetDict(
    test=Dataset.from_pandas(
        pd.read_json(fpath, lines=True)
    )
)
ds_direct = load_dataset(
    'json', data_files={
        'test': fpath
    }, features=Features(
        text_input=Value(dtype="string", id=None),
        text_output=Value(dtype="string", id=None)
    )
)
len(ds_panda['test']), len(ds_direct['test'])
```
and behold, I get 
```python
Using custom data configuration default-f679b32ab0008520
Downloading and preparing dataset json/default to /data/junwang/.cache/huggingface/datasets/json/default-f679b32ab0008520/0.0.0/e6070c77f18f01a5ad4551a8b7edfba20b8438b7cad4d94e6ad9378022ce4aab...
Dataset json downloaded and prepared to /data/junwang/.cache/huggingface/datasets/json/default-f679b32ab0008520/0.0.0/e6070c77f18f01a5ad4551a8b7edfba20b8438b7cad4d94e6ad9378022ce4aab. Subsequent calls will reuse this data.
(1, 1)
```
They match now !

This problem happens regardless of the shell I use (VScode jupyter extension or plain old Python REPL). 

I attached the `json.gz` here for reference: [test.json.gz](https://github.com/huggingface/datasets/files/9734843/test.json.gz)