Error when loading a HUGE json file (pyarrow.lib.ArrowInvalid: straddling object straddles two block boundaries)
State: closed | Author: hwijeen (NONE)

Hi, thanks for the great library. I have used the brilliant library for a couple of small projects, and now using it for a fairly big project.
When loading a huge json file of 500GB, pyarrow complains as follows:
```
Traceback (most recent call last):
  File "/home/user/.pyenv/versions/3.7.9/lib/python3.7/site-packages/datasets/builder.py", line 531, in incomplete_dir
    yield tmp_dir
  File "/home/user/.pyenv/versions/3.7.9/lib/python3.7/site-packages/datasets/builder.py", line 573, in download_and_prepare
    dl_manager=dl_manager, verify_infos=verify_infos, **download_and_prepare_kwargs
  File "/home/user/.pyenv/versions/3.7.9/lib/python3.7/site-packages/datasets/builder.py", line 650, in _download_and_prepare
    self._prepare_split(split_generator, **prepare_split_kwargs)
  File "/home/user/.pyenv/versions/3.7.9/lib/python3.7/site-packages/datasets/builder.py", line 1027, in _prepare_split
    for key, table in utils.tqdm(generator, unit=" tables", leave=False, disable=not_verbose):
  File "/home/user/.pyenv/versions/3.7.9/lib/python3.7/site-packages/tqdm/std.py", line 1133, in __iter__
    for obj in iterable:
  File "/app/.cache/huggingface/modules/datasets_modules/datasets/json/9498524fd296a6cca99c66d6c5be507d1c0991f5a814e535b507f4a66096a641/json.py", line 83, in _generate_tables
    parse_options=self.config.pa_parse_options,
  File "pyarrow/_json.pyx", line 247, in pyarrow._json.read_json
  File "pyarrow/error.pxi", line 122, in pyarrow.lib.pyarrow_internal_check_status
  File "pyarrow/error.pxi", line 84, in pyarrow.lib.check_status
pyarrow.lib.ArrowInvalid: straddling object straddles two block boundaries (try to increase block size?)
```
When using only a small portion of the sample file, say first 100 lines, it works perfectly well..

I see that it is the error from pyarrow, but could you give me a hint or possible solutions?
#369 describes the same error and #372 claims to have fixed the issue, but I have no clue why I am still getting this one. Thanks in advance!

Comments: Hi ! Can you try to increase the block size ? For example
```python
block_size_10MB = 10<<20
load_dataset("json", ..., block_size=block_size_10MB)
```
The block size corresponds to how much bytes to process at a time from the input stream.
This will determine multi-threading granularity as well as the size of individual chunks in the dataset.

You can also try with bigger block sizes if needed

Hi @lhoestq! Thank you for your prompt reply.
I have experimented with (10<<20, 10<<28, 10<<30, 10<<33, 10<<34), since my machine has 192G of memory, but it's either the above-mentioned error or processed killed because of OOM.

Could you give me a bit of background on why block size needs to be exactly calibrated?
To my understanding, small block sized should run just fine despite its slowness..




We're using the JSON loader of pyarrow. It parses the file chunk by chunk to load the dataset.
This issue happens when there's no delimiter in one chunk of data. For json line, the delimiter is the end of line.
So with a big value for chunk_size this should have worked unless you have one extremely long line in your file.

Also what version of pyarrow are you using ?

FInally I wonder if it could be an issue on pyarrow's side when using big json files. (I haven't tested big json files like yours)

I'm using `pyarrow==3.0.0` with `datasets==1.5.0`.

Your point totally makes sense. I will check if my jsonl file contains an extremely long file and let you know. 

Here are some different error messages that I got when tweaking `block_size`. I also suspect that this is related to the pyarrow... but I guess it would be wonderful if datasesets could give a clear guide on how to play with large datasets! (I am suddenly experiencing various issue when working with large datasets.. e.g. #1992 )
```python
    return paj.ReadOptions(use_threads=self.use_threads, block_size=self.block_size)
  File "pyarrow/_json.pyx", line 56, in pyarrow._json.ReadOptions.__init__
  File "pyarrow/_json.pyx", line 81, in pyarrow._json.ReadOptions.block_size.__set__
OverflowError: value too large to convert to int32_t
```

```python

line 83, in _generate_tables
    parse_options=self.config.pa_parse_options,
  File "pyarrow/_json.pyx", line 247, in pyarrow._json.read_json
  File "pyarrow/error.pxi", line 122, in pyarrow.lib.pyarrow_internal_check_status
  File "pyarrow/error.pxi", line 84, in pyarrow.lib.check_status
pyarrow.lib.ArrowInvalid: Exceeded maximum rows
```

I am getting the same error. When I tweak the block_size, I also find:
`OverflowError: value too large to convert to int32_t`
and 
`pyarrow.lib.ArrowInvalid: Exceeded maximum rows`


I made more tests. I used a smaller dataset and I was getting the same error, which means that it was not necessarily linked to the dataset size. To make both my smaller and larger datasets work, I got rid of lists with the json file. I had the following data format:
```python
[
  {'key': "a", 'value': ['one', 'two', 'three']},
  {'key': "b", 'value': ['four', 'five', 'six']}
]
```
I changed to:

```python
  {'key': "a", 'value': 'one\ntwo\nthree'},
  {'key': "b", 'value': 'four\nfive\nsix']}
```
and that worked!

I used the following to reformat my json file:
```python
with open(file_name, "w", encoding="utf-8") as f:
    for item in list_:
        f.write(json.dumps(item) + "\n")
```
This works with `block_size_10MB = 10 << 20` or without specifying `block_size`.

Thanks @hwijeen for reporting and thanks @jpilaul for pointing this out.

Indeed, those are different JSON-like formats:
- the first one is the **standard JSON** format: all the file content is JSON-valid, thus all content is either a JSON object (between curly brackets `{...}`) or a JSON array (between square brackets `[...]`)
- the second one is called **JSON Lines**: the entire file content is not JSON-valid, but only every line (newline-delimited) is JSON-valid

Currently PyArrow only supports **JSON Lines** format: 
- https://arrow.apache.org/docs/python/generated/pyarrow.json.read_json.html
  > Currently only the line-delimited JSON format is supported.
- https://arrow.apache.org/docs/python/json.html
  > Arrow supports reading columnar data from line-delimited JSON files.

Thanks @albertvillanova for your explanation, it is helpful to know (maybe add to docs?)!
However, the problem I described above happened when I was dealing with jsonl files 😿
Although I did not thoroughly inspect, I suspect the cause was the one extremely long document in my case.

I see... I guess there is another problem going one then, related to the size.