When calling load_dataset, raise error: pyarrow.lib.ArrowInvalid: offset overflow while concatenating arrays 
Opened by aihao2000 (closed)

### Describe the bug

When calling load_dataset, raise error
```
Traceback (most recent call last):                                                                             
  File "/home/aihao/miniconda3/envs/torch/lib/python3.11/site-packages/datasets/builder.py", line 1694, in _pre
pare_split_single                                                                                              
    writer.write(example, key)                                                                                 
  File "/home/aihao/miniconda3/envs/torch/lib/python3.11/site-packages/datasets/arrow_writer.py", line 490, in 
write                                                                                                          
    self.write_examples_on_file()                                                                              
  File "/home/aihao/miniconda3/envs/torch/lib/python3.11/site-packages/datasets/arrow_writer.py", line 448, in 
write_examples_on_file                                                                                         
    self.write_batch(batch_examples=batch_examples)                                                            
  File "/home/aihao/miniconda3/envs/torch/lib/python3.11/site-packages/datasets/arrow_writer.py", line 559, in 
write_batch 
    self.write_table(pa_table, writer_batch_size)      
  File "/home/aihao/miniconda3/envs/torch/lib/python3.11/site-packages/datasets/arrow_writer.py", line 571, in 
write_table                
    pa_table = pa_table.combine_chunks()               
               ^^^^^^^^^^^^^^^^^^^^^^^^^               
  File "pyarrow/table.pxi", line 3439, in pyarrow.lib.Table.combine_chunks                                     
  File "pyarrow/error.pxi", line 144, in pyarrow.lib.pyarrow_internal_check_status                             
  File "pyarrow/error.pxi", line 100, in pyarrow.lib.check_status                                              
pyarrow.lib.ArrowInvalid: offset overflow while concatenating arrays
The above exception was the direct cause of the following exception:                                           
                                                                                                               
Traceback (most recent call last):
dataset = load_dataset(                            
              ^^^^^^^^^^^^^                            
  File "/home/aihao/miniconda3/envs/torch/lib/python3.11/site-packages/datasets/load.py", line 2133, in load_da
taset                      
    builder_instance.download_and_prepare(             
  File "/home/aihao/miniconda3/envs/torch/lib/python3.11/site-packages/datasets/builder.py", line 954, in downl
oad_and_prepare            
    self._download_and_prepare(                        
  File "/home/aihao/miniconda3/envs/torch/lib/python3.11/site-packages/datasets/builder.py", line 1717, in _dow
nload_and_prepare          
    super()._download_and_prepare(                     
  File "/home/aihao/miniconda3/envs/torch/lib/python3.11/site-packages/datasets/builder.py", line 1049, in _dow
nload_and_prepare          
    self._prepare_split(split_generator, **prepare_split_kwargs)                                               
  File "/home/aihao/miniconda3/envs/torch/lib/python3.11/site-packages/datasets/builder.py", line 1555, in _pre
pare_split                 
    for job_id, done, content in self._prepare_split_single(                                                   
  File "/home/aihao/miniconda3/envs/torch/lib/python3.11/site-packages/datasets/builder.py", line 1712, in _pre
pare_split_single          
    raise DatasetGenerationError("An error occurred while generating the dataset") from e                      
datasets.builder.DatasetGenerationError: An error occurred while generating the dataset                        
Setting num_proc from 8 back to 1 for the train split to disable multiprocessing as it only contains one shard.
09/04/2023 12:02:04 - WARNING - datasets.builder - Setting num_proc from 8 back to 1 for the train split to dis
able multiprocessing as it only contains one shard.
```

### Steps to reproduce the bug

Call load_dataset with the large image as feature

### Expected behavior

no error

### Environment info

- `datasets` version: 2.14.3
- Platform: Linux-6.2.0-31-generic-x86_64-with-glibc2.35
- Python version: 3.11.4
- Huggingface_hub version: 0.16.4
- PyArrow version: 12.0.1
- Pandas version: 2.0.3

Comments:
I solved the problem by modifying the "self DEFAULT_WRITER_BATCH_SIZE" in "class MyDataset (datasets. GeneratorBasedBuilder) : __init__"

same problem, and this solution worked me also - you can set this var by setting the keyword argument `writer_batch_size=...` in `load_dataset(...,writer_batch_size=...)`