GitHub issue #6007: Get an error "OverflowError: Python int too large to convert to C long" when loading a large dataset
State: open | Created: 2023-07-05T15:16:50Z

### Describe the bug

When load a large dataset with the following code

```python
from datasets import load_dataset
dataset = load_dataset("liwu/MNBVC", 'news_peoples_daily', split='train')
```

We encountered the error: "OverflowError: Python int too large to convert to C long"
The error look something like:

```
OverflowError: Python int too large to convert to C long

During handling of the above exception, another exception occurred:

OverflowError                             Traceback (most recent call last)
<ipython-input-7-0ed8700e662d> in <module>
----> 1 dataset = load_dataset("liwu/MNBVC", 'news_peoples_daily', split='train', cache_dir='/sfs/MNBVC/.cache/')

/sfs/MNBVC/venv/lib64/python3.6/site-packages/datasets/load.py in load_dataset(path, name, data_dir, data_files, split, cache_dir, features, download_config, download_mode, ignore_verifications, keep_in_memory, save_infos, revision, use_auth_token, task, streaming, **config_kwargs)
   1749         ignore_verifications=ignore_verifications,
   1750         try_from_hf_gcs=try_from_hf_gcs,
-> 1751         use_auth_token=use_auth_token,
   1752     )
   1753 

/sfs/MNBVC/venv/lib64/python3.6/site-packages/datasets/builder.py in download_and_prepare(self, download_config, download_mode, ignore_verifications, try_from_hf_gcs, dl_manager, base_path, use_auth_token, **download_and_prepare_kwargs)
    703                     if not downloaded_from_gcs:
    704                         self._download_and_prepare(
--> 705                             dl_manager=dl_manager, verify_infos=verify_infos, **download_and_prepare_kwargs
    706                         )
    707                     # Sync info

/sfs/MNBVC/venv/lib64/python3.6/site-packages/datasets/builder.py in _download_and_prepare(self, dl_manager, verify_infos)
   1225 
   1226     def _download_and_prepare(self, dl_manager, verify_infos):
-> 1227         super()._download_and_prepare(dl_manager, verify_infos, check_duplicate_keys=verify_infos)
   1228 
   1229     def _get_examples_iterable_for_split(self, split_generator: SplitGenerator) -> ExamplesIterable:

/sfs/MNBVC/venv/lib64/python3.6/site-packages/datasets/builder.py in _download_and_prepare(self, dl_manager, verify_infos, **prepare_split_kwargs)
    791             try:
    792                 # Prepare split will record examples associated to the split
--> 793                 self._prepare_split(split_generator, **prepare_split_kwargs)
    794             except OSError as e:
    795                 raise OSError(

/sfs/MNBVC/venv/lib64/python3.6/site-packages/datasets/builder.py in _prepare_split(self, split_generator, check_duplicate_keys)
   1219                     writer.write(example, key)
   1220             finally:
-> 1221                 num_examples, num_bytes = writer.finalize()
   1222 
   1223         split_generator.split_info.num_examples = num_examples

/sfs/MNBVC/venv/lib64/python3.6/site-packages/datasets/arrow_writer.py in finalize(self, close_stream)
    536             # Re-intializing to empty list for next batch
    537             self.hkey_record = []
--> 538         self.write_examples_on_file()
    539         if self.pa_writer is None:
    540             if self.schema:

/sfs/MNBVC/venv/lib64/python3.6/site-packages/datasets/arrow_writer.py in write_examples_on_file(self)
    407             # Since current_examples contains (example, key) tuples
    408             batch_examples[col] = [row[0][col] for row in self.current_examples]
--> 409         self.write_batch(batch_examples=batch_examples)
    410         self.current_examples = []
    411 

/sfs/MNBVC/venv/lib64/python3.6/site-packages/datasets/arrow_writer.py in write_batch(self, batch_examples, writer_batch_size)
    506             col_try_type = try_features[col] if try_features is not None and col in try_features else None
    507             typed_sequence = OptimizedTypedSequence(batch_examples[col], type=col_type, try_type=col_try_type, col=col)
--> 508             arrays.append(pa.array(typed_sequence))
    509             inferred_features[col] = typed_sequence.get_inferred_type()
    510         schema = inferred_features.arrow_schema if self.pa_writer is None else self.schema

/sfs/MNBVC/venv/lib64/python3.6/site-packages/pyarrow/array.pxi in pyarrow.lib.array()

/sfs/MNBVC/venv/lib64/python3.6/site-packages/pyarrow/array.pxi in pyarrow.lib._handle_arrow_array_protocol()

/sfs/MNBVC/venv/lib64/python3.6/site-packages/datasets/arrow_writer.py in __arrow_array__(self, type)
    180             else:
    181                 trying_cast_to_python_objects = True
--> 182                 out = pa.array(cast_to_python_objects(data, only_1d_for_numpy=True))
    183             # use smaller integer precisions if possible
    184             if self.trying_int_optimization:

/sfs/MNBVC/venv/lib64/python3.6/site-packages/pyarrow/array.pxi in pyarrow.lib.array()

/sfs/MNBVC/venv/lib64/python3.6/site-packages/pyarrow/array.pxi in pyarrow.lib._sequence_to_array()

/sfs/MNBVC/venv/lib64/python3.6/site-packages/pyarrow/error.pxi in pyarrow.lib.pyarrow_internal_check_status()

OverflowError: Python int too large to convert to C long
```

However, that dataset can be loaded in a streaming manner:

```python
from datasets import load_dataset
dataset = load_dataset("liwu/MNBVC", 'news_peoples_daily', split='train', streaming=True)

for i in dataset:
    pass  # it work well
```

Another issue is reported in our dataset hub:
https://huggingface.co/datasets/liwu/MNBVC/discussions/2


### Steps to reproduce the bug

from datasets import load_dataset
dataset = load_dataset("liwu/MNBVC", 'news_peoples_daily', split='train')

### Expected behavior

the dataset can be safely loaded

### Environment info

- `datasets` version: 2.4.0
- Platform: Linux-3.10.0-1160.an7.x86_64-x86_64-with-centos-7.9
- Python version: 3.6.8
- PyArrow version: 6.0.1
- Pandas version: 1.1.5