PyArrow 'Memory mapping file failed: Cannot allocate memory' bug
Opened by lasuomela (open)

### Describe the bug

Hello,

I've been struggling with a problem using Huggingface datasets caused by PyArrow memory allocation. I finally managed to solve it, and thought to document it since similar issues have been raised here before (https://github.com/huggingface/datasets/issues/5710, https://github.com/huggingface/datasets/issues/6176).

In my case, I was trying to load ~70k dataset files from disk using `datasets.load_from_disk(data_path)` (meaning 70k repeated calls to load_from_disk). This triggered an (uninformative) exception around 64k loaded files:
```
  File "pyarrow/io.pxi", line 1053, in pyarrow.lib.memory_map
  File "pyarrow/io.pxi", line 1000, in pyarrow.lib.MemoryMappedFile._open
  File "pyarrow/error.pxi", line 154, in pyarrow.lib.pyarrow_internal_check_status
  File "pyarrow/error.pxi", line 91, in pyarrow.lib.check_status
OSError: Memory mapping file failed: Cannot allocate memory
```
Despite system RAM usage being very low. After a lot of digging around, I discovered that my Ubuntu machine had a limit on the maximum number of memory mapped files in `/proc/sys/vm/max_map_count` set to 65530, which was causing my data loader to crash. Increasing the limit in the file (`echo <new_mmap_size> | sudo tee /proc/sys/vm/max_map_count`) made the issue go away.

While this isn't a bug as such in either Datasets or PyArrow, this behavior can be very confusing to users. Maybe this should be mentioned in documentation? I suspect the other issues raised here about memory mapping OOM errors could actually be consequence of system configuration.

Br,
Lauri

### Steps to reproduce the bug

```
import numpy as np
import pyarrow as pa
import tqdm

# Write some data to disk
arr = pa.array(np.arange(100))
schema = pa.schema([
    pa.field('nums', arr.type)
])
with pa.OSFile('arraydata.arrow', 'wb') as sink:
    with pa.ipc.new_file(sink, schema=schema) as writer:
        batch = pa.record_batch([arr], schema=schema)
        writer.write(batch)

# Number of times to open the memory map
nums = 70000

# Read the data back
arrays = [pa.memory_map('arraydata.arrow', 'r') for _ in tqdm.tqdm(range(nums))]
```

### Expected behavior

No errors.

### Environment info

datasets: 2.18.0
pyarrow: 15.0.0

Comments:
Thanks for a very clean explanation. This happened to me too, and I don't have sudo access to update the value. I wonder if there might be another workaround.

One option is to just have more data in each file - /proc/sys/vm/max_map_count limits the maximum number of concurrently open files, but I don't know if the size of a single file is restricted in any way. E.g. 5000 files with 1GB each is 5TB of data. https://huggingface.co/docs/datasets/v2.18.0/en/package_reference/main_classes#datasets.concatenate_datasets can come in handy.

> One option is to just have more data in each file - /proc/sys/vm/max_map_count limits the maximum number of concurrently open files, but I don't know if the size of a single file is restricted in any way. E.g. 5000 files with 1GB each is 5TB of data. https://huggingface.co/docs/datasets/v2.18.0/en/package_reference/main_classes#datasets.concatenate_datasets can come in handy.

I still got the same error even if i only got 25k arrow files and i found this error will raise when each process of "load_from_disk" was allocated vram more than 128T

Another option is to use `load_from_disk` for a certain limit (let's say, 50,000 for safety), and the other calls for the function could include the argument `keep_in_memory=True` (e.g. disable mmap).

Example:
```python
import os
from datasets import load_from_disk

path = "datasets_dir"
files = os.listdir(path)
mmap_threshold = 50000  # how many `load_from_disk` calls should include mmap
for i, file in enumerate(files):
    full_path = os.path.join(path, file)
    
    disable_mmap = i >= mmap_threshold
    _ = load_from_disk(full_path, keep_in_memory=disable_mmap)
```