Large WebDataset: pyarrow.lib.ArrowCapacityError on load() even with streaming
Opened by cleong110 (open)

### Describe the bug

I am creating a large WebDataset-format dataset for sign language processing research, and a number of the videos are over 2GB. The instant I hit one of the shards with one of those videos, I get a ArrowCapacityError, even with streaming. 



I made a config for the dataset that specifically includes just one problem shard, and the error triggers the instant you even run load_dataset(), even with streaming=True
```
ds = load_dataset("bible-nlp/sign-bibles", "ase_chronological_bible_translation_in_american_sign_language_119_introductions_and_passages_debugging_problem_shard", streaming=True, split="train")
```

This gives: 
```
File "/opt/home/cleong/projects/semantic_and_visual_similarity/sign-bibles-dataset/sign_bibles_dataset/tasks/test_iteration.py", line 13, in iterate_keys
    ds = load_dataset("bible-nlp/sign-bibles", language_subset, streaming=True, split="train")
  File "/opt/home/cleong/envs/sign-bibles-dataset/lib/python3.13/site-packages/datasets/load.py", line 1409, in load_dataset
    return builder_instance.as_streaming_dataset(split=split)
           ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^
  File "/opt/home/cleong/envs/sign-bibles-dataset/lib/python3.13/site-packages/datasets/builder.py", line 1225, in as_streaming_dataset
    splits_generators = {sg.name: sg for sg in self._split_generators(dl_manager)}
                                               ~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^
  File "/opt/home/cleong/envs/sign-bibles-dataset/lib/python3.13/site-packages/datasets/packaged_modules/webdataset/webdataset.py", line 88, in _split_generators
    pa.Table.from_pylist(cast_to_python_objects([example], only_1d_for_numpy=True))
    ~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "pyarrow/table.pxi", line 2046, in pyarrow.lib._Tabular.from_pylist
  File "pyarrow/table.pxi", line 6431, in pyarrow.lib._from_pylist
  File "pyarrow/table.pxi", line 4893, in pyarrow.lib.Table.from_arrays
  File "pyarrow/table.pxi", line 1607, in pyarrow.lib._sanitize_arrays
  File "pyarrow/table.pxi", line 1588, in pyarrow.lib._schema_from_arrays
  File "pyarrow/array.pxi", line 375, in pyarrow.lib.array
  File "pyarrow/array.pxi", line 45, in pyarrow.lib._sequence_to_array
  File "pyarrow/error.pxi", line 155, in pyarrow.lib.pyarrow_internal_check_status
  File "pyarrow/error.pxi", line 92, in pyarrow.lib.check_status
pyarrow.lib.ArrowCapacityError: array cannot contain more than 2147483646 bytes, have 3980158992
```

### Steps to reproduce the bug

```python
#!/usr/bin/env python

import argparse
from datasets import get_dataset_config_names, load_dataset
from tqdm import tqdm
from pyarrow.lib import ArrowCapacityError, ArrowInvalid


def iterate_keys(language_subset: str) -> None:
    """Iterate over all samples in the Sign Bibles dataset and print idx and sample key."""

    # https://huggingface.co/docs/datasets/v4.0.0/en/package_reference/loading_methods#datasets.load_dataset
    ds = load_dataset("bible-nlp/sign-bibles", language_subset, streaming=True, split="train")

    print(f"\n==> Loaded dataset config '{language_subset}'")

    idx = 0
    estimated_shard_index = 0
    samples_per_shard = 5
    with tqdm(desc=f"{language_subset} samples") as pbar:
        iterator = iter(ds)
        while True:
            try:
                if idx % samples_per_shard == 0 and idx > 0:  # 5 samples per shard: 0, 1, 2, 3, 4
                    print(f"Estimated Shard idx (starting at 0, {samples_per_shard}/shard): {estimated_shard_index}")
                    estimated_shard_index += 1
                sample = next(iterator)
                sample_key = sample.get("__key__", "missing-key")
                print(f"[{language_subset}] idx={idx}, key={sample_key}")
                idx += 1

                pbar.update(1)

            except StopIteration:
                print(f"Finished iterating through {idx} samples of {language_subset}")
                break

            except (ArrowCapacityError, ArrowInvalid) as e:
                print(f"PyArrow error on idx={idx}, config={language_subset}: {e}")
                idx += 1
                pbar.update(1)
                continue

            except KeyError as e:
                print(f"Missing key error on idx={idx}, config={language_subset}: {e}")
                idx += 1
                pbar.update(1)
                continue


def main():
    configs = get_dataset_config_names("bible-nlp/sign-bibles")
    print(f"Available configs: {configs}")
    configs = [
        "ase_chronological_bible_translation_in_american_sign_language_119_introductions_and_passages_debugging_problem_shard"
    ]

    for language_subset in configs:
        print(f"TESTING CONFIG {language_subset}")
        iterate_keys(language_subset)
        # try:

        # except (ArrowCapacityError, ArrowInvalid) as e:
        #     print(f"PyArrow error at config level for {language_subset}: {e}")
        #     continue
        # except RuntimeError as e:
        #     print(f"RuntimeError at config level for {language_subset}: {e}")
        #     continue


if __name__ == "__main__":
    parser = argparse.ArgumentParser(description="Iterate through Sign Bibles dataset and print sample keys.")
    args = parser.parse_args()

    main()
```



### Expected behavior

I expect, when I load with streaming=True, that there should not be any data loaded or anything like that.
https://huggingface.co/docs/datasets/main/en/package_reference/loading_methods#datasets.load_dataset says that with streaming=true, 

I did expect to have some trouble with large files, but that the streaming mode would not actually try to load them unless requested, e.g. with sample["mp4"]

>In the streaming case:
>    Don’t download or cache anything. Instead, the dataset is lazily loaded and will be streamed on-the-fly when iterating on it.

### Environment info

Local setup: Conda environment on Ubuntu, pip list includes the following
datasets                  4.0.0
pyarrow                   20.0.0

Verified on Colab: https://colab.research.google.com/drive/1HdN8stlROWrLSYXUoNeV0vQ9pClhIVM8?usp=sharing, though there it crashes by using up all available RAM


Comments:
It seems the error occurs right here, as it tries to infer the Features: https://github.com/huggingface/datasets/blob/main/src/datasets/packaged_modules/webdataset/webdataset.py#L78-L90

It seems to me that if we have something that is so large that it cannot fit in pa.table, the fallback method should be to just set it as "binary" type, perhaps?

I also tried creating a dataset_info.json but the webdataset builder didn't seem to look for it and load it

Workaround on my end, removed all videos larger than 2GB for now. The dataset no longer crashes.

Potential patch to webdataset.py could be like so: 
```python
LARGE_THRESHOLD = 2 * 1024 * 1024 * 1024  # 2 GB
large_fields = set()

# Replace large binary fields with None for schema inference
processed_examples = []
for example in first_examples:
    new_example = {}
    for k, v in example.items():
        if isinstance(v, bytes) and len(v) > LARGE_THRESHOLD:
            large_fields.add(k)
            new_example[k] = None  # Replace with None to avoid Arrow errors
        else:
            new_example[k] = v
    processed_examples.append(new_example)

# Proceed to infer schema
pa_tables = [
    pa.Table.from_pylist(cast_to_python_objects([example], only_1d_for_numpy=True))
    for example in processed_examples
]
inferred_arrow_schema = pa.concat_tables(pa_tables, promote_options="default").schema

# Patch features to reflect large_binary
features = datasets.Features.from_arrow_schema(inferred_arrow_schema)
for field in large_fields:
    features[field] = datasets.Value("large_binary")

```