`from_list` fails while `from_generator` works for large datasets
Opened by abdulfatir (open)

### Describe the bug

I am constructing a large time series dataset and observed that first constructing a list of entries and then using `Dataset.from_list` led to a crash as the number of items became large. However, this is not a problem when using `Dataset.from_generator`.

### Steps to reproduce the bug

#### Snippet A (crashes)

```py
from tqdm.auto import tqdm
import numpy as np
import datasets


def data_generator():
    for i in tqdm(range(10_000_000)):
        length = np.random.randint(2048)
        series = np.random.rand(length)
        yield {"target": series, "item_id": str(i), "start": np.datetime64("2000", "ms")}

data_list = list(data_generator())
ds = datasets.Dataset.from_list(data_list)
```
The last line crashes with
```
ArrowInvalid: Value 2147483761 too large to fit in C integer type
```

#### Snippet B (works)

```py
from tqdm.auto import tqdm
import numpy as np
import datasets


def data_generator():
    for i in tqdm(range(10_000_000)):
        length = np.random.randint(2048)
        series = np.random.rand(length)
        yield {"target": series, "item_id": str(i), "start": np.datetime64("2000", "ms")}

ds = datasets.Dataset.from_generator(data_generator)
```

### Expected behavior

I expected both the approaches to work or to fail similarly. 

### Environment info

```
- `datasets` version: 3.6.0
- Platform: Linux-6.8.0-1029-aws-x86_64-with-glibc2.35
- Python version: 3.11.11
- `huggingface_hub` version: 0.32.2
- PyArrow version: 19.0.1
- Pandas version: 2.2.3
- `fsspec` version: 2025.3.0
```

Comments:
@lhoestq any thoughts on this? 

Thanks for the report! This behavior is expected due to how `from_list()` and `from_generator()` differ internally.

- `from_list()` builds the entire dataset in memory at once, which can easily exceed limits (especially with variable-length arrays or millions of rows). The Arrow error you're seeing (`Value too large to fit in C integer type`) is related to that memory overload.
- `from_generator()` avoids this issue by batching and streaming data incrementally, which is much more memory-efficient.

So for large datasets like time series or NLP data with large arrays, `from_generator()` (or `datasets.IterableDataset`) is the recommended approach.

Hope this helps clarify the behavior — let me know if you'd like me to point to prior issues/discussions where similar tradeoffs came up!


@ArjunJagdale Yes, it is related to using large dataset but not in the way that you have described. As I understand, the problem here is that `datasets` does not use `LargeList` with 64-bit offsets from PyArrow when using `from_list`. However, with `from_generator` this seems to work okay, likely due to batching. As such, this is more like a bug than an expected outcome. If this is indeed "expected", `datasets` should fail more gracefully in these cases with a recommendation to use `from_generator`. 

Thanks for the clarification — you're absolutely right, this seems tied to the use of 32-bit list offsets in from_list() under the hood. That distinction between List and LargeList in PyArrow is a crucial one, and definitely worth highlighting in the docs or error message. Happy to help if a check or fallback to LargeList makes sense here.