`Sequence(Features(...))` causes PyArrow cast error in `load_dataset` despite correct schema.
Opened by AHS-uni (closed)

### Description

When loading a dataset with a field declared as a list of structs using `Sequence(Features(...))`, `load_dataset` incorrectly infers the field as a plain `struct<...>` instead of a `list<struct<...>>`. This leads to the following error:

```
ArrowNotImplementedError: Unsupported cast from list<item: struct<id: string, data: string>> to struct using function cast_struct
```

This occurs even when the `features` schema is explicitly provided and the dataset format supports nested structures natively (e.g., JSON, JSONL).

---

###  Minimal Reproduction

[Colab Link.](https://colab.research.google.com/drive/1FZPQy6TP3jVd4B3mYKyfQaWNuOAvljUq?usp=sharing)

#### Dataset

```python
data = [
    {
        "list": [
            {"id": "example1", "data": "text"},
        ]
    },
]
```

#### Schema

```python
from datasets import Features, Sequence, Value

item = Features({
    "id":   Value("string"),
    "data": Value("string"),
})

features = Features({
    "list": Sequence(item),
})
```

---

###  Tested File Formats

The same schema was tested across different formats:

| Format    | Method                      | Result              |
| --------- | --------------------------- | ------------------- |
| JSONL     | `load_dataset("json", ...)` |  Arrow cast error  |
| JSON      | `load_dataset("json", ...)` |  Arrow cast error  |
| In-memory | `Dataset.from_list(...)`    | Works as expected |

The issue seems not to be in the schema or the data, but in how `load_dataset()` handles the `Sequence(Features(...))` pattern when parsing from files (specifically JSON and JSONL).

---

### Expected Behavior

If `features` is explicitly defined as:

```python
Features({"list": Sequence(Features({...}))})
```

Then the data should load correctly across all backends — including from JSON and JSONL — without any Arrow casting errors. This works correctly when loading from memory via `Dataset.from_list`.

---

###  Environment

* `datasets`: 3.6.0
* `pyarrow`: 20.0.0
* Python: 3.12.10
* OS: Ubuntu 24.04.2 LTS
* Notebook: \[Colab test notebook available]

---


Comments:
Hi @lhoestq 

Could you help confirm whether this qualifies as a bug?

It looks like the issue stems from how `Sequence(Features(...))` is interpreted as a plain struct during schema inference, which leads to a mismatch when casting with PyArrow (especially with nested structs inside lists). From the description, this seems like an inconsistency with expected behavior.

If confirmed, I’d be happy to take a shot at investigating and potentially submitting a fix.

Also looping in @AHS-uni  — could you kindly share a minimal JSONL example that reproduces this?

Thanks!

Hello @Flink-ddd 

I updated the minimal example and included both JSON and JSONL minimal examples in the Colab notebook. 

Here is the minimal  JSON file for convenience (can't upload JSONL files).

[mini.json](https://github.com/user-attachments/files/20535145/mini.json)

I've also found a number of issues which describe a similar problem:

[7569](https://github.com/huggingface/datasets/issues/7569) (Open)
[7137](https://github.com/huggingface/datasets/issues/7137) (Open)
[7501](https://github.com/huggingface/datasets/issues/7501) (Closed)
[2434](https://github.com/huggingface/datasets/issues/2434) (Closed)

The closed issues don't really address the problem (IMO). [7501](https://github.com/huggingface/datasets/issues/7501) provides a workaround (using a Python list instead of `Sequence`), but it seem precarious. 

Hi ! `Sequence({...})` corresponds to a struct of lists ([docs](https://huggingface.co/docs/datasets/v3.6.0/en/package_reference/main_classes#datasets.Features)). This come from Tensorflow Datasets.

If you want to use a list of structs, you should use `[{...}]`, e.g.

```python
item = {
    "id":   Value("string"),
    "data": Value("string"),
}

features = Features({
    "list": [item],
})
```

@lhoestq Thanks for your explanation, which helps me understand the logic behind. But I'm confused how to define that in `README.md`?

My jsonl data is: 
```
{"answers": [{"text": "text1", "label": "label1"}, {"text": "text2", "label": "label2"},]}
{"answers": [{"text": "text1", "label": "label1"}, {"text": "text2", "label": "label2"},]}
...
```

My README.md look like
```
dataset_info:
- config_name: default
  features:
  - name: answers
    sequence:
    - name: text
      dtype: string
    - name: label
      dtype: string
```
I understand `sequence` here is not correct, but what's the correct format? I tried following (`sequence -> dtype`)and seems not the case:
```
dataset_info:
- config_name: default
  features:
  - name: answers
    dtype:
    - name: text
      sequence: string
    - name: label
      sequence: string
```

The `List` type which doesn't have the weird dict behavior of `Sequence` has been added for `datasets` 4.0 (to be released next week). Feel free to install `datasets` from source to try it out :)
EDIT: it's out !

You can fix the issue using `List` instead of `Sequence`, e.g. in the case of the original post:

```python
# Feature spec with List of structs
item = {
    "id":   Value("string"),
    "data": Value("string"),
}

features = Features({
    "list": List(item),
})
```

for which the README.md is

```yaml
dataset_info:
- config_name: default
  features:
  - name: list
    list:
    - name: id
      dtype: string
    - name: data
      dtype: string
```

@lhoestq Thanks! I didn't realize there is a `list` keyword I could use. I thought I had to use `dtype` or something. Hope there could be better documentation on the `README.md` formats. I've closed my issue #7137