Cannot load data with different schemas from different parquet files
Opened by li-plus (closed)

### Describe the bug

Cannot load samples with optional fields from different files. The schema cannot be correctly derived.

### Steps to reproduce the bug

When I place two samples with an optional field `some_extra_field` within a single parquet file, it can be loaded via `load_dataset`.

```python
import pandas as pd
from datasets import load_dataset

data = [
    {'conversations': {'role': 'user', 'content': 'hello'}},
    {'conversations': {'role': 'user', 'content': 'hi', 'some_extra_field': 'some_value'}}
]
df = pd.DataFrame(data)
df.to_parquet('data.parquet')

dataset = load_dataset('parquet', data_files='data.parquet', split='train')
print(dataset.features)
```

The schema can be derived. `some_extra_field` is set to None for the first row where it is absent.
```
{'conversations': {'content': Value(dtype='string', id=None), 'role': Value(dtype='string', id=None), 'some_extra_field': Value(dtype='string', id=None)}}
```

However, when I separate the samples into different files, it cannot be loaded.
```python
import pandas as pd
from datasets import load_dataset

data1 = [{'conversations': {'role': 'user', 'content': 'hello'}}]
pd.DataFrame(data1).to_parquet('data1.parquet')
data2 = [{'conversations': {'role': 'user', 'content': 'hi', 'some_extra_field': 'some_value'}}]
pd.DataFrame(data2).to_parquet('data2.parquet')

dataset = load_dataset('parquet', data_files=['data1.parquet', 'data2.parquet'], split='train')
print(dataset.features)
```

Traceback:
```
Traceback (most recent call last):
  File "/home/tiger/.local/lib/python3.9/site-packages/datasets/builder.py", line 1854, in _prepare_split_single
    for _, table in generator:
  File "/home/tiger/.local/lib/python3.9/site-packages/datasets/packaged_modules/parquet/parquet.py", line 106, in _generate_tables
    yield f"{file_idx}_{batch_idx}", self._cast_table(pa_table)
  File "/home/tiger/.local/lib/python3.9/site-packages/datasets/packaged_modules/parquet/parquet.py", line 73, in _cast_table
    pa_table = table_cast(pa_table, self.info.features.arrow_schema)
  File "/home/tiger/.local/lib/python3.9/site-packages/datasets/table.py", line 2292, in table_cast
    return cast_table_to_schema(table, schema)
  File "/home/tiger/.local/lib/python3.9/site-packages/datasets/table.py", line 2245, in cast_table_to_schema
    arrays = [
  File "/home/tiger/.local/lib/python3.9/site-packages/datasets/table.py", line 2246, in <listcomp>
    cast_array_to_feature(
  File "/home/tiger/.local/lib/python3.9/site-packages/datasets/table.py", line 1795, in wrapper
    return pa.chunked_array([func(chunk, *args, **kwargs) for chunk in array.chunks])
  File "/home/tiger/.local/lib/python3.9/site-packages/datasets/table.py", line 1795, in <listcomp>
    return pa.chunked_array([func(chunk, *args, **kwargs) for chunk in array.chunks])
  File "/home/tiger/.local/lib/python3.9/site-packages/datasets/table.py", line 2108, in cast_array_to_feature
    raise TypeError(f"Couldn't cast array of type\n{_short_str(array.type)}\nto\n{_short_str(feature)}")
TypeError: Couldn't cast array of type
struct<content: string, role: string, some_extra_field: string>
to
{'content': Value(dtype='string', id=None), 'role': Value(dtype='string', id=None)}
```


### Expected behavior

Correctly load data with optional fields from different parquet files.

### Environment info

- `datasets` version: 3.3.2
- Platform: Linux-5.10.135.bsk.4-amd64-x86_64-with-glibc2.31
- Python version: 3.9.2
- `huggingface_hub` version: 0.28.1
- PyArrow version: 17.0.0
- Pandas version: 2.2.2
- `fsspec` version: 2024.3.1

Comments:
Hi ! `load_dataset` expects all the data_files to have the same schema.

Maybe you can try enforcing certain `features` using:

```python
features = Features({"conversations": {'content': Value('string'), 'role': Value('string',)}})
ds = load_dataset(..., features=features)
```

Thanks! It works if I explicitly specify all nested fields of the data.