GitHub issue: Better cast error when generating dataset
Author: lhoestq (MEMBER)
State: closed | Created: 2023-12-18 13:57:24

I want to improve the error message for datasets like https://huggingface.co/datasets/m-a-p/COIG-CQIA

Cc @albertvillanova @severo is this new error ok ? Or should I use a dedicated error class ?

New:

```python
Traceback (most recent call last):
  File "/Users/quentinlhoest/hf/datasets/src/datasets/builder.py", line 1920, in _prepare_split_single
    writer.write_table(table)
  File "/Users/quentinlhoest/hf/datasets/src/datasets/arrow_writer.py", line 574, in write_table
    pa_table = table_cast(pa_table, self._schema)
  File "/Users/quentinlhoest/hf/datasets/src/datasets/table.py", line 2322, in table_cast
    return cast_table_to_schema(table, schema)
  File "/Users/quentinlhoest/hf/datasets/src/datasets/table.py", line 2276, in cast_table_to_schema
    raise CastError(
datasets.table.CastError: Couldn't cast
instruction: string
other: string
index: string
domain: list<item: string>
  child 0, item: string
output: string
task_type: struct<major: list<item: string>, minor: list<item: string>>
  child 0, major: list<item: string>
      child 0, item: string
  child 1, minor: list<item: string>
      child 0, item: string
task_name_in_eng: string
input: string
to
{'answer_from': Value(dtype='string', id=None), 'instruction': Value(dtype='string', id=None), 'human_verified': Value(dtype='bool', id=None), 'domain': Sequence(feature=Value(dtype='string', id=None), length=-1, id=None), 'output': Value(dtype='string', id=None), 'task_type': {'major': Sequence(feature=Value(dtype='string', id=None), length=-1, id=None), 'minor': Sequence(feature=Value(dtype='string', id=None), length=-1, id=None)}, 'copyright': Value(dtype='string', id=None), 'input': Value(dtype='string', id=None)}
because column names don't match

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/quentinlhoest/hf/datasets/playground/ttest.py", line 74, in <module>
    load_dataset("m-a-p/COIG-CQIA")
  File "/Users/quentinlhoest/hf/datasets/src/datasets/load.py", line 2529, in load_dataset
    builder_instance.download_and_prepare(
  File "/Users/quentinlhoest/hf/datasets/src/datasets/builder.py", line 936, in download_and_prepare
    self._download_and_prepare(
  File "/Users/quentinlhoest/hf/datasets/src/datasets/builder.py", line 1031, in _download_and_prepare
    self._prepare_split(split_generator, **prepare_split_kwargs)
  File "/Users/quentinlhoest/hf/datasets/src/datasets/builder.py", line 1791, in _prepare_split
    for job_id, done, content in self._prepare_split_single(
  File "/Users/quentinlhoest/hf/datasets/src/datasets/builder.py", line 1922, in _prepare_split_single
    raise DatasetGenerationCastError.from_cast_error(
datasets.exceptions.DatasetGenerationCastError: An error occurred while generating the dataset

All the data files must have the same columns, but at some point there are 3 new columns (other, index, task_name_in_eng) and 3 missing columns (answer_from, copyright, human_verified).

This happened while the json dataset builder was generating data using

hf://datasets/m-a-p/COIG-CQIA/coig_pc/coig_pc_core_sample.json (at revision b7b7ecf290f6515036c7c04bd8537228ac2eb474)

Please either edit the data files to have matching columns, or separate them into different configurations (see docs at https://hf.co/docs/hub/datasets-manual-configuration#multiple-configurations)
```

Previously:

```python
Traceback (most recent call last):
  File "/Users/quentinlhoest/hf/datasets/src/datasets/builder.py", line 1931, in _prepare_split_single
    writer.write_table(table)
  File "/Users/quentinlhoest/hf/datasets/src/datasets/arrow_writer.py", line 574, in write_table
    pa_table = table_cast(pa_table, self._schema)
  File "/Users/quentinlhoest/hf/datasets/src/datasets/table.py", line 2295, in table_cast
    return cast_table_to_schema(table, schema)
  File "/Users/quentinlhoest/hf/datasets/src/datasets/table.py", line 2253, in cast_table_to_schema
    raise ValueError(f"Couldn't cast\n{table.schema}\nto\n{features}\nbecause column names don't match")
ValueError: Couldn't cast
task_type: struct<major: list<item: string>, minor: list<item: string>>
  child 0, major: list<item: string>
      child 0, item: string
  child 1, minor: list<item: string>
      child 0, item: string
other: string
instruction: string
task_name_in_eng: string
domain: list<item: string>
  child 0, item: string
index: string
output: string
input: string
to
{'human_verified': Value(dtype='bool', id=None), 'task_type': {'major': Sequence(feature=Value(dtype='string', id=None), length=-1, id=None), 'minor': Sequence(feature=Value(dtype='string', id=None), length=-1, id=None)}, 'answer_from': Value(dtype='string', id=None), 'copyright': Value(dtype='string', id=None), 'instruction': Value(dtype='string', id=None), 'domain': Sequence(feature=Value(dtype='string', id=None), length=-1, id=None), 'output': Value(dtype='string', id=None), 'input': Value(dtype='string', id=None)}
because column names don't match

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/Users/quentinlhoest/hf/datasets/playground/ttest.py", line 74, in <module>
    load_dataset("m-a-p/COIG-CQIA")
  File "/Users/quentinlhoest/hf/datasets/src/datasets/load.py", line 2529, in load_dataset
    builder_instance.download_and_prepare(
  File "/Users/quentinlhoest/hf/datasets/src/datasets/builder.py", line 949, in download_and_prepare
    self._download_and_prepare(
  File "/Users/quentinlhoest/hf/datasets/src/datasets/builder.py", line 1044, in _download_and_prepare
    self._prepare_split(split_generator, **prepare_split_kwargs)
  File "/Users/quentinlhoest/hf/datasets/src/datasets/builder.py", line 1804, in _prepare_split
    for job_id, done, content in self._prepare_split_single(
  File "/Users/quentinlhoest/hf/datasets/src/datasets/builder.py", line 1949, in _prepare_split_single
    raise DatasetGenerationError("An error occurred while generating the dataset") from e
datasets.builder.DatasetGenerationError: An error occurred while generating the dataset
```