Issues with large Datasets
State: open

### Describe the bug

If the coco annotation file is too large the dataset will not be able to load it, not entirely sure were the issue is but I am guessing it is due to the code trying to load it all as one line into a dataframe. This was for object detections.

My current work around is the following code but would be nice to be able to do it without worrying about it also probably there is a better way of doing it:
`
            dataset_dict = json.load(open("./local_data/annotations/train.json"))
            df = pd.DataFrame(columns=['images', 'annotations', 'categories'])
            df = df._append({'images': dataset_dict['images'], 'annotations': dataset_dict['annotations'], 'categories': dataset_dict['categories']}, ignore_index=True)
            train=Dataset.from_pandas(df)

            dataset_dict = json.load(open("./local_data/annotations/validation.json"))
            df = pd.DataFrame(columns=['images', 'annotations', 'categories'])
            df = df._append({'images': dataset_dict['images'], 'annotations': dataset_dict['annotations'],
                             'categories': dataset_dict['categories']}, ignore_index=True)
            val = Dataset.from_pandas(df)
            dataset_dict = json.load(open("./local_data/annotations/test.json"))
            df = pd.DataFrame(columns=['images', 'annotations', 'categories'])
            df = df._append({'images': dataset_dict['images'], 'annotations': dataset_dict['annotations'],
                             'categories': dataset_dict['categories']}, ignore_index=True)
            test = Dataset.from_pandas(df)
            dataset = DatasetDict({'train': train, 'validation': val, 'test': test})
`

### Steps to reproduce the bug

1) step up directory in and have the json files in coco format

-local_data
|-images
|---1.jpg
|---2.jpg
|---....
|---n.jpg
|-annotations
|---test.json
|---train.json
|---validation.json

2) try to load local_data into a dataset if the file is larger than about 300kb it will cause an error.

### Expected behavior

That it loads the jsons preferably in the same format as it has done with a smaller size.

### Environment info

- `datasets` version: 3.3.3.dev0
- Platform: Linux-6.11.0-17-generic-x86_64-with-glibc2.39
- Python version: 3.12.3
- `huggingface_hub` version: 0.29.0
- PyArrow version: 19.0.1
- Pandas version: 2.2.3
- `fsspec` version: 2024.12.0


Comments:
what's the error message ?

This was the final error message that it was giving pyarrow.lib.ArrowInvalid: JSON parse error: Column() changed from object to string in row 0

Here is the list of errors:

Traceback (most recent call last):
  File ".venv/lib/python3.12/site-packages/datasets/packaged_modules/json/json.py", line 160, in _generate_tables
    df = pandas_read_json(f)
         ^^^^^^^^^^^^^^^^^^^
  File ".venv/lib/python3.12/site-packages/datasets/packaged_modules/json/json.py", line 38, in pandas_read_json
    return pd.read_json(path_or_buf, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File ".venv/lib/python3.12/site-packages/pandas/io/json/_json.py", line 815, in read_json
    return json_reader.read()
           ^^^^^^^^^^^^^^^^^^
  File ".venv/lib/python3.12/site-packages/pandas/io/json/_json.py", line 1025, in read
    obj = self._get_object_parser(self.data)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File ".venv/lib/python3.12/site-packages/pandas/io/json/_json.py", line 1051, in _get_object_parser
    obj = FrameParser(json, **kwargs).parse()
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File ".venv/lib/python3.12/site-packages/pandas/io/json/_json.py", line 1187, in parse
    self._parse()
  File ".venv/lib/python3.12/site-packages/pandas/io/json/_json.py", line 1402, in _parse
    self.obj = DataFrame(
               ^^^^^^^^^^
  File ".venv/lib/python3.12/site-packages/pandas/core/frame.py", line 778, in __init__
    mgr = dict_to_mgr(data, index, columns, dtype=dtype, copy=copy, typ=manager)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File ".venv/lib/python3.12/site-packages/pandas/core/internals/construction.py", line 503, in dict_to_mgr
    return arrays_to_mgr(arrays, columns, index, dtype=dtype, typ=typ, consolidate=copy)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File ".venv/lib/python3.12/site-packages/pandas/core/internals/construction.py", line 114, in arrays_to_mgr
    index = _extract_index(arrays)
            ^^^^^^^^^^^^^^^^^^^^^^
  File ".venv/lib/python3.12/site-packages/pandas/core/internals/construction.py", line 677, in _extract_index
    raise ValueError("All arrays must be of the same length")
ValueError: All arrays must be of the same length

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File ".venv/lib/python3.12/site-packages/datasets/builder.py", line 1854, in _prepare_split_single
    for _, table in generator:
  File ".venv/lib/python3.12/site-packages/datasets/packaged_modules/json/json.py", line 163, in _generate_tables
    raise e
  File ".venv/lib/python3.12/site-packages/datasets/packaged_modules/json/json.py", line 137, in _generate_tables
    pa_table = paj.read_json(
               ^^^^^^^^^^^^^^
  File "pyarrow/_json.pyx", line 308, in pyarrow._json.read_json
  File "pyarrow/error.pxi", line 155, in pyarrow.lib.pyarrow_internal_check_status
  File "pyarrow/error.pxi", line 92, in pyarrow.lib.check_status
pyarrow.lib.ArrowInvalid: JSON parse error: Column() changed from object to number in row 0

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "run_object_detection.py", line 582, in <module>
    main()
  File "run_object_detection.py", line 407, in main
    dataset = load_dataset(
              ^^^^^^^^^^^^^
  File "venv/lib/python3.12/site-packages/datasets/load.py", line 2151, in load_dataset
    builder_instance.download_and_prepare(
  File ".venv/lib/python3.12/site-packages/datasets/builder.py", line 924, in download_and_prepare
    self._download_and_prepare(
  File ".venv/lib/python3.12/site-packages/datasets/builder.py", line 1000, in _download_and_prepare
    self._prepare_split(split_generator, **prepare_split_kwargs)
  File ".venv/lib/python3.12/site-packages/datasets/builder.py", line 1741, in _prepare_split
    for job_id, done, content in self._prepare_split_single(
  File ".venv/lib/python3.12/site-packages/datasets/builder.py", line 1897, in _prepare_split_single
    raise DatasetGenerationError("An error occurred while generating the dataset") from e
datasets.exceptions.DatasetGenerationError: An error occurred while generating the dataset

`datasets` is based on Arrow which expects all the lists inside the data to be of fixed type. Arrow can't load lists that contain a mix of integers and strings for example. In your case it looks like one of the lists contains numbers and JSON objects.

I'd suggest you to reformat the data to end up with list of fixed types, otherwise you won't be able to load the data in `datasets`