Title: ArrowInvalid: JSON parse error: Column() changed from object to array in row 0
State: open

### Describe the bug

Encountering an error while loading the ```liuhaotian/LLaVA-Instruct-150K dataset```.

### Steps to reproduce the bug

```
from datasets import load_dataset
fw =load_dataset("liuhaotian/LLaVA-Instruct-150K")
```
Error:
```
ArrowInvalid                              Traceback (most recent call last)
[/usr/local/lib/python3.10/dist-packages/datasets/packaged_modules/json/json.py](https://localhost:8080/#) in _generate_tables(self, files)
    136                                 try:
--> 137                                     pa_table = paj.read_json(
    138                                         io.BytesIO(batch), read_options=paj.ReadOptions(block_size=block_size)

20 frames
ArrowInvalid: JSON parse error: Column() changed from object to array in row 0

During handling of the above exception, another exception occurred:

ArrowTypeError                            Traceback (most recent call last)
ArrowTypeError: ("Expected bytes, got a 'int' object", 'Conversion failed for column id with type object')

The above exception was the direct cause of the following exception:

DatasetGenerationError                    Traceback (most recent call last)
[/usr/local/lib/python3.10/dist-packages/datasets/builder.py](https://localhost:8080/#) in _prepare_split_single(self, gen_kwargs, fpath, file_format, max_shard_size, job_id)
   1895             if isinstance(e, DatasetGenerationError):
   1896                 raise
-> 1897             raise DatasetGenerationError("An error occurred while generating the dataset") from e
   1898 
   1899         yield job_id, True, (total_num_examples, total_num_bytes, writer._features, num_shards, shard_lengths)

DatasetGenerationError: An error occurred while generating the dataset
```
### Expected behavior

I have tried loading the dataset both on my own server and on Colab, and encountered errors in both instances.

### Environment info

```
- `datasets` version: 3.2.0
- Platform: Linux-6.1.85+-x86_64-with-glibc2.35
- Python version: 3.10.12
- `huggingface_hub` version: 0.26.3
- PyArrow version: 17.0.0
- Pandas version: 2.2.2
- `fsspec` version: 2024.9.0
```


Comments: Hi ! `datasets` uses Arrow under the hood which expects each column and array to have fixed types that don't change across rows of a dataset, which is why we get this error. This dataset in particular doesn't have a format compatible with Arrow unfortunately. Don't hesitate to open a discussion or PR on HF to fix the dataset@lhoestq Is it correct to assume that most multimodal datasets with variable number of images across conversations are not compatible with Arrow ? 

I’m running into a problem while trying to format multimodal datasets (image + text) using the Hugging Face datasets library. Specifically, I’m working with a structure where conversations include both images and text messages. When I convert my dataset from a Python dict to a Hugging Face Dataset, I’m seeing unexpected None values being inserted for some fields that aren’t relevant for a given message (e.g., "text": None in image messages). Here’s what seems to be happening:

🔍 What’s going wrong

Hugging Face datasets (backed by Apache Arrow) tries to flatten the schema across all samples. That means it enforces a fixed set of fields across the dataset – even if some of them are None for a given entry. Since my dataset contains heterogeneous conversation messages (some with images, others with text), Arrow is injecting None for whichever attributes don’t exist in each message type to preserve a unified schema.

This results in a lot of meaningless or misleading Nones across the dataset, and breaks logic further down the pipeline – in particular, it causes the qwen_vl_utils functions (like process_vision_info) to crash or misbehave.


🤔 Workarounds considered

I could stick with raw JSON, which preserves the heterogeneous structure properly. But that means giving up all the nice features of datasets and Arrow (e.g. streaming, map/filter, etc.), which feels like a shame.

❓My question

Do you know of a clean way to define a flexible schema with Hugging Face Datasets – maybe using nested structures or dynamic fields – so that each conversation message doesn’t get forced into a flat structure with irrelevant keys? Or should I just stick with JSON for this kind of multimodal case?

Let me know if you’ve run into this before or have any tips!
Datasets without fixed types are harder to use in many data frameworks unfortunately. You will have to handle the case with None values if you want to use Arrow/`datasets` (or any other framework using Arrow like spark, ray, dask). **IMO the short term solution is to fix qwen_vl_utils / process_vision_info.**

Alternatively we can explore adding the Arrow `Json` type to `datasets`, but doesn't allow having image types in the Json object so I don't think this is the right solution.I totally share the analysis, I am happy to try to help with this! I am currently using Unsloth so it is the `UnslothVisionDataCollator`  which has almost the same logic as `qwen_vl_utils.process_vision_info`. Is there a Huggingface class I could patch too ?