Support for data with instance-wise dictionary as features
State: open
Labels: description: New feature or request
name: enhancement

### Feature request

I notice that when loading data instances with feature type of python dictionary, the dictionary keys would be broadcast so that every instance has the same set of keys. Please see an example in the Motivation section.

It is possible to avoid this behavior, i.e., load dictionary features as it is and do not broadcast the keys among instances? Please note that these dictionaries would have to be processed dynamically at each training iteration into strings (and tokenized).

### Motivation

I am trying to load a dataset from a json file. Each instance of the dataset has a feature that is a dictionary but its keys depend on the instance. Every two instances may have different keys. For example, imagine a dataset that contains a set of math expressions from a bunch of mutually redundant expressions:
```
{
    "index": 0,
    "feature": {
        "2 * x + y >= 3": ["2 * x + y >= 3", "4 * x + 2 * y >= 6"],
        ...
    }
},
...
{
    "index": 9999,
    "feature": {
        "x >= 6": ["x >= 6", "x >= 0", "x >= -1"],
        ...
    }
},
...
```
When directly loading the dataset using `data = load_dataset("json", data_files=file_paths, split='train')`, each instance would have all the keys from other instances and None as values. That is, instance of index 0 becomes:
```
{
    "index": 0,
    "feature": {
        "2 * x + y >= 3": ["2 * x + y >= 3", "4 * x + 2 * y >= 6"],
        ...
        "x >= 6": None,  # keys from other instances
        ...
    }
},
```
This is not desirable. Moreover, issue would be raised if I attempt to combine two such datasets using `data = concatenate_datasets(multi_datasets)`, perhaps because their dictionary features contain different keys.

A solution I can think of is to store the dictionary features as a long string, and evaluate it later. Please kindly suggest any other solution using existing methods of datasets.

### Your contribution

N/A

Comments:
Hi ! We use the Arrow columnar format under the hood, which doesn't support such dictionaries: each field must have a fixed type and exist in each sample.

Instead you can restructure your data like
```
{
    "index": 0,
    "keys": ["2 * x + y >= 3"],
    "values": [["2 * x + y >= 3", "4 * x + 2 * y >= 6"]],
    }
},
...
{
    "index": 9999,
    "keys": ["x >= 6"],
    "values": [["x >= 6", "x >= 0", "x >= -1"]],
},
...
```

Maybe there could be some type of automated conversion from dicts to tuples.  I am also trying to wrangle a json-based dataset into `datasets` and it's awful because of this issue.

Alternatively we can maybe support the [Json extension type](https://arrow.apache.org/docs/python/generated/pyarrow.JsonType.html#pyarrow.JsonType) in `pyarrow` ?

btw `datasets` is open to contributions on this subject if you'd like to take a look

Hmm, I'll think about this a bit.

https://arrow.apache.org/docs/python/json.html

> Nested JSON objects convert to a struct type, and inference proceeds recursively on the JSON objects’ values.

https://arrow.apache.org/docs/dev/python/generated/pyarrow.JsonType.html

Hmm... AFAICT from reading the docs, the `JsonType` seems like a string that is annotated as JSON.  So when you read it, it's literally just the encoded JSON.  So that's not ideal.

I guess there are conceptually two components to using this:
1. Modifying schema inference to use JsonType when "appropriate".  More on that below.
2. Handling JsonType when reading.  I guess we would want to call `json.loads` for the user on any JsonType columns?

### Schema inference

I can think of a few ways forward:
1. Add a JSON builder option that converts nested JSON objects to JsonTypes instead.
2. Use some heuristic to detect when a struct type is bad (e.g., average number of `None` values) and convert those to JsonTypes instead.

The first option is the easiest, but also would remove the nested structure from the arrow schema.  Does this matter?

The first option sounds good indeed, and more explicit / flexible.

> but also would remove the nested structure from the arrow schema. Does this matter?

Well I expect some `pyarrow.compute` functions for nested data to not work for JsonType, and maybe exporting to a pandas / polars dataframe can have a few issues. But it's ok as a first step and we can iterate imo

So this will be harder than I thought.  I thought pyarrow would provide a separate type inference function, but it seems like it doesn't.  Type inference is wrapped into the conversion/loading functions, which are, of course, failing.

Since we use pandas to load the JSON before converting to arrow, maybe we can do some conversion there.

```
In [29]: df = pd.read_json("/tmp/wtf.json")

In [30]: df
Out[30]: 
        lol
0  [42, []]

In [31]: pa.Table.from_pandas(df)
---------------------------------------------------------------------------
ArrowInvalid                              Traceback (most recent call last)
```

But if we convert the problem column to a string, we can proceed:

```
In [38]: df['lol'] = df['lol'].astype(str)

In [39]: df
Out[39]: 
        lol
0  [42, []]

In [40]: pa.Table.from_pandas(df)
Out[40]: 
pyarrow.Table
lol: string
----
lol: [["[42, []]"]]
```

So I guess we could coerce columns to strings until we're able to convert to arrow, and then convert those coerced columns to JsonType in the final arrow.  This feels kind of icky to me though.  But I think it might work.

Makes sense, yes it's maybe the way to go to have it working short term.

Longer term `pyarrow` should handle it though IMO via its JSON reader, have you opened an issue there already by any chance ?

> Longer term pyarrow should handle it though IMO via its JSON reader, have you opened an issue there already by any chance ?

I agree that `pyarrow` changes are a better solution long term.  I haven't opened any issues yet.  Were you thinking:
 
* To separate the inference function
* Open an issue with an incompatible JSON file and see what they suggest?

I was thinking of seeing what they suggest in case of incompatible JSON, it's also possible that other members of the community have asked for for help / requested such a feature already. But sharing about the inference function idea can be interesting as well

Thanks, I will do this and see what they suggest.

On Mon, Apr 7, 2025, 9:18 AM Quentin Lhoest - ***@***.***
***@***.***> wrote:

> I was thinking of seeing what they suggest in case of incompatible JSON,
> it's also possible that other members of the community have asked for for
> help / requested such a feature already. But sharing about the inference
> function idea can be interesting as well
>
> —
> Reply to this email directly, view it on GitHub
> <https://github.com/huggingface/datasets/issues/5950#issuecomment-2783312654>,
> or unsubscribe
> <https://github.com/notifications/unsubscribe-auth/AAHYKZKUPOK5IBQBMMVDCTT2YJ3LXAVCNFSM6AAAAAB2H43DQKVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDOOBTGMYTENRVGQ>
> .
> You are receiving this because you are subscribed to this thread.Message
> ID: ***@***.***>
> [image: lhoestq]*lhoestq* left a comment (huggingface/datasets#5950)
> <https://github.com/huggingface/datasets/issues/5950#issuecomment-2783312654>
>
> I was thinking of seeing what they suggest in case of incompatible JSON,
> it's also possible that other members of the community have asked for for
> help / requested such a feature already. But sharing about the inference
> function idea can be interesting as well
>
> —
> Reply to this email directly, view it on GitHub
> <https://github.com/huggingface/datasets/issues/5950#issuecomment-2783312654>,
> or unsubscribe
> <https://github.com/notifications/unsubscribe-auth/AAHYKZKUPOK5IBQBMMVDCTT2YJ3LXAVCNFSM6AAAAAB2H43DQKVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDOOBTGMYTENRVGQ>
> .
> You are receiving this because you are subscribed to this thread.Message
> ID: ***@***.***>
>