DatasetDict containing Datasets with different features when pushed to hub gets remapped features
Opened by pietrolesci (closed)

Hi there,

I am trying to load a dataset to the Hub. This dataset is a `DatasetDict` composed of various splits. Some splits have a different `Feature` mapping. Locally, the DatasetDict preserves the individual features but if I `push_to_hub` and then `load_dataset`, the features are all the same.

Dataset and code to reproduce available [here](https://huggingface.co/datasets/pietrolesci/robust_nli).

In short:

I have 3 feature mapping
```python
Tri_features = Features(
    {
        "idx": Value(dtype="int64"),
        "premise": Value(dtype="string"),
        "hypothesis": Value(dtype="string"),
        "label": ClassLabel(num_classes=3, names=["entailment", "neutral", "contradiction"]),
    }
)

Ent_features = Features(
    {
        "idx": Value(dtype="int64"),
        "premise": Value(dtype="string"),
        "hypothesis": Value(dtype="string"),
        "label": ClassLabel(num_classes=2, names=["non-entailment", "entailment"]),
    }
)

Con_features = Features(
    {
        "idx": Value(dtype="int64"),
        "premise": Value(dtype="string"),
        "hypothesis": Value(dtype="string"),
        "label": ClassLabel(num_classes=2, names=["non-contradiction", "contradiction"]),
    }
)
```

Then I create different datasets

```python
dataset_splits = {}

for split in df["split"].unique():
    print(split)
    df_split = df.loc[df["split"] == split].copy()
    
    if split in Tri_dataset:
        df_split["label"] = df_split["label"].map({"entailment": 0, "neutral": 1, "contradiction": 2})
        ds = Dataset.from_pandas(df_split, features=Tri_features)
    
    elif split in Ent_bin_dataset:
        df_split["label"] = df_split["label"].map({"non-entailment": 0, "entailment": 1})
        ds = Dataset.from_pandas(df_split, features=Ent_features)
    
    elif split in Con_bin_dataset:
        df_split["label"] = df_split["label"].map({"non-contradiction": 0, "contradiction": 1})
        ds = Dataset.from_pandas(df_split, features=Con_features)

    else:
        print("ERROR:", split)
    dataset_splits[split] = ds
datasets = DatasetDict(dataset_splits)
```

I then push to hub

```python
datasets.push_to_hub("pietrolesci/robust_nli", token="<token>")
```

Finally, I load it from the hub

```python
datasets_loaded_from_hub = load_dataset("pietrolesci/robust_nli")
```

And I get that

```python
datasets["LI_TS"].features != datasets_loaded_from_hub["LI_TS"].features
```

since 

```python
"label": ClassLabel(num_classes=2, names=["non-contradiction", "contradiction"])
```

gets remapped to 

```python
 "label": ClassLabel(num_classes=3, names=["entailment", "neutral", "contradiction"])
```

Comments:
Hi @pietrolesci, thanks for reporting.

Please note that this is a design purpose: a `DatasetDict` has the same features for all its datasets. Normally, a `DatasetDict` is composed of several sub-datasets each corresponding to a different **split**.

To handle sub-datasets with different features, we use another approach: use different **configurations** instead of **splits**.

However, for the moment `push_to_hub` does not support specifying different configurations. IMHO, we should implement this.

Hi @albertvillanova,

Thanks a lot for your reply! I got it now. The strange thing for me was to have it correctly working (i.e., DatasetDict with different features in some datasets) locally and not on the Hub. It would be great to have configuration supported by `push_to_hub`. Personally, this latter functionality allowed me to iterate rather quickly on dataset curation.

Again, thanks for your time @albertvillanova!

Best,
Pietro

Hi! Yes, we should override `DatasetDict.__setitem__` and throw an error if features dictionaries are different. `DatasetDict` is a subclass of `dict`, so `DatasetDict.{update/setdefault}` need to be overridden as well. We could avoid this by subclassing `UserDict`, but then we would get the name collision - `DatasetDict.data` vs. `UserDict.data`. This makes me think we should rename the `data` attribute of `DatasetDict`/`Dataset` for easier dict subclassing (would also simplify https://github.com/huggingface/datasets/pull/3997) and to follow good Python practices. Another option is to have a custom `UserDict` class in `py_utils`, but it can be hard to keep this class consistent with the built-in `UserDict`. 

@albertvillanova @lhoestq wdyt?

I would keep things simple and keep subclassing dict. Regarding the features check, I guess this can be done only for `push_to_hub` right ? It is the only function right now that requires the underlying datasets to be splits (e.g. train/test) and have the same features.

Note that later you will be able to push datasets with different features as different dataset **configurations** (similarly to the [GLUE subsets](https://huggingface.co/datasets/glue) for example). We will work on this soon

Hi @lhoestq,

Returning to this thread to ask whether the possibility to create `DatasetDict` with different configurations will be supported in the future.

Best,
Pietro

DatasetDict is likely to always require the datasets to have the same columns and types, while different configurations may have different columns and types.

Why would you like to see that ?
If it's related to push_to_hub, we plan to allow pushing several configs, but not using DatasetDict

Hi @lhoestq and @pietrolesci,

I have been curious about this question as well. I don't have experience working with different configurations, but I can give a bit more detail on the work flow that I have been using with `Dataset_dict`.

As @pietrolesci mentions, I have been using `push_to_hub` to quickly iterate on dataset curation for different ML experiments - locally I create a set of dataset splits e.g. `train/val/test/inference`, then convert them to `HF_Datasets` and finally a to `Dataset_Dict` to `push_to_hub`. Where I have run into issues is when I want to include different metadata for different splits. For example, I have situations where I only have meta-data for one of the splits (e.g. test) or situations where I am working with `inference` data that does not have labels. Currently I use a rather hacky work around by adding "dummy" columns for missing columns to avoid the error:

```
ValueError: All datasets in `DatasetDict` should have the same features
```

I am curious why `DatasetDict` will likely not support this functionality? I don't know much about working with different configurations, but allowing for different columns between datasets / splits would be a very helpful use-case for me. Are there any docs for using different configuration OR a more info about incorporating it with `push_to_hub`.

Best wishes,
Jonathan



+1

> I am curious why DatasetDict will likely not support this functionality?

There's a possibility we may merge the Dataset and DatasetDict classes. The DatasetDict purpose was to define a way to get the train/test splits of a dataset.

see the discussions at https://github.com/huggingface/datasets/issues/5189

> Are there any docs for using different configuration OR a more info about incorporating it with push_to_hub.

There's a PR open to allow to upload a dataset with a certain configuration name. Then later you can reload this specific configuration using `load_dataset(ds_name, config_name)`

see the PR at https://github.com/huggingface/datasets/pull/5213

Hi, regarding the following information:

> Please note that this is a design purpose: a `DatasetDict` has the same features for all its datasets. Normally, a `DatasetDict` is composed of several sub-datasets each corresponding to a different **split**.
> 
> To handle sub-datasets with different features, we use another approach: use different **configurations** instead of **splits**.

Altough this is often implied (such as how else would `DatasetDict` be able to process multiple splits in the same way?), I would expect it to be written somewhere in the docs plainly and maybe even in bold. Also I would expect to see it in multiple places such as:

- in docstring of `DatasetDict`
- in nlp/image/audio guides on how to create a dataset
- [in conceptual guide on how to create a loading script](https://huggingface.co/docs/datasets/main/en/about_dataset_load)


I think this addition would benefit the docs, especially when you guide a newbie (such as me) through the process of creating a dataset. As I said, you somehow suspect that this is in fact the case, but without reading it in the docs you cannot be sure.