YAML integer keys are not preserved Hub server-side
State: closed | Author: albertvillanova (MEMBER)
Labels: name: bug

After an internal discussion (https://github.com/huggingface/moon-landing/issues/4563):
- YAML integer keys are not preserved server-side: they are transformed to strings
  - See for example this Hub PR: https://huggingface.co/datasets/acronym_identification/discussions/1/files
    - Original:
      ```yaml
      class_label:
        names:
          0: B-long
          1: B-short
      ```
    - Returned by the server:
      ```yaml
      class_label:
        names:
          '0': B-long
          '1': B-short
      ```
- They are planning to enforce only string keys
- Other projects already use interger-transformed-to string keys: e.g. `transformers` models `id2label`: https://huggingface.co/roberta-large-mnli/blob/main/config.json
  ```yaml
  "id2label": {
    "0": "CONTRADICTION",
    "1": "NEUTRAL",
    "2": "ENTAILMENT"
  }
  ``` 

On the other hand, at `datasets` we are currently using YAML integer keys for `dataset_info` `class_label`.

Please note (thanks @lhoestq for pointing out) that previous versions (2.6 and 2.7) of `datasets` need being patched:
```python
In [18]: Features._from_yaml_list([{'dtype': {'class_label': {'names': {'0': 'neg', '1': 'pos'}}}, 'name': 'label'}])                                                                                      
---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
<ipython-input-18-974f07eea526> in <module>
----> 1 Features._from_yaml_list(ry)

~/Desktop/hf/nlp/src/datasets/features/features.py in _from_yaml_list(cls, yaml_data)
   1743                 raise TypeError(f"Expected a dict or a list but got {type(obj)}: {obj}")
   1744 
-> 1745         return cls.from_dict(from_yaml_inner(yaml_data))
   1746 
   1747     def encode_example(self, example):

~/Desktop/hf/nlp/src/datasets/features/features.py in from_yaml_inner(obj)
   1739             elif isinstance(obj, list):
   1740                 names = [_feature.pop("name") for _feature in obj]
-> 1741                 return {name: from_yaml_inner(_feature) for name, _feature in zip(names, obj)}
   1742             else:
   1743                 raise TypeError(f"Expected a dict or a list but got {type(obj)}: {obj}")

~/Desktop/hf/nlp/src/datasets/features/features.py in <dictcomp>(.0)
   1739             elif isinstance(obj, list):
   1740                 names = [_feature.pop("name") for _feature in obj]
-> 1741                 return {name: from_yaml_inner(_feature) for name, _feature in zip(names, obj)}
   1742             else:
   1743                 raise TypeError(f"Expected a dict or a list but got {type(obj)}: {obj}")

~/Desktop/hf/nlp/src/datasets/features/features.py in from_yaml_inner(obj)
   1734                             return {"_type": snakecase_to_camelcase(obj["dtype"])}
   1735                     else:
-> 1736                         return from_yaml_inner(obj["dtype"])
   1737                 else:
   1738                     return {"_type": snakecase_to_camelcase(_type), **unsimplify(obj)[_type]}

~/Desktop/hf/nlp/src/datasets/features/features.py in from_yaml_inner(obj)
   1736                         return from_yaml_inner(obj["dtype"])
   1737                 else:
-> 1738                     return {"_type": snakecase_to_camelcase(_type), **unsimplify(obj)[_type]}
   1739             elif isinstance(obj, list):
   1740                 names = [_feature.pop("name") for _feature in obj]

~/Desktop/hf/nlp/src/datasets/features/features.py in unsimplify(feature)
   1704             if isinstance(feature.get("class_label"), dict) and isinstance(feature["class_label"].get("names"), dict):
   1705                 label_ids = sorted(feature["class_label"]["names"])
-> 1706                 if label_ids and label_ids != list(range(label_ids[-1] + 1)):
   1707                     raise ValueError(
   1708                         f"ClassLabel expected a value for all label ids [0:{label_ids[-1] + 1}] but some ids are missing."

TypeError: can only concatenate str (not "int") to str
```

TODO:
- [x] Remove YAML integer keys from `dataset_info` metadata
- [x] Make a patch release for affected `datasets` versions: 2.6 and 2.7
- [x] Communicate on the fix
- [x] Wait for adoption
- [x] Bulk edit the Hub to fix this in all canonical datasets

Comments: @huggingface/datasets if you agree, I can make the bulk edit on the Hub to fix integer keys into strings.

Ok for me, and we can merge (internal) https://github.com/huggingface/moon-landing/pull/4609

FYI there are still 2k+ weekly users on `datasets` 2.6.1 which doesn't support the string label format for class labels. And among those, some are using datasets with class labels like imdb (60 users), conllpp (40), msra_ner (40), peoples_daily_enr (40), weibo_ner (30), conll2003 (20), etc. And renaming to string would break these users code.

but isn't `datasets 2.6.1` downloading files from the Hub with the corresponding tag? I thought we had something like this before

We're using `main` as models do. Some datasets need to be updated from time to time, e.g. when a link to download the data is dead.

But yea a year ago we had those tags, we just ended up not using them

I opened https://github.com/huggingface/datasets/issues/5406 to communicate on this. Let me know what you think, and if it sounds good to you I can pin this issue

So, is it OK to make the bulk edit on the Hub now or should we wait longer? If the latter, how long?

I think we can do it. If you want to be extra cautious you can do it for all datasets except imdb and conllpp for now which are actively used by 2.6.1 users. For those two we can keep the YAML like this for some more time, or alternatively use the old dataset_infos.json file

The bulk edit of canonical datasets (except imdb and conllpp) is running. 

See e.g.: https://huggingface.co/datasets/acronym_identification/discussions/3

EDITED: 
Done, except for "universal_morphologies", where I get
```
HTTPError: 413 Client Error: Payload Too Large for url: https://huggingface.co/api/validate-yaml
```

Also not done for the datasets missing matadata "dataset_info":
- mc4: https://huggingface.co/datasets/mc4/discussions/3
- the_pile: https://huggingface.co/datasets/the_pile/discussions/6
- timit_asr: https://huggingface.co/datasets/timit_asr/discussions/1

Thank you !

@lhoestq, there are 6 community datasets with YAML integer keys in their `dataset_info` `class_label`:
- indonlp/indonlu
- rcds/swiss_judgment_prediction
- Jean-Baptiste/wikiner_fr
- Bingsu/Cat_and_Dog
- taskydata/tasky_or_not
- RCC-MSU/collection3

Maybe we could open a PR on them as well?

Let's do this then:

- [x] [indonlp/indonlu](https://huggingface.co/datasets/indonlp/indonlu/discussions/3)
- [x] rcds/swiss_judgment_prediction
- [x] Jean-Baptiste/wikiner_fr
- [x] Bingsu/Cat_and_Dog -> merged
- [x] taskydata/tasky_or_not (was already using quotes)
- [x] RCC-MSU/collection3

EDIT: all done :)

@lhoestq I was not asking you to do it, but asking if you agree me to do it... :man_facepalming: 
As I self-assigned this issue... :sweat_smile: