Speech Commands v2 dataset doesn't match AST-v2 config
Opened by vymao (closed)

### Describe the bug

[According](https://huggingface.co/MIT/ast-finetuned-speech-commands-v2) to `MIT/ast-finetuned-speech-commands-v2`, the model was trained on the Speech Commands v2 dataset. However, while the model config says the model should have 35 class labels, the dataset itself has 36 class labels. Moreover, the class labels themselves don't match between the model config and the dataset. It is difficult to reproduce the data used to fine tune `MIT/ast-finetuned-speech-commands-v2`.

### Steps to reproduce the bug

```
>>> model = ASTForAudioClassification.from_pretrained("MIT/ast-finetuned-speech-commands-v2")
>>> model.config.id2label
{0: 'backward', 1: 'follow', 2: 'five', 3: 'bed', 4: 'zero', 5: 'on', 6: 'learn', 7: 'two', 8: 'house', 9: 'tree', 10: 'dog', 11: 'stop', 12: 'seven', 13: 'eight', 14: 'down', 15: 'six', 16: 'forward', 17: 'cat', 18: 'right', 19: 'visual', 20: 'four', 21: 'wow', 22: 'no', 23: 'nine', 24: 'off', 25: 'three', 26: 'left', 27: 'marvin', 28: 'yes', 29: 'up', 30: 'sheila', 31: 'happy', 32: 'bird', 33: 'go', 34: 'one'}

>>> dataset = load_dataset("speech_commands", "v0.02", split="test")
>>> torch.unique(torch.Tensor(dataset['label']))
tensor([ 0.,  1.,  2.,  3.,  4.,  5.,  6.,  7.,  8.,  9., 10., 11., 12., 13.,
        14., 15., 16., 17., 18., 19., 20., 21., 22., 23., 24., 25., 26., 27.,
        28., 29., 30., 31., 32., 33., 34., 35.])
```
If you try to explore the [dataset itself](https://huggingface.co/datasets/speech_commands/viewer/v0.02/test), you can see that the id to label does not match what is provided by `model.config.id2label`.


### Expected behavior

The labels should match completely and there should be the same number of label classes between the model config and the dataset itself.

### Environment info

datasets = 2.14.6, transformers =  4.33.3

Comments:
You can use `.align_labels_with_mapping` on the dataset to align the labels with the model config.

Regarding the number of labels, only the special `_silence_` label corresponding to noise is missing, which is consistent with the model paper (reports training on 35 labels).  You can run a `.filter` to drop it.

PS: You should create a discussion on a model/dataset repo (on the Hub) for these kinds of questions

Thanks, will keep that in mind. But I tried running `dataset_aligned = dataset.align_labels_with_mapping(model.config.id2label, 'label')`, and received this error: 

```
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/Users/victor/anaconda3/envs/transformers-v2/lib/python3.9/site-packages/datasets/arrow_dataset.py", line 5928, in align_labels_with_mapping
    label2id = {k.lower(): v for k, v in label2id.items()}
  File "/Users/victor/anaconda3/envs/transformers-v2/lib/python3.9/site-packages/datasets/arrow_dataset.py", line 5928, in <dictcomp>
    label2id = {k.lower(): v for k, v in label2id.items()}
AttributeError: 'int' object has no attribute 'lower'
```
My guess is that the dataset `label` column is purely an int ID, and I'm not sure there's a way to identify which class label the ID belongs to in the dataset easily.

Replacing `model.config.id2label` with `model.config.label2id` should fix the issue.

So, the full code to align the labels with the model config is as follows:
```python
from datasets import load_dataset
from transformers import AutoFeatureExtractor, AutoModelForAudioClassification

# extractor = AutoFeatureExtractor.from_pretrained("MIT/ast-finetuned-speech-commands-v2")
model = AutoModelForAudioClassification.from_pretrained("MIT/ast-finetuned-speech-commands-v2")

ds = load_dataset("speech_commands", "v0.02")
ds = ds.filter(lambda label: label != ds["train"].features["label"].str2int("_silence_"), input_columns="label")
ds = ds.align_labels_with_mapping(model.config.label2id, "label")
```