Finetuning AudioFrameClassification model
Author: MorenoLaQuatra
Pull request: no
Labels: name: bug

### System Info

```shell
- `transformers` version: 4.19.2
- Platform: Linux-5.10.0-051000-generic-x86_64-with-glibc2.32
- Python version: 3.9.12
- Huggingface_hub version: 0.7.0
- PyTorch version (GPU?): 1.7.1+cu110 (True)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: <fill in>
- Using distributed or parallel set-up in script?: No
```


### Who can help?

@patrickvonplaten @anton-l 

### Information

- [ ] The official example scripts
- [X] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [X] My own task or dataset (give details below)

### Reproduction

I'm trying to finetune a WavLMForAudioFrameClassification model using Trainer and a custom dataset. 
It is not my first project with transformers. 

When I tried running the training of the model I got this very **strange** warning:

`The following columns in the training set don't have a corresponding argument in WavLMForAudioFrameClassification.forward and have been ignored: labels. If labels are not expected by WavLMForAudioFrameClassification.forward,  you can safely ignore this message.`

and then the following error:

```python
  File "XXX/lib/python3.9/site-packages/transformers/utils/generic.py", line 220, in __getitem__
    return inner_dict[k]
KeyError: 'loss'
```

Looking at the code [here](https://github.com/huggingface/transformers/blob/6e535425feae20ca61a8b10ae5e8a7fab4d394ba/src/transformers/models/wavlm/modeling_wavlm.py#L1648) it seems that `labels` are not used and loss function is not computed. Is it possible to finetune an AudioFrameClassification model? Is `labels` the wrong keyword?

### Expected behavior

```shell
Standard finetuning.
```


Comments:
Hi, if possible could you share your code for how to fine-tune WavLMForAudioFrameClassification for custom dataset.
Thank you very much.

I can share with you some code I actually use to train an AudioFrameClassification model. It is not intended for Speaker Diarization but the concept behind is the same.

Here you can find the training loop using HF Trainer: https://github.com/MorenoLaQuatra/ComParE2022_MED/blob/master/train.py  

Here instead you can find the dataset class: https://github.com/MorenoLaQuatra/ComParE2022_MED/blob/master/MosTimestampDataset.py - in this specific case, the `__get_item__` function is much more complex than what you need for the simple diarization case. What you should consider is the return value. For the AudioFrameClassification case, you should use as labels something like the following:
```python
labels = [
    [0, 1, 0, 0], # for each frame it contains the one-hot encoded class, 2nd speaker in this case
    [0, 0, 1, 0], # 3rd speaker in this case
    […],
    […],
]
```

Let me know if you have any issues.

I was wondering if this could have been used to the `AutoModelForAudioFrameClassification` class-based speaker diarisation task as well. Given that huggingface Transformers library currently have no official training script nor Jupyter notebook included.