Wav2vec2 finetuned model's strange truncated predictions
Author: BakuDev
Pull request: no

### What is your question?
I'm getting strange truncation of prediction at different steps of training. Please help to understand what is the issue?
At the first steps of training like 800-1600 (2-3 epochs) I'm getting predictions of valid length and words count but with low accuracy (which is ok at the first steps), After steps > ~8000 things begin getting strange - accuracy of word prediction getting better, WER respectfully getting lower but an overall sentences' lengths getting truncated to the right side of an utterances. For example:

Target:
Dərbəndin caxır-konyak kombinatı ərazisində yanğın qeydə alınıb. Hadisə axşam saatlarında baş verib. İlkin məlumata görə, insidentə spirt məhlulunun yerə dağılması səbəb olub

Prediction @ 400 step (length is correct, WER 60+)
dərbəndin çaxır kona kombinantı erazisində yanğın qeydə alınıb harisi axşam satlarında baş verb ilki məlumata görə insidentəs birt məxlunun yerə dağılması səbəb olub

Prediction @ 800 step (length is correct, WER 50+)
dərbəndin çaxırkonakombinanta ərazisində yanğın qeydə alınıb hadisə axşamsaatlarında baş verib ilki məlumata görə insidentəs birt məhlullunun yerə dağılması səbəb olub

Prediction @ 1600 step (length getting truncated, words joining each other, WER 40+)
dərbədinçıki əazisdə ynğqdını hadişıa veiklumagörə insidentspirt məlun yerə dağılması səbəb olub

Prediction @ > 20000 step (around 30 to 100 epochs, almost no changes in WER, sentence completely truncated to the right part, WER keep around 16-27 depending on audio quality)

ndəyaninsidentəspirtməluunun yerə dağılması səbəb olub
insidntə spirt məhlulunun yerə dağılması səbəb olub
insidentə spürt məhlulunun yerə dağılması səbəb olub
nsientə spirt məhlulunun yerə dağılması səbəb olub
Code
[](https://huggingface.co/blog/fine-tune-xlsr-wav2vec2)
Exactly the same code but with different epoch param (num_train_epochs 30 to 100)...

What have you tried?
Training data: 30 hours of labeled data, single spoken person per clip, around 5 to 10 words per clip
Looks like after training a model tries to fit prediction into the dataset's training data length? Can it skew this way?

I've used https://huggingface.co/blog/fine-tune-xlsr-wav2vec2 to train very similar to Turkish language. It differs only for a few characters in alphabet so I used exactly the same params for the first training. Then removed return_attention_mask but nothing changed at all. Then I tried to fine-tune Turkish finetuned model from tutorial itself from Patrick's hub repo - got the same results.

What's your environment?
fairseq Version (e.g., 1.0 or main): current master branch
PyTorch Version (e.g., 1.0): the one which comes with Python 3.8
OS (e.g., Linux): Linux
How you installed fairseq (pip, source): clone and installed
Python version: 3.8
CUDA/cuDNN version: 10.2
GPU models and configuration: 1 x V100S (32 GB)


Comments:
Hi @BakuDev! Indeed, it looks like the model overfits to the length of sentences in the training data. Try augmenting the training dataset by randomly concatenating 2 or more clips together to roughly match the length of your validation data, or add some long examples from CommonVoice.

> Hi @BakuDev! Indeed, it looks like the model overfits to the length of sentences in the training data. Try augmenting the training dataset by randomly concatenating 2 or more clips together to roughly match the length of your validation data, or add some long examples from CommonVoice.



> Hi @BakuDev! Indeed, it looks like the model overfits to the length of sentences in the training data. Try augmenting the training dataset by randomly concatenating 2 or more clips together to roughly match the length of your validation data, or add some long examples from CommonVoice.

Forgot to mention that I get [PAD] after each predicted symbol. Maybe this can be related to described issue as well? Example:

[PAD]i[PAD]n[PAD]s[PAD]i[PAD]d[PAD]e[PAD]n[PAD]t[PAD]ə[PAD] [PAD]s[PAD]pü[PAD]r[PAD]t[PAD] m[PAD]ə[PAD]h[PAD]lul[PAD]un[PAD]un[PAD] [PAD]y[PAD]e[PAD]r[PAD]ə [PAD]d[PAD]a[PAD]ğ[PAD]ı[PAD]l[PAD]m[PAD]as[PAD]ı[PAD] [PAD]s[PAD]ə[PAD]bə[PAD]b[PAD] olub[PAD]


@BakuDev in Wav2Vec2's CTC decoder the `<pad>` is also used as a special *blank token* (see the **Alignment** section about it in this article: https://distill.pub/2017/ctc/). 

@anton-l Вы не согласились бы помочь с этой проблемой на платной основе если возможно? Если напишите куда писать в ЛС, свяжусь для обсуждения.

@BakuDev feel free to ask any specific questions about training strategies (i.e. not necessarily about the `transformers` library) on our [forums](https://discuss.huggingface.co/) or on [Discord](https://hf.co/join/discord), someone from the community will gladly help you out :)

We also have a paid support program which you might be interested in: https://huggingface.co/support