M2M100 generates repetitive text as <eos> token not produced while decoding
Author: reyha
Pull request: no
Labels: name: bug

### System Info

- `transformers` version: 4.11.3
- Platform: Linux-3.10.0-1160.59.1.el7.x86_64-x86_64-with-debian-stretch-sid
- Python version: 3.6.13
- PyTorch version (GPU?): 1.7.1+cu101 (True)
- Tensorflow version (GPU?): 2.5.0 (False)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: Yes (one V100 GPU)
- Using distributed or parallel set-up in script?: No

### Who can help?

@patrickvonplaten @patil-suraj 

### Information

- [ ] The official example scripts
- [X] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [X] My own task or dataset (give details below)

### Reproduction

Hi. I am using M2M100_418M to translate Spanish and French samples to English. However, I observe that for some samples output contains repetitive text.

While debugging, I observe that `<eos>` token is not being produced for these samples early on as expected and same token gets predicted. The model totally ignores the beginning or ending of the input sentence as well in a few scenarios.

Note: max char limit for any input is less than 256 characters.

Code:
```
def translate_m2m(text, src_lang, tgt_lg):
    tokenizer.src_lang = src_lang
    encoded_src = tokenizer(text, return_tensors="pt", padding=True, truncation=True, max_length=1024).to(device)
    generated_tokens = model.generate(**encoded_src, forced_bos_token_id=tokenizer.get_lang_id(tgt_lg))
    translated = tokenizer.batch_decode(generated_tokens, skip_special_tokens=True)    
    return translated[0]

# load model and tokenizer from checkpoints available on HF
model = M2M100ForConditionalGeneration.from_pretrained(m2m_hf_ckpt)
tokenizer = M2M100Tokenizer.from_pretrained(m2m_hf_ckpt)
model = model.to(device)
```

Adding all such samples with their outputs below:

| Source Language      | Target Language   | Input      | Output | 
| ----------- | ----------- |----------- | ----------- | 
| es      | en       | no ya ya tengo ya tengo nueva contraseña ya la ya ya tengo nueva contraseña pero no no entrar mire lo que pasa es que me sale donde voy a voy a decir como me sale ya ya con nueva contraseña mire pongo mi contraseña ingreso con mi y    | I have a new password I have a new password I have a new password I have a new password I have a new password I have a new password I have a new password I have a new password I have a new password I have a new password I have a new password I have a new password I have a new password I have a new password I have a new password I have a new password |  
| es   | en        |sea en esta la tarjeta de doble click doble click aparecer confirmar tarjeta      | Double-click double-click double-click double-click double-click double-click double-click double-click double-click double-click double-click double-click double-click double-click double-click double-click     |
| es | en | que lopues ya yo recibími servicio ya nosotros lo libra vamos o si no es posible pues usted | I’m going to get you, I’m going to get you, I’m going to get you, I’m going to get you, I’m going to get you, I’m going to get you, I’m going to get you. |
| fr   | en        | non ca va ca des choses qui mais je me rappelle moi j'avais pas que mon compte assez on donc c'est vous a pas de problème par rapport | I don't know what I'm doing, but I don't know what I'm doing, I don't know what I'm doing, I don't know what I'm doing |
| en   | es        | hey there hey there hey there hey there hey there hey there hey there hey there hey there hey there hey there hey there hey there hey there hey there hey there hey there hey there hey there hey there hey there hey there | Hay hay hay hay hay hay hay hay hay hay hay hay hay hay hay hay hay hay hay hay hay hay hay hay hay hay hay hay hay |

**Additional Information:**
- issue goes away while randomly removing any words or even adding spaces etc. but there is no consistent pattern or token where always holds true, hence, making it difficult to identify the root cause of this problem.
- same issue occurs when no. of beams = [1,2,3,4,5]
- can be reproduced for different languages (not only true for the ones listed above)

### Expected behavior

I expect M2M100 model to not produce repetitive text and that too randomly on any token for almost all major languages. This is a major problem as there no pattern or token that can be assumed to be causing this. 

Comments:
Hello Reyha (My answer doesn’t represent Hugging Face)

I think there is no bug here, it’s just the output of the raw model. I think you may have to do some post-processing, or you should implement some frequency penalties for each generated token.

However, I recommend looking at this article https://huggingface.co/blog/how-to-generate by Patrick von Platen, in the conclusion he states: 

> "As ad-hoc decoding methods, top-p and top-K sampling seem to produce more fluent text than traditional greedy - and beam search on open-ended language generation. Recently, there has been more evidence though that the apparent flaws of greedy and beam search - mainly generating repetitive word sequences - are caused by the model (especially the way the model is trained), rather than the decoding method, cf. [Welleck et al. (2019)](https://arxiv.org/pdf/1908.04319.pdf). Also, as demonstrated in [Welleck et al. (2020)](https://arxiv.org/abs/2002.02492), it looks as top-K and top-p sampling also suffer from generating repetitive word sequences."

Thanks a lot for the in-detail answer @Mustapha-AJEGHRIR - can only agree :-) 

 @Mustapha-AJEGHRIR @patrickvonplaten - Thanks for confirming, really appreciate it.