model.generate() leads to poor generation quality when providing a batch of sample with various length with some translation models
Author: MaxLSB
Pull request: no
Labels: name: bug

## System Info

Name: transformers
Version: 4.49.0
Summary: State-of-the-art Machine Learning for JAX, PyTorch and TensorFlow
Home-page: https://github.com/huggingface/transformers
Author: The Hugging Face team (past and future) with the help of all our contributors (https://github.com/huggingface/transformers/graphs/contributors)
Author-email: transformers@huggingface.co
License: Apache 2.0 License
Location: c:\users\mlasb\anaconda3\envs\triton_env\lib\site-packages
Requires: filelock, huggingface-hub, numpy, packaging, pyyaml, regex, requests, safetensors, tokenizers, tqdm
Required-by: peft

## Who can help?

@ArthurZucker @gante 

## Information

- [ ] The official example scripts
- [ ] My own modified scripts

## Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

## Reproduction

`from transformers import AutoTokenizer, AutoModelForSeq2SeqLM, BitsAndBytesConfig
import torch

device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
print(f"> Using device: {device}")
model_name = "facebook/nllb-200-1.3B"

tokenizer = AutoTokenizer.from_pretrained(model_name)
tokenizer.src_lang = "eng_Latn"
tokenizer.tgt_lang = "fra_Latn"
tokenizer.padding_side = "right"
print("> Tokenizer loaded.")

bnb_config = BitsAndBytesConfig(
    load_in_8bit=True,
    llm_int8_enable_fp32_cpu_offload=True,
)
model = AutoModelForSeq2SeqLM.from_pretrained(
    model_name,
    device_map="auto",
    torch_dtype=torch.float16,
    quantization_config=bnb_config,
)
model.eval() 
print("> Model loaded.")

texts = [
    """Once upon a time, in a big forest, there lived a rhinoceros named Roxy. Roxy loved to climb. She climbed trees, rocks, and hills. One day, Roxy found an icy hill. She had never seen anything like it before. It was shiny and cold, and she wanted to climb it. Roxy tried to climb the icy hill, but it was very slippery. She tried again and again, but she kept falling down. Roxy was sad. She wanted to climb the icy hill so much. Then, she saw a little bird named Billy. Billy saw that Roxy was sad and asked, "Why are you sad, Roxy?" Roxy told Billy about the icy hill and how she couldn't climb it. Billy said, "I have an idea! Let's find some big leaves to put under your feet. They will help you climb the icy hill." Roxy and Billy looked for big leaves and found some. Roxy put the leaves under her feet and tried to climb the icy hill again.""",
    """Spot. Spot saw the shiny car and said, "Wow, Kitty, your car is so bright and clean!" Kitty smiled and replied, "Thank you, Spot. I polish it every day." After playing with the car, Kitty and Spot felt thirsty. They found a small pond with clear water. They drank the water and felt very happy. They played together all day and became best friends.""",
    """Hello how are you doing? I am doing well. I am learning how to code in Python. It is a lot of fun! I hope you are having a great day too!""",
]
inputs = tokenizer(
    texts,
    return_tensors="pt",
    truncation=True,
    padding=True,
    max_length=256,
)
inputs = inputs.to(device)
target_lang_token = tokenizer.convert_tokens_to_ids(tokenizer.tgt_lang)
outputs = model.generate(
    input_ids=inputs["input_ids"],
    attention_mask=inputs["attention_mask"],
    forced_bos_token_id=target_lang_token,
    max_length=256,
)

print(tokenizer.batch_decode(outputs, skip_special_tokens=True))
`

## Expected behavior

Hello,

When using the "facebook/nllb-200-1.3B" or "facebook/nllb-200-distilled-1.3B" translation models (and possibly other models as well), I noticed that providing a batch of inputs with highly varying lengths leads to poor translations (when translating from english to french at least), especially for the shorter samples. However, when I translate these short samples individually, the results are accurate. So I think something is not working correctly...

Basically it work properly with a single sample. And it doesn't work properly when given a batch of samples of various length (for the shorter ones).

Is there a known way to address this issue? Could it be related to padding, to the models themselves or how model.generate() handles batches?

## Examples

- The 2nd and 3rd translations are the problem in the example:

['Il était une fois, dans une grande forêt, un rhinocéros nommé Roxy. Roxy aimait grimper. Elle grimpait des arbres, des rochers et des collines. Un jour, Roxy trouva une colline glacée. Elle n\'avait jamais rien vu de tel auparavant. C\'était brillant et froid, et elle voulait y grimper. Roxy essaya de grimper la colline glacée, mais elle était très glissante. Elle essaya encore et encore, mais elle continuait à tomber. Roxy était triste. Elle voulait tellement grimper la colline glacée. Puis, elle a vu un petit oiseau nommé Billy. Billy a vu que Roxy était triste et a demandé: "Pourquoi es-tu triste, Roxy?" Roxy a parlé à Billy de la colline glacée et de la façon dont elle ne pouvait pas l\'escalader. Billy a dit: "J\'ai une idée! Allons trouver des grandes feuilles à mettre sous tes pieds. Ils vont t\'aider à grimper la colline glacée". Billy a cherché de nouveau et Roxy a essayé de', 
'On peut aussi lui demander de lui donner un coup de main. " Et si vous aviez un coup de main, vous auriez pu le faire. " On pourrait lui demander de lui donner un coup de main. " Et si vous aviez un coup de main, vous auriez pu le faire. "', 'Wiwiwiwiwiwiwiwiwiwiwiwiwiwiwiwiwiwiwiwiwiwiwiwiwiwiwiwiwiwiwiwiwiwiwiwiwiwiwiwiwiwiwiwiwiwiwiwiwiwiwiwiwiwiwiwiwiwiwiwiwiwiwiwiwiwiwiwiwiwiwiwiwiwiwiwiwiwiwiwiwiwiwiwiwiwiwiwiwiwiwiwiwiwiwiwiwiwiwiwiwiwiwiwiwiwiwiwiwiwiwiwiwiwiwiwiwiwiwiwiwiwiwiwiwiwiwiwiwiwiwiwiwiwiwiwiwiwiwiwiwiwiwiwiwiwiwiwiwiwiwiwiwiwiwiwiwiwiwiwiwiwiwiwiwiwiwiwiwiwiwiwiwiwiwiwiwiwiwiwiwiwiwiwiwiwiwiwiwiwiwiwiwiwiwiwiwiwiwiwiwiwiwiwiwiwiwiwiwiwiwiwiwiwiwiwiwiwiwiwiwiwiwiwiwiwiwiwiwiwiwiwiwiwiwiwiwiwiwiwiwiwiwiwiwiwiwiwiwiwiwiwiwiwi']

- Here are the correct translations obtained when giving only these samples as input (batch of size 1):

Second sample: ['Spot a vu la voiture brillante et a dit: " Wow, Kitty, ta voiture est si brillante et propre ! " Kitty a souri et a répondu: " Merci, Spot. Je la polisse tous les jours. " Après avoir joué avec la voiture, Kitty et Spot ont eu soif. Ils ont trouvé un petit étang avec de l\'eau claire. Ils ont bu l\'eau et se sont sentis très heureux. Ils ont joué ensemble toute la journée et sont devenus les meilleurs amis.']

Third sample: ["Bonjour, comment allez-vous ? Je vais bien. J'apprends à coder en Python. C'est très amusant ! J'espère que vous passez une bonne journée aussi !"]




Comments:
@MaxLSB hey! For encoder decoder models `v4.49` broke batched generation and it was fixed later (https://github.com/huggingface/transformers/issues/36461). Can you try to update to earlier or later versions?

@zucchini-nlp  Thank you very much, it works properly with the latest version !