[Question]Why ther exist a big gap in evaluation step and actual inference test?
Author: daehuikim
Pull request: no

Hello!
Sorry for noob question. I am doing lora fine tuning based on llama-2-13B model. I am adjusting my model using metric "BLEU".
While training, my BLEU score shows incredibly high which is much higher than i expected. For example, evaluation bleu for my model scores about 98 scores which is unbelievable like in the graph.
![image](https://github.com/huggingface/transformers/assets/40377750/46d91e10-aab1-4494-961a-59244caa26a5)

So, I suspected my code at first, however whole codes seems no matter about bleu metrics like below. (```<SOO> ```token means start of output that indicates beginning point of output to exclude input prompts in calculating bleu score in evaluation.)

```
import area

def compute_metrics_bleu(pred):
    references = pred.label_ids
    generated_texts = pred.predictions
    bleu_scores = []
    for reference, generated in zip(references, generated_texts):
        generated = np.where(generated != -100, generated, tokenizer.pad_token_id)
        generated_text = tokenizer.decode(generated, skip_special_tokens=False)
        generated_text = generated_text.split("<SOO>")[-1]
        reference = np.where(reference != -100, reference, tokenizer.pad_token_id)
        reference_text = tokenizer.decode(reference, skip_special_tokens=False)
        reference_text = reference_text.split("<SOO>")[-1]
        bleu_score = sentence_bleu([reference_text], generated_text)
        bleu_scores.append(bleu_score)
    return {
        'bleu': sum(bleu_scores) / len(bleu_scores)
    }

(skip codes)

trainer = SFTTrainer(
        model=model,
        train_dataset=train_dataset,
        eval_dataset=valid_dataset,  # Pass validation dataset here
        peft_config=peft_config,
        dataset_text_field="text",
        max_seq_length=max_seq_length,
        tokenizer=tokenizer,
        args=training_arguments,
        packing=packing,
        compute_metrics=compute_metrics_bleu,
        preprocess_logits_for_metrics=preprocess_logits_for_metrics,
        neftune_noise_alpha=5
    )

```

so I checked the codes on ```trainer.py``` to checkout why the predictions during evaluation loop score such a high bleu score here.
https://github.com/huggingface/transformers/blob/3cefac1d974db5e2825a0cb2b842883a628be7a0/src/transformers/trainer.py#L3381
When I try my test, i use code like below.
```
model = AutoModelForCausalLM.from_pretrained(params same as training)
tokenizer = AutoTokenizer.from_pretrained(params same as training)
inputs = tokenizer(prompt_template.format(prompt=input), return_tensors="pt").input_ids.to("cuda:0")
result = model.generate(inputs, generation arguments)
```
I tried adjust many hyperparameters on ```model.generate()``` including just overriding ```generation_config.json``` on the model, I failed to get such a high bleu score in the test. (the score was around 30~40 even i tested with same dataset)

What could be the reason for this situation?
Thanks for reading my question!

Comments:
This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.

Hey 🤗  thanks for opening an issue! We try to keep the github issues for bugs/feature requests.
Could you ask your question on the [forum](https://discuss.huggingface.co/) instead? I'm sure the community will be of help!

Thanks!

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.

@daehuikim @ArthurZucker 
I met the same issue, please consider talking on this topic.
I believe it is very important for users.

sorry both I thought this was about why:
1. you train the model on the training set
2. you eval on the eval set (same data distribution as the train set)
3. you test on a potentially different data distribution, unseen by the model. 

But if it's related to generate, you have to make sure you are properly passing the `bos_token` to the generate function!