run_qa.py script does not compute eval_loss and gives KeyError: 'eval_loss' with load_best_model_at_end 
Author: ftesser
Pull request: no
Labels: name: Examples

name: Good Second Issue

name: bug

### System Info

- `transformers` version: 4.39.0
- Platform: Linux-5.15.0-100-lowlatency-x86_64-with-glibc2.35
- Python version: 3.10.12
- Huggingface_hub version: 0.20.3
- Safetensors version: 0.4.2
- Accelerate version: 0.27.2
- Accelerate config:    not found
- PyTorch version (GPU?): 2.2.0+cu121 (False)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: no
- Using distributed or parallel set-up in script?: no


### Who can help?

The official QA training script `run_qa.py` return the following error with `--load_best_model_at_end` and `--metric_for_best_model "loss"`.
```
Traceback (most recent call last):
  File "/home/fabio/repos/transformers/examples/pytorch/question-answering/run_qa.py", line 716, in <module>
    main()
  File "/home/fabio/repos/transformers/examples/pytorch/question-answering/run_qa.py", line 657, in main
    train_result = trainer.train(resume_from_checkpoint=checkpoint)
  File "/home/fabio/repos/transformers/src/transformers/trainer.py", line 1780, in train
    return inner_training_loop(
  File "/home/fabio/repos/transformers/src/transformers/trainer.py", line 2213, in _inner_training_loop
    self._maybe_log_save_evaluate(tr_loss, grad_norm, model, trial, epoch, ignore_keys_for_eval)
  File "/home/fabio/repos/transformers/src/transformers/trainer.py", line 2588, in _maybe_log_save_evaluate
    self._save_checkpoint(model, trial, metrics=metrics)
  File "/home/fabio/repos/transformers/src/transformers/trainer.py", line 2669, in _save_checkpoint
    metric_value = metrics[metric_to_check]
KeyError: 'eval_loss'
```
[full_log.txt](https://github.com/huggingface/transformers/files/14719855/full_log.txt)

@ArthurZucker @sgugger 

### Information

- [X] The official example scripts
- [ ] My own modified scripts

### Tasks

- [X] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

To reproduce the error in quick way you can use a distilled model and limit the max_train_samples and max_eval_samples:

```bash
python examples/pytorch/question-answering/run_qa.py \
--model_name_or_path deepset/roberta-base-squad2-distilled  \
--dataset_name squad \
--do_train \
--do_eval \
--max_seq_length 384 \
--doc_stride 128 \
--max_train_samples 5 \
--max_eval_samples 2 \
--num_train_epochs 3 \
--load_best_model_at_end \
--metric_for_best_model "loss" \
--evaluation_strategy "epoch" \
--save_strategy "epoch" \
--overwrite_output_dir \
--output_dir ~/tmp/debug_squad/
```

I have tested this bug also with a normal model and a full dataset but the error is always there.

### Expected behavior

During the evaluation phase the eval_loss should be computed and the best model should be saved using the loss metric.



Comments:
that is more a `trainer` issue cc @muellerzr and @SunMarc. Given that it is an example and we usually don't maintain examples, I'll set this as a good second issue. 

@ArthurZucker @jla524 I saw your pull request https://github.com/huggingface/transformers/pull/29867 , and I agree that adding the message for unsupported metrics is a great idea.

However, I don't understand why loss is not supported in the case of squad2 (https://github.com/huggingface/transformers/pull/29867#issue-2207121923): the use case is to have the loss metric available in the validation set in order to use that metric to determine the best model.

Furthermore, the loss is calculated in the training set . Why isn't it possible to calculate it in the valuation set too?

@ftesser that is a question for `evaluate`! It's probably that the loss is not really `metric` like `f1` and it's log elsewhere.