Best model selection changed
Author: creisle
Pull request: no

### System Info

```
OS: Ubuntu 18.04.2 LTS
python: 3.8.0
torch               1.10.1+cu113
torchaudio          0.10.1+cu113
torchvision         0.11.2+cu113
transformers        4.15.0
```

### Who can help?

@sgugger 

### Information

- [ ] The official example scripts
- [X] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [X] My own task or dataset (give details below)

### Reproduction

I am training an NLI model (but I am pretty sure this is task independent). Here are my training arguments

```
training_args = TrainingArguments(
        output_dir=args.output,  # output directory
        num_train_epochs=1,  # total # of training epochs
        per_device_train_batch_size=args.batch_size,  # batch size per device during training
        per_device_eval_batch_size=args.batch_size,  # batch size for evaluation
        warmup_steps=0,  # number of warmup steps for learning rate scheduler
        weight_decay=0.01,  # strength of weight decay
        logging_dir=os.path.join(args.output, 'logs'),  # directory for storing logs
        logging_steps=args.eval_steps,
        learning_rate=1e-05,
        evaluation_strategy="steps",
        save_strategy="steps",
        eval_steps=args.eval_steps,
        save_steps=args.eval_steps,
        gradient_checkpointing=True,
        load_best_model_at_end=True,
        metric_for_best_model='loss',  # default changed from this at some point?
        disable_tqdm=True,
        save_total_limit=args.save_total_limit,
    )
```

With no argument for `metric_for_best_model` or with as above setting it explicitly to `loss` the results are the same and the trainer always chooses the best model based on eval loss not training loss

### Expected behavior

I expected it to save the model based n the best training loss, is there any way to do this? from earlier versions of ths library I had used that used to be the default behaviour but after looking the details it doesn't seem like that is possible any more. Is that intentional? Is there an option here that would get the training loss that I am just not using correctly?

https://github.com/huggingface/transformers/blame/e3cc4487fe66e03ec85970ea2db8e5fb34c455f4/src/transformers/trainer.py#L2228

Comments:
It was never possible, it has always been on the evaluation loss (or any other metric).

@sgugger thanks for getting back to me so quickly! Did the default metric change at any point? I am just trying to determine why the "best model" being saved is different when I haven't changed anything in my data/training arguments. I thought maybe it was selecting on the training loss before because that is the metric that looked closest to the checkpoint it selected before

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.