Add "save_best_only" parameter in "transformers.PushToHubCallback" class
Author: MUmairAB
Pull request: no
Labels: name: Feature request

### Feature request

When utilizing Keras callbacks, we have the ability to specify when the model should be saved during training. The **transformers.PushToHubCallback()** class already incorporates similar functionality through the use of the **"save_strategy"** parameter. This parameter accepts the following values:

- "no": Saving is performed at the conclusion of training.
- "epoch": Saving is performed at the end of each epoch.
- "steps": Saving is performed every "save_steps" interval.

However, these options do not take into account accuracy (or any other specified metric) improvement. In contrast, the Keras callback provides the **"save_best_only"** parameter, which exclusively saves the model when there is an enhancement in accuracy or the specified metric. The code snippet below demonstrates its usage:

```
#Define the callback
callbacks = [
             keras.callbacks.ModelCheckpoint(
                 filepath="directory_name/model.keras",
                 monitor="val_loss",
                 save_best_only=True,
                 )
            ]

#Start the training
history = model.fit(
                    train_dataset,
                    epochs=5,
                    validation_data=validation_dataset,
                    monitor="val_loss",
                    callbacks=callbacks)
```

The model mentioned above will undergo training for a total of 5 epochs. However, the model will only be saved when there is an improvement in the "validation loss" metric.

**transformers.PushToHubCallback()** class must incorporate this feature as well.


### Motivation

This feature is indeed quite valuable, and it is readily accessible through [Keras callbacks](https://keras.io/api/callbacks/model_checkpoint/#:~:text=save_best_only%3A%20if%20save_best_only%3DTrue%20%2C,by%20each%20new%20better%20model.). By utilizing this feature, significant processing power and bandwidth can be saved, particularly when dealing with large transformers models. It ensures that only the best-performing models, based on the specified metric (such as validation loss), are saved, resulting in more efficient storage and reduced computational resources.

### Your contribution

This [source code](https://github.com/keras-team/keras/blob/v2.12.0/keras/callbacks.py) can be helpful.

Comments:
cc @sgugger to see if we want to support the **save/push only the best model during training**. It seems the trainer currently only support load the best model at the end (with a specified metric).

The whole goal of pushing the model to the hub is to be able to resume training from a different machine if there is a problem. If we push only the best model while checkpointing, this is not going to be possible anymore.

Note that the best model will be pushed at the end of the training, so you will have the correct result once the training is finished.

Thank you for your consideration and feedback on my feature request.

I understand the goal of pushing the model to the hub using **Transformers.PushToHubCallback()** is to enable resuming training from a different machine if necessary. I appreciate the point you made about the potential impact on that capability if only the best model is saved and pushed during training.

Keeping this in mind, would it be possible to explore a solution that balances both requirements?

- Perhaps a configuration option that allows users to choose between saving and pushing only the best model, or 
- Saving and pushing the best model at specific intervals during training?

This way, users can have the flexibility to optimize storage and computational resources while still maintaining the ability to resume training from different machines if needed.

Thank you for your time and consideration. I look forward to hearing your thoughts on this matter.


_**A workaround**_

_I would like to state that even if the PushToHubCallback does not incorporate this feature, there is still a workaround if we are constrained by the bandwidth._

_We should train the model without **PushToHubCallback**. In order to save the best model locally, use the Keras **save_best_only** callback as shown above. Finally, at the end of the training, we can use **model.push_to_hub()** to save the best model stored on the local machine to the Hub._

_Nonetheless, it would be better if this feature is incorporated in the **Transformers.PushToHubCallback()**._