Repository: huggingface/transformers

Issue:
Model is saved every eval_steps steps if eval_steps < save_steps. Is this expected behavior?
## Environment info
<!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->

- `transformers` version: 4.6.1
- Platform: macOS-10.16-x86_64-i386-64bit
- Python version: 3.8.5
- PyTorch version (GPU?): 1.7.1 (False)
- Tensorflow version (GPU?): not installed (NA)
- Using GPU in script?: No
- Using distributed or parallel set-up in script?: No

### Who can help

@sgugger
<!-- Your issue will be replied to more quickly if you can figure out the right person to tag with @
 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.

Models:

- albert, bert, xlm: @LysandreJik
- blenderbot, bart, marian, pegasus, encoderdecoder,  t5: @patrickvonplaten, @patil-suraj
- longformer, reformer, transfoxl, xlnet: @patrickvonplaten
- fsmt: @stas00
- funnel: @sgugger
- gpt2: @patrickvonplaten, @LysandreJik
- rag: @patrickvonplaten, @lhoestq
- tensorflow: @Rocketknight1

Library:

- benchmarks: @patrickvonplaten
- deepspeed: @stas00
- ray/raytune: @richardliaw, @amogkam
- text generation: @patrickvonplaten
- tokenizers: @LysandreJik
- trainer: @sgugger
- pipelines: @LysandreJik

Documentation: @sgugger

Model hub:

- for issues with a model report at https://discuss.huggingface.co/ and tag the model's creator.

HF projects:

- datasets: [different repo](https://github.com/huggingface/datasets)
- rust tokenizers: [different repo](https://github.com/huggingface/tokenizers)

Examples:

- maintained examples (not research project or legacy): @sgugger, @patil-suraj
- research_projects/bert-loses-patience: @JetRunner
- research_projects/distillation: @VictorSanh

 -->

## Information

Model I am using (Bert, XLNet ...): Bert, but I don't think that is relevant

The problem arises when using:
* [ ] the official example scripts: (give details below)
* [x] my own modified scripts: (give details below)

The tasks I am working on is:
* [ ] an official GLUE/SQUaD task: (give the name)
* [x] my own task or dataset: (give details below)

## To reproduce

Steps to reproduce the behavior:

1. Make a `TrainingArgs` object with `eval_steps < save_steps` and `eval_strategy` and `save_strategy` both set to `"steps"`
2. Pass those to a `Trainer`
3. Model checkpoints every `eval_steps` steps, not every `save_steps` steps

<!-- If you have code snippets, error messages, stack traces please provide them here as well.
     Important! Use code tags to correctly format your code. See https://help.github.com/en/github/writing-on-github/creating-and-highlighting-code-blocks#syntax-highlighting
     Do not use screenshots, as they are hard to read and (more importantly) don't allow others to copy-and-paste your code.-->

Here is my `TrainingArguments` code:

```python
args = TrainingArguments(
    output_dir=outpath,
    save_total_limit=10,
    load_best_model_at_end=True,
    save_strategy="steps" if cli_args.save_steps is not None else "epoch",
    save_steps=cli_args.save_steps,
    evaluation_strategy="steps" if cli_args.eval_steps is not None else "epoch",
    eval_steps=cli_args.eval_steps,
    metric_for_best_model="loss",
    learning_rate=cli_args.learning_rate,
    per_device_train_batch_size=cli_args.batch_size,
    per_device_eval_batch_size=cli_args.batch_size,
    num_train_epochs=cli_args.num_train_epochs,
    weight_decay=cli_args.weight_decay,
    fp16=cli_args.fp16,
    deepspeed=deepspeed,
    local_rank=cli_args.local_rank,
)
```

with the values I am using filled in, this is:

```python
args = TrainingArguments(
    output_dir="ten_m/model",
    save_total_limit=10,
    load_best_model_at_end=True,
    save_strategy="steps",
    save_steps=6,  # for testing
    evaluation_strategy="steps",
    eval_steps=2,  # for testing
    metric_for_best_model="loss",
    learning_rate=2e-5,
    per_device_train_batch_size=16,
    per_device_eval_batch_size=16,
    num_train_epochs=3,
    weight_decay=0.01,
    fp16=False,
    deepspeed=None,
    local_rank=-1,
)
```

## Expected behavior

<!-- A clear and concise description of what you would expect to happen. -->

Well, maybe this is expected? But if so, I feel like it should be documented more obviously.

I wrote a callback to upload the saved checkpoint to GCS, but the eval step is very quick, so I was going to do those much more frequently. However, if evaluating means I have to upload to GCS, then I will evaluate less often. However, I verified that even if I don't use the GCS save callback, with the above settings, a checkpoint is saved every 2 steps, not every 6.

If this is expected behavior, then is the correct way to change it to write a Callback that `on_evaluate` sets the argument of type `transformers.TrainerControl` to have property `should_save` to `False`?

Thank you


Discussion:
You can't have different evaluation and save intervals when using `load_best_model_at_end=True` (save need to be synchronized with evaluation otherwise we can't keep track of the best model). Remove that option and you will have the evaluation and save disconnected as requested.
Thank you, that makes sense.

Also, now that I know it's related I immediately noticed
![image](https://user-images.githubusercontent.com/47401552/123009944-8a013700-d372-11eb-9120-67d8115ce581.png)

Might be worth mentioning under `save_strategy` as well? But maybe it was just me.
Sure! Do you want to make a PR with that change?
sure!
haha it's been a while! 
![image](https://user-images.githubusercontent.com/47401552/123132567-badd7c80-d403-11eb-8a90-b44c52cfe8da.png)

Oh indeed! :sweat_smile: 

Resolving patch:
diff --git a/src/transformers/training_args.py b/src/transformers/training_args.py
--- a/src/transformers/training_args.py
+++ b/src/transformers/training_args.py
@@ -171,12 +171,12 @@ class TrainingArguments:
         logging_steps (:obj:`int`, `optional`, defaults to 500):
             Number of update steps between two logs if :obj:`logging_strategy="steps"`.
         save_strategy (:obj:`str` or :class:`~transformers.trainer_utils.IntervalStrategy`, `optional`, defaults to :obj:`"steps"`):
-            The checkpoint save strategy to adopt during training. Possible values are:
+            The checkpoint save strategy to adopt during training (Note that when :obj:`load_best_model_at_end=True`,
+            this parameter is ignored and the model is saved after each evaluation). Possible values are:
 
                 * :obj:`"no"`: No save is done during training.
                 * :obj:`"epoch"`: Save is done at the end of each epoch.
                 * :obj:`"steps"`: Save is done every :obj:`save_steps`.
-
         save_steps (:obj:`int`, `optional`, defaults to 500):
             Number of updates steps before two checkpoint saves if :obj:`save_strategy="steps"`.
         save_total_limit (:obj:`int`, `optional`):


Test patch: