Repository: huggingface/transformers
Pull request: Trainer evaluation delay

# What does this PR do?

Attempts to implement #16327 by adding two new `TrainingArguments` arguments, one to delay evaluation until a certain number of epochs have passed and another for steps (which overrides the former). The aim is to delay evaluation to free up more time for training at the initial stages.

## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [x] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case. [Link](https://github.com/huggingface/transformers/issues/16327)
- [x] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/master/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/master/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?


## Who can review?
Let me know your thoughts @sgugger


Changed files:
filename: src/transformers/trainer_callback.py
patch: @@ -416,7 +416,7 @@ def on_step_end(self, args: TrainingArguments, state: TrainerState, control: Tra
             control.should_log = True
 
         # Evaluate
-        if args.evaluation_strategy == IntervalStrategy.STEPS and state.global_step % args.eval_steps == 0:
+        if args.evaluation_strategy == IntervalStrategy.STEPS and state.global_step % args.eval_steps == 0 and args.eval_delay > state.global_step:
             control.should_evaluate = True
 
         # Save
@@ -439,7 +439,7 @@ def on_epoch_end(self, args: TrainingArguments, state: TrainerState, control: Tr
             control.should_log = True
 
         # Evaluate
-        if args.evaluation_strategy == IntervalStrategy.EPOCH:
+        if args.evaluation_strategy == IntervalStrategy.EPOCH and args.eval_delay > state.epoch:
             control.should_evaluate = True
 
         # Save

filename: src/transformers/training_args.py
patch: @@ -138,6 +138,8 @@ class TrainingArguments:
             Number of predictions steps to accumulate the output tensors for, before moving the results to the CPU. If
             left unset, the whole predictions are accumulated on GPU/TPU before being moved to the CPU (faster but
             requires more memory).
+        eval_delay (`float`, *optional*):
+            Number of epochs or steps to wait for before the first evaluation can be performed, depending on the evaluation_strategy.
         learning_rate (`float`, *optional*, defaults to 5e-5):
             The initial learning rate for [`AdamW`] optimizer.
         weight_decay (`float`, *optional*, defaults to 0):
@@ -472,6 +474,11 @@ class TrainingArguments:
         metadata={"help": "Number of predictions steps to accumulate before moving the tensors to the CPU."},
     )
 
+    eval_delay: Optional[float] = field(
+        default=0,
+        metadata={"help": "Number of epochs or steps to wait for before the first evaluation can be performed, depending on the evaluation_strategy."},
+    )            
+
     learning_rate: float = field(default=5e-5, metadata={"help": "The initial learning rate for AdamW."})
     weight_decay: float = field(default=0.0, metadata={"help": "Weight decay for AdamW if we apply some."})
     adam_beta1: float = field(default=0.9, metadata={"help": "Beta1 for AdamW optimizer"})