Training arguments are not applied when resuming from a checkpoint
Author: jonflynng
Pull request: no
Labels: name: Feature request

### System Info

I noticed when resuming the training of a model from a checkpoint changing properties like `save_steps` and `per_device_train_batch_size` has no effect. I'm wondering if there's something syntactically wrong here or technically the config of the model checkpoint overrides everything? I've seen a thread started about this [here](https://discuss.huggingface.co/t/if-train-resume-from-checkpoint-cant-change-trainerarguments/70715/4)

```
import transformers
from datetime import datetime

tokenizer.pad_token = tokenizer.eos_token

learning_rate = 5e-5  
warmup_steps = 100

gradient_accumulation_steps = 2  

trainer = transformers.Trainer(
    model=model,
    callbacks=[upload_checkpoint_callback],
    train_dataset=tokenized_train_dataset,
    eval_dataset=tokenized_val_dataset,
    args=transformers.TrainingArguments(
        output_dir=output_dir,
        warmup_steps=warmup_steps,
        per_device_train_batch_size=8,
        gradient_checkpointing=True,
        gradient_accumulation_steps=gradient_accumulation_steps,
        max_steps=5000,
        learning_rate=learning_rate,
        logging_steps=10,
        fp16=True,
        optim="paged_adamw_8bit",
        logging_dir="/content/logs",       
        save_strategy="steps",      
        save_steps=10,              
        evaluation_strategy="steps", 
        eval_steps=10,               
        load_best_model_at_end=True,
        report_to="wandb",           
        run_name=f"{run_name}-{datetime.now().strftime('%Y-%m-%d-%H-%M')}"          # Name of the W&B run (optional)
    ),
    data_collator=transformers.DataCollatorForLanguageModeling(tokenizer, mlm=False),
)

model.config.use_cache = False
trainer.train(resume_from_checkpoint="/content/latest_checkpoint/")
```

### Who can help?

_No response_

### Information

- [ ] The official example scripts
- [X] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [X] My own task or dataset (give details below)

### Reproduction

Train a model from a checkpoint and adjust some training arguments, these won't have any effect

### Expected behavior

The training arguments should have effect when training off a checkpointed model

Comments:
https://github.com/huggingface/transformers/issues/7198

"That feature is not supported, you should resume training with the exact same hyperparameters or start a new training if you want to change them."

I understand certain hyperparameters that shouldn't be changed (or would be difficult to change within the library) but shouldn't things like `save_steps` be possible. This could be raised as a feature request?

FYI @muellerzr, might be worth raising an issue / warning that the new setting will not be used! 

> FYI @muellerzr, might be worth raising an issue / warning that the new setting will not be used!

@ArthurZucker  I can raise a PR to ensure the training_args override whatever  is in `traning_args.json` unless this is by design?

I am not sure TBH. Both choices would make sense, let's aime for the least breaking : a warning 

Have created a PR here: https://github.com/huggingface/transformers/pull/29255