The network does not change if I resume training from checkpoint while also providing custom optimizer and scheduler
Author: vmaryasin
Pull request: no

## Environment info

- `transformers` version: 4.3.2
- Platform: Linux-5.4.0-77-generic-x86_64-with-glibc2.10
- Python version: 3.8.8
- PyTorch version (GPU?): 1.9.0a0+df837d0 (True)
- Tensorflow version (GPU?): not installed (NA)
- Using GPU in script?: yes
- Using distributed or parallel set-up in script?: no, 1 gpu

### Who can help

@sgugger

## Information

Model I am using (Bert, XLNet ...):
FlaubertForSequenceClassification
with some serious modifications in the model

The problem arises when using:
* [x] my own modified scripts: (give details below)
Resuming training from a saved checkpoint+scheduler+optimizer with the custom optimizer-scheduler pair 

The tasks I am working on is:
* [x] my own task or dataset: (give details below)
Custom sequence classification task. 
The model and the script should be working OK, I have been continuously working on the project for a year, and one particular functionality bit of the script stopped working after the upgrade of Transformers 3.0.2 -> 4.3.2

## To reproduce
Here is the essential part of the code:
```
#...
    model = FlaubertForMultipleSequenceClassification.from_pretrained(
        model_name_or_path, config=config, cache_dir=model_args.cache_dir
    )

    # 4. Training
    trainer = Trainer(
        model=model,
        args=training_args,
        train_dataset=train_dataset,
        eval_dataset=dev_dataset,
        compute_metrics=metrics.compute_multihead_metrics,
#        optimizers=optimizers,
        data_collator=data_collator,
    )

    num_training_steps = training_args.num_train_epochs * len(train_dataset) / training_args.per_device_train_batch_size

    trainer.create_optimizer_and_scheduler(num_training_steps=int(num_training_steps))

#    trainer.optimizer = None

    # Check if scheduler and optimizer states should be restored or reinitialized
    if model_args.resume_training:
        log.info("Resuming training from {}".format(model_name_or_path))
        trainer.train(model_name_or_path)
    else:
        log.info("Starting a new training from {}".format(model_name_or_path))
        trainer.train()
```

## Expected behaviour
The network does not change if I resume training from checkpoint (i.e. when loading a saved optimizer and lr_scheduler), while also providing a custom optimizer and scheduler. It means that the weights of the model don't change at all, consecutive saved pytorch_model.bin files match exactly, and so do logged metrics and eval_losses. (The logged learning rate behaves as expected according to the set lr_scheduler.)

I expect the weights of the model to change upon training the model.

## Additional information

In the code above I may resume the entire training run, or simply initialize the model's weight. And it is the former, which does not work.

I know that the manual call to `trainer.create_optimizer_and_scheduler()` is very strange. In my real code, I create an optimizer+scheduler tuple and pass it to `Trainer` constructor as in the commented line above. This manual call is just to emulate bugging behaviour without providing my own functions. I.e. even the default functions behave not like expected.

The main thing is that if I don't provide the optimizer (for example by uncommenting `trainer.optinizer=None`), the training goes on fine. If I don't load the optimizer and just initialize the weights of the model - too. But using custom `lr_scheduler` is required by my code, so I need to specify the optimizer.

Now, one more clue, this approach worked in my previous transformers v3.0.2. The code was the same, modulo some changes due to API upgrade: I defined my tuple of optimizers, initialized `Trainer(optimizers=optimizers)`, then started the training exactly the same way as above, and it worked. I know that 4.3.2 is not the most recent version, so I am sorry if the issue is already solved in newer versions.

Sorry for the long post, I tried to provide all information that I managed to find out myself, but I am giving up. Could someone point me towards the solution to my problem? I would be very grateful for any ideas and hints. Also sorry for not providing a minimal working example. There are a lot of small things to unwind in my model, but I will certainly do this if absolutely necessary.


Comments:
Yes, this has been solved since then I believe, as the `Trainer` used to recreate the model when loading a checkpoint (thus disconnecting it from a custom optimizer). This is no longer the case, so I think your code will work if you upgrade to the latest version of Transformers.

Thank you very much for the reply. 
Indeed, this has been corrected and resuming the training works in 4.8.2.