How to use its own custom Optimizer (GLUE Example)
Author: BelhalK
Pull request: no

# ❓ Questions & Help
I am referring to the example on GLUE text-classification/run_glue.py.
I would like to change the default optimizer (which I believe it to be ADAMW) to my own one.
Should be in the imported Trainer or TrainingArguments, but I have not found an example doing so.

Is it possible with any optimizer? (as long as they are written as a Torch Optimizer of course)
Thanks a lot!

Comments:
Hi, the Trainer takes an optional `optimizers` arg which is a two-tuple of (optimizer, scheduler):

https://github.com/huggingface/transformers/blob/95a26fcf2d8d7072e4e63129cea8605f756bba1d/src/transformers/trainer.py#L152-L181

I created the optimizers using the same way. But the model is not getting trained, because the training loss is not decreasing with time.

import transformers
grouped_params = model.parameters()
optimizer=transformers.AdamW(grouped_params,
                             lr=0.00025)
scheduler=transformers.get_cosine_schedule_with_warmup(optimizer=optimizer, 
                                                       num_warmup_steps=2000, 
                                                       num_training_steps=60000)
optimizers = optimizer, scheduler

training_args = TrainingArguments(
    output_dir="./test_checkpoint",
    overwrite_output_dir=True,
    num_train_epochs=15,
    per_device_train_batch_size=8,
    save_steps=1000,
    save_total_limit=3,
    logging_steps=50,
    dataloader_drop_last=True,
)

trainer = Trainer(
    model=model,
    args=training_args,
    data_collator=data_collator,
    train_dataset=dataset,
    prediction_loss_only=True,
    optimizers=optimizers
)

Same issue here. Any solution?

I have the same issue as well.


> I have the same issue as well.

Hey Chris, I'd like to know how you actually found out that you cannot pass the custom optimizer to Trainer?

In my case, I create custom optim and lr scheduler by:

```python
training_steps = training_step_calc(                  # self-defined func
    encoded_dataset['train'],
    PER_DEVICE_TRAIN_BATCH_SIZE,
    gpu_count,
    NUM_TRAIN_EPOCHS
)

warmup_steps = (training_steps * WARMUP_RATIO)

optimizer = bnb.optim.AdamW8bit(
    model.parameters(),
    lr=LEARNING_RATE,
    weight_decay=WEIGHT_DECAY,    
)

scheduler = transformers.get_cosine_schedule_with_warmup(
    optimizer, 
    num_warmup_steps=warmup_steps, 
    num_training_steps=training_steps, 
```
Then I didn't specify the optim-related args in `TrainingArguments()`:

```python
training_args = TrainingArguments(
    output_dir=SAVE_PATH,
    # basic hp
    num_train_epochs=NUM_TRAIN_EPOCHS,
    # auto_find_batch_size=True,
    per_device_train_batch_size=PER_DEVICE_TRAIN_BATCH_SIZE,
    per_device_eval_batch_size=PER_DEVICE_EVAL_BATCH_SIZE,
    gradient_checkpointing=GRADIENT_CHECKPOINTING,
    # optim-related, comment if use a custom optimiser
    # optim="adamw_hf" if OPTIM is None else OPTIM,
    # learning_rate=LEARNING_RATE,
    # weight_decay=WEIGHT_DECAY,    
    # lr_scheduler_type=LR_SCHEDULER_TYPE,
    # warmup_ratio=WARMUP_RATIO,
    # data related
    data_seed=DATA_SEED,
    dataloader_num_workers=DATALOADER_NUM_WORKERS,
```
After passing all the parameters to `Trainer()`, it ended up with this:
```python
trainer = Trainer(
    model = model,
    tokenizer = tokenizer,
    args = training_args,
    train_dataset = encoded_dataset["train"],
    eval_dataset = encoded_dataset["test"],
    data_collator = data_collator,
    optimizers=(optimizer, scheduler),
    compute_metrics = compute_metrics,
)
```
When I check the `trainer.args`, the optim in the args seems to be the default, and so it's shown on wandb run page. But the  `trainer.optimizer` is shown as:

```python
AdamW8bit (
Parameter Group 0
    betas: (0.9, 0.999)
    eps: 1e-08
    initial_lr: 7e-06
    lr: 0.0
    weight_decay: 0.01
)
```
In fact, by manipulating the optimizer settings of the Trainer, even though the default adamw_hf optimizer is still displayed in the args in wandb and trainer, the optimizer is overridden by the custom optimizer and scheduler at training time.