Issue with trainer.py Line#1022,1025,1035,1043,1051,1059,1061
Author: avmodi
Pull request: no
Labels: name: bug

### System Info

```shell
Copy-and-paste the text below in your GitHub issue and FILL OUT the two last points.

- `transformers` version: 4.18.0
- Platform: Linux-4.14.252-131.483.amzn1.x86_64-x86_64-with-glibc2.9
- Python version: 3.6.13
- Huggingface_hub version: 0.4.0
- PyTorch version (GPU?): 1.5.0 (True)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: <fill in>
- Using distributed or parallel set-up in script?: <fill in>
```


### Who can help?

@sgugger 



### Information

- [ ] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

```python
import os
os.environ["CUDA_DEVICE_ORDER"]="PCI_BUS_ID" 
os.environ["CUDA_VISIBLE_DEVICES"]="0"
os.environ['COMET_MODE'] = 'DISABLED'
logging.basicConfig(
    format="%(asctime)s - %(levelname)s - %(name)s -   %(message)s",
    level=logging.INFO,
    datefmt="%m/%d/%Y %H:%M:%S",
    handlers=[stream_handler, file_handler]
)
logger = logging.getLogger(__name__)
training_args.log_level = 'INFO'

trainer = Trainer(
            model=model,
            args=training_args,
            train_dataset=train_dataset,
            eval_dataset=val_dataset,
            compute_metrics=build_compute_metrics_fn(data_args.task))

if training_args.do_train:
    trainer.train(model_path=model_args.model_name_or_path if os.path.isdir(model_args.model_name_or_path) else None)
    trainer.save_model()

```

---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
<ipython-input-29-feef8f1bc697> in <module>
      1 if training_args.do_train:
----> 2     trainer.train(model_path=model_args.model_name_or_path if os.path.isdir(model_args.model_name_or_path) else None)
      3     trainer.save_model()

~/anaconda3/envs/pytorch_p36/lib/python3.6/site-packages/transformers/trainer.py in train(self, resume_from_checkpoint, trial, ignore_keys_for_eval, **kwargs)
   1262             self.lr_scheduler = lr_scheduler
   1263         elif not delay_optimizer_creation:
-> 1264             self.create_optimizer_and_scheduler(num_training_steps=max_steps)
   1265 
   1266         self.state = TrainerState()

~/anaconda3/envs/pytorch_p36/lib/python3.6/site-packages/transformers/trainer.py in create_optimizer_and_scheduler(self, num_training_steps)
    827         `create_scheduler`) in a subclass.
    828         """
--> 829         self.create_optimizer()
    830         self.create_scheduler(num_training_steps=num_training_steps, optimizer=self.optimizer)
    831 

~/anaconda3/envs/pytorch_p36/lib/python3.6/site-packages/transformers/trainer.py in create_optimizer(self)
    851             ]
    852 
--> 853             optimizer_cls, optimizer_kwargs = Trainer.get_optimizer_cls_and_kwargs(self.args)
    854 
    855             if self.sharded_ddp == ShardedDDPOption.SIMPLE:

~/anaconda3/envs/pytorch_p36/lib/python3.6/site-packages/transformers/trainer.py in get_optimizer_cls_and_kwargs(args)
    912                 raise ValueError("Trainer tried to instantiate apex FusedAdam but apex is not installed!")
    913         else:
--> 914             raise ValueError(f"Trainer cannot instantiate unsupported optimizer: {args.optim}")
    915         return optimizer_cls, optimizer_kwargs
    916 

**ValueError: Trainer cannot instantiate unsupported optimizer: adamw_hf**



### Expected behavior

```shell
if args.optim == OptimizerNames.ADAFACTOR:
            optimizer_cls = Adafactor
            optimizer_kwargs.update({"scale_parameter": False, "relative_step": False})
        elif args.optim == OptimizerNames.ADAMW_HF:
            from .optimization import AdamW

            optimizer_cls = AdamW
            optimizer_kwargs.update(adam_kwargs)
        elif args.optim == OptimizerNames.ADAMW_TORCH:
            from torch.optim import AdamW

            optimizer_cls = AdamW
            optimizer_kwargs.update(adam_kwargs)
        elif args.optim == OptimizerNames.ADAMW_TORCH_XLA:
            try:
                from torch_xla.amp.syncfree import AdamW

                optimizer_cls = AdamW
                optimizer_kwargs.update(adam_kwargs)
            except ImportError:
                raise ValueError("Trainer failed to import syncfree AdamW from torch_xla.")
        elif args.optim == OptimizerNames.ADAMW_APEX_FUSED:
            try:
                from apex.optimizers import FusedAdam

                optimizer_cls = FusedAdam
                optimizer_kwargs.update(adam_kwargs)
            except ImportError:
                raise ValueError("Trainer tried to instantiate apex FusedAdam but apex is not installed!")
        elif args.optim == OptimizerNames.ADAMW_BNB:
            try:
                from bitsandbytes.optim import Adam8bit

                optimizer_cls = Adam8bit
                optimizer_kwargs.update(adam_kwargs)
            except ImportError:
                raise ValueError("Trainer tried to instantiate bnb Adam8bit but bnb is not installed!")
        elif args.optim == OptimizerNames.SGD:
            optimizer_cls = torch.optim.SGD
        elif args.optim == OptimizerNames.ADAGRAD:
            optimizer_cls = torch.optim.Adagrad
        else:
            raise ValueError(f"Trainer cannot instantiate unsupported optimizer: {args.optim}")
        return optimizer_cls, optimizer_kwargs


We can use OptimizerNames.ADAMW_HF.name == args.optim 
instead of OptimizerNames.ADAMW_HF == args.optim
```


Comments:
Please include a *complete* reproducer for the bug you are raising. Your code sample does not include many things, in particular the `TrainingArguments`.

```python
import logging
import os
from statistics import mean, stdev
import sys
from typing import Callable, Dict

import numpy as np
from pprint import pformat
from scipy.special import softmax
import torch

from transformers import (
    AutoTokenizer,
    AutoConfig,
    HfArgumentParser,
    Trainer,
    EvalPrediction,
    set_seed
)

from utils import calc_classification_metrics, calc_regression_metrics, create_dir_if_not_exists
from data  import load_data_from_folder
from model import TabularConfig, AutoModelWithTabular
from multimodal_args import MultiModalDataArguments, ModelArguments, MultiModalTrainingArguments
from transformers.debug_utils import DebugOption
from transformers.training_args import OptimizerNames
print(MultiModalDataArguments)
parser = HfArgumentParser((ModelArguments,MultiModalDataArguments,MultiModalTrainingArguments))
model_args, data_args, training_args = parser.parse_json_file(json_file=os.path.abspath('config.json'))

# training_args.debug = [DebugOption.UNDERFLOW_OVERFLOW]
# training_args.optim = OptimizerNames.ADAMW_HF

stream_handler = logging.StreamHandler(sys.stdout)
file_handler = logging.FileHandler(filename=os.path.join(training_args.output_dir, 'eval_log.txt'),
                                       mode='w+')
logging.basicConfig(
    format="%(asctime)s - %(levelname)s - %(name)s -   %(message)s",
    level=logging.INFO,
    datefmt="%m/%d/%Y %H:%M:%S",
    handlers=[stream_handler, file_handler]
)
logger = logging.getLogger(__name__)


set_seed(training_args.seed)
if (
        os.path.exists(training_args.output_dir)
        and os.listdir(training_args.output_dir)
        and training_args.do_train
        and not training_args.overwrite_output_dir
    ):
        raise ValueError(
            f"Output directory ({training_args.output_dir}) already exists and is not empty. Use --overwrite_output_dir to overcome."
        )
        

create_dir_if_not_exists(training_args.output_dir)

tokenizer = AutoTokenizer.from_pretrained(
        model_args.tokenizer_name if model_args.tokenizer_name else model_args.model_name_or_path,
        cache_dir=model_args.cache_dir,
    )
train_dataset, val_dataset, test_dataset = load_data_from_folder(
            data_args.data_path,
            data_args.text_cols,
            tokenizer,
            label_col=data_args.label_col,
            label_list=data_args.label_list,
            categorical_cols=data_args.cat_cols,
            numerical_cols=data_args.num_cols,
            categorical_encode_type=data_args.categorical_encoding,
            numerical_transformer_method=data_args.numerical_encoding,
            sep_text_token_str=tokenizer.sep_token,
            do_train = training_args.do_train,
            do_eval = training_args.do_eval,
            do_predict = training_args.do_predict
        )
train_datasets = [train_dataset]
val_datasets = [val_dataset]
test_datasets = [test_dataset]

def build_compute_metrics_fn(task_name: str) -> Callable[[EvalPrediction], Dict]:
    def compute_metrics_fn(p: EvalPrediction):
        if task_name == "classification":
            preds_labels = np.argmax(p.predictions, axis=1)
            if p.predictions.shape[-1] == 2:
                pred_scores = softmax(p.predictions, axis=1)[:, 1]
            else:
                pred_scores = softmax(p.predictions, axis=1)
            return calc_classification_metrics(pred_scores, preds_labels,
                                               p.label_ids)
        elif task_name == "regression":
            preds = np.squeeze(p.predictions)
            return calc_regression_metrics(preds, p.label_ids)
        else:
            return {}
    return compute_metrics_fn

config = AutoConfig.from_pretrained(
            model_args.config_name if model_args.config_name else model_args.model_name_or_path,
            cache_dir=model_args.cache_dir,
        )
tabular_config = TabularConfig(num_labels=len(data_args.label_list),
                               cat_feat_dim=test_dataset.cat_feats.shape[
                                   1] if test_dataset.cat_feats is not None else 0,
                               numerical_feat_dim=test_dataset.numerical_feats.shape[
                                   1] if test_dataset.numerical_feats is not None else 0,
                               **vars(data_args))
config.tabular_config = tabular_config

model = AutoModelWithTabular.from_pretrained(
    model_args.config_name if model_args.config_name else model_args.model_name_or_path,
    config=config,
    cache_dir=model_args.cache_dir
)

import os
os.environ["CUDA_DEVICE_ORDER"]="PCI_BUS_ID"   # see issue #152
os.environ["CUDA_VISIBLE_DEVICES"]="0"
os.environ['COMET_MODE'] = 'DISABLED'
logging.basicConfig(
    format="%(asctime)s - %(levelname)s - %(name)s -   %(message)s",
    level=logging.INFO,
    datefmt="%m/%d/%Y %H:%M:%S",
    handlers=[stream_handler, file_handler]
)
logger = logging.getLogger(__name__)
training_args.log_level = 'INFO'

trainer = Trainer(
            model=model,
            args=training_args,
            train_dataset=train_dataset,
            eval_dataset=val_dataset,
            compute_metrics=build_compute_metrics_fn(data_args.task))

if training_args.do_train:
    trainer.train(model_path=model_args.model_name_or_path if os.path.isdir(model_args.model_name_or_path) else None)
    trainer.save_model()
```

Config

```json
{"text_cols" : [],
"num_cols" : [],
"label_col" : "label",
"label_list" : [],
"model_name_or_path" : "bert-base-uncased",
"data_path" : "input",
"combine_feat_method" : "gating",
"task" : "classification",
"create_folds" : false,
"num_classes" : 3,
"numerical_transformer_method" : "min_max",
"output_dir" : "run/output",
"logging_dir" : "run/log",
"overwrite_output_dir" : true,
"do_train" : true,
"do_eval" : true,
"do_predict" : true,
"per_device_train_batch_size" : 256,
"per_device_eval_batch_size" : 256,
"num_train_epochs" : 10,
"evaluate_during_training" : true,
"logging_steps" : 25,
"eval_steps" : 50,
"save_steps" : 50,
"log_level"  : "INFO",
"report_to" : []
}
```
The training_args.optim = "adamw_hf" ( default choice )

Same as in the other issue, this is not a reproducer. We have no idea how you defined the class `MultiModalTrainingArguments` in particular, which is probably where the problem lies.

Please use the [forums](https://discuss.huggingface.co/) to get helps from the community to debug your code (and find a small reproducer of the bug if it is indeed a bug in the lbirary).

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.