adamw_bnb_8bit is actually Adam8bit and doesn't respect TrainingArguments weight_decay
Author: n9Mtq4
Pull request: no
Labels: name: bug

### System Info

- `transformers` version: 4.22.0.dev0
- Platform: Linux-5.15.60-1-MANJARO-x86_64-with-glibc2.36
- Python version: 3.10.5
- Huggingface_hub version: 0.9.1
- PyTorch version (GPU?): 1.13.0.dev20220903+cu116 (True)
- Tensorflow version (GPU?): 2.8.0 (True)
- Flax version (CPU?/GPU?/TPU?): 0.4.2 (gpu)
- Jax version: 0.3.10
- JaxLib version: 0.3.10
- Using GPU in script?: Yes (Ampere)
- Using distributed or parallel set-up in script?: No

### Who can help?

@sgugger

### Information

- [ ] The official example scripts
- [X] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [X] My own task or dataset (give details below)

### Reproduction

https://github.com/huggingface/transformers/blob/bb6f6d53386bf2340eead6a8f9320ce61add3e96/src/transformers/trainer.py#L1137-L1142

That snippet sets the optimizer when using adamw_bnb_8bit, but it uses Adam8bit instead of AdamW8bit. This isn't a problem in and of itself as both implementations in bitsandbytes are the same except for the weight_decay parameter (see [AdamW8bit](https://github.com/TimDettmers/bitsandbytes/blob/2e630b55f51d454f3bd723dffda68a07ef93190c/bitsandbytes/optim/adamw.py#L38-L64), [Adam8bit](https://github.com/TimDettmers/bitsandbytes/blob/2e630b55f51d454f3bd723dffda68a07ef93190c/bitsandbytes/optim/adam.py#L46-L72)).

However, the trainer doesn't correctly set the weight_decay parameter for Adam8bit leaving it at 0 and the behavior as Adam instead of AdamW.

```
from transformers import TrainingArguments, Trainer, AutoModelForMaskedLM, AutoTokenizer, DataCollatorForLanguageModeling
from datasets import Dataset

tokenizer = AutoTokenizer.from_pretrained("roberta-base")
model = AutoModelForMaskedLM.from_pretrained("roberta-base")

toy_dataset = Dataset.from_dict({'text': ['a', 'b', 'c']})
toy_dataset = toy_dataset.map(lambda examples: tokenizer(examples['text']))

args = TrainingArguments(output_dir='/tmp/outdir', optim='adamw_bnb_8bit', weight_decay=-0.1)
trainer = Trainer(args=args, model=model, train_dataset=toy_dataset,
                  data_collator=DataCollatorForLanguageModeling(tokenizer=tokenizer))
trainer.train()
```
The code works even with a negative weight_decay, indicating weight_decay isn't being set. I've confirmed this with a print statement inside the optimizer.


### Expected behavior

I expect the weight_decay parameter to be passed to the constructor of `Adam8bit`.

In the case of the reproduction code, an exception `ValueError: Invalid weight_decay value: -0.1` should be raised from the check [here](https://github.com/TimDettmers/bitsandbytes/blob/2e630b55f51d454f3bd723dffda68a07ef93190c/bitsandbytes/optim/optimizer.py#L325). But `weight_decay` isn't set correctly and remains set at 0.



Comments:
Would you like to make a PR to fix this?

Actually upon further review, the weight decay is being set correctly. I didn't fully understand the purpose of these lines that set the weight decay for only some parameters. This bypasses the non-negative check in the constructor, but does use the correct value when updating the parameters.

https://github.com/huggingface/transformers/blob/e6f221c8d4829c9a3bca699c18a32043ab21f7a0/src/transformers/trainer.py#L1057-L1066