can't use load_in_8bit in rinna/japanese-gpt-neox-3.6b-instruction-ppo
Author: 50516017
Pull request: no

### System Info

Hi!

I want to use "rinna/japanese-gpt-neox-3.6b-instruction-ppo" model & want to use load_in_8bit.
but AutoModelForCausalLM.from_pretrained throw error
ValueError: The model you want to train is loaded in 8-bit precision. Training an 8-bit model is not supported yet.

when i tried training without load_in_8bit ,RuntimeError: No executable batch size found, reached zero. occured.
Is this happening due to insufficient memory?

I am a beginner who is new to deep learning and environment building.

I would like to fine tune a model on the following PC spec.
Does anyone have a solution to this?
please help me.


## enviroment 
```
OS: ubuntu22.04 on windows11 using WSL2
GPU:NVIDIA Geforce RTX 4060Ti(16GB)
CPU:AMN Rzen 5 4500 6-core (16GB)

nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2022 NVIDIA Corporation
Built on Wed_Sep_21_10:33:58_PDT_2022
Cuda compilation tools, release 11.8, V11.8.89
Build cuda_11.8.r11.8/compiler.31833905_0

+---------------------------------------------------------------------------------------+
| NVIDIA-SMI 535.104.05             Driver Version: 537.13       CUDA Version: 12.2     |
|-----------------------------------------+----------------------+----------------------+
| GPU  Name                 Persistence-M | Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp   Perf          Pwr:Usage/Cap |         Memory-Usage | GPU-Util  Compute M. |
|                                         |                      |               MIG M. |
|=========================================+======================+======================|
|   0  NVIDIA GeForce RTX 4060 Ti     On  | 00000000:01:00.0  On |                  N/A |
|  0%   43C    P8              10W / 165W |    478MiB / 16380MiB |     10%      Default |
|                                         |                      |                  N/A |
+-----------------------------------------+----------------------+----------------------+

+---------------------------------------------------------------------------------------+
| Processes:                                                                            |
|  GPU   GI   CI        PID   Type   Process name                            GPU Memory |
|        ID   ID                                                             Usage      |
|=======================================================================================|
|    0   N/A  N/A       276      G   /Xwayland                                 N/A      |
```




### Who can help?

@pacman100
@muellerz

### Information

- [ ] The official example scripts
- [X] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [X] My own task or dataset (give details below)

### Reproduction

1. running Training_code
## source code
```
model_name = "rinna/japanese-gpt-neox-3.6b-instruction-ppo"

tokenizer = AutoTokenizer.from_pretrained(model_name, use_fast=False)
config = AutoConfig.from_pretrained(model_name,use_fast=False)

model = AutoModelForCausalLM.from_pretrained(
    model_name,
    config=config,
    device_map="auto",
    #torch_dtype=torch.bfloat16,
    load_in_8bit=True
)

eval_steps = 11
save_steps = 33
logging_steps = 3
MICRO_BATCH_SIZE = 2
BATCH_SIZE = 16

trainer = transformers.Trainer(
    
    model = model,
    data_collator=collator,
    train_dataset=tokenized_train,
    eval_dataset=tokenized_val,
    args=transformers.TrainingArguments(
        num_train_epochs=1,
        #learning_rate=3e-5,
        evaluation_strategy="steps",
        save_strategy="steps",
        eval_steps=eval_steps,
        save_steps=save_steps,
        #warmup_ratio=0.15,
        per_device_train_batch_size=MICRO_BATCH_SIZE,
        per_device_eval_batch_size=MICRO_BATCH_SIZE,
        gradient_accumulation_steps=BATCH_SIZE // MICRO_BATCH_SIZE,
        #bf16=True,
        dataloader_num_workers=12,
        logging_steps=logging_steps,
        output_dir="./output",
        #report_to="wandb",
        save_total_limit=1,
        load_best_model_at_end=True,
        greater_is_better=False,
        metric_for_best_model="eval_loss",
        fp16=True,
        auto_find_batch_size=True
    )
)
```

### Expected behavior

## error message(using load_in_8bit)
```
bin /venv/lib/python3.10/site-packages/bitsandbytes/libbitsandbytes_cpu.so
/home/venv/lib/python3.10/site-packages/bitsandbytes/cuda_setup/main.py:149: UserWarning: WARNING: The following directories listed in your path were found to be non-existent: {PosixPath('unix')}
  warn(msg)
/venv/lib/python3.10/site-packages/bitsandbytes/cuda_setup/main.py:149: UserWarning: Found duplicate ['libcudart.so', 'libcudart.so.11.0', 'libcudart.so.12.0'] files: {PosixPath('/usr/local/cuda/lib64/libcudart.so'), PosixPath('/usr/local/cuda/lib64/libcudart.so.11.0')}.. We'll flip a coin and try one of these, in order to fail forward.
Either way, this might cause trouble in the future:
If you get `CUDA error: invalid device function` errors, the above might be the cause and the solution is to make sure only one ['libcudart.so', 'libcudart.so.11.0', 'libcudart.so.12.0'] in the paths that we search based on your env.
  warn(msg)
CUDA exception! Error code: no CUDA-capable device is detected
CUDA exception! Error code: initialization error
CUDA SETUP: CUDA runtime path found: /usr/local/cuda/lib64/libcudart.so
/venv/lib/python3.10/site-packages/bitsandbytes/cuda_setup/main.py:149: UserWarning: WARNING: No GPU detected! Check your CUDA paths. Proceeding to load CPU-only library...
  warn(msg)
CUDA SETUP: Detected CUDA version 118
CUDA SETUP: Loading binary /venv/lib/python3.10/site-packages/bitsandbytes/libbitsandbytes_cpu.so...
0
The model weights are not tied. Please use the `tie_weights` method before using the `infer_auto_device` function.
/venv/lib/python3.10/site-packages/transformers/models/t5/tokenization_t5.py:226: UserWarning: This sequence already has 
. In future versions this behavior may lead to duplicated eos tokens being added.
  warnings.warn(
Traceback (most recent call last):
  File "/source.py", line 207, in <module>
    trainer = transformers.Trainer(
  File "/venv/lib/python3.10/site-packages/transformers/trainer.py", line 362, in __init__
    raise ValueError(
ValueError: The model you want to train is loaded in 8-bit precision. Training an 8-bit model is not supported yet.

```


## error message( without load_in_8bit)
```
bin /lib/python3.10/site-packages/bitsandbytes/libbitsandbytes_cpu.so
/lib/python3.10/site-packages/bitsandbytes/cuda_setup/main.py:149: UserWarning: WARNING: The following directories listed in your path were found to be non-existent: {PosixPath('unix')}
  warn(msg)
/lib/python3.10/site-packages/bitsandbytes/cuda_setup/main.py:149: UserWarning: Found duplicate ['libcudart.so', 'libcudart.so.11.0', 'libcudart.so.12.0'] files: {PosixPath('/usr/local/cuda/lib64/libcudart.so.11.0'), PosixPath('/usr/local/cuda/lib64/libcudart.so')}.. We'll flip a coin and try one of these, in order to fail forward.
Either way, this might cause trouble in the future:
If you get `CUDA error: invalid device function` errors, the above might be the cause and the solution is to make sure only one ['libcudart.so', 'libcudart.so.11.0', 'libcudart.so.12.0'] in the paths that we search based on your env.
  warn(msg)
CUDA exception! Error code: no CUDA-capable device is detected
CUDA exception! Error code: initialization error
CUDA SETUP: CUDA runtime path found: /usr/local/cuda/lib64/libcudart.so.11.0
/lib/python3.10/site-packages/bitsandbytes/cuda_setup/main.py:149: UserWarning: WARNING: No GPU detected! Check your CUDA paths. Proceeding to load CPU-only library...
  warn(msg)
CUDA SETUP: Detected CUDA version 118
CUDA SETUP: Loading binary /lib/python3.10/site-packages/bitsandbytes/libbitsandbytes_cpu.so...

  warnings.warn(
Using cuda_amp half precision backend
/lib/python3.10/site-packages/transformers/optimization.py:306: FutureWarning: This implementation of AdamW is deprecated and will be removed in a future version. Use the PyTorch implementation torch.optim.AdamW instead, or set `no_deprecation_warning=True` to disable this warning
  warnings.warn(
***** Running training *****
  Num examples = 226
  Num Epochs = 1
  Instantaneous batch size per device = 2
  Total train batch size (w. parallel, distributed & accumulation) = 16
  Gradient Accumulation steps = 8
  Total optimization steps = 14
  Number of trainable parameters = 3607245312
Automatic Weights & Biases logging enabled, to disable set os.environ["WANDB_DISABLED"] = "true"

  0%|                                                                                            | 0/14 [00:00<?, ?it/s]***** Running training *****
  Num examples = 226
  Num Epochs = 1
  Instantaneous batch size per device = 2
  Total train batch size (w. parallel, distributed & accumulation) = 16
  Gradient Accumulation steps = 8
  Total optimization steps = 28
  Number of trainable parameters = 3607245312
  0%|                                                                                            | 0/14 [00:38<?, ?it/s]Traceback (most recent call last):                                                               | 0/28 [00:00<?, ?it/s]  File "/home/shimizu/create_LLM/macin/work2/prepare_dataset.py", line 237, in <module>
    trainer.train()
  File "/lib/python3.10/site-packages/transformers/trainer.py", line 1543, in train
    return inner_training_loop(
  File "/lib/python3.10/site-packages/accelerate/utils/memory.py", line 130, in decorator
    raise RuntimeError("No executable batch size found, reached zero.")
RuntimeError: No executable batch size found, reached zero.
```

Comments: