From the first step, loss is always 0.0 and evel_loss becomes NaN.
Author: 50516017
Pull request: no

### System Info

- `transformers` version: 4.34.0.dev0
- Platform: Linux-5.15.90.1-microsoft-standard-WSL2-x86_64-with-glibc2.35
- Python version: 3.10.12
- Huggingface_hub version: 0.16.4
- Safetensors version: 0.3.3
- Accelerate version: 0.20.3
- Accelerate config:    not found
- PyTorch version (GPU?): 2.0.1+cu117 (True)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: <fill in>
- Using distributed or parallel set-up in script?: <fill in>

### Who can help?

@pacman100,@muellerz,@younesbelkada

### Information

- [ ] The official example scripts
- [X] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [X] My own task or dataset (give details below)

### Reproduction

I want to create a chatbot with a certain character using transformer. I prepared a simple dataset for teaching tone and started training a model with the code below. However, loss became 0 and eval_loss became NaN. What approach should I take?

```
model_name = "rinna/japanese-gpt-neox-3.6b-instruction-ppo"
tokenizer = AutoTokenizer.from_pretrained(model_name, use_fast=False)
config = AutoConfig.from_pretrained(model_name,use_fast=False)
model = AutoModelForCausalLM.from_pretrained(
    model_name,
    config=config,
    device_map="auto",
    load_in_8bit=True
)
new_dataset = transform_dialogues(dataset)
VAL_SET_SIZE = int(len(new_dataset) * 0.05)
collator = InstructCollator(tokenizer)


trainer = transformers.Trainer(
    model = model,
    data_collator=collator,
    train_dataset=tokenized_train,
    eval_dataset=tokenized_val,
    args=transformers.TrainingArguments(
        num_train_epochs=10,
        learning_rate=3e-5,
        evaluation_strategy="steps",
        save_strategy="steps",
        eval_steps=eval_steps,
        save_steps=save_steps,
        per_device_train_batch_size=MICRO_BATCH_SIZE,
        per_device_eval_batch_size=MICRO_BATCH_SIZE,
        gradient_accumulation_steps=BATCH_SIZE // MICRO_BATCH_SIZE,
        dataloader_num_workers=12,
        logging_steps=logging_steps,
        output_dir=f"{output_dir}/{learn_start_time}",
        report_to="wandb",
        save_total_limit=1,
        load_best_model_at_end=True,
        greater_is_better=False,
        metric_for_best_model="eval_loss",
        auto_find_batch_size=True
    )
)
model.config.use_cache = False
    trainer.train()



```

My own collator is below. Could this be causing the problem?


```
class InstructCollator():
    def __init__(self, tokenizer, ignore_index=-100):
        self.tokenizer = tokenizer
        self.ignore_index = -100

    def __call__(self, examples):
        input_batch = []
        label_batch = []
        for example in examples:
            input_batch.append(example['input_ids'])
            label_batch.append(example['labels'])
        input_ids = pad_sequence(
            input_batch, batch_first=True, padding_value=self.tokenizer.pad_token_id
        )
        
        labels = pad_sequence(
            label_batch, batch_first=True, padding_value=self.ignore_index
        )
       
        attention_mask = input_ids.ne(self.tokenizer.pad_token_id)
        return {
            'input_ids': input_ids,
            'labels': labels,
            'attention_mask': attention_mask
        }

```


### Expected behavior

A large loss is recorded in the first step, and after that, loss = 0, evel_loss = Nan.

```
train start
  0%|                                                                                           | 0/600 [01:01<?, ?it/s]{'loss': 76656.0521, 'learning_rate': 2.9925619834710744e-05, 'epoch': 0.02}                   | 0/1210 [00:00<?, ?it/s]{'loss': 0.0, 'learning_rate': 2.9851239669421488e-05, 'epoch': 0.05}
{'loss': 0.0, 'learning_rate': 2.9776859504132232e-05, 'epoch': 0.07}
{'eval_loss': nan, 'eval_runtime': 14.0042, 'eval_samples_per_second': 7.284, 'eval_steps_per_second': 3.642, 'epoch': 0.08}
{'loss': 0.0, 'learning_rate': 2.9702479338842976e-05, 'epoch': 0.1}
{'loss': 0.0, 'learning_rate': 2.962809917355372e-05, 'epoch': 0.12}
{'loss': 0.0, 'learning_rate': 2.9553719008264463e-05, 'epoch': 0.15}
...

```

Comments:
Hi @50516017 
Thanks for the issue! 
You can't perform pure fine-tuning on 8bit/4bit models, please use LoRA adapters that you can easily attach to the model before passing it to `Trainer`:

```python
from peft import LoraConfig


model_name = "rinna/japanese-gpt-neox-3.6b-instruction-ppo"
tokenizer = AutoTokenizer.from_pretrained(model_name, use_fast=False)
config = AutoConfig.from_pretrained(model_name,use_fast=False)
model = AutoModelForCausalLM.from_pretrained(
    model_name,
    config=config,
    device_map="auto",
    load_in_8bit=True
)

lora_config = LoraConfig(
    r=8
)

model.add_adapter(lora_config)

new_dataset = transform_dialogues(dataset)
VAL_SET_SIZE = int(len(new_dataset) * 0.05)
collator = InstructCollator(tokenizer)

trainer = transformers.Trainer(
    model = model,
    data_collator=collator,
    train_dataset=tokenized_train,
    eval_dataset=tokenized_val,
    args=transformers.TrainingArguments(
        num_train_epochs=10,
        learning_rate=3e-5,
        evaluation_strategy="steps",
        save_strategy="steps",
        eval_steps=eval_steps,
        save_steps=save_steps,
        per_device_train_batch_size=MICRO_BATCH_SIZE,
        per_device_eval_batch_size=MICRO_BATCH_SIZE,
        gradient_accumulation_steps=BATCH_SIZE // MICRO_BATCH_SIZE,
        dataloader_num_workers=12,
        logging_steps=logging_steps,
        output_dir=f"{output_dir}/{learn_start_time}",
        report_to="wandb",
        save_total_limit=1,
        load_best_model_at_end=True,
        greater_is_better=False,
        metric_for_best_model="eval_loss",
        auto_find_batch_size=True
    )
)
model.config.use_cache = False
trainer.train()
```

@ArthurZucker we actually do pass a logger.info here: https://github.com/huggingface/transformers/blob/main/src/transformers/trainer.py#L408 maybe we should set a stronger check and raise a value error instead as it is very confusing for some users, what do you think also @SunMarc ?

Yes, I think that it is reasonable to use a value error in that case. 

Awesome, I'll work on it asap!

Given how many issues we have had on this, yes, let's raise an error! 

Sorry, I misunderstood. Thank you for your reply!