LLAMA for sequence classification
Author: lathashree01
Pull request: no

### System Info

@ArthurZucker and @younesbelkada

I am trying to perform sequence classification for text using LLAMA 7B model leveraging LORA training. I have 2 classes.
Tokeniser and models are loading fine. But loss is zero after the first batch; when I check the logits, of model outputs, they are nan. 

I am getting ‘NaN’ loss after the first batch.

Experiments tried (but did not work):

- Tried clip grad value and clip grad norm (values from 1.0 to 5.0)
  ```torch.nn.utils.clip_grad_value_(model.parameters(), 5.0)```
- Tried changing lr too
- loading in 8bit and float16

Any help would be greatly appreciated. Thanks

### Who can help?

_No response_

### Information

- [ ] The official example scripts
- [X] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [X] My own task or dataset (give details below)

### Reproduction

Starting seq classification training using the above code.

### Expected behavior

I see the loss calculated for the first batch only. From the next batch the logits become NaN and in turn loss and everything else is NaN

Comments:
hi @Lathashree01 
thanks for reporting, can you try to load the model in `bfloat16`, also what is the GPU hardware you are using? 

Hi @younesbelkada , 
I am using Quadro RTX 6000 node with 8 GPUs of 24GB memory.

When I run using - bfloat16, I am getting below error:
`TypeError: Got unsupported ScalarType BFloat16`
    @  preds = output['logits'].detach().cpu().numpy()


Can you replace the lines that causing that error to:
```python
            preds = output['logits'].detach().cpu().float().numpy()
            labels = b_labels.to('cpu').float().numpy()
```
from what I can they are used to compute the accuracy only so it should be fine
Also can you share your bitsandbytes version?

Hi @younesbelkada ,

I tried removing the above lines and ran with bfloat16; I see loss values normally, hope everything works as expected. 
Thank you so much. I was stuck in this and was trying out so many other things. 

Also, I changed the accuracy calculation to:
```
            preds = output['logits'].detach().cpu().to(torch.float16)
            labels = b_labels.to('cpu').numpy()
```

My bitsandBytes version is  - bitsandbytes             0.39.0
However, I do see some errors when I run `python -m bitsandbytes`. 

UserWarning: /home/anaconda3/envs/finetuneenv did not contain ['libcudart.so', 'libcudart.so.11.0', 'libcudart.so.12.0'] as expected! Searchingfurther paths...
..
things related to posixpath
....
raise RuntimeError('Something when wrong when trying to find file. Maybe you do not have a linux system?')
RuntimeError: Something when wrong when trying to find file. Maybe you do not have a linux system?

Thanks ! Does training works with PEFT + int8 as showed in the script you shared? i.e. do you get that error only if you do `python -m bitsandbytes` ?
Also, there should be no need to call `model.cuda()` after you have quantized the model

No, the training does not work fine with int8; logits and loss goes to NaN.

Above mentioned error about bitsandbytes appears when I start training (even in bfloat16)  and when I do `python -m bitsandbytes`.

Since I am not using any components on bnb and I am also not loading the model in int8, I ignored the error while training with bfloat16. Is that fine? Please let me know what I can do if that's a problem.



I see now thanks ! 
for the bnb issue probably your CUDA + bnb installation might be broken, you can post the issue on the bitsandbytes repository by stating what hardware and operating system you are using
Regarding your hotfix, which is to fine-tune in bf16 I think it is fine to do so, bfloat16 training is recommended over float16 training

> Regarding your hotfix, which is to fine-tune in bf16 I think it is fine to do so, bfloat16 training is recommended over float16 training

Oh, that's relieving. 

Thank you, I will check on bitsandbytes error.

I have the same errors.

> I have the same errors.

Mine got resolved when trained with bf16. 
If you could please elaborate on where your error is or maybe post your error trace. It will be helpful for the team or others to provide any suggestions.


Hope you're doing well! 👋 I'm currently working on a project similar to yours. Unfortunately, I cannot get model's outputs:
```
Exception: Please convert all Tensors to FakeTensors first or instantiate FakeTensorMode with 'allow_non_fake_inputs'. Found in aten.embedding.default(*(Parameter containing:
tensor([[ 9.8884e-05, -2.3329e-04,  5.8460e-04,  ..., -3.4237e-04,
          5.9724e-05, -1.1957e-04],
        [ 1.5289e-02, -1.2154e-02,  1.2512e-02,  ...,  1.3092e-02,
          7.2174e-03, -6.8045e-04],
        [ 1.7433e-03,  1.7633e-03, -1.4465e-02,  ..., -1.1444e-02,
         -1.2665e-02,  3.7289e-04],
        ...,
        [-9.0179e-03,  3.0807e-02, -1.6708e-02,  ..., -1.2680e-02,
          1.0437e-02,  4.2343e-03],
        [-1.1368e-02, -1.4801e-02, -3.5667e-03,  ...,  6.5308e-03,
         -2.2263e-02, -6.1455e-03],
        [-1.3992e-02,  1.6985e-03, -2.1469e-02,  ...,  1.3527e-02,
          2.8290e-02, -8.9111e-03]], device='cuda:0', dtype=torch.float16), FakeTensor(FakeTensor(..., device='meta', size=(1, 111), dtype=torch.int64), cuda:0), 31999), **{}) 
```
My code is really simple:
```python
import os
import sys

import fire
import torch
import transformers
from peft import PeftModel
from transformers import GenerationConfig, LlamaForSequenceClassification, LlamaTokenizer

from utils.callbacks import Iteratorize, Stream
from utils.prompter import Prompter
from datasets import load_dataset
from sklearn.metrics import precision_score, recall_score, f1_score
import csv
import gradio as gr
from peft import (
    LoraConfig,
    get_peft_model,
    get_peft_model_state_dict,
    prepare_model_for_int8_training,
    set_peft_model_state_dict,
)
if torch.cuda.is_available():
    device = "cuda"
else:
    device = "cpu"

try:
    if torch.backends.mps.is_available():
        device = "mps"
except:  # noqa: E722
    pass


def main(
    load_8bit: bool = True, 
    base_model: str = "/home/fyli/pretrain/llama-7b-hf",
    lora_weights: str = "/home/fyli/alpaca-lora/trained_weight",
    prompt_template: str = "",  # The prompt template to use, will default to alpaca.
    data_path: str = "/home/fyli/datasets/machamp/rel-heter/test.json",
):
    base_model = base_model or os.environ.get("BASE_MODEL", "")
    assert (
        base_model
    ), "Please specify a --base_model, e.g. --base_model='huggyllama/llama-7b'"

    tokenizer = LlamaTokenizer.from_pretrained(base_model)

    if device == "cuda":
        model = LlamaForSequenceClassification.from_pretrained(
            base_model,
            load_in_8bit=load_8bit,
            torch_dtype=torch.float16,
            device_map="auto",
        )
        model = PeftModel.from_pretrained(
            model,
            lora_weights,
            torch_dtype=torch.float16,
        )
    else:
        assert False, "CUDA is not available."
    print('[INFO]model: '+str(model)+"[INFO]\n")

    # unwind broken decapoda-research config
    model.config.pad_token_id = tokenizer.pad_token_id = 0  # unk
    model.config.bos_token_id = 1
    model.config.eos_token_id = 2

    if not load_8bit:
        model.half()  # seems to fix bugs for some users.

    model.eval()
    if torch.__version__ >= "2" and sys.platform != "win32":
        model = torch.compile(model)

    def evaluate(
        instruction,
        input=None,
        **kwargs,
    ):
        full_prompt = input+instruction
        tokenizer = LlamaTokenizer.from_pretrained(base_model)
        tokenizer_full_prompt = tokenizer(full_prompt, return_tensors="pt").to("cuda:0")
        # input_ids = tokenizer_full_prompt["input_ids"].to(device)

        with torch.no_grad():
            output = model(**tokenizer_full_prompt) # error occur here
            
        print("\n[INFO]output: "+str(output))


    Test = load_dataset('json',data_files=data_path)
    for test in Test["train"]:
        gen = evaluate(test["instruction"], test["input"])
        print("\n[INFO] gen: "+str(gen))

if __name__ == "__main__":
    fire.Fire(main)
```
Any help would be greatly appreciated. Thanks

@paulthewineguy can you open a different issue for this? It is not related. You can also try to get help on[ the forum ](https://discuss.huggingface.co/) as it seems you need help debugging your code. 

I have the same problem when trying to run_glue.py (for text classification) using the Lllam 7b, loaded with load_in_8bit=True and also setting training_args.fp16 = True