Text generation does not complete after tuning with QLora
Author: 50516017
Pull request: no

### System Info

I use Google colab V100 instance
`
 `transformers` version: 4.33.2
- Platform: Linux-5.15.120+-x86_64-with-glibc2.35
- Python version: 3.10.12
- Huggingface_hub version: 0.17.2
- Safetensors version: 0.3.3
- Accelerate version: 0.23.0
- Accelerate config: 	not found
- PyTorch version (GPU?): 2.0.1+cu118 (True)
- Tensorflow version (GPU?): 2.13.0 (True)
- Flax version (CPU?/GPU?/TPU?): 0.7.2 (gpu)
- Jax version: 0.4.14
- JaxLib version: 0.4.14
- Using GPU in script?: <fill in>
- Using distributed or parallel set-up in script?: <fill in>
`

### Who can help?

@younesbelkada,@ArthurZucker,@gante

### Information

- [ ] The official example scripts
- [X] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [X] My own task or dataset (give details below)

### Reproduction

I fine-tuned "elyza/ELYZA-japanese-Llama-2-7b-fast-instruct" using QLora. The train loss and eval loss were also output within a range that seemed to be OK. After fine-tuning, I tried to generate sentences based on the user's input using the code below, but model.generate did not complete even after more than 10 minutes. When I did model.generate without QLora, a response was returned immediately. Was my training method bad? Is there any way to find the cause?

toraining code 
```
model_name= "elyza/ELYZA-japanese-Llama-2-7b-fast-instruct"
tokenizer = AutoTokenizer.from_pretrained(model_name, use_fast=True)
config = AutoConfig.from_pretrained(model_name,use_fast=True)
bnb_config = BitsAndBytesConfig(
        load_in_4bit=True,
        bnb_4bit_use_double_quant=True,
        bnb_4bit_quant_type="nf4",
        bnb_4bit_compute_dtype=torch.bfloat16
    )
model = AutoModelForCausalLM.from_pretrained(
    model_name,
    config=config,
    device_map="auto",
    #load_in_8bit=True,
    quantization_config=bnb_config
    
)
tokenized_train = tokenize_dataset(train_data, tokenizer)
tokenized_val = tokenize_dataset(val_data, tokenizer)
collator = InstructCollator(tokenizer)
loader = DataLoader(tokenized_train, collate_fn=collator, batch_size=8, shuffle=True)

eval_steps = 10
save_steps = 30
logging_steps = 3
MICRO_BATCH_SIZE = 2
BATCH_SIZE = 32

def find_all_linear_names(model): 
    cls = bnb.nn.Linear4bit # if args.bits == 4 else (bnb.nn.Linear8bitLt if args.bits == 8 else torch.nn.Linear)
    lora_module_names = set()
    for name, module in model.named_modules():
        if isinstance(module, cls):
            names = name.split('.')
            lora_module_names.add(names[0] if len(names) == 1 else names[-1])


    if 'lm_head' in lora_module_names: # needed for 16-bit
        lora_module_names.remove('lm_head')
    return list(lora_module_names)

linear_name = find_all_linear_names(model) 
lora_config = LoraConfig(
    r= 8,
    lora_alpha=16,
    #target_modules=["query_key_value"],
    lora_dropout=0.05,
    bias="none",
    task_type=TaskType.CAUSAL_LM,
    target_modules = linear_name
)

model = prepare_model_for_int8_training(model)
model = get_peft_model(model, lora_config)

trainer = transformers.Trainer(

    model = model,
    data_collator=collator,
    train_dataset=tokenized_train,
    eval_dataset=tokenized_val,
    args=transformers.TrainingArguments(
        num_train_epochs=7,
        learning_rate=3e-5,
        evaluation_strategy="steps",
        save_strategy="steps",
        eval_steps=eval_steps,
        #bf16=True,
        save_steps=save_steps,
        per_device_train_batch_size=MICRO_BATCH_SIZE,
        per_device_eval_batch_size=MICRO_BATCH_SIZE,
        gradient_accumulation_steps=BATCH_SIZE // MICRO_BATCH_SIZE,
        dataloader_num_workers=12,
        logging_steps=logging_steps,
        output_dir=f"{output_dir}/{learn_start_time}",
        report_to="wandb",
        save_total_limit=1,
        load_best_model_at_end=True,
        greater_is_better=False,
        metric_for_best_model="eval_loss",
        auto_find_batch_size=True
    )
)

model.config.use_cache = False
trainer.train()
model.config.use_cache = True

```

inference code

```
def generate(text,tokenizer,model,history=None):
 

  prompt_no_output = "instruction"
  token_ids = tokenizer.encode(prompt_no_output, add_special_tokens=False, return_tensors="pt")
  with torch.no_grad():
      output_ids = model.generate(
        input_ids=token_ids.to(model.device),
        do_sample=True,
        max_new_tokens=2000,
        # temperature=temperature,
        top_p=0.95,
        repetition_penalty=1.0,
        pad_token_id=tokenizer.pad_token_id,
        bos_token_id=tokenizer.bos_token_id,
        eos_token_id=tokenizer.eos_token_id
    )
  output = tokenizer.decode(output_ids.tolist()[0])

  return "========\n" + tokenizer.decode(output_ids.tolist()[0])



def main(model_name,output_dir):
    LORA_WEIGHTS=output_dir
    MODEL_NAME = model_name
    device_map = "auto"
    bnb_config = BitsAndBytesConfig(
        load_in_4bit=True,
        bnb_4bit_use_double_quant=True,
        bnb_4bit_quant_type="nf4",
        bnb_4bit_compute_dtype=torch.bfloat16
    )
    base_model = AutoModelForCausalLM.from_pretrained(
        MODEL_NAME,
        device_map=device_map,
        #load_in_8bit=True,
        quantization_config=bnb_config
    )
    tokenizer = AutoTokenizer.from_pretrained(MODEL_NAME, use_fast=True)
    
    model = PeftModel.from_pretrained(
        base_model,
        LORA_WEIGHTS,
        device_map="auto"
    )

    path = "./sample_text_small.txt"
    base_model.eval()

  
            (generate("input text",tokenizer,))
         

if __name__ == "__main__":
	main("elyza/ELYZA-japanese-Llama-2-7b-fast-instruct","output checkpoint file")



```
If you need anything, such as tokenized data, fine-tuned data format, wandb loss graph, etc., we will provide it immediately. Please let me know how to solve it!




### Expected behavior

model.generate runs for more than 10 minutes. It does not output any errors. GPU doesn't seem to be a memory issue with usage around 6GB

Comments:
postscript
When I ran it with the number of data set to 1/100 and the number of epochs halved, it worked fine, although it was slow. Is there a case where generation becomes slow due to overtraining?

> Is there a case where generation becomes slow due to overtraining?

@50516017 The text generation speed (measured in tokens per second, for outputs of similar lengths) should not change with fine-tuning :)