Repository: unslothai/unsloth

Issue #2009: Applying LoRA Doesn't Change Model Output
Created: 2025-03-13T19:38:47Z
Greetings,
I am extremely confused why my model generates consistent result w/o my rank=64 LoRA. What's even more confusing is, the LoRA works in my notebook after training. But whenn I start fresh, loading the LoRA again, it gives the output of the original model.
model_name=="Qwen/Qwen2.5-1.5B-Instruct",
vllm==0.7.3
trl==0.15.2
unsloth==2025.2.14
unsloth_zoo==2025.2.7

So I followed the tutorial notebook, trained the model using GRPO trainer. Then within the notebook, I model.save_lora("grpo_saved_lora"), then I do this using vllm with greedy decoding:
sampling_params = SamplingParams(
    max_tokens=32,  # Maximum number of tokens to generate
    temperature=0,  # Temperature of 0 means no randomness (deterministic)
    top_p=1.0,      # Consider all tokens in the distribution
    top_k=1         # Only select the most likely token at each step
)

outputs = model.fast_generate(
    test_split,
    sampling_params = sampling_params,
    lora_request = model.load_lora("grpo_saved_lora"),
)

These are all fine, I have different set of output compare with before training. But then, when I try to load the model lora with a separate script. The output is suddenly consistent with the original model, even I loaded the lora in this way.
cfg = load_config(args.config)
    model, tokenizer = FastLanguageModel.from_pretrained(
        model_name=cfg.model_name,
        max_seq_length=cfg.max_seq_length,
        load_in_4bit=cfg.load_in_4bit,
        fast_inference=cfg.fast_inference,
        max_lora_rank=cfg.lora_rank,
        gpu_memory_utilization=cfg.gpu_memory_utilization,
    )
    lora = None
    
    if args.lora_path is not None:
        model = FastLanguageModel.get_peft_model(
            model,
            r=cfg.lora_rank,
            target_modules=cfg.target_modules,
            lora_alpha=cfg.lora_rank,
            use_gradient_checkpointing="unsloth",
            random_state=3407,
        )
        lora = model.load_lora(args.lora_path)
        print('using lora ' + args.lora_path)
            
    _, _, test_dataset = get_dataset("ecqa")
    
    sampling_params = SamplingParams( # Greedy decoding
        max_tokens=32,  # Maximum number of tokens to generate
        temperature=0,  # Temperature of 0 means no randomness (deterministic)
        top_p=1.0,      # Consider all tokens in the distribution
        top_k=1         # Only select the most likely token at each step
    )
    
    vllm_prompts = [x['prompt'] for x in test_dataset]
    
    outputs = model.fast_generate(
        vllm_prompts,
        sampling_params = sampling_params,
        lora_request = lora
    )

Could you please help me? I am really confused.
Thank you very much.

Duplicate issue #2187: A question about how to save models directly from checkpoint
Created: 2025-03-25T10:11:25Z
If I use the following method to load LoRA adapter:
model = FastModel.get_peft_model(
    model,
    checkpoint_path = "/content/drive/MyDrive/outputs/checkpoint-900", 
......
It seems that the checkpoint will not load when I try to run the model via Unsloth native inference:
from unsloth.chat_templates import get_chat_template
tokenizer = get_chat_template(
    tokenizer,
    chat_template = "gemma-3",
)
messages = [{
    "role": "user",
    "content": [{
        "type" : "text",
......
Therefore, I tried to use the original peft way:
checkpoint_path = "/content/drive/MyDrive/outputs/checkpoint-900"

from peft import PeftModel
model = PeftModel.from_pretrained(
    model,
    checkpoint_path,
    is_trainable=False
)
Now from the chat inference it seems that the checkpoint was loaded.
However, when I trying to use unsloth's method to save gguf file, what I got is only the original untrained model:
model.save_pretrained_merged("gemma-3-finetune", tokenizer)
model.save_pretrained_gguf(
    "gemma-3-finetune",
    quantization_type = "Q8_0", # For now only Q8_0, BF16, F16 supported
)
So now I have to use a very inelegant approach, which is to set argsmax_steps = checkpoint_step and use trainer.train(resume from checkpoint) to finish training, to save the model

Closing comment (2025-03-25T10:20:03Z): I don't know if this issue is related to #2009