Repository: unslothai/unsloth

Issue #1521: OSError: does not appear to have a file named pytorch_model.bin, model.safetensors, tf_model.h5, model.ckpt or flax_model.msgpack.
Created: 2025-01-08T01:13:59Z
Hello, I m following the code in "Llama 3.2 Vision Finetuning Unsloth - Radiography.ipynb" for finetuning on my dataset.
but after pushing the merged model to hugging face I got the error when trying to load it:
/usr/local/lib/python3.10/dist-packages/transformers/modeling_utils.py:3491: FutureWarning: The `use_auth_token` argument is deprecated and will be removed in v5 of Transformers. Please use `token` instead.
  warnings.warn(
Unused kwargs: ['quant_method']. These kwargs are not used in <class 'transformers.utils.quantization_config.BitsAndBytesConfig'>.
`low_cpu_mem_usage` was None, now default to True since model is quantized.
---------------------------------------------------------------------------
OSError                                   Traceback (most recent call last)
[<ipython-input-2-01d76ccf350d>](https://localhost:8080/#) in <cell line: 4>()
      2 from transformers import MllamaForConditionalGeneration
      3 
----> 4 model, tokenizer = MllamaForConditionalGeneration.from_pretrained("Hessa/llama3.2_10k", use_auth_token="hf_***", torch_dtype=torch.float16) #torch_dtype=torch.bfloat16

[/usr/local/lib/python3.10/dist-packages/transformers/modeling_utils.py](https://localhost:8080/#) in from_pretrained(cls, pretrained_model_name_or_path, config, cache_dir, ignore_mismatched_sizes, force_download, local_files_only, token, revision, use_safetensors, weights_only, *model_args, **kwargs)
   3927                                 )
   3928                             else:
-> 3929                                 raise EnvironmentError(
   3930                                     f"{pretrained_model_name_or_path} does not appear to have a file named"
   3931                                     f" {_add_variant(WEIGHTS_NAME, variant)}, {_add_variant(SAFE_WEIGHTS_NAME, variant)},"

OSError: Hessa/llama3.2_10k does not appear to have a file named pytorch_model.bin, model.safetensors, tf_model.h5, model.ckpt or flax_model.msgpack.

Duplicate issue #2009: Applying LoRA Doesn't Change Model Output
Created: 2025-03-13T19:38:47Z
Greetings,
I am extremely confused why my model generates consistent result w/o my rank=64 LoRA. What's even more confusing is, the LoRA works in my notebook after training. But whenn I start fresh, loading the LoRA again, it gives the output of the original model.
model_name=="Qwen/Qwen2.5-1.5B-Instruct",
vllm==0.7.3
trl==0.15.2
unsloth==2025.2.14
unsloth_zoo==2025.2.7

So I followed the tutorial notebook, trained the model using GRPO trainer. Then within the notebook, I model.save_lora("grpo_saved_lora"), then I do this using vllm with greedy decoding:
sampling_params = SamplingParams(
    max_tokens=32,  # Maximum number of tokens to generate
    temperature=0,  # Temperature of 0 means no randomness (deterministic)
    top_p=1.0,      # Consider all tokens in the distribution
    top_k=1         # Only select the most likely token at each step
)

outputs = model.fast_generate(
    test_split,
    sampling_params = sampling_params,
    lora_request = model.load_lora("grpo_saved_lora"),
)

These are all fine, I have different set of output compare with before training. But then, when I try to load the model lora with a separate script. The output is suddenly consistent with the original model, even I loaded the lora in this way.
cfg = load_config(args.config)
    model, tokenizer = FastLanguageModel.from_pretrained(
        model_name=cfg.model_name,
        max_seq_length=cfg.max_seq_length,
        load_in_4bit=cfg.load_in_4bit,
        fast_inference=cfg.fast_inference,
        max_lora_rank=cfg.lora_rank,
        gpu_memory_utilization=cfg.gpu_memory_utilization,
    )
    lora = None
    
    if args.lora_path is not None:
        model = FastLanguageModel.get_peft_model(
            model,
            r=cfg.lora_rank,
            target_modules=cfg.target_modules,
            lora_alpha=cfg.lora_rank,
            use_gradient_checkpointing="unsloth",
            random_state=3407,
        )
        lora = model.load_lora(args.lora_path)
        print('using lora ' + args.lora_path)
            
    _, _, test_dataset = get_dataset("ecqa")
    
    sampling_params = SamplingParams( # Greedy decoding
        max_tokens=32,  # Maximum number of tokens to generate
        temperature=0,  # Temperature of 0 means no randomness (deterministic)
        top_p=1.0,      # Consider all tokens in the distribution
        top_k=1         # Only select the most likely token at each step
    )
    
    vllm_prompts = [x['prompt'] for x in test_dataset]
    
    outputs = model.fast_generate(
        vllm_prompts,
        sampling_params = sampling_params,
        lora_request = lora
    )

Could you please help me? I am really confused.
Thank you very much.

Closing comment (2025-04-07T10:59:06Z): You are not alone, it is related to #1521

So merging LoRA to the base model is an issue, and even for those who successfully merge and try to inference the merged model, they still face an issue where performance deteriorates ???

Yes. I have spent hundreds of dollars on A100 in order to finetune with GRPO Phi-4 and now that model is stuck on the virtual runtime because I cannot merge the model and push it to HF, and neither pushing only the lora works, because applying that lora gives only vanilla outputs.
I also tried retraining with latest versions of unsloth using smaller models (phi-4-mini and gemma-3), also using their own colabs without touching anything. Nothing.
For now, it is a waste of money, I'm sorry to say that.