`Trainer.push_to_hub()` with PEFT doesn't work when the base model is loaded from local disk
Author: valayDave
Pull request: no
Labels: name: trainer

name: bug

name: PEFT

### System Info


- `transformers` version: 4.44.2
- Platform: Linux-5.15.0-1066-aws-x86_64-with-glibc2.31
- Python version: 3.12.0
- Huggingface_hub version: 0.25.1
- Safetensors version: 0.4.5
- Accelerate version: 0.34.2
- Accelerate config:    not found
- PyTorch version (GPU?): 2.4.1+cu121 (False)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using distributed or parallel set-up in script?: False

### Who can help?

@muellerzr @SunMarc

### Information

- [ ] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

```python
model_path = "path/to/my/model/on/disk"
model = AutoModelForCausalLM.from_pretrained(
    model_path,
    quantization_config=bnb_config,
    device_map=device_map,
    use_auth_token=True,
)

tokenizer = AutoTokenizer.from_pretrained(model_path, trust_remote_code=True)
# some code to create PeftConfig 
...

# You can even use a regular trainer instead 
trainer = SFTTrainer(
        model=model,
        args=training_arguments,
        tokenizer=tokenizer,
        train_dataset=train_dataset,
        peft_config=peft_config,
        dataset_text_field="text",
        max_seq_length=args.max_seq_length,
        packing=args.packing,
        callbacks=callbacks,
    )

trainer.train()
trainer.push_to_hub()
```

I have a `hub_model_id` present in the `TrainingArgs` when I run my scripts. If I set `push_to_hub=True` in TrainingArgs, it ends up throwing a ValueError like 
```
 File "/home/ob-workspace/metaflow-checkpoint-examples/nim_lora/finetune_hf_peft.py", line 127, in sft
    trainer.push_to_hub(
  File "/home/ob-workspace/micromamba/envs/metaflow/linux-64/51041216a07a03b/lib/python3.12/site-packages/trl/trainer/sft_trainer.py", line 481, in push_to_hub
    return super().push_to_hub(commit_message=commit_message, blocking=blocking, kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/ob-workspace/micromamba/envs/metaflow/linux-64/51041216a07a03b/lib/python3.12/site-packages/transformers/trainer.py", line 4353, in push_to_hub
    return upload_folder(
           ^^^^^^^^^^^^^^
  File "/home/ob-workspace/micromamba/envs/metaflow/linux-64/51041216a07a03b/lib/python3.12/site-packages/huggingface_hub/utils/_validators.py", line 114, in _inner_fn
    return fn(args, kwargs)
           ^^^^^^^^^^^^^^^^^^^
  File "/home/ob-workspace/micromamba/envs/metaflow/linux-64/51041216a07a03b/lib/python3.12/site-packages/huggingface_hub/hf_api.py", line 1485, in _inner
    return fn(self, args, kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/ob-workspace/micromamba/envs/metaflow/linux-64/51041216a07a03b/lib/python3.12/site-packages/huggingface_hub/hf_api.py", line 4972, in upload_folder
    add_operations = self._prepare_upload_folder_additions(
                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/ob-workspace/micromamba/envs/metaflow/linux-64/51041216a07a03b/lib/python3.12/site-packages/huggingface_hub/hf_api.py", line 9478, in _prepare_upload_folder_additions
    self._validate_yaml(
  File "/home/ob-workspace/micromamba/envs/metaflow/linux-64/51041216a07a03b/lib/python3.12/site-packages/huggingface_hub/hf_api.py", line 9542, in _validate_yaml
    raise ValueError(f"Invalid metadata in README.md.\n{message}") from e
ValueError: Invalid metadata in README.md.
- "base_model" with value "/tmp/metaflow_models_model_reference_c5stocp_" is not valid. Use a model id from https://hf.co/models.
```

While reading the code I also noticed that if we passed a `finetuned_from` argument to the `push_to_hub` function, the trainer passes them down to the `create_model_card` [function](https://github.com/huggingface/transformers/blob/15a4d24805a8d003ab7ce0758a7ede4142f011a9/src/transformers/trainer.py#L4584), but that function ends up letting Peft [change the generated card with a new one](https://github.com/huggingface/transformers/blob/15a4d24805a8d003ab7ce0758a7ede4142f011a9/src/transformers/trainer.py#L4446-L4447). The problem here is that PeftModel's [create_or_update_model_card](https://github.com/huggingface/peft/blob/8d9ecbed080eed79f32b279bdec211767919bd94/src/peft/peft_model.py#L1302)  is not accounting for the value of `base_model` set in the card and because of that the card is invalid and Huggingface doesn't allow pushing the model to the hub. 

I have a fix in the PeFT library to fix this : huggingface/peft#2124 



### Expected behavior

Overall the expected behavior for me when the model being loaded from disk and pushed to hub is : 

If a model is loaded from local disk and then trained with Peft (or any other HF extensions/trainer), push to hub should work. For accommodating this the library should : 
1. Distinguish between names and paths when setting certain information such as `base_model`. 
2. Allow passing information like `base_model` in the TrainingArguments so that `push_to_hub=True` can work with the trainer. Currently `push_to_hub=True` wont work in the PEFT scenarios because the README.md created by the PeftModel's [create_or_update_model_card](https://github.com/huggingface/peft/blob/8d9ecbed080eed79f32b279bdec211767919bd94/src/peft/peft_model.py#L1302) Overrides the value with something which can be a name or a path. And if it is a path (like this issue), it will just crash!
3. Ensure that `finetuned_from` can be passed down explicitly to extensions of the library like PEFT or does something like huggingface/peft#2124 (don't write a `base_model` value in the card if there is a `base_model` set in it)

Comments:
Maybe @qgallouedec for TRL and @BenjaminBossan for PEFT can give advice :)

Thanks for reporting @valayDave. Which trl version do you use?
For the future, it's easier for us when we have a minimal code that we can directly copy-paste and run. Something like this:

```python
from transformers import AutoModelForCausalLM, AutoTokenizer
from trl import SFTTrainer, SFTConfig
from peft import LoraConfig, get_peft_model
from datasets import load_dataset

model_id = "gpt2"
model = AutoModelForCausalLM.from_pretrained(model_id)
tokenizer = AutoTokenizer.from_pretrained(model_id)

local_path = "local-gpt2"
model.save_pretrained(local_path)
tokenizer.save_pretrained(local_path)

del model, tokenizer

model = AutoModelForCausalLM.from_pretrained(local_path)
tokenizer = AutoTokenizer.from_pretrained(local_path)
tokenizer.pad_token = tokenizer.eos_token

peft_config = LoraConfig()
model = get_peft_model(model, peft_config)

train_dataset = load_dataset("trl-lib/zen", "standard_language_modeling", split="train")

trainer = SFTTrainer(
    model=model,
    args=SFTConfig("gpt2-zen", push_to_hub=True),
    tokenizer=tokenizer,
    train_dataset=train_dataset,
    peft_config=peft_config,
)

trainer.train()
trainer.push_to_hub()
```

At this point, I'm not able to reproduce the bug you're reporting. The pushed model for the example is here: https://huggingface.co/qgallouedec/gpt2-zen (not 100% satisfying since we end up with "This model is a fine-tuned version of [None](https://huggingface.co/None).").

Note that we've recently changed the way trl generates model cards: https://github.com/huggingface/trl/pull/2123. So it's possible that we've solved this problem without knowing it.

I am using `0.10.1` . Is https://github.com/huggingface/trl/pull/2123 a part of a recent release ? Then I can give it a try ? 

Not released yet. But it will be part of v0.12. You can give it a try with dev version though

Okay so the good news is that it works but it requires the latest head of the `transformers` repo otherwise I can't even instantiate the trainer because of this error : 

```
    trainer = SFTTrainer(
              ^^^^^^^^^^^
  File "/opt/outerbounds/fastbakery/lib/python/site-packages/huggingface_hub/utils/_deprecation.py", line 101, in inner_f
    return f(args, kwargs)
           ^^^^^^^^^^^^^^^^^^
  File "/opt/outerbounds/fastbakery/lib/python/site-packages/transformers/utils/deprecation.py", line 165, in wrapped_func
    return func(args, kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/opt/outerbounds/fastbakery/lib/python/site-packages/trl/trainer/sft_trainer.py", line 409, in __init__
    super().__init__(
TypeError: Trainer.__init__() got an unexpected keyword argument 'processing_class'
```

Overall, it will require the very latest transformers that [contain this commit ](https://github.com/huggingface/transformers/commit/b7474f211d4d1b6f83ba3b6cb9f942a18d3b02a1) ; I was able to create a model card and use `trainer.push_to_hub` without modifying anything. Thanks for the help! we can close this issue here and this issue : https://github.com/huggingface/peft/pull/2124