[Question] What is the main difference between "AutoModelForCasualLM" and "PeftModelForCausalLM"?
Author: daehuikim
Pull request: no

I also wrote it down in peft repo. However this issue is also related to transformers. So i write my question here again.
issue is here in peft(https://github.com/huggingface/peft/issues/1245)

Hello, Sorry for naive question.
I noticed that the``model.generate()`` function performed differently when inferrence right after train with ```trainer.model``` and after merge and unload. (Every params are the same.)
So I checked two different object with simple print function.
Difference was the object that contains model.

1. ```model = trainer.model```
```
PeftModelForCausalLM(
  (base_model): LoraModel(
    (model): LlamaForCausalLM(
      (model): LlamaModel(
        (embed_tokens): ModulesToSaveWrapper(
          (original_module): Embedding(32008, 5120)
          (modules_to_save): ModuleDict(
            (default): Embedding(32008, 5120)
          )
        )
        (layers): ModuleList(
          (0-39): 40 x LlamaDecoderLayer(
            (self_attn): LlamaAttention(
              (q_proj): Linear4bit(
                (lora_dropout): ModuleDict(
                  (default): Dropout(p=0.1, inplace=False)
                )
                (lora_A): ModuleDict(
                  (default): Linear(in_features=5120, out_features=64, bias=False)
                )
                (lora_B): ModuleDict(
                  (default): Linear(in_features=64, out_features=5120, bias=False)
                )
                (lora_embedding_A): ParameterDict()
                (lora_embedding_B): ParameterDict()
                (base_layer): Linear4bit(in_features=5120, out_features=5120, bias=False)
              )
              (k_proj): Linear4bit(
                (lora_dropout): ModuleDict(
                  (default): Dropout(p=0.1, inplace=False)
                )
                (lora_A): ModuleDict(
                  (default): Linear(in_features=5120, out_features=64, bias=False)
                )
                (lora_B): ModuleDict(
                  (default): Linear(in_features=64, out_features=5120, bias=False)
                )
                (lora_embedding_A): ParameterDict()
                (lora_embedding_B): ParameterDict()
                (base_layer): Linear4bit(in_features=5120, out_features=5120, bias=False)
              )
              (v_proj): Linear4bit(
                (lora_dropout): ModuleDict(
                  (default): Dropout(p=0.1, inplace=False)
                )
                (lora_A): ModuleDict(
                  (default): Linear(in_features=5120, out_features=64, bias=False)
                )
                (lora_B): ModuleDict(
                  (default): Linear(in_features=64, out_features=5120, bias=False)
                )
                (lora_embedding_A): ParameterDict()
                (lora_embedding_B): ParameterDict()
                (base_layer): Linear4bit(in_features=5120, out_features=5120, bias=False)
              )
              (o_proj): Linear4bit(
                (lora_dropout): ModuleDict(
                  (default): Dropout(p=0.1, inplace=False)
                )
                (lora_A): ModuleDict(
                  (default): Linear(in_features=5120, out_features=64, bias=False)
                )
                (lora_B): ModuleDict(
                  (default): Linear(in_features=64, out_features=5120, bias=False)
                )
                (lora_embedding_A): ParameterDict()
                (lora_embedding_B): ParameterDict()
                (base_layer): Linear4bit(in_features=5120, out_features=5120, bias=False)
              )
              (rotary_emb): LlamaRotaryEmbedding()
            )
            (mlp): LlamaMLP(
              (gate_proj): Linear4bit(
                (lora_dropout): ModuleDict(
                  (default): Dropout(p=0.1, inplace=False)
                )
                (lora_A): ModuleDict(
                  (default): Linear(in_features=5120, out_features=64, bias=False)
                )
                (lora_B): ModuleDict(
                  (default): Linear(in_features=64, out_features=13824, bias=False)
                )
                (lora_embedding_A): ParameterDict()
                (lora_embedding_B): ParameterDict()
                (base_layer): Linear4bit(in_features=5120, out_features=13824, bias=False)
              )
              (up_proj): Linear4bit(
                (lora_dropout): ModuleDict(
                  (default): Dropout(p=0.1, inplace=False)
                )
                (lora_A): ModuleDict(
                  (default): Linear(in_features=5120, out_features=64, bias=False)
                )
                (lora_B): ModuleDict(
                  (default): Linear(in_features=64, out_features=13824, bias=False)
                )
                (lora_embedding_A): ParameterDict()
                (lora_embedding_B): ParameterDict()
                (base_layer): Linear4bit(in_features=5120, out_features=13824, bias=False)
              )
              (down_proj): Linear4bit(
                (lora_dropout): ModuleDict(
                  (default): Dropout(p=0.1, inplace=False)
                )
                (lora_A): ModuleDict(
                  (default): Linear(in_features=13824, out_features=64, bias=False)
                )
                (lora_B): ModuleDict(
                  (default): Linear(in_features=64, out_features=5120, bias=False)
                )
                (lora_embedding_A): ParameterDict()
                (lora_embedding_B): ParameterDict()
                (base_layer): Linear4bit(in_features=13824, out_features=5120, bias=False)
              )
              (act_fn): SiLUActivation()
            )
            (input_layernorm): LlamaRMSNorm()
            (post_attention_layernorm): LlamaRMSNorm()
          )
        )
        (norm): LlamaRMSNorm()
      )
      (lm_head): ModulesToSaveWrapper(
        (original_module): Linear(in_features=5120, out_features=32008, bias=False)
        (modules_to_save): ModuleDict(
          (default): Linear(in_features=5120, out_features=32008, bias=False)
        )
      )
    )
  )
)
```
2. ```AutoModelForCasualLM.from_pretrained(after merging lora adapter)```
```
LlamaForCausalLM(
  (model): LlamaModel(
    (embed_tokens): Embedding(32008, 5120)
    (layers): ModuleList(
      (0-39): 40 x LlamaDecoderLayer(
        (self_attn): LlamaAttention(
          (q_proj): Linear4bit(in_features=5120, out_features=5120, bias=False)
          (k_proj): Linear4bit(in_features=5120, out_features=5120, bias=False)
          (v_proj): Linear4bit(in_features=5120, out_features=5120, bias=False)
          (o_proj): Linear4bit(in_features=5120, out_features=5120, bias=False)
          (rotary_emb): LlamaRotaryEmbedding()
        )
        (mlp): LlamaMLP(
          (gate_proj): Linear4bit(in_features=5120, out_features=13824, bias=False)
          (up_proj): Linear4bit(in_features=5120, out_features=13824, bias=False)
          (down_proj): Linear4bit(in_features=13824, out_features=5120, bias=False)
          (act_fn): SiLUActivation()
        )
        (input_layernorm): LlamaRMSNorm()
        (post_attention_layernorm): LlamaRMSNorm()
      )
    )
    (norm): LlamaRMSNorm()
  )
  (lm_head): Linear(in_features=5120, out_features=32008, bias=False)
)
```

I think both modes should work exactly the same way, but when I inferred with the model.generate function, I found that #1 (PeftModelForCausalLM) works much more accurately. I'd like to know why, is there a theoretical or engineering reason for this?
Thanks for watching my long long question!

Comments:
cc @younesbelkada 

Hmm looks interesting. I think, when you print it out, first thing is considering original+ adapter and last thing is considering only structure of it’s backbone. 
Did you considered a random state, before inference?

Sometimes, same input’s result can be different when you inference it. 

> Hmm looks interesting. I think, when you print it out, first thing is considering original+ adapter and last thing is considering only structure of it’s backbone. Did you considered a random state, before inference?
> 
> Sometimes, same input’s result can be different when you inference it.

The reason why the structure of the last one feels exactly the same as the structure of the backbone model is because the lora adapter of this one has already been merged, so there is no numerical difference.
The only difference is the Object they are carried (```PeftModelForCausalLM```, ```AutoModelForCasualLM```).

And 2 models are initialized from different sources. one from ```PeftModelForCausalLM.from_pretrained``` and one for from ```AutoModelForCasualLM.from_pretrained```. However, every params, configuration files are the same except the object.

Hi @daehuikim 
Thanks for your issue, recall the formulation of the LoRA adapters in the figure below:
![lora-animated.gif](https://cdn-uploads.huggingface.co/production/uploads/62441d1d9fdefb55a0b7d12c/EMtsDOf4Wh7aveX0wy-OZ.gif)

The fundamental difference between `AutoModelForCausalLM` and `PeftModelForCausalLM`, in your case is that the `AutoModelForCausalLM` contains the merged adapter whereas the second model contains the model with LoRA attached on them. Technically the inference results should be the same as the LoRA operations can be simply rewritten as a refactorization of simple matrix multiplication:

![Screenshot 2023-11-02 at 20.12.17.png](https://cdn-uploads.huggingface.co/production/uploads/62441d1d9fdefb55a0b7d12c/Oc_cbRvTyGZIOp_SyAX7b.png)

Now regarding your issue, it seems you have further trained your `lm_head` as well. I think `merge_and_unload` does not properly take care of that, but I am not sure. Can you confirm which PEFT version are you using now?

> Hi @daehuikim Thanks for your issue, recall the formulation of the LoRA adapters in the figure below: ![lora-animated.gif](https://camo.githubusercontent.com/0c4168380b2d5a2d93f963838ccd920bc6384e56ebea50f4904031289db70673/68747470733a2f2f63646e2d75706c6f6164732e68756767696e67666163652e636f2f70726f64756374696f6e2f75706c6f6164732f3632343431643164396664656662353561306237643132632f454d7473444f6634576837617665583077792d4f5a2e676966) [ ![lora-animated.gif](https://camo.githubusercontent.com/0c4168380b2d5a2d93f963838ccd920bc6384e56ebea50f4904031289db70673/68747470733a2f2f63646e2d75706c6f6164732e68756767696e67666163652e636f2f70726f64756374696f6e2f75706c6f6164732f3632343431643164396664656662353561306237643132632f454d7473444f6634576837617665583077792d4f5a2e676966) ](https://camo.githubusercontent.com/0c4168380b2d5a2d93f963838ccd920bc6384e56ebea50f4904031289db70673/68747470733a2f2f63646e2d75706c6f6164732e68756767696e67666163652e636f2f70726f64756374696f6e2f75706c6f6164732f3632343431643164396664656662353561306237643132632f454d7473444f6634576837617665583077792d4f5a2e676966) [ ](https://camo.githubusercontent.com/0c4168380b2d5a2d93f963838ccd920bc6384e56ebea50f4904031289db70673/68747470733a2f2f63646e2d75706c6f6164732e68756767696e67666163652e636f2f70726f64756374696f6e2f75706c6f6164732f3632343431643164396664656662353561306237643132632f454d7473444f6634576837617665583077792d4f5a2e676966)
> 
> The fundamental difference between `AutoModelForCausalLM` and `PeftModelForCausalLM`, in your case is that the `AutoModelForCausalLM` contains the merged adapter whereas the second model contains the model with LoRA attached on them. Technically the inference results should be the same as the LoRA operations can be simply rewritten as a refactorization of simple matrix multiplication:
> 
> ![Screenshot 2023-11-02 at 20.12.17.png](https://camo.githubusercontent.com/212b6ba2d32f654813e329fc0caf83873681a059ea11300c383f42905411b7db/68747470733a2f2f63646e2d75706c6f6164732e68756767696e67666163652e636f2f70726f64756374696f6e2f75706c6f6164732f3632343431643164396664656662353561306237643132632f4f635f636252765479475a494f705f5379415837622e706e67)
> 
> Now regarding your issue, it seems you have further trained your `lm_head` as well. I think `merge_and_unload` does not properly take care of that, but I am not sure. Can you confirm which PEFT version are you using now?

Thanks for describing the issue correctly.
My peftconfig for training is like below: (```lm_head``` is not trained by trainer) 
```
peft_config = LoraConfig(
        lora_alpha=lora_alpha,
        lora_dropout=lora_dropout,
        r=lora_r,
        bias="none",
        task_type="CAUSAL_LM",
        target_modules= [
        "q_proj",
        "k_proj",
        "v_proj",
        "o_proj",
        "gate_proj",
        "up_proj",
        "down_proj",
        ],
        modules_to_save=[
            "embed_tokens",
            "lm_head"
        ]
    )
```
And this is my peft version
```
$ pip freeze | grep peft
peft==0.6.2
```

Thanks for answering! @younesbelkada 

Thanks @daehuikim 
You are adding 

```py
        modules_to_save=[
            "embed_tokens",
            "lm_head"
        ]
```
in the lora config, this will make the lm head and the input embedding layer trainable, hence the strange behaviour you are facing (I am not sure we support merge and unload for models that have `modules_to_save`). Can you try without `modules_to_save`?

> Thanks @daehuikim You are adding
> 
> ```python
>         modules_to_save=[
>             "embed_tokens",
>             "lm_head"
>         ]
> ```
> 
> in the lora config, this will make the lm head and the input embedding layer trainable, hence the strange behaviour you are facing (I am not sure we support merge and unload for models that have `modules_to_save`). Can you try without `modules_to_save`?

Thanks again @younesbelkada 

```modules_to_save``` wrapper save the modules not to be trained from trainer.
Also, ```embed_tokens, lm_head``` must to be saved when there exist added_tokens.
Therefore, ```modules_to_save``` block is necessary for my task.
I will see what happens in ```merge_and_unload``` for finding this out.
Thank you again!

https://github.com/huggingface/peft/blob/21c304f6f6ea62c7dcbef8e201d178a7575b471d/src/peft/tuners/lora/model.py#L419

```
elif isinstance(target, ModulesToSaveWrapper):
                # save any additional trainable modules part of `modules_to_save`
                setattr(parent, target_name, target.modules_to_save[target.active_adapter])
```
@younesbelkada 
```modules_to_save``` preserve modules from parent model.
Theoretically both model should reproduce same outputs.
But why this still happen? I am curious.

Hi, I'm having the same problem as you, in my scenario I also need to add some tokens for training and I've also set the
```
modules_to_save=[
            "embed_tokens",
            "lm_head"
      ]
```
May I ask if the problem is solved now? How did you end up solving it, looking forward to your reply.

> Hi, I'm having the same problem as you, in my scenario I also need to add some tokens for training and I've also set the
> 
> ```
> modules_to_save=[
>             "embed_tokens",
>             "lm_head"
>       ]
> ```
> 
> May I ask if the problem is solved now? How did you end up solving it, looking forward to your reply.

I have no idea yet. whole codes seems same in peft and transformers. I guess it could be some weight initialization problem

> > Hi, I'm having the same problem as you, in my scenario I also need to add some tokens for training and I've also set the
> > ```
> > modules_to_save=[
> >             "embed_tokens",
> >             "lm_head"
> >       ]
> > ```
> > 
> > 
> >     
> >       
> >     
> > 
> >       
> >     
> > 
> >     
> >   
> > May I ask if the problem is solved now? How did you end up solving it, looking forward to your reply.
> 
> I have no idea yet. whole codes seems same in peft and transformers. I guess it could be some weight initialization problem

Roger that. Thank you for your reply.

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.

Hi there! Is the issue solved? Let me know if you need more clarification

@younesbelkada I think we can close this issue by now. Thanks :)

Thanks @daehuikim !