ValueError: Target module QuantLinear() is not supported. Currently, only `torch.nn.Linear` and `Conv1D` are supported.
Author: ghost
Pull request: no

### System Info

docker/python 3.9

### Who can help?

@muellerzr; @pacman100

### Information

- [ ] The official example scripts
- [X] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

Noticed upon updating transformers from ` 4.34.1` to `4.35.0` while on `peft version 0.5.0` that upon invoking `peft.get_peft_model` I get the error:

`ValueError: Target module QuantLinear() is not supported. Currently, only `torch.nn.Linear` and `Conv1D` are supported.`

i was able to produce this error with some code i grabbed from a tutorial:



```
from datasets import Dataset
dataset = Dataset.from_pandas(train_df)
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer, BitsAndBytesConfig

# sharded model path in hugging face
# model_name = "TinyPixel/Llama-2-7B-bf16-sharded"


# model_name = 'NousResearch/Llama-2-7b-hf'

# Quantization config
# bnb_config = BitsAndBytesConfig(
#     load_in_4bit=True,
#     bnb_4bit_quant_type="nf4",
#     bnb_4bit_compute_dtype="float16",
# )
# take pre quantized model from hugging face
model_id = "TheBloke/Llama-2-7B-GPTQ"
# model = AutoModelForCausalLM.from_pretrained(model_id, device_map="auto")
# tokenizer = AutoTokenizer.from_pretrained(model_id)

# loading the model with quantization config
# model = AutoModelForCausalLM.from_pretrained(
#     model_name,
#     quantization_config=bnb_config,
#     trust_remote_code=True,
#     device_map='auto'
# )


# can change to False if need the newest model update.

# %%
from peft import prepare_model_for_kbit_training
from transformers import GPTQConfig

# model_id = "TheBloke/Llama-2-7B-GPTQ"
# model_id = "TheBloke/Llama-2-7b-Chat-GPTQ"
quantization_config_loading = GPTQConfig(bits=4, disable_exllama=True)
model = AutoModelForCausalLM.from_pretrained(
    model_id, quantization_config=quantization_config_loading, device_map="auto"
)
tokenizer = AutoTokenizer.from_pretrained(model_id)
model.config.use_cache = False
model.config.pretraining_tp = 1
# %%
model.gradient_checkpointing_enable()
model = prepare_model_for_kbit_training(model)
from peft import LoraConfig, get_peft_model
config = LoraConfig(
    r=8,
    lora_alpha=32,
    target_modules=["k_proj","o_proj","q_proj","v_proj"],
    lora_dropout=0.05,
    bias="none",
    task_type="CAUSAL_LM"
)

print('error here expected')
model = get_peft_model(model, config)
print('dont expect to get here')
model.print_trainable_parameters()
# %%
from transformers import Trainer, TrainingArguments, DataCollatorForLanguageModeling

# needed for llama 2 tokenizer
tokenizer.pad_token = tokenizer.eos_token
tokenizer.padding_side = "right"
model.config.use_cache = False # silence the warnings. Please re-enable for inference!
args=TrainingArguments(
        per_device_train_batch_size=4,
        gradient_accumulation_steps=4,
        warmup_steps=2,
        max_steps=100,
        learning_rate=2e-4,
        fp16=True, #use mixed precision training
        logging_steps=1,
        output_dir="outputs_gptq_training",
        optim="adamw_hf",
        save_strategy="epoch",
        report_to="none")

from trl import SFTTrainer
trainer = SFTTrainer(
    model=model,
    args=args,
    train_dataset=dataset,
    peft_config=config,
    dataset_text_field="text",
    tokenizer=tokenizer,
    packing=False,
    max_seq_length=512)


# %%
train_result = trainer.train()

# %%
checkpoint_name ="final_checkpoints_gptqsummarizer_7b_peft"
#to merge and save the model
output_dir = os.path.join(args.output_dir, checkpoint_name)
trainer.model.save_pretrained(output_dir)


from peft import AutoPeftModelForCausalLM
# To perform inference on the test dataset example load the model from the checkpoint
persisted_model = AutoPeftModelForCausalLM.from_pretrained(
    output_dir,
    low_cpu_mem_usage=True,
    return_dict=True,
    torch_dtype=torch.float16,
    device_map="cuda",
)

# %%
#inference on test data example
from time import perf_counter
from rich import print
from transformers import GenerationConfig
text = test_df['text'][4]
inputs = tokenizer(text, return_tensors="pt").to('cuda')
generation_config = GenerationConfig(
    penalty_alpha=0.6, 
    do_sample = True, 
    top_k=5, 
    temperature=0.5, 
    repetition_penalty=1.2,
    max_new_tokens=100
)
start_time = perf_counter()
outputs = persisted_model.generate(**inputs, generation_config=generation_config)
print("output:::::::::")
print(tokenizer.decode(outputs[0], skip_special_tokens=True))
end_time = perf_counter()
output_time = end_time - start_time
print(f"Time taken for inference: {round(output_time,2)} seconds")
```

i was able to eliminate the error by downgrading my container build to 4.34.1 and reproduce it by rebuilding with 4.35.0.  peft release 0.6.0 was released a few days ago and i tried it with 4.35.0 but 0.6.0/4.35.0 did not resolve the issue.

i scanned the release notes for both the 4.35.0 and 0.6.0 versions and didn't notice any breaking changes that would predict this.

will continue to dig but i thought i'd throw this out in case anyone else sees it.

### Expected behavior

4.35.0 works like 4.34.1 with peft 0.5.0/0.6.0

Comments:
cc @younesbelkada 

Hi @jason-brian-anderson 
Thanks for the issue, there might be a regression but not sure
Are you able to run this small snippet? The snippet below worked fine on my end (transformers main + PEFT 0.6.0)
```python
from transformers import AutoModelForCausalLM, GPTQConfig
from peft import LoraConfig, get_peft_model

model_id = "TheBloke/Llama-2-7B-GPTQ"
quantization_config = GPTQConfig(bits=4, disable_exllama=True)

model = AutoModelForCausalLM.from_pretrained(model_id, low_cpu_mem_usage=True, device_map={"": 0}, quantization_config=quantization_config)

lora_config = LoraConfig(
    r=8,
    target_modules=["k_proj","o_proj","q_proj","v_proj"],
    task_type="CAUSAL_LM"
)

model = get_peft_model(model, lora_config)
print(model)
```
Also I noticed that you pass a `PeftModel` to `SFTTrainer` together with a `peft_config` argument, this might lead to create a nested peft model which can lead to some bugs.

> Hi @jason-brian-anderson Thanks for the issue, there might be a regression but not sure Are you able to run this small snippet? The snippet below worked fine on my end (transformers main + PEFT 0.6.0)
> 
> ```python
> from transformers import AutoModelForCausalLM, GPTQConfig
> from peft import LoraConfig, get_peft_model
> 
> model_id = "TheBloke/Llama-2-7B-GPTQ"
> quantization_config = GPTQConfig(bits=4, disable_exllama=True)
> 
> model = AutoModelForCausalLM.from_pretrained(model_id, low_cpu_mem_usage=True, device_map={"": 0}, quantization_config=quantization_config)
> 
> lora_config = LoraConfig(
>     r=8,
>     target_modules=["k_proj","o_proj","q_proj","v_proj"],
>     task_type="CAUSAL_LM"
> )
> 
> model = get_peft_model(model, lora_config)
> print(model)
> ```
> 
> Also I noticed that you pass a `PeftModel` to `SFTTrainer` together with a `peft_config` argument, this might lead to create a nested peft model which can lead to some bugs.

First thanks for the tip about sending both `peft_config` and `PeftModel` to `SFTTrainer`, my results are now much better by not passing the `peft_config `arg to `SFTTrainer` along with the `PeftModel`  :-)

i copy/pasted/ran your code and i do still get the error: on 4.35.0 whereas on 4.34.1 i do see the model 

```
Using `disable_exllama` is deprecated and will be removed in version 4.37. Use `use_exllama` instead and specify the version with `exllama_config`.The value of `use_exllama` will be overwritten by `disable_exllama` passed in `GPTQConfig` or stored in your config file.
You passed `quantization_config` to `from_pretrained` but the model you're loading already has a `quantization_config` attribute and has already quantized weights. However, loading attributes (e.g. use_exllama, exllama_config, use_cuda_fp16, max_input_length) will be overwritten with the one you passed to `from_pretrained`. The rest will be ignored.
---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
Cell In[1], line 15
      7 model = AutoModelForCausalLM.from_pretrained(model_id, low_cpu_mem_usage=True, device_map={"": 0}, quantization_config=quantization_config)
      9 lora_config = LoraConfig(
     10     r=8,
     11     target_modules=["k_proj","o_proj","q_proj","v_proj"],
     12     task_type="CAUSAL_LM"
     13 )
---> 15 model = get_peft_model(model, lora_config)
     16 print(model)

File /app/venv/lib/python3.10/site-packages/peft/mapping.py:106, in get_peft_model(model, peft_config, adapter_name)
    104 if peft_config.is_prompt_learning:
    105     peft_config = _prepare_prompt_learning_config(peft_config, model_config)
--> 106 return MODEL_TYPE_TO_PEFT_MODEL_MAPPING[peft_config.task_type](model, peft_config, adapter_name=adapter_name)

File /app/venv/lib/python3.10/site-packages/peft/peft_model.py:889, in PeftModelForCausalLM.__init__(self, model, peft_config, adapter_name)
    888 def __init__(self, model, peft_config: PeftConfig, adapter_name="default"):
--> 889     super().__init__(model, peft_config, adapter_name)
    890     self.base_model_prepare_inputs_for_generation = self.base_model.prepare_inputs_for_generation

File /app/venv/lib/python3.10/site-packages/peft/peft_model.py:111, in PeftModel.__init__(self, model, peft_config, adapter_name)
    109 if not peft_config.is_prompt_learning:
    110     self.peft_config[adapter_name] = peft_config
--> 111     self.base_model = PEFT_TYPE_TO_MODEL_MAPPING[peft_config.peft_type](
    112         self.base_model, self.peft_config, adapter_name
    113     )
    114     self.set_additional_trainable_modules(peft_config, adapter_name)
    115 else:

File /app/venv/lib/python3.10/site-packages/peft/tuners/lora.py:274, in LoraModel.__init__(self, model, config, adapter_name)
    273 def __init__(self, model, config, adapter_name) -> None:
--> 274     super().__init__(model, config, adapter_name)

File /app/venv/lib/python3.10/site-packages/peft/tuners/tuners_utils.py:88, in BaseTuner.__init__(self, model, peft_config, adapter_name)
     85 if not hasattr(self, "config"):
     86     self.config = {"model_type": "custom"}
---> 88 self.inject_adapter(self.model, adapter_name)
     90 # Copy the peft_config in the injected model.
     91 self.model.peft_config = self.peft_config

File /app/venv/lib/python3.10/site-packages/peft/tuners/tuners_utils.py:219, in BaseTuner.inject_adapter(self, model, adapter_name)
    212     parent, target, target_name = _get_submodules(model, key)
    214     optionnal_kwargs = {
    215         "loaded_in_8bit": getattr(model, "is_loaded_in_8bit", False),
    216         "loaded_in_4bit": getattr(model, "is_loaded_in_4bit", False),
    217         "current_key": key,
    218     }
--> 219     self._create_and_replace(peft_config, adapter_name, target, target_name, parent, **optionnal_kwargs)
    221 if not is_target_modules_in_base_model:
    222     raise ValueError(
    223         f"Target modules {peft_config.target_modules} not found in the base model. "
    224         f"Please check the target modules and try again."
    225     )

File /app/venv/lib/python3.10/site-packages/peft/tuners/lora.py:372, in LoraModel._create_and_replace(self, lora_config, adapter_name, target, target_name, parent, **optionnal_kwargs)
    364     target.update_layer(
    365         adapter_name,
    366         lora_config.r,
   (...)
    369         lora_config.init_lora_weights,
    370     )
    371 else:
--> 372     new_module = self._create_new_module(lora_config, adapter_name, target, **kwargs)
    373     self._replace_module(parent, target_name, new_module, target)

File /app/venv/lib/python3.10/site-packages/peft/tuners/lora.py:481, in LoraModel._create_new_module(lora_config, adapter_name, target, **kwargs)
    479             kwargs["fan_in_fan_out"] = lora_config.fan_in_fan_out = True
    480     else:
--> 481         raise ValueError(
    482             f"Target module {target} is not supported. "
    483             f"Currently, only `torch.nn.Linear` and `Conv1D` are supported."
    484         )
    485     new_module = Linear(adapter_name, in_features, out_features, bias=bias, **kwargs)
    487 return new_module

ValueError: Target module QuantLinear() is not supported. Currently, only `torch.nn.Linear` and `Conv1D` are supported.
```

my container versions were as follows during the test:


```
import numpy as np
import wandb
import transformers
import IPython
import peft
print(np.__version__)
print(wandb.__version__)
print(transformers.__version__)
print(IPython.__version__)
print(peft.__version__)

1.26.1
0.16.0
4.35.0
8.16.1
0.5.0
```

also, to correct original assertion, i'm running

```
!python
Python 3.10.12 (main, Jun 11 2023, 05:26:28) [GCC 11.4.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> 
```
not 3.9.  this is a dev env so had tons of other libraries installed at various versions, perhaps the issue lies there.

OK interesting, thanks for running the experiments, a similar issue also on PEFT: https://github.com/huggingface/peft/issues/1097 there might be a regression on transformers @BenjaminBossan did also not managed to reproduce, will have a deeper look tomorrow

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.