Skip some weights for load_in_8bit and keep them as fp16/32?
Author: gregor-ge
Pull request: no

### Feature request

Hello,

I am looking for a way to load a checkpoint where I only load some of the weights in 8 bit and keep others in 16/32 bit.

### Motivation

My motivation is for vision-language models like Llava or BLIP2 where I want to load the LLM part in 8 bit but the image encoder should stay in 16 bit because I notice performance degradations with CLIP in 8 bit and also want to be able to train this part without LoRA.

As far as I can see in the documentation, issues and with Google (both here and for bitsandbytes), there is currently no way to do this.

### Your contribution

I can in theory help implement something like this but I don't know where and how in the code this should be done.

Comments:
Is this what you are looking for:
https://github.com/huggingface/transformers/blob/a3b00030c96be4c66e818cf9a22040dbe41006d8/src/transformers/modeling_utils.py#L1165


Not quite, no, but it is similar.  `_keep_in_fp32_modules ` is set on the library side (e.g. for the T5 modules) and it also is limited to fp32. I am looking for a solution similar to the initial suggestion in #20683, that is for example `Blip2ForConditionalGeneration.from_pretrained(..., load_in_8bit=True, skip_for_8bit_modules=["qformer", "vision_encoder", "language_projection"])`  which loads those modules in the checkpoint format (fp16, fp32, bf16, ...) without quantization.

I think the following could work as a (dirty) way of implementing what I want with it though  but I have to check later:
```python
from transformers import Blip2ForConditionalGeneration
# add modules to list so they are skipped for quantization
Blip2ForConditionalGeneration._keep_in_fp32_modules = ["qformer", "vision_encoder", "language_projection"]
model = Blip2ForConditionalGeneration.from_pretrained(..., load_in_8bit=True)

# manually cast back to fp16 if wanted
Blip2ForConditionalGeneration._keep_in_fp32_modules = []
model.qformer = model.qformer.half()
model.vision_encoder= model.vision_encoder.half()
...
```

I dug around in `load_pretrained()` and realized there *is* a config option `quantization_config.llm_int8_skip_modules` [here](https://github.com/huggingface/transformers/blob/a3b00030c96be4c66e818cf9a22040dbe41006d8/src/transformers/modeling_utils.py#L3510) which seems to do exactly what I want. I will test this out on the weekend and report back. One thing I noticed is that when you set this parameter, you have to also manually add parameters like the `lm_head`which are skipped on default otherwise.

Yes, it works just as I want. It also work for int4. The parameter is documented [here](https://huggingface.co/docs/transformers/main_classes/quantization#transformers.BitsAndBytesConfig.llm_int8_skip_modules) which I seem to have missed in my search before.

For posterity, the following snippet is how to use it:
```python
from transformers import Blip2ForConditionalGeneration
skip_int8_modules = ["lm_head", "vision_encoder", "language_projection", "qformer"] if skip_vision_int8 else None
self.model = Blip2ForConditionalGeneration.from_pretrained(
    checkpoint,
    load_in_8bit=True,
    llm_int8_skip_modules=skip_int8_modules
)
´´´

Thanks for helping me!