Fix hqq skipped modules and dynamic quant
Author: mobicham
Pull request: yes

# What does this PR do?
This PR tries to fix a couple of issues with hqq that popped up lately:
* `skip_modules`: skip modules was not working properly. For example, if you specify the vision tower to skip during the quantization step, it was ignored. Now it works.
* Some skipped modules were creating issues when the quantized model is loaded. 
* `dynamic_config=True` has been broken lately after some changes in transformers. This is now fixed. 

You can now skip hqq quantization for the vision tower in VLMs as follows: 
```Python
import torch
device        = 'cuda:0'
compute_dtype = torch.bfloat16
cache_dir     = None
model_id      = 'google/gemma-3-12b-it'

########################################################################
#Load model
from transformers import HqqConfig, Gemma3ForConditionalGeneration, AutoProcessor

processor = AutoProcessor.from_pretrained(model_id, cache_dir=cache_dir)

#quant_config = HqqConfig(nbits=4, group_size=64, axis=1, skip_modules=['lm_head', 'vision_tower'])

q4_config = {'nbits':4, 'group_size':64}
q3_config = {'nbits':3, 'group_size':64}
quant_config  = HqqConfig(dynamic_config={
  'self_attn.q_proj':q4_config,
  'self_attn.k_proj':q4_config,
  'self_attn.v_proj':q4_config,
  'self_attn.o_proj':q4_config,

  'mlp.gate_proj':q3_config,
  'mlp.up_proj'  :q3_config,
  'mlp.down_proj':q3_config,
}, skip_modules=['lm_head', 'vision_tower'])

model = Gemma3ForConditionalGeneration.from_pretrained(
    model_id,
    torch_dtype=compute_dtype,
    attn_implementation="sdpa",
    cache_dir=cache_dir,
    quantization_config=quant_config,
    device_map="cuda",
)

messages = [
    {
        "role": "system",
        "content": [{"type": "text", "text": "You are a helpful assistant."}]
    },
    {
        "role": "user",
        "content": [
            {"type": "image", "image": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/bee.jpg"},
            {"type": "text", "text": "Describe this image in detail."}
        ]
    }
]

inputs = processor.apply_chat_template(messages, add_generation_prompt=True, tokenize=True, return_dict=True, return_tensors="pt").to(model.device, dtype=compute_dtype)

input_len = inputs["input_ids"].shape[-1]

with torch.inference_mode():
    generation = model.generate(**inputs, max_new_tokens=128, do_sample=False)[0][input_len:]
    decoded    = processor.decode(generation, skip_special_tokens=True)

print(decoded)
```

⚠️ There's still currently an issue related to saving/loading some quantized hqq models because transformers doesn't make sure the loaded state dict of a certain module contains all the necessary attributes. 
Currently, I am using <a href="https://github.com/mobiusml/hqq/blob/master/hqq/models/base.py#L545">my custom solution</a> which works well - it makes sure that a safetensors chunk contains all the attributes for a given module before saving. We should have a open a separate issue for this though. 

## Who can review?
@ArthurZucker @SunMarc 



Comments:
Hi 👋, thank you for opening this pull request! The pull request is converted to draft by default. When it is ready for review, please click the `Ready for review` button (at the bottom of the PR page).

> Thanks ! Can you add a test to make sure that `skip_modules` works ?

Thank you @SunMarc! Sure, added `test_model_serialization_dynamic_quant_with_skip` which tests both dynamic quant and the skip.