Repository: huggingface/transformers
Pull request: Fix some failing AWQ tests

This PR fixes some failures observed in the AWQ quantization tests (`tests/quantization/autoawq/test_awq.py`) 

Also, some notes:

1.  `test_quantized_model_bf16`
    *   Error: `RuntimeError: expected scalar type Half but found BFloat16`
    *   Reason: `transformers>=4.50` preserves `bfloat16` dtype for scales when loading with `torch_dtype=torch.bfloat16`. However, AWQ implementation expects `float16` scales.
    *   Solution: Cast the scales to `torch.float16` if the model's `torch_dtype` is `bfloat16` in AWQ quantizer.
    *   Note: This resolves the `RuntimeError` but causes an `AssertionError` comparing output to a `bfloat16` baseline. This is expected, as the fix makes AWQ operate correctly in its supported `float16` mode, which differs from `bfloat16` output.

2.  `test_quantized_model_no_k_proj_quantized`
    *   Error: `RuntimeError: expected scalar type Int but found Half`
    *   Reason: The `lm_head` module was being incorrectly identified for quantization and converted to `WQLinear_GEMM`.
    *   Solution: Adjusted logic within the AWQ quantizer that determines the set of modules to be excluded from quantization.

3.  `test_generation_fused` & `test_generation_fused_batched`
    *   Error (CI - T4 GPU): `RuntimeError: FlashAttention only supports Ampere GPUs or newer.`
    *   Reason (CI): The CI environment uses T4 GPUs (Compute Capability 7.5), which lack hardware support for FlashAttention
    *   Error (Ampere+ GPU - A100): `ValueError: too many values to unpack (expected 2)` during the attention forward pass.
    *   Reason (Ampere+): The `QuantAttentionFused` module provided by `AutoAWQ` returns 3 values. (PR #35235) have modified the expected return signature to 2 values, removing `present_key_value`, also cache implementation seems to have been changed.
    *   Solution (Ampere+): Further investigation needed

4.  `test_generation_llava_fused`
    *   Error: `ValueError: Image features and image tokens do not match: tokens: 1, features 576`
    *   Reason: It also seems related to (PR #35235), which seems to be first failing commit.
    *   Solution:  Further investigation needed


Changed files:
filename: src/transformers/quantizers/base.py
patch: @@ -263,14 +263,17 @@ def get_modules_to_not_convert(
         model: "PreTrainedModel",
         skip_modules: Optional[List[str]] = None,
         keep_in_fp32_modules: Optional[List[str]] = None,
+        add_default_skips: bool = False,
     ):
         from ..integrations import get_keys_to_not_convert
 
-        modules_to_not_convert = []
-        if skip_modules is None:
+        if skip_modules is None or add_default_skips:
             modules_to_not_convert = get_keys_to_not_convert(model)
         else:
-            modules_to_not_convert = skip_modules
+            modules_to_not_convert = []
+
+        if skip_modules is not None:
+            modules_to_not_convert.extend(skip_modules)
 
         if keep_in_fp32_modules is not None:
             modules_to_not_convert.extend(keep_in_fp32_modules)

filename: src/transformers/quantizers/quantizer_awq.py
patch: @@ -92,6 +92,9 @@ def update_torch_dtype(self, torch_dtype):
         if torch_dtype is None:
             torch_dtype = torch.float16
             logger.info("Loading the model in `torch.float16`. To overwrite it, set `torch_dtype` manually.")
+        elif torch_dtype == torch.bfloat16:
+            logger.warning("`torch.bfloat16` is not supported for AWQ kernels yet. Casting to `torch.float16`.")
+            torch_dtype = torch.float16
         elif torch_dtype != torch.float16:
             logger.warning("We suggest you to set `torch_dtype=torch.float16` for better efficiency with AWQ.")
         return torch_dtype
@@ -102,7 +105,7 @@ def _process_model_before_weight_loading(
         from ..integrations import replace_quantization_scales, replace_with_awq_linear
 
         self.modules_to_not_convert = self.get_modules_to_not_convert(
-            model, self.quantization_config.modules_to_not_convert, keep_in_fp32_modules
+            model, self.quantization_config.modules_to_not_convert, keep_in_fp32_modules, add_default_skips=True
         )
 
         model, has_been_replaced = replace_with_awq_linear(