Enable BNB multi-backend support
Author: jiqing-feng
Pull request: yes
Labels: name: Quantization

Refer to [1227](https://github.com/TimDettmers/bitsandbytes/pull/1227), and [1178](https://github.com/TimDettmers/bitsandbytes/pull/1178), [1206](https://github.com/TimDettmers/bitsandbytes/pull/1206). The bitsandbytes now support CPU backend, so we can remove the cuda restriction in transformers.

Comments:
This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.

Hi @younesbelkada @SunMarc @ArthurZucker . I think this PR is ready to be reviewed, as the bitsandbytes already gives the installation guide for CPU. See [here](https://github.com/TimDettmers/bitsandbytes?tab=readme-ov-file#alpha-testers-wanted-multi-backend-refactor-amd-gpu--intel-cpugpu-specific-bnb-backend-implementations).

cc @Titus-von-Koeller 

@jiqing-feng, we decided this likely still needs some work. I'll look into it more this week.

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_31098). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

Refer to [1243](https://github.com/bitsandbytes-foundation/bitsandbytes/pull/1243). I updated the check method when cuda is not available, please review it. Thx! @SunMarc 

 @Titus-von-Koeller please help to check if it is enough for multi-backend-refactor branch. 


Ok, I started looking into this a bit on the Intel machine that we have access to and based on the code of this PR, I'm still getting other errors when running the tests, disabling the skipping of tests due to CUDA having been required before.

Basically almost all the tests fail with the same below error and none of them pass.

```
______________________________________________________ Bnb4BitGPT2Test.test_rwkv_4bit _______________________________________________________

self = <bnb.test_4bit.Bnb4BitGPT2Test testMethod=test_rwkv_4bit>

    def setUp(self):
        super().setUp()
    
        # Models and tokenizer
        self.model_fp16 = AutoModelForCausalLM.from_pretrained(
            self.model_name, torch_dtype=torch.float16, device_map="auto"
        )
>       self.model_4bit = AutoModelForCausalLM.from_pretrained(self.model_name, load_in_4bit=True, device_map="auto")

tests/quantization/bnb/test_4bit.py:122: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
src/transformers/models/auto/auto_factory.py:564: in from_pretrained
    return model_class.from_pretrained(
src/transformers/modeling_utils.py:3834: in from_pretrained
    hf_quantizer.validate_environment(device_map=device_map)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

self = <transformers.quantizers.quantizer_bnb_4bit.Bnb4BitHfQuantizer object at 0x7f0219252070>, args = ()
kwargs = {'device_map': OrderedDict([('', 'cpu')])}, device_map_without_lm_head = {'': 'cpu'}

    def validate_environment(self, *args, **kwargs):
        if not is_accelerate_available():
            raise ImportError("Using `bitsandbytes` 4-bit quantization requires Accelerate: `pip install accelerate`")
        if not is_bitsandbytes_available():
            raise ImportError(
                "Using `bitsandbytes` 4-bit quantization requires the latest version of bitsandbytes: `pip install -U bitsandbytes`"
            )
        # if not torch.cuda.is_available():
        #    import bitsandbytes as bnb
    
        #    if not getattr(bnb, "is_multi_backend_refactor_preview", False):
        #        raise RuntimeError(
        #            "Current bitsandbytes only support cuda, please switch to multi_backend_refactor to support multi backends."
        #        )
    
        if kwargs.get("from_tf", False) or kwargs.get("from_flax", False):
            raise ValueError(
                "Converting into 4-bit or 8-bit weights from tf/flax weights is currently not supported, please make"
                " sure the weights are in PyTorch format."
            )
    
        device_map = kwargs.get("device_map", None)
        if (
            device_map is not None
            and isinstance(device_map, dict)
            and not self.quantization_config.llm_int8_enable_fp32_cpu_offload
        ):
            device_map_without_lm_head = {
                key: device_map[key] for key in device_map.keys() if key not in self.modules_to_not_convert
            }
            if "cpu" in device_map_without_lm_head.values() or "disk" in device_map_without_lm_head.values():
>               raise ValueError(
                    "Some modules are dispatched on the CPU or the disk. Make sure you have enough GPU RAM to fit the "
                    "quantized model. If you want to dispatch the model on the CPU or the disk while keeping these modules "
                    "in 32-bit, you need to set `load_in_8bit_fp32_cpu_offload=True` and pass a custom `device_map` to "
                    "`from_pretrained`. Check "
                    "https://huggingface.co/docs/transformers/main/en/main_classes/quantization#offload-between-cpu-and-gpu "
                    "for more details. "
                )
E               ValueError: Some modules are dispatched on the CPU or the disk. Make sure you have enough GPU RAM to fit the quantized model. If you want to dispatch the model on the CPU or the disk while keeping these modules in 32-bit, you need to set `load_in_8bit_fp32_cpu_offload=True` and pass a custom `device_map` to `from_pretrained`. Check https://huggingface.co/docs/transformers/main/en/main_classes/quantization#offload-between-cpu-and-gpu for more details.

src/transformers/quantizers/quantizer_bnb_4bit.py:91: ValueError
```

> Ok, I started looking into this a bit on the Intel machine that we have access to and based on the code of this PR, I'm still getting other errors when running the tests, disabling the skipping of tests due to CUDA having been required before.
> 
> Basically almost all the tests fail with the same below error and none of them pass.

Could you try to pass `None` or `cpu` into `device_map` since we are testing it on CPU? I suppose device_map="auto" is for cuda, am I right? @SunMarc 

Yeah, I'm working on a solution.

To see if there are further issues when running the [test script from the original Intel 4bit PR](https://github.com/bitsandbytes-foundation/bitsandbytes/pull/1206#issuecomment-2126386067) or when running the Transformer bnb-related integration test suite, I modified it temporarily like this:

```
 90             if "disk" in device_map_without_lm_head.values():
 91             #if "cpu" in device_map_without_lm_head.values() or "disk" in device_map_without_lm_head.values():
```

My idea is to check if BNB is multi-backend enabled and in that case skip all the GPU related checks and allow CPU as well.

@SunMarc In this case, do you think it's save to sth like:

```
if ("cpu" in device_map_without_lm_head.values() or bnb_is_multibackend_enabled) or "disk" in device_map_without_lm_head.values():
```

?

Currently, for the `tests/quantization/bnb/test_4bit.py` I'm still getting a bunch of failures, but most are related to the model being moved to `cuda(0)` or NotImplementedError, for which I'm working on a fix and is not related to the Intel implementation in BNB.

However, there's one failing test that caught my attention and that I couldn't understand the root cause of, yet.. Could you please help me figure out what's going on with that one, @jiqing-feng ? See below:

```
______________________________________________ Pipeline4BitTest.test_pipeline _______________________________________________
                                                                                                                             
self = <bnb.test_4bit.Pipeline4BitTest testMethod=test_pipeline>                                                             
                                                                                                                             
    def test_pipeline(self):                                  
        r"""                                                                                                                 
        The aim of this test is to verify that the mixed 4bit is compatible with `pipeline` from transformers. Since         
        we used pipline for inference speed benchmarking we want to make sure that this feature does not break anything      
        on pipline.                                                                                                          
        """                                                                                                                  
        # self._clear_cuda_cache()                                                                                           
        self.pipe = pipeline(                                                                                                
            "text-generation",                                                                                               
            model=self.model_name,                                                                                           
            model_kwargs={"device_map": "auto", "load_in_4bit": True, "torch_dtype": torch.float16},                         
            max_new_tokens=self.MAX_NEW_TOKENS,                                                                              
        )                                                                                                                    
                                                                                                                             
        # Real second forward pass                                                                                           
>       pipeline_output = self.pipe(self.input_text)                                                                         
                                                                                                                             
tests/quantization/bnb/test_4bit.py:462:                                                                                     
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
src/transformers/pipelines/text_generation.py:262: in __call__                                                               
    return super().__call__(text_inputs, **kwargs)                                                                           
src/transformers/pipelines/base.py:1254: in __call__                                                                         
    return self.run_single(inputs, preprocess_params, forward_params, postprocess_params)                                    
src/transformers/pipelines/base.py:1261: in run_single                                                                       
    model_outputs = self.forward(model_inputs, **forward_params)                                                             
src/transformers/pipelines/base.py:1161: in forward                                                                          
    model_outputs = self._forward(model_inputs, **forward_params)  
src/transformers/pipelines/text_generation.py:351: in _forward                                         (11 results) [443/925]
    generated_sequence = self.model.generate(input_ids=input_ids, attention_mask=attention_mask, **generate_kwargs)          
../../.condax/mamba/envs/bnb/lib/python3.8/site-packages/torch/utils/_contextlib.py:115: in decorate_context                 
    return func(*args, **kwargs)                                                                                             
src/transformers/generation/utils.py:1969: in generate                                                                       
    result = self._sample(                                                                                                   
src/transformers/generation/utils.py:2912: in _sample                                                                        
    outputs = self(**model_inputs, return_dict=True)                                                                         
../../.condax/mamba/envs/bnb/lib/python3.8/site-packages/torch/nn/modules/module.py:1532: in _wrapped_call_impl              
    return self._call_impl(*args, **kwargs)                                                                                  
../../.condax/mamba/envs/bnb/lib/python3.8/site-packages/torch/nn/modules/module.py:1541: in _call_impl                      
    return forward_call(*args, **kwargs)                                                                                     
../../.condax/mamba/envs/bnb/lib/python3.8/site-packages/accelerate/hooks.py:169: in new_forward                             
    output = module._old_forward(*args, **kwargs)                                                                            
src/transformers/models/bloom/modeling_bloom.py:848: in forward                                                              
    transformer_outputs = self.transformer(                                                                                  
../../.condax/mamba/envs/bnb/lib/python3.8/site-packages/torch/nn/modules/module.py:1532: in _wrapped_call_impl              
    return self._call_impl(*args, **kwargs)                                                                                  
../../.condax/mamba/envs/bnb/lib/python3.8/site-packages/torch/nn/modules/module.py:1541: in _call_impl                      
    return forward_call(*args, **kwargs)                                                                                     
../../.condax/mamba/envs/bnb/lib/python3.8/site-packages/accelerate/hooks.py:169: in new_forward                             
    output = module._old_forward(*args, **kwargs)                                                                            
src/transformers/models/bloom/modeling_bloom.py:712: in forward                                                              
    outputs = block(                                                                                                         
../../.condax/mamba/envs/bnb/lib/python3.8/site-packages/torch/nn/modules/module.py:1532: in _wrapped_call_impl              
    return self._call_impl(*args, **kwargs)                   
../../.condax/mamba/envs/bnb/lib/python3.8/site-packages/torch/nn/modules/module.py:1541: in _call_impl                      
    return forward_call(*args, **kwargs)                                                                                     
../../.condax/mamba/envs/bnb/lib/python3.8/site-packages/accelerate/hooks.py:169: in new_forward                             
    output = module._old_forward(*args, **kwargs)                                                                            
src/transformers/models/bloom/modeling_bloom.py:400: in forward                                                              
    attn_outputs = self.self_attention(                                                                                      
../../.condax/mamba/envs/bnb/lib/python3.8/site-packages/torch/nn/modules/module.py:1532: in _wrapped_call_impl              
    return self._call_impl(*args, **kwargs)                                                                                  
../../.condax/mamba/envs/bnb/lib/python3.8/site-packages/torch/nn/modules/module.py:1541: in _call_impl                      
    return forward_call(*args, **kwargs)                                                                                     
../../.condax/mamba/envs/bnb/lib/python3.8/site-packages/accelerate/hooks.py:169: in new_forward                             
    output = module._old_forward(*args, **kwargs)                                                                            
src/transformers/models/bloom/modeling_bloom.py:251: in forward                                                              
    fused_qkv = self.query_key_value(hidden_states)  # [batch_size, seq_length, 3 x hidden_size]                             
../../.condax/mamba/envs/bnb/lib/python3.8/site-packages/torch/nn/modules/module.py:1532: in _wrapped_call_impl              
    return self._call_impl(*args, **kwargs)                                                                                  
../../.condax/mamba/envs/bnb/lib/python3.8/site-packages/torch/nn/modules/module.py:1541: in _call_impl                      
    return forward_call(*args, **kwargs)                                                                                     
../../.condax/mamba/envs/bnb/lib/python3.8/site-packages/accelerate/hooks.py:169: in new_forward                             
    output = module._old_forward(*args, **kwargs)                                                                            
../bnb/bitsandbytes/nn/modules.py:475: in forward                                                                            
    out = bnb.matmul_4bit(x, self.weight.t(), bias=bias, quant_state=self.weight.quant_state)                                
../bnb/bitsandbytes/autograd/_functions.py:586: in matmul_4bit                                                               
    out = F.gemv_4bit(A, B.t(), out, state=quant_state)                                                                      
../bnb/bitsandbytes/functional.py:1506: in gemv_4bit                                                                         
    return backends[A.device.type].gemv_4bit(                                                                                
../bnb/bitsandbytes/backends/cpu.py:171: in gemv_4bit                                                                        
    return gemm_4bit_impl(A, B, out, transposed_A, transposed_B, state)                                                      
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
                                                                                                                             
A = tensor([[[ 0.0030,  0.2297,  0.1515,  ..., -0.7241, -0.2993, -0.6665],                                                   
         [-0.8872, -0.4702,  0.6714,  ...,  0.2...62,  0.1533,  ..., -0.0767, -0.4731,  0.0826],                             
         [-0.1199,  0.2257,  0.4138,  ...,  0.1516,  0.3250, -0.5029]]])                                                     
B = tensor([231, 102, 109,  ...,  44, 100, 203], dtype=torch.uint8), out = None, transposed_A = False, transposed_B = False  
state = <bitsandbytes.utils.QuantState object at 0x7f6d1f90d760>                                                             
                                                                                                                             
    def gemm_4bit_impl(                                                                                                      
        A: torch.Tensor,                                                                                                     
        B: torch.Tensor,                                                                                                     
        out: Optional[torch.Tensor] = None,                                                                                  
        transposed_A=False,                                                                                                  
        transposed_B=False,                                                                                                  
        state: QuantState = None,                                                                                            
    ) -> torch.Tensor:                                                                                                       
        """                                                                                                                  
        Matrix-matrix multiplication with 4-bit quantization.                                                                
                                                                                                                             
        Parameters                                                                                                           
        ----------                                                                                                           
        A : torch.Tensor
            The first input tensor. Usually the activation tensor.
        B : torch.Tensor
            The second input tensor. Usually the weight tensor.
        out : torch.Tensor
            The output tensor.
        transposed_A : bool
            Whether A is transposed
        transposed_B : bool
            Whether B is transposed
        state : QuantState
            Contains quantization info, such as blocksize and dtype
     
        Returns
        -------
        torch.Tensor:
            GEMM output tensor.
        """
        if ipex_cpu and _ipex_cpu_version_prereq(2, 3) and hasattr(state, "op_context"):
            assert state.op_context is not None
            output = torch.ops.torch_ipex.ipex_woq_linear(A, state.op_context.get_data_handle())
        else:
            dqB = dequantize_4bit_impl(B, state, blocksize=state.blocksize)
>           output = torch.matmul(A, dqB)
E           RuntimeError: expected m1 and m2 to have the same dtype, but got: float != c10::Half

../bnb/bitsandbytes/backends/cpu_xpu_common.py:527: RuntimeError
```

Hi @Titus-von-Koeller , as we have fixed the issue [1285](https://github.com/bitsandbytes-foundation/bitsandbytes/pull/1285), do you have any updates?

Hey @jiqing-feng,

Yes, I'm actively working on it and currently struggling with these remaining failing tests. Do any of these catch your eye, could you help fixing them?

```
=============================================================================================== FAILURES ================================================================================================
_____________________________________________________________________________ Bnb4BitTest.test_generate_quality_dequantize ______________________________________________________________________________

self = <bnb.test_4bit.Bnb4BitTest testMethod=test_generate_quality_dequantize>

    def test_generate_quality_dequantize(self):
        r"""
        Test that loading the model and unquantize it produce correct results
        """
        bnb_config = BitsAndBytesConfig(load_in_4bit=True)
    
        model_4bit = AutoModelForCausalLM.from_pretrained(
            self.model_name, quantization_config=bnb_config, device_map="auto"
        )
    
        model_4bit.dequantize()
    
        encoded_input = self.tokenizer(self.input_text, return_tensors="pt")
>       output_sequences = model_4bit.generate(input_ids=encoded_input["input_ids"].to(self.device), max_new_tokens=10)

tests/quantization/bnb/test_4bit.py:285: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
../../.condax/mamba/envs/bnb/lib/python3.8/site-packages/torch/utils/_contextlib.py:115: in decorate_context
    return func(*args, **kwargs)
src/transformers/generation/utils.py:1969: in generate
    result = self._sample(
src/transformers/generation/utils.py:2912: in _sample
    outputs = self(**model_inputs, return_dict=True)
../../.condax/mamba/envs/bnb/lib/python3.8/site-packages/torch/nn/modules/module.py:1532: in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
../../.condax/mamba/envs/bnb/lib/python3.8/site-packages/torch/nn/modules/module.py:1541: in _call_impl
    return forward_call(*args, **kwargs)
../../.condax/mamba/envs/bnb/lib/python3.8/site-packages/accelerate/hooks.py:169: in new_forward
    output = module._old_forward(*args, **kwargs)
src/transformers/models/bloom/modeling_bloom.py:848: in forward
    transformer_outputs = self.transformer(
../../.condax/mamba/envs/bnb/lib/python3.8/site-packages/torch/nn/modules/module.py:1532: in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
../../.condax/mamba/envs/bnb/lib/python3.8/site-packages/torch/nn/modules/module.py:1541: in _call_impl
    return forward_call(*args, **kwargs)
../../.condax/mamba/envs/bnb/lib/python3.8/site-packages/accelerate/hooks.py:169: in new_forward
    output = module._old_forward(*args, **kwargs)
src/transformers/models/bloom/modeling_bloom.py:712: in forward
    outputs = block(
../../.condax/mamba/envs/bnb/lib/python3.8/site-packages/torch/nn/modules/module.py:1532: in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
../../.condax/mamba/envs/bnb/lib/python3.8/site-packages/torch/nn/modules/module.py:1541: in _call_impl
    return forward_call(*args, **kwargs)
../../.condax/mamba/envs/bnb/lib/python3.8/site-packages/accelerate/hooks.py:169: in new_forward
    output = module._old_forward(*args, **kwargs)
src/transformers/models/bloom/modeling_bloom.py:400: in forward
    attn_outputs = self.self_attention(
../../.condax/mamba/envs/bnb/lib/python3.8/site-packages/torch/nn/modules/module.py:1532: in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
../../.condax/mamba/envs/bnb/lib/python3.8/site-packages/torch/nn/modules/module.py:1541: in _call_impl
    return forward_call(*args, **kwargs)
../../.condax/mamba/envs/bnb/lib/python3.8/site-packages/accelerate/hooks.py:169: in new_forward
    output = module._old_forward(*args, **kwargs)
src/transformers/models/bloom/modeling_bloom.py:251: in forward
    fused_qkv = self.query_key_value(hidden_states)  # [batch_size, seq_length, 3 x hidden_size]
../../.condax/mamba/envs/bnb/lib/python3.8/site-packages/torch/nn/modules/module.py:1532: in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
../../.condax/mamba/envs/bnb/lib/python3.8/site-packages/torch/nn/modules/module.py:1541: in _call_impl
    return forward_call(*args, **kwargs)
../../.condax/mamba/envs/bnb/lib/python3.8/site-packages/accelerate/hooks.py:169: in new_forward
    output = module._old_forward(*args, **kwargs)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

self = Linear(in_features=2048, out_features=6144, bias=True)
input = tensor([[[ 0.0030,  0.2297,  0.1515,  ..., -0.7241, -0.2993, -0.6665],
         [-0.8872, -0.4702,  0.6714,  ...,  0.2...-0.4731,  0.0826],
         [-0.1199,  0.2257,  0.4138,  ...,  0.1516,  0.3250, -0.5029]]],
       dtype=torch.float16)

    def forward(self, input: Tensor) -> Tensor:
>       return F.linear(input, self.weight, self.bias)
E       RuntimeError: mat1 and mat2 shapes cannot be multiplied (4x2048 and 6144x2048)

../../.condax/mamba/envs/bnb/lib/python3.8/site-packages/torch/nn/modules/linear.py:116: RuntimeError
___________________________________________________________________________________ Bnb4BitTestTraining.test_training ___________________________________________________________________________________

self = <bnb.test_4bit.Bnb4BitTestTraining testMethod=test_training>

    def test_training(self):
        if version.parse(importlib.metadata.version("bitsandbytes")) < version.parse("0.37.0"):
            self.skipTest(reason="This test requires bitsandbytes >= 0.37.0")
    
        # Step 1: freeze all parameters
        model = AutoModelForCausalLM.from_pretrained(self.model_name, load_in_4bit=True)
    
>       self.assertEqual(set(model.hf_device_map.values()), {torch.cuda.current_device()})

tests/quantization/bnb/test_4bit.py:544: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

self = OPTForCausalLM(
  (model): OPTModel(
    (decoder): OPTDecoder(
      (embed_tokens): Embedding(50272, 512, padding_id...entwise_affine=True)
        )
      )
    )
  )
  (lm_head): Linear(in_features=512, out_features=50272, bias=False)
)
name = 'hf_device_map'

    def __getattr__(self, name: str) -> Any:
        if '_parameters' in self.__dict__:
            _parameters = self.__dict__['_parameters']
            if name in _parameters:
                return _parameters[name]
        if '_buffers' in self.__dict__:
            _buffers = self.__dict__['_buffers']
            if name in _buffers:
                return _buffers[name]
        if '_modules' in self.__dict__:
            modules = self.__dict__['_modules']
            if name in modules:
                return modules[name]
>       raise AttributeError(f"'{type(self).__name__}' object has no attribute '{name}'")
E       AttributeError: 'OPTForCausalLM' object has no attribute 'hf_device_map'

../../.condax/mamba/envs/bnb/lib/python3.8/site-packages/torch/nn/modules/module.py:1709: AttributeError
_______________________________________________________________________________ Bnb4BitGPT2Test.test_fp32_4bit_conversion _______________________________________________________________________________

self = <bnb.test_4bit.Bnb4BitGPT2Test testMethod=test_fp32_4bit_conversion>

    def test_fp32_4bit_conversion(self):
        r"""
        Test whether it is possible to mix both `4bit` and `fp32` weights when using `keep_in_fp32_modules` correctly.
        """
        model = AutoModelForSeq2SeqLM.from_pretrained("google-t5/t5-small", load_in_4bit=True, device_map="auto")
>       self.assertTrue(model.decoder.block[0].layer[2].DenseReluDense.wo.weight.dtype == torch.float32)
E       AssertionError: False is not true

tests/quantization/bnb/test_4bit.py:334: AssertionError
___________________________________________________________________________ Bnb4BitGPT2Test.test_generate_quality_dequantize ____________________________________________________________________________

self = <bnb.test_4bit.Bnb4BitGPT2Test testMethod=test_generate_quality_dequantize>

    def test_generate_quality_dequantize(self):
        r"""
        Test that loading the model and unquantize it produce correct results
        """
        bnb_config = BitsAndBytesConfig(load_in_4bit=True)
    
        model_4bit = AutoModelForCausalLM.from_pretrained(
            self.model_name, quantization_config=bnb_config, device_map="auto"
        )
    
        model_4bit.dequantize()
    
        encoded_input = self.tokenizer(self.input_text, return_tensors="pt")
>       output_sequences = model_4bit.generate(input_ids=encoded_input["input_ids"].to(self.device), max_new_tokens=10)

tests/quantization/bnb/test_4bit.py:285: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
../../.condax/mamba/envs/bnb/lib/python3.8/site-packages/torch/utils/_contextlib.py:115: in decorate_context
    return func(*args, **kwargs)
src/transformers/generation/utils.py:1969: in generate
    result = self._sample(
src/transformers/generation/utils.py:2912: in _sample
    outputs = self(**model_inputs, return_dict=True)
../../.condax/mamba/envs/bnb/lib/python3.8/site-packages/torch/nn/modules/module.py:1532: in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
../../.condax/mamba/envs/bnb/lib/python3.8/site-packages/torch/nn/modules/module.py:1541: in _call_impl
    return forward_call(*args, **kwargs)
../../.condax/mamba/envs/bnb/lib/python3.8/site-packages/accelerate/hooks.py:169: in new_forward
    output = module._old_forward(*args, **kwargs)
src/transformers/models/gpt2/modeling_gpt2.py:1315: in forward
    transformer_outputs = self.transformer(
../../.condax/mamba/envs/bnb/lib/python3.8/site-packages/torch/nn/modules/module.py:1532: in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
../../.condax/mamba/envs/bnb/lib/python3.8/site-packages/torch/nn/modules/module.py:1541: in _call_impl
    return forward_call(*args, **kwargs)
../../.condax/mamba/envs/bnb/lib/python3.8/site-packages/accelerate/hooks.py:169: in new_forward
    output = module._old_forward(*args, **kwargs)
src/transformers/models/gpt2/modeling_gpt2.py:1129: in forward
    outputs = block(
../../.condax/mamba/envs/bnb/lib/python3.8/site-packages/torch/nn/modules/module.py:1532: in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
../../.condax/mamba/envs/bnb/lib/python3.8/site-packages/torch/nn/modules/module.py:1541: in _call_impl
    return forward_call(*args, **kwargs)
../../.condax/mamba/envs/bnb/lib/python3.8/site-packages/accelerate/hooks.py:169: in new_forward
    output = module._old_forward(*args, **kwargs)
src/transformers/models/gpt2/modeling_gpt2.py:614: in forward
    attn_outputs = self.attn(
../../.condax/mamba/envs/bnb/lib/python3.8/site-packages/torch/nn/modules/module.py:1532: in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
../../.condax/mamba/envs/bnb/lib/python3.8/site-packages/torch/nn/modules/module.py:1541: in _call_impl
    return forward_call(*args, **kwargs)
../../.condax/mamba/envs/bnb/lib/python3.8/site-packages/accelerate/hooks.py:169: in new_forward
    output = module._old_forward(*args, **kwargs)
src/transformers/models/gpt2/modeling_gpt2.py:517: in forward
    query, key, value = self.c_attn(hidden_states).split(self.split_size, dim=2)
../../.condax/mamba/envs/bnb/lib/python3.8/site-packages/torch/nn/modules/module.py:1532: in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
../../.condax/mamba/envs/bnb/lib/python3.8/site-packages/torch/nn/modules/module.py:1541: in _call_impl
    return forward_call(*args, **kwargs)
../../.condax/mamba/envs/bnb/lib/python3.8/site-packages/accelerate/hooks.py:169: in new_forward
    output = module._old_forward(*args, **kwargs)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

self = Linear(in_features=1600, out_features=4800, bias=True)
input = tensor([[[ 0.1334, -0.0541, -0.0432,  ...,  0.0113,  0.1115,  0.0245],
         [-0.1196,  0.0271, -0.1114,  ...,  0.0...-0.3340,  0.0841],
         [-0.1744, -0.0175, -0.0184,  ..., -0.5654, -0.4402, -0.0274]]],
       dtype=torch.float16)

    def forward(self, input: Tensor) -> Tensor:
>       return F.linear(input, self.weight, self.bias)
E       RuntimeError: mat1 and mat2 shapes cannot be multiplied (4x1600 and 4800x1600)

../../.condax/mamba/envs/bnb/lib/python3.8/site-packages/torch/nn/modules/linear.py:116: RuntimeError
```

Particularly the tests with

```
RuntimeError: mat1 and mat2 shapes cannot be multiplied (4x1600 and 4800x1600)
```

seem to me that they might be related to the Intel backend logic.

Ok, so I just pushed my changes with which I ran all the tests. Find attached also the 8-bit related test results. There are a bunch of failures, essentially three types (just search the logs for `E   `):

1. `RuntimeError: self and mat2 must have the same dtype, but got Half and BFloat16`
2. `AttributeError: 'NoneType' object has no attribute 'device'`
3. `AttributeError: 'OPTForCausalLM' object has no attribute 'hf_device_map'`

No idea about the root causes yet.

@jiqing-feng I saw you also changed the HF 8bit quantizer. Not sure to what extent these tests should be working [based on the Intel 8bit BNB PR](https://github.com/bitsandbytes-foundation/bitsandbytes/pull/1178) (cc @Xia-Weiwen) and which parts show an issue on the BNB side and which might be related to further fixes needed in the tests on the Transformer side. Would extremely appreciate your input on that.

See [transf_multi-backend_8bit-tests.log](https://github.com/user-attachments/files/16433845/transf_multi-backend_8bit-tests.log).

Do the other changes I introduced seem sensible to you? Would be happy to hear your opinion.

cc @SunMarc do you have an idea of how error type 3 could come to be? looks more like sth on the Transformer side to me..

> RuntimeError: mat1 and mat2 shapes cannot be multiplied (4x1600 and 4800x1600)
> seem to me that they might be related to the Intel backend logic.

This [PR](https://github.com/bitsandbytes-foundation/bitsandbytes/pull/1300) could fix this error 

@jiqing-feng Unfortunately, it doesn't, see the [transf_multi-backend_4bit-tests.log](https://github.com/user-attachments/files/16440618/transf_multi-backend_4bit-tests.log):

```
❯ rg -o 'E\s\s+(.*)' ~/Downloads/transf_multi-backend_4bit-tests.log -r '$1' | sort | uniq -c | sort -rn

   4 RuntimeError: mat1 and mat2 shapes cannot be multiplied (7x768 and 3072x768)
   3 RuntimeError: mat1 and mat2 shapes cannot be multiplied (4x2048 and 6144x2048)
   2 RuntimeError: mat1 and mat2 shapes cannot be multiplied (4x1600 and 4800x1600)
   2 RuntimeError: mat1 and mat2 shapes cannot be multiplied (12x512 and 2048x512)
   1 AttributeError: 'OPTForCausalLM' object has no attribute 'hf_device_map'
   1 AssertionError: False is not true
```

You can validate yourself with the following commands:
```
RUN_SLOW=1 pytest tests/quantization/bnb/test_4bit.py -rsx -v
RUN_SLOW=1 pytest tests/quantization/bnb/test_mixed_int8.py -rsx -v
```

Since it's practical to have this overview of failed tests, I'm also pasting the current status of failing tests for the 8-bit tests:
```
❯ rg -o 'E\s\s+(.*)' ~/Downloads/transf_multi-backend_8bit-tests.log -r '$1' | sort | uniq -c | sort -rn

   9 AttributeError: 'NoneType' object has no attribute 'device'
   2 RuntimeError: self and mat2 must have the same dtype, but got Half and BFloat16
   1 AttributeError: 'OPTForCausalLM' object has no attribute 'hf_device_map'
```

Who could help me in figuring out how to get the 8-bit tests working from the Intel side? Would that be @Xia-Weiwen I invited him to the new #bitsandbytes-intel-collab Slack channel. Could you communicate internally with him that he accepts the invite and/or let me know who else I should invite?

1. For `self and mat2 must have the same dtype, but got Half and BFloat16`: The 4bit has been fixed by [1285](https://github.com/bitsandbytes-foundation/bitsandbytes/pull/1285). The 8bit comes from [here](https://github.com/huggingface/transformers/blob/main/src/transformers/integrations/bitsandbytes.py#L334-L364). I think we should pass the original dtype so we can dequant the model's type to the original dtype. WDYT? I fixed it by this [commit](https://github.com/huggingface/transformers/pull/31098/commits/48578048d5a5bf619ea9d18f57ae4d04c5085b17), please let me know your opinion.(https://github.com/bitsandbytes-foundation/bitsandbytes/pull/1285)
2. For `mat1 and mat2 shapes cannot be multiplied`: The 4bit has been fixed by [1300](https://github.com/bitsandbytes-foundation/bitsandbytes/pull/1300). 
3. For `'OPTForCausalLM' object has no attribute 'hf_device_map'`: I suppose the CPU model doesn't have `device_map` or `hf_device_map`, we need to discuss with the transformers maintainer whether we should enable device_map in the CPU model, or we should disable `device_map` check in CPU model. And we cannot pass `device_map='auto'` in CPU models.
4. For `'NoneType' object has no attribute 'device'`: I cannot produce the error if I set `device_map="cpu"`.

> Who could help me in figuring out how to get the 8-bit tests working from the Intel side? Would that be @Xia-Weiwen I invited him to the new #bitsandbytes-intel-collab Slack channel. Could you communicate internally with him that he accepts the invite and/or let me know who else I should invite?

I have communicated with him and he should be in the group.


Hey @SunMarc 🤗 

> For 'OPTForCausalLM' object has no attribute 'hf_device_map': I suppose the CPU model doesn't have device_map or hf_device_map, we need to discuss with the transformers maintainer whether we should enable device_map in the CPU model, or we should disable device_map check in CPU model. And we cannot pass device_map='auto' in CPU models.

Do you have an opinion on this or do you have a suggestion whom of our colleagues we could pull in to help answer this question?

@jiqing-feng

Current status of open failing tests:
```
4-bit Test Error Summary:
      4 RuntimeError: mat1 and mat2 shapes cannot be multiplied (7x768 and 3072x768)
      1 AttributeError: 'OPTForCausalLM' object has no attribute 'hf_device_map'

8-bit Test Error Summary:
      9 AttributeError: 'NoneType' object has no attribute 'device'
      1 AttributeError: 'OPTForCausalLM' object has no attribute 'hf_device_map'
```


> 1. For `self and mat2 must have the same dtype, but got Half and BFloat16` [...]

YES, these are all fixed, thanks 🤗 !

> 2. For `mat1 and mat2 shapes cannot be multiplied`: The 4bit has been fixed by [1300](https://github.com/bitsandbytes-foundation/bitsandbytes/pull/1300).

That PR fixed most of them, but not all:

all fixed
```
3x RuntimeError: mat1 and mat2 shapes cannot be multiplied (4x2048 and 6144x2048)
2x RuntimeError: mat1 and mat2 shapes cannot be multiplied (4x1600 and 4800x1600)
2x RuntimeError: mat1 and mat2 shapes cannot be multiplied (12x512 and 2048x512)
```

remaining in 4-bit, even with current fix:
```
4x RuntimeError: mat1 and mat2 shapes cannot be multiplied (7x768 and 3072x768)
```

> 3. For `'OPTForCausalLM' object has no attribute 'hf_device_map'`:

Agreed, also not an expert on this, let's see what Marc says.

> 4. For `'NoneType' object has no attribute 'device'`: I cannot produce the error if I set `device_map="cpu"`.

I tried replacing `device_map="auto"` in `transformers/src/transformers/quantizers/quantizer_bnb_8bit.py` with `device_map="auto" if torch.cuda.is_available() else "cpu",` but that didn't change the test results regarding NoneType at all:

`9x AttributeError: 'NoneType' object has no attribute 'device'` for the 8 bit test before or after the replacement. Is that what you meant?

If it's sth else, please try changing the tests accordingly and see for yourself if the tests pass or not. On the Intel dev VM that you also have access to I created a command `integration` that you can run from any directory and that will output the test logs to stdout and also to file in the ~/src/workbench/multi-backend/logs dir. Please feel free to work there and share the results here. The below style of summary will be written out as `summary.log` in the just-mentioned directory.

Hi @Titus-von-Koeller 

For 4bit matmul dim mismatch: Fixed by [1301](https://github.com/bitsandbytes-foundation/bitsandbytes/pull/1301)

For 8bit None type error: Fixed by [1303](https://github.com/bitsandbytes-foundation/bitsandbytes/pull/1303)

Ok, all tests passing now, but the below:

```
=================================== FAILURES ===================================
______________________ Bnb4BitTestTraining.test_training _______________________

self = <bnb.test_4bit.Bnb4BitTestTraining testMethod=test_training>

    def test_training(self):
        if version.parse(importlib.metadata.version("bitsandbytes")) < version.parse("0.37.0"):
            self.skipTest(reason="This test requires bitsandbytes >= 0.37.0")
    
        # Step 1: freeze all parameters
        model = AutoModelForCausalLM.from_pretrained(self.model_name, load_in_4bit=True)
    
>       self.assertEqual(set(model.hf_device_map.values()), {torch.cuda.current_device()})

tests/quantization/bnb/test_4bit.py:523: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = OPTForCausalLM(
  (model): OPTModel(
    (decoder): OPTDecoder(
      (embed_tokens): Embedding(50272, 512, padding_id...entwise_affine=True)
        )
      )
    )
  )
  (lm_head): Linear(in_features=512, out_features=50272, bias=False)
)
name = 'hf_device_map'

    def __getattr__(self, name: str) -> Any:
        if '_parameters' in self.__dict__:
            _parameters = self.__dict__['_parameters']
            if name in _parameters:
                return _parameters[name]
        if '_buffers' in self.__dict__:
            _buffers = self.__dict__['_buffers']
            if name in _buffers:
                return _buffers[name]
        if '_modules' in self.__dict__:
            modules = self.__dict__['_modules']
            if name in modules:
                return modules[name]
>       raise AttributeError(f"'{type(self).__name__}' object has no attribute '{name}'")
E       AttributeError: 'OPTForCausalLM' object has no attribute 'hf_device_map'

../../.condax/mamba/envs/bnb/lib/python3.8/site-packages/torch/nn/modules/module.py:1709: AttributeError


_____________________ MixedInt8TestTraining.test_training ______________________

self = <bnb.test_mixed_int8.MixedInt8TestTraining testMethod=test_training>

    def test_training(self):
        if version.parse(importlib.metadata.version("bitsandbytes")) < version.parse("0.37.0"):
            self.skipTest(reason="This test requires bitsandbytes>=0.37.0")
    
        # Step 1: freeze all parameters
        model = AutoModelForCausalLM.from_pretrained(self.model_name, load_in_8bit=True)
    
>       self.assertEqual(set(model.hf_device_map.values()), {torch.cuda.current_device()})

tests/quantization/bnb/test_mixed_int8.py:858: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = OPTForCausalLM(
  (model): OPTModel(
    (decoder): OPTDecoder(
      (embed_tokens): Embedding(50272, 512, padding_id...entwise_affine=True)
        )
      )
    )
  )
  (lm_head): Linear(in_features=512, out_features=50272, bias=False)
)
name = 'hf_device_map'

    def __getattr__(self, name: str) -> Any:
        if '_parameters' in self.__dict__:
            _parameters = self.__dict__['_parameters']
            if name in _parameters:
                return _parameters[name]
        if '_buffers' in self.__dict__:
            _buffers = self.__dict__['_buffers']
            if name in _buffers:
                return _buffers[name]
        if '_modules' in self.__dict__:
            modules = self.__dict__['_modules']
            if name in modules:
                return modules[name]
>       raise AttributeError(f"'{type(self).__name__}' object has no attribute '{name}'")
E       AttributeError: 'OPTForCausalLM' object has no attribute 'hf_device_map'

../../.condax/mamba/envs/bnb/lib/python3.8/site-packages/torch/nn/modules/module.py:1709: AttributeError
```

Ok, so with this change I could avoid the device map error:

```diff
-        self.assertEqual(set(model.hf_device_map.values()), {torch.cuda.current_device()})
+        if torch.cuda.is_available():
+            self.assertEqual(set(model.hf_device_map.values()), {torch.cuda.current_device()})
+        else:
+            self.assertTrue(all(param.device.type == "cpu" for param in model.parameters()))
```
It seems for GPU it triggers Accelerate and therefore has `hf_device_map` and for CPU it doesn't so you need to do another check..

However, this brought to light two more failures:

```
4-bit Test Error Summary:
      1 RuntimeError: expected m1 and m2 to have the same dtype, but got: c10::Half != float

8-bit Test Error Summary:
      1 RuntimeError: expected m1 and m2 to have the same dtype, but got: c10::Half != float
```

[Find the detailed test failure logs here](https://github.com/bitsandbytes-foundation/workbench/tree/main/multi-backend/logs)

Reproduce with:
```bash
export RUN_SLOW=1 && pytest tests/quantization/bnb/test_4bit.py::Bnb4BitTestTraining::test_training -rsx -v && pytest tests/quantization/bnb/test_mixed_int8.py::MixedInt8TestTraining::test_training
```

All other tests are passing now.

Hi @Titus-von-Koeller 

For `RuntimeError: expected m1 and m2 to have the same dtype, but got: c10::Half != float`. The error comes from the default type. The default type of opt model is float16 which is the same as cuda bnb dtype, but bfloat16 in CPU. In that case, the error will be in cuda if you replace the model with a bf16 model like [mistral](https://huggingface.co/mistralai/Mistral-7B-Instruct-v0.2). 

So we can fix the model dtype to fp32 by replacing `model = AutoModelForCausalLM.from_pretrained(self.model_name, load_in_8bit=True)` to `model = AutoModelForCausalLM.from_pretrained(self.model_name, load_in_8bit=True, torch_dtype=torch.float32)`. Then, the error will disappear.


x-posting here what I already wrote on Slack. cc @SunMarc Do you have any idea what might be going on here?

---

In the BNB integration tests, which I modified in the PR for Intel CPU, I encountered an error in the test `Bnb4BitTestTraining::test_training` with the line `model = AutoModelForCausalLM.from_pretrained(self.model_name, load_in_8bit=True)`. The error was as follows:

```
self = Linear(in_features=1024, out_features=16, bias=False)
input = tensor([[[ 0.0041, -0.0679, -0.0397,  ...,  0.0058,  0.0221, -0.0017],
         [-0.0280,  0.0693,  0.0341,  ..., -0.0...-0.0358,  0.0205],
         [-0.0345,  0.0459,  0.0049,  ..., -0.0307,  0.0067, -0.0451]]],
       dtype=torch.float16)

def forward(self, input: Tensor) -> Tensor:
    return F.linear(input, self.weight, self.bias)
RuntimeError: expected m1 and m2 to have the same dtype, but got: c10::Half != float
```

When using `model = AutoModelForCausalLM.from_pretrained(..., torch_dtype=torch.bfloat16 if not torch.cuda.is_available() else None)`, I received the following error:

```
self = Linear(in_features=1024, out_features=16, bias=False)
input = tensor([[[ 0.0032, -0.0679, -0.0405,  ...,  0.0045,  0.0244, -0.0016],
         [-0.0276,  0.0708,  0.0344,  ..., -0.0...0.0322,  0.0205],
         [-0.0339,  0.0447,  0.0045,  ..., -0.0334,  0.0092, -0.0452]]],
       dtype=torch.bfloat16)

def forward(self, input: Tensor) -> Tensor:
    return F.linear(input, self.weight, self.bias)
RuntimeError: expected m1 and m2 to have the same dtype, but got: c10::BFloat16 != float
```

However, when setting `torch_dtype=torch.float32`, there is no error. I find this behavior very weird.

The strange thing is that the failing test becomes passing if I remove [the torch.amp line](https://github.com/huggingface/transformers/blob/458b0cd2c544cdd6c700f9b0c21077c889bcee6c/tests/quantization/bnb/test_4bit.py#L523)

You can see that the input is actually changed and not the weight in the test error output `input = tensor(...., dtype=torch.bfloat16)` and the dtype of the weight is visible in the RuntimeError.

Hi @Titus-von-Koeller . I just recognized we use the wrong autocast for CPU, the correct usage should be `with torch.autocast(device_type):` which the `device_type` should be `"cpu"` or `"cuda"`.

The failed CI is not related to my changes.

Ok, I think we're through with this PR from our side, could you please review @SunMarc?

We're looking forward to merge this, so we can announce the alpha release.

This is where we stand regarding integration test suite execution ([see script](https://github.com/bitsandbytes-foundation/workbench/blob/main/multi-backend/run_quant_tests.sh)) on the 4 different backends ([logs here](https://github.com/bitsandbytes-foundation/workbench/tree/main/multi-backend/logs); including CUDA for reference and checking that we didn't break anything via the test refactor):

---

- CUDA: no tests failing, neither with bnb `main` nor `multi-backend-refactor` branches
- Intel CPU: all tests pass!
- ROCm:
  - 4-bit: all tests pass!
  - 8-bit:
    - 20 passed
    - 24 Exception: cublasLt ran into an error! (needs fixing in upcoming ROCm release)
    - 1 AssertionError: False is not true (needs fixing on Python side, communicated to AMD team)
- XPU: WIP, however integration test execution itself now successfully executes; failing tests need further implementation work on BNB XPU Backend side.

The failing tests in XPU and ROCm are not related to the Transformer code base and are therefore not a hold-up in merging this PR. They need fixing/implementing on the ROCm and BNB side. Also, we're already catching "Not Implemented" errors and skipping those tests, but for the ROCm exceptions this wasn't yet possible as they will need to be made more specific; which is currently being worked on by AMD on the BNB side.

---

In conclusion, in my view at this time there's no apparent reason that would keep us from merging, other than now needing a thorough look by you and fixing whatever comes up in the review :)

Let us know what you think, thanks 🤗 

cc @jiqing-feng @faaany @pnunna93 @matthewdouglas 

P.S. I'm getting some errors when trying to fix the failing CI checks, they seem to come from elsewhere and the merge from `main` didn't fix it.

How would you advise to proceed? For now I didn't commit the changed files:

```
❯ make style; make quality
ruff check examples tests src utils setup.py conftest.py --fix --exclude ""
ruff format examples tests src utils setup.py conftest.py --exclude ""
22 files reformatted, 3042 files left unchanged
make autogenerate_code
make[1]: Entering directory '/home/sdp/src/transformers'
running deps_table_update
updating src/transformers/dependency_versions_table.py
make[1]: Leaving directory '/home/sdp/src/transformers'
make extra_style_checks
make[1]: Entering directory '/home/sdp/src/transformers'
python utils/custom_init_isort.py
python utils/sort_auto_mappings.py
python utils/check_doc_toc.py --fix_and_overwrite
make[1]: Leaving directory '/home/sdp/src/transformers'
/home/sdp/.condax/mamba/envs/bnb/lib/python3.10/site-packages/torchvision/io/image.py:13: UserWarning: Failed to load image Python extension: '/home/sdp/.condax/mamba/envs/bnb/lib/python3.10/site-packages/torchvision/image.so: undefined symbol: _ZN3c1017RegisterOperatorsD1Ev'If you don't plan on using image functionality from `torchvision.io`, you can ignore this warning. Otherwise, there might be something wrong with your environment. Did you have `libjpeg` or `libpng` installed before building `torchvision` from source?
  warn(
/home/sdp/src/transformers/src/transformers/deepspeed.py:24: FutureWarning: transformers.deepspeed module is deprecated and will be removed in a future version. Please import deepspeed modules directly from transformers.integrations
  warnings.warn(
Traceback (most recent call last):
  File "/home/sdp/src/transformers/src/transformers/utils/import_utils.py", line 1636, in _get_module
    return importlib.import_module("." + module_name, self.__name__)
  File "/home/sdp/.condax/mamba/envs/bnb/lib/python3.10/importlib/__init__.py", line 126, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1050, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1027, in _find_and_load
  File "<frozen importlib._bootstrap>", line 1006, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 688, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 883, in exec_module
  File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed
  File "/home/sdp/src/transformers/src/transformers/models/audio_spectrogram_transformer/feature_extraction_audio_spectrogram_transformer.py", line 30, in <module>
    import torchaudio.compliance.kaldi as ta_kaldi
  File "/home/sdp/.condax/mamba/envs/bnb/lib/python3.10/site-packages/torchaudio/__init__.py", line 1, in <module>
    from . import (  # noqa: F401
  File "/home/sdp/.condax/mamba/envs/bnb/lib/python3.10/site-packages/torchaudio/_extension/__init__.py", line 45, in <module>
    _load_lib("libtorchaudio")
  File "/home/sdp/.condax/mamba/envs/bnb/lib/python3.10/site-packages/torchaudio/_extension/utils.py", line 64, in _load_lib
    torch.ops.load_library(path)
  File "/home/sdp/.condax/mamba/envs/bnb/lib/python3.10/site-packages/torch/_ops.py", line 1295, in load_library
    ctypes.CDLL(path)
  File "/home/sdp/.condax/mamba/envs/bnb/lib/python3.10/ctypes/__init__.py", line 374, in __init__
    self._handle = _dlopen(self._name, mode)
OSError: /home/sdp/.condax/mamba/envs/bnb/lib/python3.10/site-packages/torchaudio/lib/libtorchaudio.so: undefined symbol: _ZNK5torch8autograd4Node4nameB5cxx11Ev

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "<frozen importlib._bootstrap>", line 1073, in _handle_fromlist
  File "<frozen importlib._bootstrap>", line 1075, in _handle_fromlist
  File "/home/sdp/src/transformers/src/transformers/utils/import_utils.py", line 1627, in __getattr__
    value = getattr(module, name)
  File "/home/sdp/src/transformers/src/transformers/utils/import_utils.py", line 1626, in __getattr__
    module = self._get_module(self._class_to_module[name])
  File "/home/sdp/src/transformers/src/transformers/utils/import_utils.py", line 1638, in _get_module
    raise RuntimeError(
RuntimeError: Failed to import transformers.models.audio_spectrogram_transformer.feature_extraction_audio_spectrogram_transformer because of the following error (look up to see its traceback):
/home/sdp/.condax/mamba/envs/bnb/lib/python3.10/site-packages/torchaudio/lib/libtorchaudio.so: undefined symbol: _ZNK5torch8autograd4Node4nameB5cxx11Ev
🚨 import failed, this means you introduced unprotected imports! 🚨
make: *** [Makefile:53: quality] Error 1

❯ git status
On branch bnb_cpu
Your branch is up to date with 'jiqing-feng/bnb_cpu'.

Changes not staged for commit:
  (use "git add <file>..." to update what will be committed)
  (use "git restore <file>..." to discard changes in working directory)
        modified:   examples/legacy/benchmarking/plot_csv_file.py
        modified:   examples/tensorflow/benchmarking/plot_csv_file.py
        modified:   src/transformers/generation/stopping_criteria.py
        modified:   src/transformers/models/big_bird/modeling_big_bird.py
        modified:   src/transformers/models/bigbird_pegasus/modeling_bigbird_pegasus.py
        modified:   src/transformers/models/convbert/modeling_convbert.py
        modified:   src/transformers/models/donut/convert_donut_to_pytorch.py
        modified:   src/transformers/models/esm/openfold_utils/feats.py
        modified:   src/transformers/models/esm/openfold_utils/tensor_utils.py
        modified:   src/transformers/models/longt5/convert_longt5x_checkpoint_to_flax.py
        modified:   src/transformers/models/mobilenet_v2/modeling_mobilenet_v2.py
        modified:   src/transformers/models/nougat/convert_nougat_to_hf.py
        modified:   src/transformers/models/swin/convert_swin_simmim_to_pytorch.py
        modified:   src/transformers/models/swin/convert_swin_timm_to_pytorch.py
        modified:   src/transformers/models/swin2sr/convert_swin2sr_original_to_pytorch.py
        modified:   src/transformers/models/swinv2/convert_swinv2_timm_to_pytorch.py
        modified:   src/transformers/models/t5/convert_t5x_checkpoint_to_flax.py
        modified:   src/transformers/models/umt5/convert_umt5_checkpoint_to_pytorch.py
        modified:   src/transformers/quantizers/base.py
        modified:   src/transformers/testing_utils.py
        modified:   src/transformers/tokenization_utils.py
        modified:   src/transformers/utils/fx.py
        modified:   utils/check_copies.py
```

Just freshly rebased onto `main`.

Hi @SunMarc . I have fixed all your comments, please take a review. Thx!
cc @Titus-von-Koeller 

Dear @SunMarc,

I made note of every single point you mentioned in your last review and the only changes since then [are all encapsulated in this last commit for you to review](https://github.com/huggingface/transformers/pull/31098/commits/463c211199e56d09612b6094974ebf6d53402250).

Everything is ready for a final review from our side. @ArthurZucker, would you be so kind to take a look? Marc said to contact you about this now. Let us know what you think 🤗 

P.S. XPU is still very much a work in progress on the BNB backend. The tests now execute successfully with that backend, too. However, we put a bit less attention to detail in the environment validation for that one, as we'll focus on that in a follow-up PR. For now we want to get the ROCm and Intel CPU out the door for the alpha testing release of these backends.

P.P.S. Please check [this post about the test coverage with the different backends](https://github.com/huggingface/transformers/pull/31098#issuecomment-2300005257) (I reran the tests on all relevant platforms, see Github repo for logs) and the post just below that regarding the failing CI checks.

cc @jiqing-feng @matthewdouglas 

Hi @SunMarc @ArthurZucker . The PR should be ready to merge, thx!

@cbensimon you may be interested here for ZeroGPU especially in combination with #33122.

> Super nice! This needs a bit of documentation!
> 
> 1. FMI but can e included in the doc: is it faster to quantize on CPU vs on the remote device
> 2. What is the use-case for CPU quantization
> 3. Can we cover the cost of quantizing by asynch sending to device orsomething like that?

1. This feature enables BNB on CPU without remote device instead of replacing the remote device. FMI, remote device will be faster than cpu in most cases. (If the remote device you mentioned is sth. like GPU)
2. The use-case in transformers is exactly the same as cuda.
3. @Titus-von-Koeller 

Hi @ArthurZucker . Thanks for your reminder. I removed the BNB cuda restriction in some docs, but there are too many docs in transformers, please let me know if I missed anywhere it appears. Thx!

@jiqing-feng I think the removal of that one sentence is not right. Instead we should add an info box about the backend refactor and link to the BNB docs. I'll handle that now, didn't have the time to finish this on Friday.

> @jiqing-feng I think the removal of that one sentence is not right. Instead we should add an info box about the backend refactor and link to the BNB docs. I'll handle that now, didn't have the time to finish this on Friday.

Hi @Titus-von-Koeller . I have attached the BNB docs now and fix all tests, but the #33122  introduces other issues like we cannot move to cuda if we installed the cpu backend bnb.

> Hi @Titus-von-Koeller . I have attached the BNB docs now and fix all tests, but the https://github.com/huggingface/transformers/pull/33122 introduces other issues like we cannot move to cuda if we installed the cpu backend bnb.

Hmm that's strange. Can you share the traceback ? This PR should allow users to actually move their model as it wasn't possible before. 

> > Hi @Titus-von-Koeller . I have attached the BNB docs now and fix all tests, but the #33122 introduces other issues like we cannot move to cuda if we installed the cpu backend bnb.
> 
> Hmm that's strange. Can you share the traceback ? This PR should allow users to actually move their model as it wasn't possible before.

Hi @SunMarc 
The cuda operation [here](https://github.com/huggingface/transformers/blob/main/tests/quantization/bnb/test_4bit.py#L318) and [here](https://github.com/huggingface/transformers/blob/main/tests/quantization/bnb/test_4bit.py#L289) is not safe because BNB has different backend. I checked cuda available before it but it is still not safe.

The failed tests are `tests/quantization/bnb/test_4bit.py::Bnb4BitTest::test_device_and_dtype_assignment` and `tests/quantization/bnb/test_4bit.py::Bnb4BitGPT2Test::test_device_and_dtype_assignment`
```
self = <bnb.test_4bit.Bnb4BitGPT2Test testMethod=test_device_and_dtype_assignment>

    def test_device_and_dtype_assignment(self):
        r"""
        Test whether attempting to change the device or cast the dtype of a model
        after converting it to 4-bit precision will raise an appropriate error.
        The test ensures that such operations are prohibited on 4-bit models
        to prevent invalid conversions.
        """
                                                                                                                                                                                                                              # Moving with `to` or `cuda` is not supported with versions < 0.43.2.
        if version.parse(importlib.metadata.version("bitsandbytes")) < version.parse("0.43.2"):
            with self.assertRaises(ValueError):
                # Tries with `str`
                self.model_4bit.to("cpu")

            with self.assertRaises(ValueError):
                # Tries with a `device`                                                                                                                                                                                               self.model_4bit.to(torch.device("cuda:0"))

            with self.assertRaises(ValueError):
                # Tries with `cuda`
                self.model_4bit.cuda()
        else:                                                                                                                                                                                                         >           with self.assertRaises(ValueError):
E           AssertionError: ValueError not raised

tests/quantization/bnb/test_4bit.py:304: AssertionError
```

It seems that I can run `model_4bit.to("cpu")`

cc @Titus-von-Koeller 

All tests have been fixed, please review again. Thx
cc @SunMarc @Titus-von-Koeller 

Hi @Titus-von-Koeller @SunMarc @ArthurZucker , can you help to take the last round review and merge it if no updates are required? Thx!

Do you know how to fix the code quality check? If I run `make style`, it will changs files that are not contained in my PR.

```
Traceback (most recent call last):
  File "/root/transformers/utils/custom_init_isort.py", line 330, in <module>
    sort_imports_in_all_inits(check_only=args.check_only)
  File "/root/transformers/utils/custom_init_isort.py", line 322, in sort_imports_in_all_inits
    raise ValueError(f"Would overwrite {len(failures)} files, run `make style`.")
ValueError: Would overwrite 1 files, run `make style`.

Exited with code exit status 1
```

@SunMarc [The preview docs aren't updating](https://moon-ci-docs.huggingface.co/docs/transformers/pr_31098/en/quantization/bitsandbytes#bitsandbytes) correctly and we still need to check correct rendering + links of commit [2be4169](https://github.com/huggingface/transformers/pull/31098/commits/2be4169773f02f2595ca4859e9693f8bebdefc61). It's strange that it does not appear in the preview, despite the [doc-build workflow going through cleanly on the last commit](https://github.com/huggingface/transformers/actions/runs/10813863675/job/29998891355?pr=31098).

This issue seems to be due to a regression in the doc-builder, see [this Slack thread](https://huggingface.slack.com/archives/C04F8N7FQNL/p1726067750730969?thread_ts=1724791099.544009&cid=C04F8N7FQNL) to discuss/follow the resolution.

@ArthurZucker Please let us know if these doc updates are enough or if you desire anything else. Thanks for the prior review! I think we addressed all your feedback now.

For me, there are still these open points and then we can merge:

- [x] wrap up [this discussion with @akx](https://github.com/huggingface/transformers/pull/31098/files#r1753145143)
- [x] the [two conversations in this commit being resolved](https://github.com/huggingface/transformers/pull/31098/commits/08f31f875751d4432261fefb2fc5f34fadfc77d8#diff-70576373928eee203a92800d5ea1f6270d45462a9bde494cde65338710a0a331R209-R213)
- [x] [put back the is_torch_available check](https://github.com/huggingface/transformers/pull/31098/commits/b506b98e115770d52df86511c7acd7431171f670), @jiqing-feng (unless you specify a good reason)
- [x] I will rerun the integration test suite once more with CUDA, ROCm and Intel CPU

Hi @SunMarc @akx , you were right. We should keep `is_bitsandbytes_avaliable()` as simple as possible and move all bnb-relevant functions to `integration/bitsandbytes.py`. The have 2 cases in `is_bitsandbytes_avaliable()`. 
1. Return torch.cuda.is_available() if bnb < 0.43.1 because this version of bnb only supports cuda
2. Return True if bnb >= 0.43.1, other checks will be performed in `validate_environment`.

> Can we rebase this PR on main also ? There are unrelated commits in the diff

Seconded. It's rather impossible to re-review now.


> > Can we rebase this PR on main also ? There are unrelated commits in the diff
> 
> Seconded. It's rather impossible to re-review now.

Unrelated changes have been reverted.

Nice @jiqing-feng  ! With that, we just need need to rerun the test in order to merge the PR. cc @Titus-von-Koeller 

I'll do a final review today and tmr and then merge. Was busy this week preparing for presenting at Pytorch conf and traveling.

Thanks everyone for the great work 🤗 

cc @jiqing-feng 

> @cbensimon you may be interested here for ZeroGPU especially in combination with https://github.com/huggingface/transformers/pull/33122.

Thanks @matthewdouglas for the pointer, I'll look at this