caching_allocator_warmup function raise TypeError on AppleSilicon M4 max
Author: theVannu
Pull request: no
Labels: name: bug

### System Info

- `transformers` version: 5.0.0
- Platform: macOS-26.2-arm64-arm-64bit-Mach-O
- Python version: 3.14.2
- Huggingface_hub version: 1.3.4
- Safetensors version: 0.7.0
- Accelerate version: 1.12.0
- Accelerate config:    not found
- DeepSpeed version: not installed
- PyTorch version (accelerator?): 2.10.0 (NA)
- Using distributed or parallel set-up in script?: <fill in>

### Who can help?

I am trying to load a model  using following code:

```
if torch.cuda.is_available():
    capability = torch.cuda.get_device_capability()
    if capability[0] >= 8:  # Ampere or newer
        dtype = torch.bfloat16
        print("   Using torch.bfloat16 (Ampere+ GPU detected)")
    else:
        dtype = torch.float16
        print("   Using torch.float16")
else:
    dtype = torch.float16
    print("   Using torch.float16 (no CUDA)")

# BitsAndBytes config for 4-bit quantization
bnb_config = BitsAndBytesConfig(
    load_in_4bit=True,
    bnb_4bit_use_double_quant=True,
    bnb_4bit_quant_type="nf4",
    bnb_4bit_dtype=dtype,
    bnb_4bit_quant_storage=dtype,
)

# Load model
model = AutoModelForCausalLM.from_pretrained(
    self.config.model,
    quantization_config=bnb_config,
    device_map="mps",
    dtype=dtype,
    attn_implementation="flash_attention_2" if self.config.use_flash_attention else "eager",
)
```

Here, dtype is set to torch.float16 because I am working on Apple Silicon.

When loading the model, I get the following exception:

```
line 4762, in caching_allocator_warmup
    _ = torch.empty(byte_count // 2, dtype=torch.float16, device=device, requires_grad=False)
TypeError: empty() received an invalid combination of arguments - got (float, requires_grad=bool, device=torch.device, dtype=torch.dtype), but expected one of:
 * (tuple of ints size, *, tuple of names names, torch.memory_format memory_format = None, torch.dtype dtype = None, torch.layout layout = None, torch.device device = None, bool pin_memory = False, bool requires_grad = False)
 * (tuple of ints size, *, torch.memory_format memory_format = None, Tensor out = None, torch.dtype dtype = None, torch.layout layout = None, torch.device device = None, bool pin_memory = False, bool requires_grad = False)
 * 
 * 
 ```
The issue seems to originate from the caching_allocator_warmup function in modeling_utils.py, when the following line is executed:

```
 _ = torch.empty(byte_count // 2, dtype=torch.float16, device=device, requires_grad=False)
```

How can I safely load the model in this setup?
Thank you,
Gian Luca

### Information

- [ ] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

1. Load qwen3-4b-Instruct-2507 model using AutoModelForCausalLM.from_pretrained with BitsAndBytesConfig 4 bit quantization
2. an error is raise if working on apple silicon

### Expected behavior

Load the model without error

Comments:
cc @Cyrilvallez 

I've reproduced and fixed this issue on Apple Silicon M3 Pro. The bug affects all MPS devices, not just M4 Max. On line 4762, `byte_count // 2` returns a float when `byte_count` is a torch/numpy scalar. This happens on MPS devices because the int casting on line 4760 only runs for CUDA/XPU (line 4748 checks `if device.type in ["cuda", "xpu"]`). To fix, I wrapped the division in `int()` to handle all device types.

- Main branch: Loading GPT-2 with BitsAndBytes quantization on MPS -> TypeError ✗
- With fix: Same setup -> Model loads successfully 

Submitting PR now.