ValueError: weight is on the meta device, we need a `value` to put in on 0. `Gemma3`
Author: akhilpandey95
Pull request: no
Labels: name: bug

### System Info

**Error trace**
```python
---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
Cell In[64], line 2
      1 tokenizer = AutoTokenizer.from_pretrained(model_id)
----> 2 model = Gemma3ForCausalLM.from_pretrained(model_id, device_map="auto")
      3 input_ids = tokenizer("Write me a poem about Machine Learning.", return_tensors="pt").to(model.device)
      4 outputs = model.generate(**input_ids, max_new_tokens=100)

File [~/.conda/envs/ai4scisci/lib/python3.12/site-packages/transformers/modeling_utils.py:273](http://localhost:3126/lab/tree/users/uig2924/notebooks/~/.conda/envs/ai4scisci/lib/python3.12/site-packages/transformers/modeling_utils.py#line=272), in restore_default_torch_dtype.<locals>._wrapper(*args, **kwargs)
    271 old_dtype = torch.get_default_dtype()
    272 try:
--> 273     return func(*args, **kwargs)
    274 finally:
    275     torch.set_default_dtype(old_dtype)

File [~/.conda/envs/ai4scisci/lib/python3.12/site-packages/transformers/modeling_utils.py:4531](http://localhost:3126/lab/tree/users/uig2924/notebooks/~/.conda/envs/ai4scisci/lib/python3.12/site-packages/transformers/modeling_utils.py#line=4530), in PreTrainedModel.from_pretrained(cls, pretrained_model_name_or_path, config, cache_dir, ignore_mismatched_sizes, force_download, local_files_only, token, revision, use_safetensors, weights_only, *model_args, **kwargs)
   4528         device_map_kwargs["offload_buffers"] = True
   4530     if not is_fsdp_enabled() and not is_deepspeed_zero3_enabled():
-> 4531         dispatch_model(model, **device_map_kwargs)
   4533 # This is needed for the RotaryEmbedding, which was not initialized on the correct device as it is
   4534 # not part of the state_dict (persistent=False)
   4535 if device_mesh is not None:

File [~/.conda/envs/ai4scisci/lib/python3.12/site-packages/accelerate/big_modeling.py:420](http://localhost:3126/lab/tree/users/uig2924/notebooks/~/.conda/envs/ai4scisci/lib/python3.12/site-packages/accelerate/big_modeling.py#line=419), in dispatch_model(model, device_map, main_device, state_dict, offload_dir, offload_index, offload_buffers, skip_keys, preload_module_classes, force_hooks)
    415         tied_params_map[data_ptr] = {}
    417         # Note: To handle the disk offloading case, we can not simply use weights_map[param_name].data_ptr() as the reference pointer,
    418         # as we have no guarantee that safetensors' `file.get_tensor()` will always give the same pointer.
--> 420 attach_align_device_hook_on_blocks(
    421     model,
    422     execution_device=execution_device,
    423     offload=offload,
    424     offload_buffers=offload_buffers,
    425     weights_map=weights_map,
    426     skip_keys=skip_keys,
    427     preload_module_classes=preload_module_classes,
    428     tied_params_map=tied_params_map,
    429 )
    431 # warn if there is any params on the meta device
    432 offloaded_devices_str = " and ".join(
    433     [device for device in set(device_map.values()) if device in ("cpu", "disk")]
    434 )

File [~/.conda/envs/ai4scisci/lib/python3.12/site-packages/accelerate/hooks.py:656](http://localhost:3126/lab/tree/users/uig2924/notebooks/~/.conda/envs/ai4scisci/lib/python3.12/site-packages/accelerate/hooks.py#line=655), in attach_align_device_hook_on_blocks(module, execution_device, offload, weights_map, offload_buffers, module_name, skip_keys, preload_module_classes, tied_params_map)
    654 for child_name, child in module.named_children():
    655     child_name = f"{module_name}.{child_name}" if len(module_name) > 0 else child_name
--> 656     attach_align_device_hook_on_blocks(
    657         child,
    658         execution_device=execution_device,
    659         offload=offload,
    660         weights_map=weights_map,
    661         offload_buffers=offload_buffers,
    662         module_name=child_name,
    663         preload_module_classes=preload_module_classes,
    664         skip_keys=skip_keys,
    665         tied_params_map=tied_params_map,
    666     )

File [~/.conda/envs/ai4scisci/lib/python3.12/site-packages/accelerate/hooks.py:656](http://localhost:3126/lab/tree/users/uig2924/notebooks/~/.conda/envs/ai4scisci/lib/python3.12/site-packages/accelerate/hooks.py#line=655), in attach_align_device_hook_on_blocks(module, execution_device, offload, weights_map, offload_buffers, module_name, skip_keys, preload_module_classes, tied_params_map)
    654 for child_name, child in module.named_children():
    655     child_name = f"{module_name}.{child_name}" if len(module_name) > 0 else child_name
--> 656     attach_align_device_hook_on_blocks(
    657         child,
    658         execution_device=execution_device,
    659         offload=offload,
    660         weights_map=weights_map,
    661         offload_buffers=offload_buffers,
    662         module_name=child_name,
    663         preload_module_classes=preload_module_classes,
    664         skip_keys=skip_keys,
    665         tied_params_map=tied_params_map,
    666     )

File [~/.conda/envs/ai4scisci/lib/python3.12/site-packages/accelerate/hooks.py:616](http://localhost:3126/lab/tree/users/uig2924/notebooks/~/.conda/envs/ai4scisci/lib/python3.12/site-packages/accelerate/hooks.py#line=615), in attach_align_device_hook_on_blocks(module, execution_device, offload, weights_map, offload_buffers, module_name, skip_keys, preload_module_classes, tied_params_map)
    607 if module_name in execution_device and module_name in offload and not offload[module_name]:
    608     hook = AlignDevicesHook(
    609         execution_device=execution_device[module_name],
    610         offload_buffers=offload_buffers,
   (...)
    614         tied_params_map=tied_params_map,
    615     )
--> 616     add_hook_to_module(module, hook)
    617     attach_execution_device_hook(module, execution_device[module_name], tied_params_map=tied_params_map)
    618 elif module_name in execution_device and module_name in offload:

File [~/.conda/envs/ai4scisci/lib/python3.12/site-packages/accelerate/hooks.py:161](http://localhost:3126/lab/tree/users/uig2924/notebooks/~/.conda/envs/ai4scisci/lib/python3.12/site-packages/accelerate/hooks.py#line=160), in add_hook_to_module(module, hook, append)
    158     old_forward = module.forward
    159     module._old_forward = old_forward
--> 161 module = hook.init_hook(module)
    162 module._hf_hook = hook
    164 def new_forward(module, *args, **kwargs):

File [~/.conda/envs/ai4scisci/lib/python3.12/site-packages/accelerate/hooks.py:283](http://localhost:3126/lab/tree/users/uig2924/notebooks/~/.conda/envs/ai4scisci/lib/python3.12/site-packages/accelerate/hooks.py#line=282), in AlignDevicesHook.init_hook(self, module)
    281 if not self.offload and self.execution_device is not None:
    282     for name, _ in named_module_tensors(module, recurse=self.place_submodules):
--> 283         set_module_tensor_to_device(module, name, self.execution_device, tied_params_map=self.tied_params_map)
    284 elif self.offload:
    285     self.original_devices = {
    286         name: param.device for name, param in named_module_tensors(module, recurse=self.place_submodules)
    287     }

File [~/.conda/envs/ai4scisci/lib/python3.12/site-packages/accelerate/utils/modeling.py:364](http://localhost:3126/lab/tree/users/uig2924/notebooks/~/.conda/envs/ai4scisci/lib/python3.12/site-packages/accelerate/utils/modeling.py#line=363), in set_module_tensor_to_device(module, tensor_name, device, value, dtype, fp16_statistics, tied_params_map)
    361     return
    363 if old_value.device == torch.device("meta") and device not in ["meta", torch.device("meta")] and value is None:
--> 364     raise ValueError(f"{tensor_name} is on the meta device, we need a `value` to put in on {device}.")
    366 param = module._parameters[tensor_name] if tensor_name in module._parameters else None
    367 param_cls = type(param)

ValueError: weight is on the meta device, we need a `value` to put in on 0.
```

**Code to reproduce**
```python
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = Gemma3ForCausalLM.from_pretrained(model_id, device_map="auto")
input_ids = tokenizer("Write me a poem about Machine Learning.", return_tensors="pt").to(model.device)
outputs = model.generate(**input_ids, max_new_tokens=100)
text = tokenizer.batch_decode(outputs, skip_special_tokens=True)
```
taken from https://huggingface.co/docs/transformers/main/en/model_doc/gemma3#transformers.Gemma3TextConfig.

**Hardware accelerator**
```python
!nvidia-smi -L
GPU 0: NVIDIA A100 80GB PCIe (UUID: GPU-cf881215-9c35-ed1c-cfc3-fb7a629e34fe)
```

**Library information**
```python
print(transformers.__version__)
print(torch.__version__)
4.50.0.dev0
2.4.1+cu121
```

**Kernel**
```shell
Linux qgpu0518 3.10.0-1160.95.1.el7.x86_64 #1 SMP Fri Jun 23 08:44:55 EDT 2023 x86_64 x86_64 x86_64 GNU/Linux
```

**OS**
```shell
LSB Version:	:core-4.1-amd64:core-4.1-noarch:cxx-4.1-amd64:cxx-4.1-noarch:desktop-4.1-amd64:desktop-4.1-noarch:languages-4.1-amd64:languages-4.1-noarch:printing-4.1-amd64:printing-4.1-noarch
Distributor ID:	RedHatEnterpriseServer
Description:	Red Hat Enterprise Linux Server release 7.9 (Maipo)
Release:	7.9
Codename:	Maipo
```

> NOTE: I've disabled the `device_map` yet the meta device issue wouldn't go away. The only circumstance the above snippet works is when i use `cpu`.

### Who can help?

_No response_

### Information

- [x] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

Run the following code

```python
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = Gemma3ForCausalLM.from_pretrained(model_id, device_map="auto")
input_ids = tokenizer("Write me a poem about Machine Learning.", return_tensors="pt").to(model.device)
outputs = model.generate(**input_ids, max_new_tokens=100)
text = tokenizer.batch_decode(outputs, skip_special_tokens=True)
```

### Expected behavior

`ValueError`

```python
ValueError: weight is on the meta device, we need a `value` to put in on 0.
```

Comments:
Hey @akhilpandey95 !

Thanks for reporting. The issue was introduced only in the model release tag and is fixed already in `main`. Can confirm it works when I install from source. Can you check out?

I tried, and it solves my issue. Thanks @zucchini-nlp !!

```shell
Successfully built transformers
Installing collected packages: transformers
  Attempting uninstall: transformers
    Found existing installation: transformers 4.50.0.dev0
    Uninstalling transformers-4.50.0.dev0:
      Successfully uninstalled transformers-4.50.0.dev0
```