Local pretrained models cannot be loaded with multithreading
Author: Crissium
Pull request: no
Labels: name: bug

### System Info

- `transformers` version: 4.56.1
- Platform: Linux-6.8.0-85-generic-x86_64-with-glibc2.39
- Python version: 3.12.11
- Huggingface_hub version: 0.34.4
- Safetensors version: 0.5.3
- Accelerate version: 1.8.0
- Accelerate config:    not found
- DeepSpeed version: not installed
- PyTorch version (accelerator?): 2.7.1 (CUDA)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using distributed or parallel set-up in script?: Parallel
- Using GPU in script?: Yes
- GPU type: NVIDIA GeForce RTX 3090

### Who can help?

@Cyrilvallez 

### Information

- [ ] The official example scripts
- [x] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [x] My own task or dataset (give details below)

### Reproduction

Simplified script to reproduce:

```python
import random
import torch
import traceback
import transformers
from concurrent.futures import ThreadPoolExecutor


model_class = transformers.T5ForConditionalGeneration


def load_model() -> None:
	device_index = random.randint(0, torch.cuda.device_count() - 1)
	device = torch.device(f'cuda:{device_index}')
	model = model_class.from_pretrained('google-t5/t5-small')
	print(f'Loaded model to {model.device}')
	try:
		model.to(device)
		print(f'Moved model to {device}')
	except:
		print(f'Failed to move model to {device}')
		traceback.print_exc()


if __name__ == '__main__':
	with ThreadPoolExecutor(max_workers=torch.cuda.device_count()) as executor:
		futures = [executor.submit(load_model) for _ in range(10)]
		for future in futures:
			future.result()
```

Here I use a locally cached Hub model, but in my actual usage, each threads loads a different checkpoint stored locally. Anyway, a no-data error will be raised:
```
Some weights of T5ForConditionalGeneration were not initialized from the model checkpoint at google-t5/t5-small and are newly initialized: ['lm_head.weight']
You should probably TRAIN this model on a down-stream task to be able to use it for predictions and inference.
Some weights of T5ForConditionalGeneration were not initialized from the model checkpoint at google-t5/t5-small and are newly initialized: ['lm_head.weight']
You should probably TRAIN this model on a down-stream task to be able to use it for predictions and inference.
Some weights of T5ForConditionalGeneration were not initialized from the model checkpoint at google-t5/t5-small and are newly initialized: ['lm_head.weight']
You should probably TRAIN this model on a down-stream task to be able to use it for predictions and inference.
Loaded model to meta
Loaded model to meta
Loaded model to meta
Failed to move model to cuda:2
Failed to move model to cuda:6
Traceback (most recent call last):
  File "/mnt/users_home/cpii.local/yxing/Workspace/ASR/Study/hf_multithreaded_loading.py", line 17, in load_model
    model.to(device)
  File "/mnt/users_home/cpii.local/yxing/miniconda3/envs/g/lib/python3.12/site-packages/transformers/modeling_utils.py", line 4459, in to
    return super().to(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/mnt/users_home/cpii.local/yxing/miniconda3/envs/g/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1355, in to
    return self._apply(convert)
           ^^^^^^^^^^^^^^^^^^^^
  File "/mnt/users_home/cpii.local/yxing/miniconda3/envs/g/lib/python3.12/site-packages/torch/nn/modules/module.py", line 915, in _apply
    module._apply(fn)
  File "/mnt/users_home/cpii.local/yxing/miniconda3/envs/g/lib/python3.12/site-packages/torch/nn/modules/module.py", line 942, in _apply
    param_applied = fn(param)
                    ^^^^^^^^^
  File "/mnt/users_home/cpii.local/yxing/miniconda3/envs/g/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1348, in convert
    raise NotImplementedError(
NotImplementedError: Cannot copy out of meta tensor; no data! Please use torch.nn.Module.to_empty() instead of torch.nn.Module.to() when moving module from meta to a different device.
```

There's a similar issue: #40357, but I haven't tried the older versions.

### Expected behavior

Pretrained models are loaded successfully with multithreading.

Edit: Just a quick note to anyone who runs into this: using multiprocessing is perfectly fine for me.

Comments:
Hi @Crissium,

The error happens because model.to(device) can’t move a meta tensor to CUDA directly. This is expected behavior in PyTorch 2.x when using meta tensors for lazy initialization.

A few ways to fix/work around this:

1. Use to_empty() instead of to() when moving from meta to a real device:

model = model.to_empty(device)


2. Avoid multithreading for loading models; PyTorch doesn’t fully support loading the same model in multiple threads when using meta tensors. Using multiprocessing instead works fine.

Reference: PyTorch Meta Tensors


> Hi [@Crissium](https://github.com/Crissium),
> 
> The error happens because model.to(device) can’t move a meta tensor to CUDA directly. This is expected behavior in PyTorch 2.x when using meta tensors for lazy initialization.
> 
> A few ways to fix/work around this:
> 
>     1. Use to_empty() instead of to() when moving from meta to a real device:
> 
> 
> model = model.to_empty(device)
> 
>     2. Avoid multithreading for loading models; PyTorch doesn’t fully support loading the same model in multiple threads when using meta tensors. Using multiprocessing instead works fine.
> 
> 
> Reference: PyTorch Meta Tensors

But why is the model loaded onto the meta device in the first place?

The model is loaded onto the meta device because PyTorch uses lazy initialization for large models. Meta tensors are placeholders without actual data, which allows memory-efficient and fast model creation. The error occurs because .to(device) cannot move meta tensors across GPUs in multiple threads. This is expected behavior: using multiprocessing or first materializing the model on a real device before multithreading avoids this issue.

> The error occurs because .to(device) cannot move meta tensors across GPUs in multiple threads.

Sorry for being obtuse, but what do you mean by moving 'meta tensors across GPUs'? Why can't a model be initialised and filled with real data inside a thread?

In fact, I have tried this code snippet and it worked perfectly well:
```python
def run():
	device = torch.device('cuda:1')
	model = transformers.T5ForConditionalGeneration.from_pretrained(<path to checkpoint>).to(device)
	print(model)

thread = threading.Thread(target=run)
thread.start()
```

The root cause of this issue is that `from_pretrained()` uses `low_cpu_mem_usage=True` by default in recent versions of transformers. This causes models to be loaded onto the "meta" device (a placeholder for lazy initialization) instead of being materialized with actual data immediately.

**The Fix:**

Add `low_cpu_mem_usage=False` when calling `from_pretrained()` in your multithreaded code:

```python
def load_model() -> None:
    device_index = random.randint(0, torch.cuda.device_count() - 1)
    device = torch.device(f'cuda:{device_index}')
    # Add low_cpu_mem_usage=False to disable meta tensor usage
    model = model_class.from_pretrained('google-t5/t5-small', low_cpu_mem_usage=False)
    print(f'Loaded model to {model.device}')
    try:
        model.to(device)
        print(f'Moved model to {device}')
    except:
        print(f'Failed to move model to {device}')
        traceback.print_exc()
```

**Why this works:**

When `low_cpu_mem_usage=True` (the default), transformers initializes models on the meta device to save memory during loading. However, this lazy initialization doesn't work properly in multithreaded contexts because PyTorch's meta tensor mechanism isn't thread-safe when moving tensors to real devices.

By setting `low_cpu_mem_usage=False`, the model is immediately loaded with real data on the CPU, and then can be safely moved to CUDA devices even within threads.

**Alternative solutions if you need to keep `low_cpu_mem_usage=True`:**
1. Use multiprocessing instead of multithreading (as you already discovered)
2. Use `model.to_empty(device)` instead of `model.to(device)` after loading

This is a legitimate issue with how transformers handles model loading in threaded contexts, and the default behavior could potentially be improved to detect threading environments.

**Working Code Example**

I've created a complete, ready-to-run Python script demonstrating the fix:

https://gist.github.com/ManishhDev/a1bd20a9e3258b495e3953ae7712cc58

The script includes:
- ❌ Broken version (showing the error)
- ✅ Fixed version with `low_cpu_mem_usage=False` 
- ✅ Alternative fix using `to_empty()`
- Full test harness with ThreadPoolExecutor

You can download and run it directly to verify the solution works. The key change is adding `low_cpu_mem_usage=False` when calling `from_pretrained()` in threaded contexts.

@uzmajamadar @ManishhDev Thank you both! I understand now.

Just FYI, `low_cpu_mem_usage` does not exist anymore and is plainly ignored, it has no effect. The above answers are wrong. What you are observing is simply the fact that `from_pretrained` is not thread-safe 😉