Offloading not working as expected with quantization
Author: edmcman
Pull request: no
Labels: name: bug

### System Info

- `transformers` version: 4.57.3
- Platform: Linux-6.8.0-94-generic-x86_64-with-glibc2.35
- Python version: 3.11.12
- Huggingface_hub version: 0.36.2
- Safetensors version: 0.7.0
- Accelerate version: 1.12.0
- Accelerate config: 	not found
- DeepSpeed version: not installed
- PyTorch version (accelerator?): 2.10.0+cu128 (CUDA)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using distributed or parallel set-up in script?: <fill in>
- Using GPU in script?: <fill in>
- GPU type: NVIDIA GeForce RTX 4070 Laptop GPU

### Who can help?

@sunmarc @MekkCyber 

### Information

- [ ] The official example scripts
- [x] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [x] My own task or dataset (give details below)

### Reproduction

1. Run with `uvx --with protobuf,sentencepiece --with "transformers==4.57.3" --with "accelerate==1.12.0" --with "bitsandbytes==0.49.1" python bug3.py`
(The versions are needed because of #43872)
``` python
import torch
from transformers import AutoTokenizer, AutoModelForCausalLM, BitsAndBytesConfig

MODEL_ID = "ejschwartz/decaf-v1-22b"

bnb_config = BitsAndBytesConfig(
    load_in_8bit=True,
    llm_int8_enable_fp32_cpu_offload=True,
)

tokenizer = AutoTokenizer.from_pretrained(MODEL_ID, use_fast=True)

model = AutoModelForCausalLM.from_pretrained(
    MODEL_ID,
    quantization_config=bnb_config,
    device_map="auto",
    low_cpu_mem_usage=True,
)

prompt = "Hello!"
inputs = tokenizer(prompt, return_tensors="pt", truncation=True, max_length=1024)
inputs = {k: v.to(model.device) for k, v in inputs.items()}

with torch.inference_mode():
    out = model.generate(**inputs, max_new_tokens=32, do_sample=False)

print(tokenizer.decode(out[0], skip_special_tokens=True))
```
2. See error `torch.OutOfMemoryError: CUDA out of memory. Tried to allocate 36.00 MiB. GPU 0 has a total capacity of 7.62 GiB of which 75.75 MiB is free. Including non-PyTorch memory, this process has 6.46 GiB memory in use. Of the allocated memory 6.20 GiB is allocated by PyTorch, and 92.81 MiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting PYTORCH_ALLOC_CONF=expandable_segments:True to avoid fragmentation.  See documentation for Memory Management  (https://pytorch.org/docs/stable/notes/cuda.html#environment-variables)`

### Expected behavior

Not to run out of memory.  Note that commenting out the `quantization_config=bnb_config` line counter-intuitively does *not* result in a CUDA error though it's very slow, as expected.

I have also tried `max_memory={0: "3GiB", "cpu": "64GiB"},` which did not help.  I have about 7 GiB of free vram, so this should have easily fit. 

Comments:
cc @Cyrilvallez, are you fine with cleaning `_is_hf_initialized` flag after we finished loading the model or there could be unexpected use case ? The issue happens later on here. 

Humm, I don't see any link to the `_is_hf_initialized` flag @SunMarc?? Moreover, this is on v4 so the flag was only applied to modules there, and loading has changed tremenduously.
The real reason here @edmcman is that you seem to be trying to load a 22B model into a 8GB gpu, so it will never fit even with quantization. Not sure why the device_map is not being respected with bnb config though, could you try on latest v5 instead?

Oh sorry, tagged you in the incorrect issue as OP opened two issues. Here's the right one where he tested on v5 : https://github.com/huggingface/transformers/issues/43872

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.

I think this is still an issue.  It's still unclear if quantization +
offloading is supported, and if so, when?  Always?

On Thu, Mar 12, 2026 at 4:09 AM github-actions[bot] -
***@***.*** <github.edmcman.99c9f1b9d0.notifications#
***@***.***> wrote:

> *github-actions[bot]* left a comment (huggingface/transformers#43873)
> <https://github.com/huggingface/transformers/issues/43873#issuecomment-4044786480>
>
> This issue has been automatically marked as stale because it has not had
> recent activity. If you think this still needs to be addressed please
> comment on this thread.
>
> Please note that issues that do not follow the contributing guidelines
> <https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md>
> are likely to be ignored.
>
> —
> Reply to this email directly, view it on GitHub
> <https://github.com/huggingface/transformers/issues/43873#issuecomment-4044786480>,
> or unsubscribe
> <https://github.com/notifications/unsubscribe-auth/AAHYKZJK2546PUXKNOWQGFT4QJWCNAVCNFSM6AAAAACURH7UVCVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHM2DANBUG44DMNBYGA>
> .
> You are receiving this because you were mentioned.Message ID:
> ***@***.***>
>


Yes I agree this is still an issue going on

On Thu, Mar 12, 2026, 1:51 PM Edward J. Schwartz ***@***.***>
wrote:

> *edmcman* left a comment (huggingface/transformers#43873)
> <https://github.com/huggingface/transformers/issues/43873#issuecomment-4046936878>
> I think this is still an issue. It's still unclear if quantization +
> offloading is supported, and if so, when? Always?
>
> On Thu, Mar 12, 2026 at 4:09 AM github-actions[bot] -
> ***@***.*** <github.edmcman.99c9f1b9d0.notifications#
> ***@***.***> wrote:
>
> > *github-actions[bot]* left a comment (huggingface/transformers#43873)
> > <
> https://github.com/huggingface/transformers/issues/43873#issuecomment-4044786480>
>
> >
> > This issue has been automatically marked as stale because it has not had
> > recent activity. If you think this still needs to be addressed please
> > comment on this thread.
> >
> > Please note that issues that do not follow the contributing guidelines
> > <https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md>
> > are likely to be ignored.
> >
> > —
> > Reply to this email directly, view it on GitHub
> > <
> https://github.com/huggingface/transformers/issues/43873#issuecomment-4044786480>,
>
> > or unsubscribe
> > <
> https://github.com/notifications/unsubscribe-auth/AAHYKZJK2546PUXKNOWQGFT4QJWCNAVCNFSM6AAAAACURH7UVCVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHM2DANBUG44DMNBYGA>
>
> > .
> > You are receiving this because you were mentioned.Message ID:
> > ***@***.***>
> >
>
> —
> Reply to this email directly, view it on GitHub
> <https://github.com/huggingface/transformers/issues/43873#issuecomment-4046936878>,
> or unsubscribe
> <https://github.com/notifications/unsubscribe-auth/B7HTEEJQXLDWHX3RVZJC6OT4QK6GJAVCNFSM6AAAAACURH7UVCVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHM2DANBWHEZTMOBXHA>
> .
> You are receiving this because you are subscribed to this thread.Message
> ID: ***@***.***>
>


This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.

not stale

On Mon, Apr 6, 2026 at 4:28 AM github-actions[bot] -
***@***.*** <github.edmcman.99c9f1b9d0.notifications#
***@***.***> wrote:

> *github-actions[bot]* left a comment (huggingface/transformers#43873)
> <https://github.com/huggingface/transformers/issues/43873#issuecomment-4191140253>
>
> This issue has been automatically marked as stale because it has not had
> recent activity. If you think this still needs to be addressed please
> comment on this thread.
>
> Please note that issues that do not follow the contributing guidelines
> <https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md>
> are likely to be ignored.
>
> —
> Reply to this email directly, view it on GitHub
> <https://github.com/huggingface/transformers/issues/43873#issuecomment-4191140253>,
> or unsubscribe
> <https://github.com/notifications/unsubscribe-auth/AAHYKZOHHEMAJMMTWC2DIK34UNTD7AVCNFSM6AAAAACURH7UVCVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHM2DCOJRGE2DAMRVGM>
> .
> You are receiving this because you were mentioned.Message ID:
> ***@***.***>
>


This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.

Not stale

On Fri, May 1, 2026, 4:36 AM github-actions[bot] - ***@***.***
***@***.***> wrote:

> *github-actions[bot]* left a comment (huggingface/transformers#43873)
> <https://github.com/huggingface/transformers/issues/43873#issuecomment-4358524904>
>
> This issue has been automatically marked as stale because it has not had
> recent activity. If you think this still needs to be addressed please
> comment on this thread.
>
> Please note that issues that do not follow the contributing guidelines
> <https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md>
> are likely to be ignored.
>
> —
> Reply to this email directly, view it on GitHub
> <https://github.com/huggingface/transformers/issues/43873#issuecomment-4358524904>,
> or unsubscribe
> <https://github.com/notifications/unsubscribe-auth/AAHYKZJASZQHZLKZ5M3HTPD4YROYDAVCNFSM6AAAAACURH7UVCVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHM2DGNJYGUZDIOJQGQ>
> .
> Triage notifications on the go with GitHub Mobile for iOS
> <https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675>
> or Android
> <https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub>.
>
> You are receiving this because you were mentioned.Message ID:
> ***@***.***>
>


This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.

Still an issue 

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.

Not stale

On Sat, Jun 20, 2026, 5:00 AM github-actions[bot] - ***@***.***
***@***.***> wrote:

> *github-actions[bot]* left a comment (huggingface/transformers#43873)
> <https://github.com/huggingface/transformers/issues/43873#issuecomment-4757073779>
>
> This issue has been automatically marked as stale because it has not had
> recent activity. If you think this still needs to be addressed please
> comment on this thread.
>
> Please note that issues that do not follow the contributing guidelines
> <https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md>
> are likely to be ignored.
>
> —
> Reply to this email directly, view it on GitHub
> <https://github.com/huggingface/transformers/issues/43873?email_source=notifications&email_token=AAHYKZLHK2MK5PDCS3M4SOL5AZHDPA5CNFSNUABFM5UWIORPF5TWS5BNNB2WEL2JONZXKZKDN5WW2ZLOOQXTINZVG4YDOMZXG442M4TFMFZW63VHNVSW45DJN5XKKZLWMVXHJLDGN5XXIZLSL5RWY2LDNM#issuecomment-4757073779>,
> or unsubscribe
> <https://github.com/notifications/unsubscribe-auth/AAHYKZIW3XM7EORY66TMFAL5AZHDPAVCNFSNUABFKJSXA33TNF2G64TZHMYTKNJSGIYDMNBRHNEXG43VMU5TGOJRHA2TMOBTGM4KC5QC>
> .
> Triage notifications, keep track of coding agent tasks and review pull
> requests on the go with GitHub Mobile for iOS
> <https://github.com/notifications/mobile/ios/AAHYKZINXPHXH5FMSPWR7LL5AZHDPA5CNFSNUABFM5UWIORPF5TWS5BNNB2WEL2JONZXKZKDN5WW2ZLOOQXTINZVG4YDOMZXG442M4TFMFZW63VHNVSW45DJN5XKKZLWMVXHJKTGN5XXIZLSL5UW64Y>
> and Android
> <https://github.com/notifications/mobile/android/AAHYKZLQNXEYOQVYIZOI3JT5AZHDPA5CNFSNUABFM5UWIORPF5TWS5BNNB2WEL2JONZXKZKDN5WW2ZLOOQXTINZVG4YDOMZXG442M4TFMFZW63VHNVSW45DJN5XKKZLWMVXHJLTGN5XXIZLSL5QW4ZDSN5UWI>.
> Download it today!
> You are receiving this because you were mentioned.Message ID:
> ***@***.***>
>


This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.

Not stale

On Wed, Jul 15, 2026, 4:44 AM github-actions[bot] - ***@***.***
***@***.***> wrote:

> *github-actions[bot]* left a comment (huggingface/transformers#43873)
> <https://github.com/huggingface/transformers/issues/43873#issuecomment-4978655669>
>
> This issue has been automatically marked as stale because it has not had
> recent activity. If you think this still needs to be addressed please
> comment on this thread.
>
> Please note that issues that do not follow the contributing guidelines
> <https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md>
> are likely to be ignored.
>
> —
> Reply to this email directly, view it on GitHub
> <https://github.com/huggingface/transformers/issues/43873?email_source=notifications&email_token=AAHYKZJHKIHEH2WXTWS5YTT5E5AANA5CNFSNUABFM5UWIORPF5TWS5BNNB2WEL2JONZXKZKDN5WW2ZLOOQXTIOJXHA3DKNJWGY42M4TFMFZW63VHNVSW45DJN5XKKZLWMVXHJLDGN5XXIZLSL5RWY2LDNM#issuecomment-4978655669>,
> or unsubscribe
> <https://github.com/notifications/unsubscribe-auth/AAHYKZI3K2WMWZYS5BI4S6T5E5AANAVCNFSNUABFKJSXA33TNF2G64TZHMYTKNJSGIYDMNBRHNEXG43VMU5TGOJRHA2TMOBTGM4KC5QC>
> .
> Triage notifications, keep track of coding agent tasks and review pull
> requests on the go with GitHub Mobile for iOS
> <https://github.com/notifications/mobile/ios/AAHYKZNQ35QBCAKIF6DVVJ35E5AANA5CNFSNUABFM5UWIORPF5TWS5BNNB2WEL2JONZXKZKDN5WW2ZLOOQXTIOJXHA3DKNJWGY42M4TFMFZW63VHNVSW45DJN5XKKZLWMVXHJKTGN5XXIZLSL5UW64Y>
> and Android
> <https://github.com/notifications/mobile/android/AAHYKZI4PSROBGAXJ624LBD5E5AANA5CNFSNUABFM5UWIORPF5TWS5BNNB2WEL2JONZXKZKDN5WW2ZLOOQXTIOJXHA3DKNJWGY42M4TFMFZW63VHNVSW45DJN5XKKZLWMVXHJLTGN5XXIZLSL5QW4ZDSN5UWI>.
> Download it today!
> You are receiving this because you were mentioned.Message ID:
> ***@***.***>
>