Assertion Error during Text Generation (model inference): assert quant_state is not None
Author: akjindal53244
Pull request: no

### System Info

(tempPy39) minimalist@minimalist-pc:~$ transformers-cli env

Copy-and-paste the text below in your GitHub issue and FILL OUT the two last points.

- `transformers` version: 4.35.0.dev0
- Platform: Linux-5.15.0-86-generic-x86_64-with-glibc2.31
- Python version: 3.9.18
- Huggingface_hub version: 0.17.3
- Safetensors version: 0.4.0
- Accelerate version: 0.24.0.dev0
- Accelerate config: 	not found
- PyTorch version (GPU?): 2.2.0.dev20231013+cu118 (True)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: RTX 4090
- Using distributed or parallel set-up in script?: No

### Who can help?

_No response_

### Information

- [x] The official example scripts
- [ ] My own modified scripts

### Tasks

- [x] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

**Steps to Reproduce:**

1. **Installations**:

```
conda create -n <env_name> python=3.9
conda activate <env_name>
pip install --pre torch --index-url https://download.pytorch.org/whl/nightly/cu118
pip install scipy
pip install -q -U bitsandbytes
pip install -q -U git+https://github.com/huggingface/transformers.git
pip install -q -U git+https://github.com/huggingface/peft.git
pip install -q -U git+https://github.com/huggingface/accelerate.git
```

2. **Code**

```python
import torch
from transformers import AutoTokenizer, AutoModelForCausalLM, BitsAndBytesConfig

bnb_config = BitsAndBytesConfig(
    load_in_4bit=True,
    bnb_4bit_use_double_quant=True,
    bnb_4bit_quant_type="nf4",
    bnb_4bit_compute_dtype=torch.bfloat16
)

model_id = "mistralai/Mistral-7B-Instruct-v0.1"
model = AutoModelForCausalLM.from_pretrained(model_id, quantization_config=bnb_config, device_map="{"": "cpu"}")
tokenizer = AutoTokenizer.from_pretrained(model_id)

PROMPT= """ ### Instruction: Act as a data science expert.
### Question:
Explain to me what is Large Language Model. Assume that I am a 5-year-old child.

### Answer:
"""

model_inputs = tokenizer(PROMPT, return_tensors="pt", add_special_tokens=True)

generated_ids = model.generate(**model_inputs, max_new_tokens=1000, do_sample=True, pad_token_id=tokenizer.eos_token_id)
```

### Expected behavior

It shouldn't fail.

Error log:
```FP4 quantization state not initialized. Please call .cuda() or .to(device) on the LinearFP4 layer first.
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/minimalist/miniconda3/envs/tempPy39/lib/python3.9/site-packages/torch/utils/_contextlib.py", line 115, in decorate_context
    return func(*args, **kwargs)
  File "/home/minimalist/miniconda3/envs/tempPy39/lib/python3.9/site-packages/transformers/generation/utils.py", line 1693, in generate
    return self.sample(
  File "/home/minimalist/miniconda3/envs/tempPy39/lib/python3.9/site-packages/transformers/generation/utils.py", line 2775, in sample
    outputs = self(
  File "/home/minimalist/miniconda3/envs/tempPy39/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1519, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "/home/minimalist/miniconda3/envs/tempPy39/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1528, in _call_impl
    return forward_call(*args, **kwargs)
  File "/home/minimalist/miniconda3/envs/tempPy39/lib/python3.9/site-packages/accelerate/hooks.py", line 164, in new_forward
    output = module._old_forward(*args, **kwargs)
  File "/home/minimalist/miniconda3/envs/tempPy39/lib/python3.9/site-packages/transformers/models/mistral/modeling_mistral.py", line 1042, in forward
    outputs = self.model(
  File "/home/minimalist/miniconda3/envs/tempPy39/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1519, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "/home/minimalist/miniconda3/envs/tempPy39/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1528, in _call_impl
    return forward_call(*args, **kwargs)
  File "/home/minimalist/miniconda3/envs/tempPy39/lib/python3.9/site-packages/accelerate/hooks.py", line 164, in new_forward
    output = module._old_forward(*args, **kwargs)
  File "/home/minimalist/miniconda3/envs/tempPy39/lib/python3.9/site-packages/transformers/models/mistral/modeling_mistral.py", line 929, in forward
    layer_outputs = decoder_layer(
  File "/home/minimalist/miniconda3/envs/tempPy39/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1519, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "/home/minimalist/miniconda3/envs/tempPy39/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1528, in _call_impl
    return forward_call(*args, **kwargs)
  File "/home/minimalist/miniconda3/envs/tempPy39/lib/python3.9/site-packages/accelerate/hooks.py", line 164, in new_forward
    output = module._old_forward(*args, **kwargs)
  File "/home/minimalist/miniconda3/envs/tempPy39/lib/python3.9/site-packages/transformers/models/mistral/modeling_mistral.py", line 618, in forward
    hidden_states, self_attn_weights, present_key_value = self.self_attn(
  File "/home/minimalist/miniconda3/envs/tempPy39/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1519, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "/home/minimalist/miniconda3/envs/tempPy39/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1528, in _call_impl
    return forward_call(*args, **kwargs)
  File "/home/minimalist/miniconda3/envs/tempPy39/lib/python3.9/site-packages/accelerate/hooks.py", line 164, in new_forward
    output = module._old_forward(*args, **kwargs)
  File "/home/minimalist/miniconda3/envs/tempPy39/lib/python3.9/site-packages/transformers/models/mistral/modeling_mistral.py", line 258, in forward
    query_states = self.q_proj(hidden_states)
  File "/home/minimalist/miniconda3/envs/tempPy39/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1519, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "/home/minimalist/miniconda3/envs/tempPy39/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1528, in _call_impl
    return forward_call(*args, **kwargs)
  File "/home/minimalist/miniconda3/envs/tempPy39/lib/python3.9/site-packages/accelerate/hooks.py", line 164, in new_forward
    output = module._old_forward(*args, **kwargs)
  File "/home/minimalist/miniconda3/envs/tempPy39/lib/python3.9/site-packages/bitsandbytes/nn/modules.py", line 248, in forward
    out = bnb.matmul_4bit(x, self.weight.t(), bias=bias, quant_state=self.weight.quant_state)
  File "/home/minimalist/miniconda3/envs/tempPy39/lib/python3.9/site-packages/bitsandbytes/autograd/_functions.py", line 567, in matmul_4bit
    assert quant_state is not None
AssertionError
```

Comments:
cc @younesbelkada and @SunMarc 

I verified that it doesn't fail, if I remove `quantization_config` parameter while calling `AutoModelForCausalLM.from_pretrained()` to load the model.
Specifically: 

- `model = AutoModelForCausalLM.from_pretrained(model_id, device_map="{"": "cpu"}")` works 
- But `model = AutoModelForCausalLM.from_pretrained(model_id, quantization_config=bnb_config, device_map="{"": "cpu"}")` fails with error mentioned above.

Hi @akjindal53244 
Thanks for the issue, you are facing that issue because quantization is not supported on CPU. Please make sure to have access to a GPU device to use quantization feature. 
We should probably add a stronger check on `from_pretrained` to properly guide users quantization is not supported on CPU.

I got the same error when I set the device to "cuda", without specifying the index of the gpu. I updated it to "cuda:0" and it worked after.

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.

This desperately needs a more informative error message.

Feel free to open a PR to update it! 🤗 Did not follow everything but let's make it better for others!