AutoGPTQ quantization stucks without any progress
Author: franchukpetro
Pull request: no

### System Info

**Hardware details**
CPU - AMD Ryzen Threadripper PRO 3955WX 16-Cores
GPU - NVIDIA RTX 4090

**Software details**
OS - Ubuntu 22.04.3 LTS
CUDA - 12.1 (I've also tried with 11.8)
Python - 3.10.13
AutoGPTQ - 0.8.0.dev0+cu1211 (build from source, but I've also tried with 0.7.0 and 0.6.0 versions using pip install, with corresponding matching CUDA and PyTorch versions)
PyTorch - 2.2.0+cu121 (also have tried different version to match requirements when using lower versions of AutoGPTQ)
transformers - I have no information about it's version for now, since I've deleted the vm, but that was the default version of transformers, which were installed together with auto-gptq installation.

### Who can help?

@SunMarc @younesbelkada 

### Information

- [ ] The official example scripts
- [X] My own modified scripts

### Tasks

- [X] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

```python
from transformers import AutoModelForCausalLM, AutoTokenizer, GPTQConfig

model_path = 'mistralai/Mistral-7B-v0.1'
quant_path = 'Mistral-7b-v0.1-gptq-int4'

print("Tokenizer initialization started...")

tokenizer = AutoTokenizer.from_pretrained(model_path)

print("Tokenizer initialization ended...")

gptq_config = GPTQConfig(bits=4, dataset="c4", tokenizer=tokenizer)

print("GPTQ config initialization ended...")

quantized_model = AutoModelForCausalLM.from_pretrained(model_path, device_map="auto", quantization_config=gptq_config)

print("Model quantization ended...")


quantized_model.to("cpu")
quantized_model.save_pretrained(quant_path)
tokenizer.save_pretrained(quant_path)
```

### Expected behavior

I'm trying to do the GPTQ quantization of Mistral 7B model on Nvidia 4090 GPU on Vast.ai platform. However, my quantization process constantly stucks after the model weights and data for quantization are loaded - just there is no progress bar or other text output.

I was trying a lot of different combinations of CUDA, PyTorch and AutoGPTQ, but none of them worked. Moreover, I've tried to quantize Falcon 1B & 7B models, but they haven't succeed as well - Falcon 1B was starting the quantization procees, but then it crashed with CUDA out of memory error (which is weird, since 4090 GPU has 24GB of VRAM, which is more than enough to load Falcon 1B).

Comments:
Hi @franchukpetro 
Hmm that shouldn't happen IMO, can you try out on the latest version of `optimum` library?

@younesbelkada thanks for the response!

I'm actually installing it from the source, as it is recommended in this [tutorial](https://huggingface.co/docs/transformers/main/en/quantization#autogptq). I guess it should be already latest version, isn't it?

`pip install git+https://github.com/huggingface/optimum.git`

Okay, so I created two new VMs, and let the quantization process stuck for more than hour, and it resulted in CUDA out of memory error in both cases. For both VMs I was using docker image pytorch/pytorch:2.2.0-cuda12.1-cudnn8-devel. Such docker image has CUDA v12.1 (verified with nvcc) and PyTorch v2.2.0.

### **First VM**
I've done next installation steps, as described in Transformers' [Quantization tutorial](https://huggingface.co/docs/transformers/main/en/quantization#autogptq):

```
pip install auto-gptq
pip install git+https://github.com/huggingface/optimum.git
pip install git+https://github.com/huggingface/transformers.git
pip install --upgrade accelerate
```
transformers-cli env output:
- `transformers` version: 4.39.0.dev0
- Platform: Linux-5.4.0-167-generic-x86_64-with-glibc2.35
- Python version: 3.10.13
- Huggingface_hub version: 0.21.4
- Safetensors version: 0.4.2
- Accelerate version: 0.27.2
- Accelerate config:    not found
- PyTorch version (GPU?): 2.2.0 (True)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed

### **Second VM**
For the second VM I uninstall torch, torchvision and torchaudio, and manually installe PyTorch 2.2.0 build specifically for CUDA 12.1

`pip install torch==2.2.0 torchvision==0.17.0 torchaudio==2.2.0 --index-url https://download.pytorch.org/whl/cu121`

Next, I've installed AutoGPTQ [from source](https://github.com/AutoGPTQ/AutoGPTQ?tab=readme-ov-file#install-from-source), and installed optimum, transformer and accelerate similarly to first VM.

transformers-cli env output:
- `transformers` version: 4.39.0.dev0
- Platform: Linux-5.4.0-172-generic-x86_64-with-glibc2.35
- Python version: 3.10.13
- Huggingface_hub version: 0.21.4
- Safetensors version: 0.4.2
- Accelerate version: 0.27.2
- Accelerate config:    not found
- PyTorch version (GPU?): 2.2.0+cu121 (True)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed

###  **Error**
   
`torch.cuda.OutOfMemoryError: CUDA out of memory. Tried to allocate 256.00 MiB. GPU 0 has a total capacity of 23.65 GiB of which 194.06 MiB is free. Process 3736189 has 23.45 GiB memory in use. Of the allocated memory 22.77 GiB is allocated by PyTorch, and 256.48 MiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True to avoid fragmentation.  See documentation for Memory Management  (https://pytorch.org/docs/stable/notes/cuda.html#environment-variables)  `


As I already mentioned, I was also facing CUDA out of memory errors with smaller models, such as Falcon 1B, which even theoretically cannot reach the max capacity of the GPU RAM (I was doing similar experiments with Falcons on 3080 with 16GB VRAM, and everything was smooth).

Hi @franchukpetro, normally, you should see a tqdm progress to follow the quantization progress. If you don't see it, it means that there might be an issue when we load the model or when we process the dataset. LMK which case it is. It will help me debug this issue ! 

I don't see the tqdm progress bar, and that's what I'm worried as well.

There appears progress bars for model weights loading, quantization dataset loading, and the last step which is visible with progress bar is training splits generation. After that there is no outputs for around 1-2h, and after that it crashes with CUDA OOM error.

Interestingly, I was observing from the vast.ai cloud platform the load of GPU, and GPU is not loaded at all, while GPU VRAM firstly is almost fully free, while after a few minutes of such stucking it becames to be ±15GB loaded for a while until crashing.

The PR above should do the trick. If you don't want to checkout to this PR, you can pass 
`gptq_config = GPTQConfig(bits=4, dataset="c4", tokenizer=tokenizer, model_seqlen=2048)`. The issue lied in the model_seqlen used to create the calibration dataset. For mistral, we have a very big value `32768` and this created some issues. LMK if this works ! 

That worked, quantization has started successfully!

Thank you, you saved me!

@SunMarc I've faced same issue with CUDA out of memory for Falcon 1B/7B and Gemma 2B/7B. Setting model_seqlen parameter is not helping, since these models have different API as far as I understand. 

I've tried to build optimum from the source, with that small fix for gptq quantization, but that didn't help as well.

Is that some common problem with GPTQ right now, or could that be some issue with setup on my side?

Hey @franchukpetro , In the falcon/gemma case, does the quantization starts (tqdm progress bar) or it just OOM ? 

@SunMarc  As far as I remember for Falcon quantization starts, but drops in the middle of the process. For Gemma I don't remember 100%, but I guess it was dropping right at the start without tqdm progress bar appearing.

Thanks for the details, I'll investigate a bit when I get the time!