GPTQ quantization throws an error with custom dataset but works perfectly fine on already existing ones.
Author: manitadayon
Pull request: no
Labels: name: bug

### System Info

transformers: 4.45.2
huggingface-hub: 0.26.1
accelerate: 1.0.1
optimum: 1.23.1
auto-gptq: 0.7.1
bitsandbytes: 0.44.1

### Who can help?

_No response_

### Information

- [ ] The official example scripts
- [X] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [X] My own task or dataset (give details below)

### Reproduction

I can successfully quantize llama model using GPTQConfig given wikitext2 dataset, however the moment I switch to a custom dataset  (literally any custom dataset), I will get all sort of errors: Here is my setup:


```
custom_data = ['example1', example2', . . . , ]

config = GPTQConfig(bits = 4, dataset = custom_data (any list of string), tokenizer  = tokenizer, group_size = 128, desc_false = False, model_seq_len = 4096)
model = AutoModelForCausalLM.from_pretrained(model_path, quantization_config = config , device_map = 'auto')
```




This would give the error message of CUDA OOM, no matter size of data, what so ever. Like if I even choose two short data points.
Now if I get rid of device_map = 'auto' and set it as follows:

```
custom_data = ['example1', example2', . . . , ]

config = GPTQConfig(bits = 4, dataset = custom_data (any list of string), tokenizer  = tokenizer, group_size = 128, desc_false = False, model_seq_len = 4096)
model = AutoModelForCausalLM.from_pretrained(model_path, quantization_config = config , device_map =0)
```
I get the following error:

`Expected all tensors to be on the same device but got cpu and cuda
`
Removing the device_map = 'auto' also gives me the same error.

Now keeping everything the same but changing the dataset to wikitext2 works just fine. 
Wondering is there any problem with the custom data?  or is any way I can format my data to be understood by the package?


### Expected behavior

I expect according to documentation, the GPTQConfig works fine with custom dataset as long as the data is a list of string. BTW, I have enough resources in terms of memory & GPU, that I think CUDA OOM is not related to resource limitation.

Comments:
cc @SunMarc @MekkCyber for quantization!

Hey @manitadayon, what model are you trying to quantize ? and how much gpu memory do you have ? 
I am asking because this script works fine for me and you can find the result [here](https://huggingface.co/medmekk/Llama3.2-1B-GPTQ-4bits)
```
from transformers import GPTQConfig
from transformers import AutoModelForCausalLM, AutoTokenizer

model_path = "meta-llama/Llama-3.2-1B"

custom_data = ['example1', 'example2']

tokenizer = AutoTokenizer.from_pretrained(model_path)

config = GPTQConfig(bits = 4, dataset = custom_data, tokenizer=tokenizer, group_size = 128, desc_false = False, model_seq_len = 4096)
model = AutoModelForCausalLM.from_pretrained(model_path, quantization_config = config , device_map = 'auto')

model.push_to_hub('medmekk/Llama3.2-1B-GPTQ-4bits')
```
I replicated the same env as you with the same versions

Hi @MekkCyber, I am trying to quantize Llama 3.1 70B. I do not think resource availability is the issue since that script works on pre-existing data like wikitext2. I have 2 A100 GPUs (each 80GB memory) 
I have tried on 4 A100 GPUs with the same problem.

Thanks for the quick reply ! Yes I see the problem and it's weird I can't reproduce it, what version of torch are you using ?

It is 2.0.1 (I can see this is not updated), May I know what is your version? I can try with new version, it is just it is strange it works for pre-existing data. 

I am using torch 2.5.1+cu121 (yes that's true it's strange about how it works with pre-existing data but I even tried on an other gpu and it works for me)

Thanks may I know about the format of your data? Is it like list of strings or something similar? 

Yeah it's a list of strings, I will try with Llama 3.1 70B next to see !

@manitadayon, FYI I tried the 70B model with a list of two sentences and it works fine https://huggingface.co/medmekk/Llama3.1-70B-GPTQ-4bits, I tried on different gpus as well

@MekkCyber wow, this is beyond strange, may I know the configuration of .from_pretrained since under one case which I originally mentioned I get expected tensor to be on the same device. 

Do you have device_map = 'auto', I am thinking it maybe my configuration for either GPTQConfig or from_pretrained. 

Well I used the very same GPTQConfig you shared in the issue description, and I used device_map = "auto", did you retry with an updated torch version ?

Trying to upgrade the torch version, now I get a problem with flash_attn. I think the problem might be this package since I did not mention the version before. My version is flash_attn = 2.3.6
May I know the version of your flash_attn? 

I didn't specify the attention implementation to be `flash_attention` so it's `SDPA` by default. But I don't understand do you get the error with `flash_attn` when trying to quantize with the same config you shared above ? If not can you share your exact code that's not working ?

Just tried this with the latest version of flash_attn and I got the same error (CUDA OOM). I don't use flash_attn, just installed the latest version of flash_attn.

I tried to update my torch version, and run the script again and then I got the error:
RuntimeError: Failed to import transformers.models.llama.modeling_llama because of the following error:

`Undefined symbol: _ZN2at4_ops9_pad_enum4callERKNS_6TensorEN3c108ArrayRefINS5_6SymIntEEElNS5_8optionalIdEE`

Which seems to be for version of flash. 

I am thinking it maybe package version conflicts (since my packages are pre installed and I am using databricks) 

> Hey [@manitadayon](https://github.com/manitadayon), what model are you trying to quantize ? and how much gpu memory do you have ? I am asking because this script works fine for me and you can find the result [here](https://huggingface.co/medmekk/Llama3.2-1B-GPTQ-4bits)
> 
> ```
> from transformers import GPTQConfig
> from transformers import AutoModelForCausalLM, AutoTokenizer
> 
> model_path = "meta-llama/Llama-3.2-1B"
> 
> custom_data = ['example1', 'example2']
> 
> tokenizer = AutoTokenizer.from_pretrained(model_path)
> 
> config = GPTQConfig(bits = 4, dataset = custom_data, tokenizer=tokenizer, group_size = 128, desc_false = False, model_seq_len = 4096)
> model = AutoModelForCausalLM.from_pretrained(model_path, quantization_config = config , device_map = 'auto')
> 
> model.push_to_hub('medmekk/Llama3.2-1B-GPTQ-4bits')
> ```
> 
> I replicated the same env as you with the same versions

How would it work for a sequence-classification model?

@DebasishDhal , sorry for the late response, I made it to work, the problem was incompatibility of package versioning and that's why I was getting errors. My script was very similar to yours only with different parameters.

@manitadayon Correct me if I'm wrong.

Did you manage to get it working for sequence classification task? I don't think this is doable, but I'd be happy to look into the code. Can you post it here?