Using 4bit or 8bit gives wrong result for COLA /Text Classification
Author: Oxi84
Pull request: no

### System Info

latest version of transfomers, ubuntu

### Who can help?

_No response_

### Information

- [ ] The official example scripts
- [x] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

    from transformers import AutoTokenizer, AutoModelForSequenceClassification
    from transformers import BitsAndBytesConfig
    quantization_config = BitsAndBytesConfig(load_in_4bit=True)    
    the_model_name1 = "textattack/roberta-base-CoLA"
    tokenizer3_rob = AutoTokenizer.from_pretrained(the_model_name1)  
    model3_rob = AutoModelForSequenceClassification.from_pretrained(the_model_name1, low_cpu_mem_usage=True,quantization_config=quantization_config).eval()
    print("loaded cola rob")



    from transformers import pipeline
    checker = pipeline("text-classification",the_model_name1)
    
    
    
    def cola_rob(sentences1):
        import torch
        import time
        beg = time.time()
        inputs = tokenizer3_rob(sentences1, return_tensors="pt", padding=True, truncation=True, max_length=50).to("cuda")
        with torch.no_grad():
            outputs = model3_rob(**inputs)
        
        print("outputs",outputs) 
        
        the_cola_scores = []
        for aout in outputs.logits:
            print(aout)
            cola_prediction = torch.nn.functional.softmax(aout)[0].item()
            print("torch.nn.functional.softmax(aout)",torch.nn.functional.softmax(aout))
            the_cola_scores.append(round(cola_prediction,3))
                
        return the_cola_scores
    
    sentences1 = ["I eat cheese."]
    res = cola_rob(sentences1)
    print(res)
    results = checker(sentences1)
    print("results",results)
    
    
    
    
    

### Expected behavior

Here I run cola for one sentence in 4 bits. And i get different results than when using pipeline, which uses 16 bit I suppose:

torch.nn.functional.softmax(aout) tensor([0.0574, 0.9424], device='cuda:0', dtype=torch.float16)
[0.057]
results [{'label': 'LABEL_1', 'score': 0.9699094295501709}]

So the result via 4bit is 0.057 while in 16 bit is 0.96

Of course they should be the same.



Comments:
Hi @Oxi84, thanks for opening this issue. 

I'm not sure why you would expect the values to be exactly the same - quantization changes the numerical representation of the weights, it's not a lossless compression and the benefits of reduced memory and faster computation aren't free. There will be some reduction in metric performance, ~but as shown in the example, it's actually not that big given the massive reduction in memory used!~ (I misread the example, the difference is actually quite large) 

cc'ing @younesbelkada, the king of quantization, who can comment more on this (and correct me if I'm wrong!)



> Hi @Oxi84, thanks for opening this issue.
> 
> I'm not sure why you would expect the values to be exactly the same - quantization changes the numerical representation of the weights, it's not a lossless compression and the benefits of reduced memory and faster computation aren't free. There will be some reduction in metric performance, but as shown in the example, it's actually not that big given the massive reduction in memory used!
> 
> cc'ing @younesbelkada, the king of quantization, who can comment more on this (and correct me if I'm wrong!)

Thanks for the answer.

I am not expecting them to be the same. i expected quite similar values, but they are completely different. This makes 4bit model not usable in this case.

For some sentences you get cola of 0.52 using 4 bits and 0.96 using 16 bits.  These are actually almost opposite values.

I am using 4bits for other models and they perform much better, such as for T5. The values are quite similar. And speed is quite improved, just around 30 percent slower than fp16.

@younesbelkada


This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.