Allow compressed-tensors quantized model to be trained
Author: xkazuka
Pull request: yes

# What does this PR do?
Using HFQuantizer, models that were quantized using `compressed-tensors` can be loaded.

The purpose of this pr is to allow quantized models to be loaded using the trainer pathway. 
Currently, if quantized (HFQuantizer is instantiated based on the quantization_config), then raise for training. 

Using llm-compressor, we have a pathway to run oneshot (to quantize) and then finetune (QAT). 


## Who can review?
@SunMarc @younesbelkada

Comments:
@SunMarc 
Yes we are quantizing the model using oneshot from compressed-tensors, loading that model using `AutoModelForCausalLM` and `HFQuantizer`. Once loaded we will be running the training - qat finetuning. The 'quantization' we run is fakequant.

We are not using LoRA adapters

> Yes we are quantizing the model using oneshot from compressed-tensors, loading that model using AutoModelForCausalLM and HFQuantizer. Once loaded we will be running the training - qat finetuning. The 'quantization' we run is fakequant.
> 
> We are not using LoRA adapters

Nice thanks for confirming ! It would be nice to add the `is_qat_trainable` in the base class (`HfQuantizer`) and set it to False by default. Feel free to ping me when the PR is ready ! 

> > Yes we are quantizing the model using oneshot from compressed-tensors, loading that model using AutoModelForCausalLM and HFQuantizer. Once loaded we will be running the training - qat finetuning. The 'quantization' we run is fakequant.
> > We are not using LoRA adapters
> 
> Nice thanks for confirming ! It would be nice to add the `is_qat_trainable` in the base class (`HfQuantizer`) and set it to False by default. Feel free to ping me when the PR is ready !

Hey Mark, 

Its ready for review. 
There is a test failure but i think its from the api timeout.
it shows

```
1 failed because `requests.exceptions.ReadTimeout: (ReadTimeoutError("HTTPSConnectionPool(host='huggingface.co', port=443): Read timed out. (read timeout=10)"), '(Request ID` -> f55e37eb-9159-4b2b-bb6e-e87085e7590b)')
```

I tried to rerun, but the option is not clickable from my end. Anything I can do from my end to rerun?

Thank you!

> I tried to rerun, but the option is not clickable from my end. Anything I can do from my end to rerun?

No, don't worry. I will rerun it 

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_34520). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

@SunMarc 
Hi Marc, 
This is ready to merge

> Thanks! Given this:
> 
> > Using llm-compressor, we have a pathway to run oneshot (to quantize) and then finetune (QAT).
> 
> Just want to make sure we are not breaking (i.e. if we need a version check on this? )

Hi @ArthurZucker where are you recommending adding a version check?

> Thanks! Given this:
> 
> > Using llm-compressor, we have a pathway to run oneshot (to quantize) and then finetune (QAT).
> 
> Just want to make sure we are not breaking (i.e. if we need a version check on this? )

This won't be breaking. We had a oneshot then finetune pathway in the past. We are now using HFQuantizer to load model instead of our own custom class (based on AutoModelForCausalLM) we used to have. So previous versions of llm-compressor (library to run oneshot and finetune) and compressed-tensors (library to compress/decompress tensors) will support this pipeline.

If for comfortability, we want to add a version check, I would be happy to add! 

Let me know!