FEAT : Adding VPTQ quantization method to HFQuantizer
Author: wejoncy
Pull request: yes

# What does this PR do?

This PR is intended to add support for Extreme Low-bit Vector Post-Training Quantization(VPTQ) to the transformers library.

VPTQ is a novel Post-Training Quantization method that leverages Vector Quantization to high accuracy on LLMs at an extremely low bit-width (<2-bit). VPTQ can compress 70B, even the 405B model, to 1-2 bits without retraining and maintain high accuracy.. More details here: https://github.com/microsoft/vptq


## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.

@SunMarc @younesbelkada @ArthurZucker 


Comments:
cc @MekkCyber 

Niiiiice! 🚀 

Awesome PR @wejoncy, Thank you for adding this amazing quantization method ! The integration looks very smooth🔥! I Just left some minor comments

> Awesome PR @wejoncy, Thank you for adding this amazing quantization method ! The integration looks very smooth🔥! I Just left some minor comments

Hi @MekkCyber   Really appreciate your review on this PR and thanks for your interest on VPTQ quantization integration into transformers.
I have addressed your comments in the new commits thanks again.

Please let me know if there is anything need to be fixed.

Thanks for iterating @wejoncy LGTM 🔥! Left only two small nits

Thanks for the updates @wejoncy ! Can't wait for the PR to be merged !

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_34770). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

> Thanks the PR ! Excited to see more VTPQ models on the hub. Just a few nits. Thanks also for the nice tests and the documentation

Thank @SunMarc, for your review! We are actively developing new models and algorithms, and we expect to release support for multimodal models (LLaMA 3.2 and Qwen2-VL) next month.

friendly ping @ArthurZucker 

Hi @ArthurZucker ,
A friendly Ping. Could you please take a look this PR?  I will be actively updating code if any suggestions. 
Thanks.

Hi @ArthurZucker,
Just a polite ping to check in—may I ask if there’s anything else we need to modify? Thank you!

Best regards,
Yang

Super sorry for the late review, I am a bit under water 😓 

> Super sorry for the late review, I am a bit under water 😓

Thanks so much! And thanks to everyone for their help! Special thanks to @xianbaoqian for the support—super grateful! 😊

Hey @YangWang92, i was going through the VPTQ paper, and I wanted to make sure that I understand correctly, so VPTQ is the same as GPTQ except that the quantization step is not linear but uses VQ instead, is that correct ? 

> Hey @YangWang92, i was going through the VPTQ paper, and I wanted to make sure that I understand correctly, so VPTQ is the same as GPTQ except that the quantization step is not linear but uses VQ instead, is that correct ?

Hi @MekkCyber, 
Sorry for the late reply! I'm quite busy recent days. VPTQ is indeed quite similar to GPTQ in terms of modeling the optimization problem, as both use second-order optimization methods for quantization. The key difference is that VPTQ specifically addresses the additional research questions introduced by Vector Quantization.