Repository: huggingface/transformers
Pull request #37159: Add "selecting a quantization method" doc
Author: login: DerekLiu35
Created: 2025-04-01T01:47:56Z | Merged: 2025-04-09T13:51:37Z

# What does this PR do?
Modify docs to help users determine which quantization methods to use

## Who can review?
@SunMarc @stevhliu


Diff:
diff --git a/docs/source/en/_toctree.yml b/docs/source/en/_toctree.yml
index ed306fac947f..0bcc6427f41e 100644
--- a/docs/source/en/_toctree.yml
+++ b/docs/source/en/_toctree.yml
@@ -161,6 +161,8 @@
   sections:
   - local: quantization/overview
     title: Overview
+  - local: quantization/selecting
+    title: Selecting a quantization method
   - local: quantization/aqlm
     title: AQLM
   - local: quantization/awq
diff --git a/docs/source/en/quantization/selecting.md b/docs/source/en/quantization/selecting.md
new file mode 100644
index 000000000000..b22097432ce8
--- /dev/null
+++ b/docs/source/en/quantization/selecting.md
@@ -0,0 +1,135 @@
+<!--Copyright 2024 The HuggingFace Team. All rights reserved.
+
+Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with
+the License. You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on
+an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the
+specific language governing permissions and limitations under the License.
+
+⚠️ Note that this file is in Markdown but contain specific syntax for our doc-builder (similar to MDX) that may not be
+rendered properly in your Markdown viewer.
+
+-->
+
+# Selecting a quantization method
+
+There are many quantization methods available in Transformers for inference and fine-tuning. This guide helps you choose the most common and production-ready quantization techniques depending on your use case, and presents the advantages and disadvantages of each technique.
+
+For a comprehensive overview of all supported methods and their features, refer back to the table in the [Overview](./overview).
+
+## Inference
+
+Consider the quantization methods below for inference.
+
+| quantization method | use case |
+|---|---|
+| bitsandbytes | ease of use and QLoRA fine-tuning on NVIDIA GPUs |
+| compressed-tensors | loading specific quantized formats (FP8, Sparse) |
+| GPTQModel or AWQ | good 4-bit accuracy with upfront calibration |
+| HQQ | fast on the fly quantization without calibration |
+| torchao | flexibility and fast inference with torch.compile |
+
+### No Calibration Required (On-the-fly Quantization)
+
+These methods are generally easier to use as they don't need a separate calibration dataset or step.
+
+#### bitsandbytes
+
+| Pros                                                         | Cons                                                    |
+|--------------------------------------------------------------|---------------------------------------------------------|
+| Very simple, no calibration dataset required for inference.  | Primarily optimized for NVIDIA GPUs (CUDA).             |
+| Good community support and widely adopted.                   | Inference speedup isn't guaranteed.                     |
+
+See the [bitsandbytes documentation](./bitsandbytes) for more details.
+
+#### HQQ (Half-Quadratic Quantization)
+
+| Pros                                                                 | Cons                                                                       |
+|----------------------------------------------------------------------|----------------------------------------------------------------------------|
+| Fast quantization process, no calibration data needed.              | Accuracy can degrade significantly at bit depths <4-bit.                     |
+| Multiple backends for fast inference.                                | Inference speed may not match others unless using `torch.compile` or backends. |
+| Compatible with `torch.compile`.                                     |                                                                            |
+| Supports wide range of bit depths (8, 4, 3, 2, 1-bit).              |                                                                            |
+
+See the [HQQ documentation](./hqq) for more details.
+
+#### torchao
+
+| Pros                                                                 | Cons                                                                 |
+|----------------------------------------------------------------------|----------------------------------------------------------------------|
+| Strong integration with `torch.compile` for potential speedups.     | Newer library, ecosystem still evolving.                             |
+| Offers decent CPU quantization support.                              | Performance depends on `torch.compile` working well.                 |
+| Flexibility in quantization schemes (int8, int4, fp8).           | 4-bit quantization (int4wo) may not match GPTQ/AWQ in accuracy.              |
+
+See the [torchao documentation](./torchao) for more details.
+
+### Calibration-based Quantization
+
+These methods require an upfront calibration step using a dataset to potentially achieve higher accuracy.
+
+#### GPTQ/GPTQModel
+
+Calibration for 8B model takes ~20 minutes on one A100 gpu.
+
+| Pros                                                                 | Cons                                                                 |
+|----------------------------------------------------------------------|----------------------------------------------------------------------|
+| Often achieves high accuracy.                                        | Requires a calibration dataset and a separate calibration step.      |
+| Can lead to inference speedups.                                      | Possible to overfit on calibration data.                             |
+| Many pre-quantized GPTQ models on [Hugging Face Hub](https://huggingface.co/models?other=gptq). |                                           |
+
+See the [GPTQ documentation](./gptq) for more details.
+
+#### AWQ (Activation-aware Weight Quantization)
+
+Calibration for 8B model takes ~10 minutes on one A100 gpu.
+
+| Pros                                                                 | Cons                                                |
+|----------------------------------------------------------------------|-----------------------------------------------------|
+| Often achieves high accuracy at 4-bit. (Sometimes surpasses GPTQ on specific tasks.) | Requires calibration if quantizing yourself.        |
+| Can lead to inference speedups.                                      |                                                     |
+| Shorter calibration time than GPTQ.                                  |                                                     |
+| Many pre-quantized AWQ models on [Hugging Face Hub](https://huggingface.co/models?other=awq). |                                                     |
+
+See the [AWQ documentation](./awq) for more details.
+
+### Loading Specific Formats
+
+#### compressed-tensors
+
+| Pros                                                         | Cons                                                        |
+|--------------------------------------------------------------|-------------------------------------------------------------|
+| Supports flexible formats including FP8 and sparsity.        | Primarily for loading pre-quantized models.                 |
+|                                                              | Doesn't perform quantization within Transformers directly.  |
+
+See the [compressed-tensors documentation](./compressed_tensors) for more details.
+
+## Fine-tuning
+
+Consider the quantization method below during fine-tuning to save memory.
+
+### bitsandbytes[[training]]
+
+*   **Description:** The standard method for QLoRA fine-tuning via PEFT.
+*   **Pros:** Enables fine-tuning large models on consumer GPUs; widely supported and documented for PEFT.
+*   **Cons:** Primarily for NVIDIA GPUs.
+
+Other methods offer PEFT compatibility, though bitsandbytes is the most established and straightforward path for QLoRA.
+
+See the [bitsandbytes documentation](./bitsandbytes#qlora) and [PEFT Docs](https://huggingface.co/docs/peft/developer_guides/quantization#aqlm-quantization) for more details. 
+
+## Research
+
+Methods like [AQLM](./aqlm), [SpQR](./spqr), [VPTQ](./vptq), [HIGGS](./higgs), etc., push the boundaries of compression (< 2-bit) or explore novel techniques.
+
+*   Consider these if:
+    *   You need extreme compression (sub-4-bit).
+    *   You are conducting research or require state-of-the-art results from their respective papers.
+    *   You have significant compute resources available for potentially complex quantization procedures.
+We recommend consulting each methods documentation and associated papers carefully before choosing one for use in production.
+
+
+> [!TIP]
+> Always benchmark the performance (accuracy and speed) of the quantized model on your specific task and hardware to ensure it meets your requirements. Refer to the individual documentation pages linked above for detailed usage instructions.
\ No newline at end of file


Reviews:
user: login: SunMarc
body: Thanks for digging into that, left a couple of comments 
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-04-01T15:25:13Z
user: login: stevhliu
body: Really nice job simplifying the selection process, this will be super helpful!
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-04-01T21:22:51Z
user: login: stevhliu
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-04-01T21:32:30Z
user: login: DerekLiu35
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-02T03:17:39Z
user: login: stevhliu
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-04-02T15:44:43Z
user: login: stevhliu
body: LGTM, thanks!
state: APPROVED
author_association: MEMBER
submitted_at: 2025-04-02T20:04:32Z
user: login: SunMarc
body: Nice ! One last thing that could be nice to add is the link to each quants method page, so that they can quickly test ! 
Also can you separate in the inference section methods that can quantize on the fly and those who requires calibration. I think this is something users are the most interested. Feel free also to include an estimate of the calibration time.  
state: APPROVED
author_association: MEMBER
submitted_at: 2025-04-07T13:32:41Z
user: login: stevhliu
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-04-07T16:16:56Z
user: login: SunMarc
state: APPROVED
author_association: MEMBER
submitted_at: 2025-04-09T13:51:22Z

Review comments:
diff_hunk: @@ -0,0 +1,114 @@
+<!--Copyright 2024 The HuggingFace Team. All rights reserved.
+
+Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with
+the License. You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on
+an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the
+specific language governing permissions and limitations under the License.
+
+⚠️ Note that this file is in Markdown but contain specific syntax for our doc-builder (similar to MDX) that may not be
+rendered properly in your Markdown viewer.
+
+-->
+
+# Selecting a Quantization Method
+
+This guide helps you navigate the most common and production-ready quantization techniques available in Transformers and associated libraries. For a comprehensive overview of all supported methods and their features, refer back to the [Quantization Overview](./overview) table.
+
+## TLDR
+
+*   For ease of use and QLoRA fine-tuning on NVIDIA GPUs: **bitsandbytes**
+*   For good 4-bit accuracy with upfront calibration: **GPTQModel** or **AWQ**
+*   For speed via `torch.compile` and flexibility: **torchao**
+*   For fast on-the-fly quantization without calibration: **HQQ** is worth considering.
+
+## Production-Ready & Widely Used Methods
+
+These methods are generally well-tested, widely adopted, and suitable for many production scenarios.
+
+### 1. bitsandbytes (`load_in_8bit` / `load_in_4bit`)
+
+*   **Description:** Enables 4-bit training/inference (QLoRA using NF4) and 8-bit inference (LLM.int8).
path: docs/source/en/quantization/selecting.md
user: login: SunMarc
body: Maybe we should do two distinct sections: 
- training (e.g fine-tuning with peft, qat) 
- inference 
created_at: 2025-04-01T15:17:10Z
author_association: MEMBER
diff_hunk: @@ -0,0 +1,114 @@
+<!--Copyright 2024 The HuggingFace Team. All rights reserved.
+
+Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with
+the License. You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on
+an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the
+specific language governing permissions and limitations under the License.
+
+⚠️ Note that this file is in Markdown but contain specific syntax for our doc-builder (similar to MDX) that may not be
+rendered properly in your Markdown viewer.
+
+-->
+
+# Selecting a Quantization Method
+
+This guide helps you navigate the most common and production-ready quantization techniques available in Transformers and associated libraries. For a comprehensive overview of all supported methods and their features, refer back to the [Quantization Overview](./overview) table.
+
+## TLDR
+
+*   For ease of use and QLoRA fine-tuning on NVIDIA GPUs: **bitsandbytes**
+*   For good 4-bit accuracy with upfront calibration: **GPTQModel** or **AWQ**
path: docs/source/en/quantization/selecting.md
user: login: SunMarc
body: compressed-tensors also offer nice quantized models
created_at: 2025-04-01T15:18:23Z
author_association: MEMBER
diff_hunk: @@ -0,0 +1,114 @@
+<!--Copyright 2024 The HuggingFace Team. All rights reserved.
+
+Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with
+the License. You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on
+an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the
+specific language governing permissions and limitations under the License.
+
+⚠️ Note that this file is in Markdown but contain specific syntax for our doc-builder (similar to MDX) that may not be
+rendered properly in your Markdown viewer.
+
+-->
+
+# Selecting a Quantization Method
+
+This guide helps you navigate the most common and production-ready quantization techniques available in Transformers and associated libraries. For a comprehensive overview of all supported methods and their features, refer back to the [Quantization Overview](./overview) table.
+
+## TLDR
+
+*   For ease of use and QLoRA fine-tuning on NVIDIA GPUs: **bitsandbytes**
+*   For good 4-bit accuracy with upfront calibration: **GPTQModel** or **AWQ**
path: docs/source/en/quantization/selecting.md
user: login: SunMarc
body: maybe having some kind of flow chart could be nice in the end. I think that you had one at some point @stevhliu 
created_at: 2025-04-01T15:22:00Z
author_association: MEMBER
diff_hunk: @@ -0,0 +1,114 @@
+<!--Copyright 2024 The HuggingFace Team. All rights reserved.
+
+Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with
+the License. You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on
+an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the
+specific language governing permissions and limitations under the License.
+
+⚠️ Note that this file is in Markdown but contain specific syntax for our doc-builder (similar to MDX) that may not be
+rendered properly in your Markdown viewer.
+
+-->
+
+# Selecting a Quantization Method
+
+This guide helps you navigate the most common and production-ready quantization techniques available in Transformers and associated libraries. For a comprehensive overview of all supported methods and their features, refer back to the [Quantization Overview](./overview) table.
+
+## TLDR
+
+*   For ease of use and QLoRA fine-tuning on NVIDIA GPUs: **bitsandbytes**
+*   For good 4-bit accuracy with upfront calibration: **GPTQModel** or **AWQ**
+*   For speed via `torch.compile` and flexibility: **torchao**
+*   For fast on-the-fly quantization without calibration: **HQQ** is worth considering.
+
+## Production-Ready & Widely Used Methods
+
+These methods are generally well-tested, widely adopted, and suitable for many production scenarios.
+
+### 1. bitsandbytes (`load_in_8bit` / `load_in_4bit`)
+
+*   **Description:** Enables 4-bit training/inference (QLoRA using NF4) and 8-bit inference (LLM.int8).
+*   **Pros:**
+    *   No calibration dataset required for basic inference quantization.
+    *   The standard and necessary backend for **QLoRA** fine-tuning.
+    *   Good community support and widely adopted.
+*   **Cons:**
+    *   Primarily optimized for NVIDIA GPUs (CUDA). There is [ongoing work](https://github.com/bitsandbytes-foundation/bitsandbytes/blob/main/docs/source/installation.mdx#multi-backend) to support other hardware (Intel/AMD, Apple M-series, etc.)
+    *   Inference speedup compared to fp16/bf16 might not always be significant.
+    *   LLM.int8 (8-bit) can sometimes have higher accuracy degradation than PTQ methods for 8-bit.
+
+See the [bitsandbytes documentation](./bitsandbytes) for more details.
path: docs/source/en/quantization/selecting.md
user: login: SunMarc
body: I feel like there might a bit too much details in general and that we should better make this doc as simple as possible. Lots of technical words that can confuse a lot the user
created_at: 2025-04-01T15:24:55Z
author_association: MEMBER
diff_hunk: @@ -0,0 +1,136 @@
+<!--Copyright 2024 The HuggingFace Team. All rights reserved.
+
+Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with
+the License. You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on
+an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the
+specific language governing permissions and limitations under the License.
+
+⚠️ Note that this file is in Markdown but contain specific syntax for our doc-builder (similar to MDX) that may not be
+rendered properly in your Markdown viewer.
+
+-->
+
+# Selecting a Quantization Method
+
+This guide helps you navigate the most common and production-ready quantization techniques available in Transformers and associated libraries. For a comprehensive overview of all supported methods and their features, refer back to the [Quantization Overview](./overview) table.
path: docs/source/en/quantization/selecting.md
user: login: stevhliu
body: ```suggestion
# Selecting a quantization method

There are many quantization methods available in Transformers for inference and fine-tuning. This guide helps you choose the most common and production-ready quantization techniques depending on your use case, and presents the advantages and disadvantages of each technique.

For a comprehensive overview of all supported methods and their features, refer back to the table in the [Overview](./overview).
```
created_at: 2025-04-01T21:07:11Z
author_association: MEMBER
diff_hunk: @@ -0,0 +1,136 @@
+<!--Copyright 2024 The HuggingFace Team. All rights reserved.
+
+Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with
+the License. You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on
+an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the
+specific language governing permissions and limitations under the License.
+
+⚠️ Note that this file is in Markdown but contain specific syntax for our doc-builder (similar to MDX) that may not be
+rendered properly in your Markdown viewer.
+
+-->
+
+# Selecting a Quantization Method
+
+This guide helps you navigate the most common and production-ready quantization techniques available in Transformers and associated libraries. For a comprehensive overview of all supported methods and their features, refer back to the [Quantization Overview](./overview) table.
+
+## Inference
path: docs/source/en/quantization/selecting.md
user: login: stevhliu
body: I think it'd be easier to present the info in a table, and it'd be nice to clarify what exactly is meant by "nice quantized models" :)

```suggestion
## Inference

Consider the quantization methods below for inference.

| quantization method | use case |
|---|---|
| bitsandbytes | ease of use and QLoRA fine-tuning on NVIDIA GPUs |
| compressed-tensors | loading specific quantized formats (FP8, Sparse) |
| GPTQModel or AWQ | good 4-bit accuracy with upfront calibration |
| HQQ | fast on the fly quantization without calibration |
| torchao | flexibility and fast inference with torch.compile |
```
created_at: 2025-04-01T21:12:29Z
author_association: MEMBER
diff_hunk: @@ -0,0 +1,136 @@
+<!--Copyright 2024 The HuggingFace Team. All rights reserved.
+
+Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with
+the License. You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on
+an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the
+specific language governing permissions and limitations under the License.
+
+⚠️ Note that this file is in Markdown but contain specific syntax for our doc-builder (similar to MDX) that may not be
+rendered properly in your Markdown viewer.
+
+-->
+
+# Selecting a Quantization Method
+
+This guide helps you navigate the most common and production-ready quantization techniques available in Transformers and associated libraries. For a comprehensive overview of all supported methods and their features, refer back to the [Quantization Overview](./overview) table.
+
+## Inference
+
+**TLDR (Inference):**
+
+*   For ease of use and QLoRA fine-tuning on NVIDIA GPUs: **bitsandbytes**
+*   For good 4-bit accuracy with upfront calibration: **GPTQModel** or **AWQ**
+*   For speed via `torch.compile` and flexibility: **torchao**
+*   For fast on-the-fly quantization without calibration: **HQQ** is worth considering.
+*   For loading specific formats (FP8, Sparse) and nice quantized models: **compressed-tensors** 
+
+
+## Production-Ready & Widely Used Inference Methods:
+These methods are generally well-tested, widely adopted, and suitable for many production scenarios.
+
+### 1. bitsandbytes (`load_in_8bit` / `load_in_4bit`)
path: docs/source/en/quantization/selecting.md
user: login: stevhliu
body: I think it'd also be easier here to compare and read if the info was presented in a table!

```suggestion
### bitsandbytes
```
created_at: 2025-04-01T21:14:30Z
author_association: MEMBER
diff_hunk: @@ -0,0 +1,136 @@
+<!--Copyright 2024 The HuggingFace Team. All rights reserved.
+
+Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with
+the License. You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on
+an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the
+specific language governing permissions and limitations under the License.
+
+⚠️ Note that this file is in Markdown but contain specific syntax for our doc-builder (similar to MDX) that may not be
+rendered properly in your Markdown viewer.
+
+-->
+
+# Selecting a Quantization Method
+
+This guide helps you navigate the most common and production-ready quantization techniques available in Transformers and associated libraries. For a comprehensive overview of all supported methods and their features, refer back to the [Quantization Overview](./overview) table.
+
+## Inference
+
+**TLDR (Inference):**
+
+*   For ease of use and QLoRA fine-tuning on NVIDIA GPUs: **bitsandbytes**
+*   For good 4-bit accuracy with upfront calibration: **GPTQModel** or **AWQ**
+*   For speed via `torch.compile` and flexibility: **torchao**
+*   For fast on-the-fly quantization without calibration: **HQQ** is worth considering.
+*   For loading specific formats (FP8, Sparse) and nice quantized models: **compressed-tensors** 
+
+
+## Production-Ready & Widely Used Inference Methods:
+These methods are generally well-tested, widely adopted, and suitable for many production scenarios.
+
+### 1. bitsandbytes (`load_in_8bit` / `load_in_4bit`)
+
+*   **Description:** Enables 4-bit and 8-bit inference.
+*   **Pros:**
+    *   Very simple, no calibration dataset required for inference
+    *   Good community support and widely adopted.
+*   **Cons:**
+    *   Primarily optimized for NVIDIA GPUs (CUDA).  There is [ongoing work](https://github.com/bitsandbytes-foundation/bitsandbytes/blob/main/docs/source/installation.mdx#multi-backend) to support other hardware.
+    *   Inference speedup isn't guaranteed
+
+See the [bitsandbytes documentation](./bitsandbytes) for more details.
+
+### 2. GPTQ / GPTQModel
+
+*   **Description:** quantizes weights (commonly to 4-bit, but supports others) by processing calibration data. 
+*   **Pros:**
+    *   Often achieves high accuracy
+    *   Can lead to inference speedups
+    *   There are many pre-quantized GPTQ models on [Hugging Face Hub](https://huggingface.co/models?other=gptq). This means you can often skip doing the quantization yourself and directly download a GPTQ-quantized model.
+*   **Cons:**
+    *   Requires a calibration dataset and a separate calibration step before inference. This takes time and compute resources.
+    *   Accuracy depends on the quality and relevance of the calibration data (possible to overfit on calibration data).
+
+See the [GPTQ documentation](./gptq) for more details.
+
+### 3. AWQ (Activation-aware Weight Quantization)
+
+*   **Description:** Similar to GPTQ in workflow, that also typically targets 4-bit quantization.
+*   **Pros:**
+    *   Often achieves high accuracy at 4-bit, sometimes surpassing GPTQ for specific tasks.
+    *   Can lead to inference speedups
+    *   Shorter calibration time than GPTQ.
+    *   There are many pre-quantized AWQ models on [Hugging Face Hub](https://huggingface.co/models?other=awq). This means you can often skip doing the quantization yourself and directly download a AWQ-quantized model.
+*   **Cons:**
+    *   Requires calibration if quantizing yourself.
+
+See the [AWQ documentation](./awq) for more details.
+
+### 4. torchao
+
+*   **Description:** Library from PyTorch focused on architecture optimization. It's designed for strong integration with `torch.compile`.
+*   **Pros:**
+    *   Strong integration with `torch.compile`, potentially leading to significant inference speedups when compiled.
+    *   Offers decent CPU quantization support.
+    *   Flexibility in quantization schemes (int8, int4, fp8).
+*   **Cons:**
+    *   Newer compared to BnB or GPTQ, so the ecosystem and best practices are still evolving.
+    *   Performance gains often rely heavily on `torch.compile` working effectively for the model and hardware.
+    *   4-bit quantized model via torchao might not reach the same accuracy as GPTQ/AWQ’s 4-bit, because torchao might just do straightforward rounding.
+
+See the [torchao documentation](./torchao) for more details.
+
+### 5. HQQ (Half-Quadratic Quantization)
+
+*   **Description:** A fast, on-the-fly quantization method (like bitsandbytes) that doesn't require calibration data.
+*   **Pros:**
+    *   Fast quantization process and no calibration data required.
+    *   Multiple backends for fast inference.
+    *   Compatible with `torch.compile` for potential inference speedups.
+    *   It supports a wide range of bit depths (8, 4, 3, 2, 1-bit)
+*   **Cons:**
+    *   Accuracy at very low bit depths (<4-bit) can degrade significantly and requires careful evaluation for the specific task.
+    *   While quantization is fast, inference speed might not be faster than other methods unless using specific backends or `torch.compile`.
+See the [HQQ documentation](./hqq) for more details.
+
+### 6. compressed-tensors
+
+*   **Description:** Allows loading models saved in specific compressed formats (like FP8, INT8/4, potentially with sparsity).
+*   **Pros:** 
+    *   Supports flexible formats including FP8 and sparsity.
+*   **Cons:** 
+    *   Primarily for loading pre-quantized models, not performing the quantization within Transformers itself.
+
+See the [compressed-tensors documentation](./compressed_tensors) for more details.
+
+## Fine-Tuning
+
+Quantization can also be used during fine-tuning to save memory, most commonly with PEFT's QLoRA.
path: docs/source/en/quantization/selecting.md
user: login: stevhliu
body: ```suggestion
## Fine-tuning

Consider the quantization method below during fine-tuning to save memory.
```
created_at: 2025-04-01T21:19:02Z
author_association: MEMBER
diff_hunk: @@ -0,0 +1,136 @@
+<!--Copyright 2024 The HuggingFace Team. All rights reserved.
+
+Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with
+the License. You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on
+an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the
+specific language governing permissions and limitations under the License.
+
+⚠️ Note that this file is in Markdown but contain specific syntax for our doc-builder (similar to MDX) that may not be
+rendered properly in your Markdown viewer.
+
+-->
+
+# Selecting a Quantization Method
+
+This guide helps you navigate the most common and production-ready quantization techniques available in Transformers and associated libraries. For a comprehensive overview of all supported methods and their features, refer back to the [Quantization Overview](./overview) table.
+
+## Inference
+
+**TLDR (Inference):**
+
+*   For ease of use and QLoRA fine-tuning on NVIDIA GPUs: **bitsandbytes**
+*   For good 4-bit accuracy with upfront calibration: **GPTQModel** or **AWQ**
+*   For speed via `torch.compile` and flexibility: **torchao**
+*   For fast on-the-fly quantization without calibration: **HQQ** is worth considering.
+*   For loading specific formats (FP8, Sparse) and nice quantized models: **compressed-tensors** 
+
+
+## Production-Ready & Widely Used Inference Methods:
+These methods are generally well-tested, widely adopted, and suitable for many production scenarios.
+
+### 1. bitsandbytes (`load_in_8bit` / `load_in_4bit`)
+
+*   **Description:** Enables 4-bit and 8-bit inference.
+*   **Pros:**
+    *   Very simple, no calibration dataset required for inference
+    *   Good community support and widely adopted.
+*   **Cons:**
+    *   Primarily optimized for NVIDIA GPUs (CUDA).  There is [ongoing work](https://github.com/bitsandbytes-foundation/bitsandbytes/blob/main/docs/source/installation.mdx#multi-backend) to support other hardware.
+    *   Inference speedup isn't guaranteed
+
+See the [bitsandbytes documentation](./bitsandbytes) for more details.
+
+### 2. GPTQ / GPTQModel
+
+*   **Description:** quantizes weights (commonly to 4-bit, but supports others) by processing calibration data. 
+*   **Pros:**
+    *   Often achieves high accuracy
+    *   Can lead to inference speedups
+    *   There are many pre-quantized GPTQ models on [Hugging Face Hub](https://huggingface.co/models?other=gptq). This means you can often skip doing the quantization yourself and directly download a GPTQ-quantized model.
+*   **Cons:**
+    *   Requires a calibration dataset and a separate calibration step before inference. This takes time and compute resources.
+    *   Accuracy depends on the quality and relevance of the calibration data (possible to overfit on calibration data).
+
+See the [GPTQ documentation](./gptq) for more details.
+
+### 3. AWQ (Activation-aware Weight Quantization)
+
+*   **Description:** Similar to GPTQ in workflow, that also typically targets 4-bit quantization.
+*   **Pros:**
+    *   Often achieves high accuracy at 4-bit, sometimes surpassing GPTQ for specific tasks.
+    *   Can lead to inference speedups
+    *   Shorter calibration time than GPTQ.
+    *   There are many pre-quantized AWQ models on [Hugging Face Hub](https://huggingface.co/models?other=awq). This means you can often skip doing the quantization yourself and directly download a AWQ-quantized model.
+*   **Cons:**
+    *   Requires calibration if quantizing yourself.
+
+See the [AWQ documentation](./awq) for more details.
+
+### 4. torchao
+
+*   **Description:** Library from PyTorch focused on architecture optimization. It's designed for strong integration with `torch.compile`.
+*   **Pros:**
+    *   Strong integration with `torch.compile`, potentially leading to significant inference speedups when compiled.
+    *   Offers decent CPU quantization support.
+    *   Flexibility in quantization schemes (int8, int4, fp8).
+*   **Cons:**
+    *   Newer compared to BnB or GPTQ, so the ecosystem and best practices are still evolving.
+    *   Performance gains often rely heavily on `torch.compile` working effectively for the model and hardware.
+    *   4-bit quantized model via torchao might not reach the same accuracy as GPTQ/AWQ’s 4-bit, because torchao might just do straightforward rounding.
+
+See the [torchao documentation](./torchao) for more details.
+
+### 5. HQQ (Half-Quadratic Quantization)
+
+*   **Description:** A fast, on-the-fly quantization method (like bitsandbytes) that doesn't require calibration data.
+*   **Pros:**
+    *   Fast quantization process and no calibration data required.
+    *   Multiple backends for fast inference.
+    *   Compatible with `torch.compile` for potential inference speedups.
+    *   It supports a wide range of bit depths (8, 4, 3, 2, 1-bit)
+*   **Cons:**
+    *   Accuracy at very low bit depths (<4-bit) can degrade significantly and requires careful evaluation for the specific task.
+    *   While quantization is fast, inference speed might not be faster than other methods unless using specific backends or `torch.compile`.
+See the [HQQ documentation](./hqq) for more details.
+
+### 6. compressed-tensors
+
+*   **Description:** Allows loading models saved in specific compressed formats (like FP8, INT8/4, potentially with sparsity).
+*   **Pros:** 
+    *   Supports flexible formats including FP8 and sparsity.
+*   **Cons:** 
+    *   Primarily for loading pre-quantized models, not performing the quantization within Transformers itself.
+
+See the [compressed-tensors documentation](./compressed_tensors) for more details.
+
+## Fine-Tuning
+
+Quantization can also be used during fine-tuning to save memory, most commonly with PEFT's QLoRA.
+
+### bitsandbytes (4-bit NF4 for QLoRA)
path: docs/source/en/quantization/selecting.md
user: login: stevhliu
body: Format for identical section headers

```suggestion
### bitsandbytes[[training]]
```
created_at: 2025-04-01T21:19:50Z
author_association: MEMBER
diff_hunk: @@ -0,0 +1,136 @@
+<!--Copyright 2024 The HuggingFace Team. All rights reserved.
+
+Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with
+the License. You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on
+an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the
+specific language governing permissions and limitations under the License.
+
+⚠️ Note that this file is in Markdown but contain specific syntax for our doc-builder (similar to MDX) that may not be
+rendered properly in your Markdown viewer.
+
+-->
+
+# Selecting a Quantization Method
+
+This guide helps you navigate the most common and production-ready quantization techniques available in Transformers and associated libraries. For a comprehensive overview of all supported methods and their features, refer back to the [Quantization Overview](./overview) table.
+
+## Inference
+
+**TLDR (Inference):**
+
+*   For ease of use and QLoRA fine-tuning on NVIDIA GPUs: **bitsandbytes**
+*   For good 4-bit accuracy with upfront calibration: **GPTQModel** or **AWQ**
+*   For speed via `torch.compile` and flexibility: **torchao**
+*   For fast on-the-fly quantization without calibration: **HQQ** is worth considering.
+*   For loading specific formats (FP8, Sparse) and nice quantized models: **compressed-tensors** 
+
+
+## Production-Ready & Widely Used Inference Methods:
+These methods are generally well-tested, widely adopted, and suitable for many production scenarios.
+
+### 1. bitsandbytes (`load_in_8bit` / `load_in_4bit`)
+
+*   **Description:** Enables 4-bit and 8-bit inference.
+*   **Pros:**
+    *   Very simple, no calibration dataset required for inference
+    *   Good community support and widely adopted.
+*   **Cons:**
+    *   Primarily optimized for NVIDIA GPUs (CUDA).  There is [ongoing work](https://github.com/bitsandbytes-foundation/bitsandbytes/blob/main/docs/source/installation.mdx#multi-backend) to support other hardware.
+    *   Inference speedup isn't guaranteed
+
+See the [bitsandbytes documentation](./bitsandbytes) for more details.
+
+### 2. GPTQ / GPTQModel
+
+*   **Description:** quantizes weights (commonly to 4-bit, but supports others) by processing calibration data. 
+*   **Pros:**
+    *   Often achieves high accuracy
+    *   Can lead to inference speedups
+    *   There are many pre-quantized GPTQ models on [Hugging Face Hub](https://huggingface.co/models?other=gptq). This means you can often skip doing the quantization yourself and directly download a GPTQ-quantized model.
+*   **Cons:**
+    *   Requires a calibration dataset and a separate calibration step before inference. This takes time and compute resources.
+    *   Accuracy depends on the quality and relevance of the calibration data (possible to overfit on calibration data).
+
+See the [GPTQ documentation](./gptq) for more details.
+
+### 3. AWQ (Activation-aware Weight Quantization)
+
+*   **Description:** Similar to GPTQ in workflow, that also typically targets 4-bit quantization.
+*   **Pros:**
+    *   Often achieves high accuracy at 4-bit, sometimes surpassing GPTQ for specific tasks.
+    *   Can lead to inference speedups
+    *   Shorter calibration time than GPTQ.
+    *   There are many pre-quantized AWQ models on [Hugging Face Hub](https://huggingface.co/models?other=awq). This means you can often skip doing the quantization yourself and directly download a AWQ-quantized model.
+*   **Cons:**
+    *   Requires calibration if quantizing yourself.
+
+See the [AWQ documentation](./awq) for more details.
+
+### 4. torchao
+
+*   **Description:** Library from PyTorch focused on architecture optimization. It's designed for strong integration with `torch.compile`.
+*   **Pros:**
+    *   Strong integration with `torch.compile`, potentially leading to significant inference speedups when compiled.
+    *   Offers decent CPU quantization support.
+    *   Flexibility in quantization schemes (int8, int4, fp8).
+*   **Cons:**
+    *   Newer compared to BnB or GPTQ, so the ecosystem and best practices are still evolving.
+    *   Performance gains often rely heavily on `torch.compile` working effectively for the model and hardware.
+    *   4-bit quantized model via torchao might not reach the same accuracy as GPTQ/AWQ’s 4-bit, because torchao might just do straightforward rounding.
+
+See the [torchao documentation](./torchao) for more details.
+
+### 5. HQQ (Half-Quadratic Quantization)
+
+*   **Description:** A fast, on-the-fly quantization method (like bitsandbytes) that doesn't require calibration data.
+*   **Pros:**
+    *   Fast quantization process and no calibration data required.
+    *   Multiple backends for fast inference.
+    *   Compatible with `torch.compile` for potential inference speedups.
+    *   It supports a wide range of bit depths (8, 4, 3, 2, 1-bit)
+*   **Cons:**
+    *   Accuracy at very low bit depths (<4-bit) can degrade significantly and requires careful evaluation for the specific task.
+    *   While quantization is fast, inference speed might not be faster than other methods unless using specific backends or `torch.compile`.
+See the [HQQ documentation](./hqq) for more details.
+
+### 6. compressed-tensors
+
+*   **Description:** Allows loading models saved in specific compressed formats (like FP8, INT8/4, potentially with sparsity).
+*   **Pros:** 
+    *   Supports flexible formats including FP8 and sparsity.
+*   **Cons:** 
+    *   Primarily for loading pre-quantized models, not performing the quantization within Transformers itself.
+
+See the [compressed-tensors documentation](./compressed_tensors) for more details.
+
+## Fine-Tuning
+
+Quantization can also be used during fine-tuning to save memory, most commonly with PEFT's QLoRA.
+
+### bitsandbytes (4-bit NF4 for QLoRA)
+
+*   **Description:** The standard method for QLoRA fine-tuning via PEFT.
+*   **Pros:** Enables fine-tuning large models on consumer GPUs; widely supported and documented for PEFT.
+*   **Cons:** Primarily for NVIDIA GPUs.
+
+Other methods offer PEFT compatibility, though bitsandbytes is the most established and straightforward path for QLoRA.
+
+See the [bitsandbytes documentation](./bitsandbytes#qlora) and [PEFT Docs](https://huggingface.co/docs/peft/developer_guides/quantization#aqlm-quantization) for more details. 
+
+## Advanced & Research-Oriented Methods
path: docs/source/en/quantization/selecting.md
user: login: stevhliu
body: ```suggestion
## Research
```
created_at: 2025-04-01T21:20:16Z
author_association: MEMBER
diff_hunk: @@ -0,0 +1,136 @@
+<!--Copyright 2024 The HuggingFace Team. All rights reserved.
+
+Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with
+the License. You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on
+an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the
+specific language governing permissions and limitations under the License.
+
+⚠️ Note that this file is in Markdown but contain specific syntax for our doc-builder (similar to MDX) that may not be
+rendered properly in your Markdown viewer.
+
+-->
+
+# Selecting a Quantization Method
+
+This guide helps you navigate the most common and production-ready quantization techniques available in Transformers and associated libraries. For a comprehensive overview of all supported methods and their features, refer back to the [Quantization Overview](./overview) table.
+
+## Inference
+
+**TLDR (Inference):**
+
+*   For ease of use and QLoRA fine-tuning on NVIDIA GPUs: **bitsandbytes**
+*   For good 4-bit accuracy with upfront calibration: **GPTQModel** or **AWQ**
+*   For speed via `torch.compile` and flexibility: **torchao**
+*   For fast on-the-fly quantization without calibration: **HQQ** is worth considering.
+*   For loading specific formats (FP8, Sparse) and nice quantized models: **compressed-tensors** 
+
+
+## Production-Ready & Widely Used Inference Methods:
+These methods are generally well-tested, widely adopted, and suitable for many production scenarios.
+
+### 1. bitsandbytes (`load_in_8bit` / `load_in_4bit`)
+
+*   **Description:** Enables 4-bit and 8-bit inference.
+*   **Pros:**
+    *   Very simple, no calibration dataset required for inference
+    *   Good community support and widely adopted.
+*   **Cons:**
+    *   Primarily optimized for NVIDIA GPUs (CUDA).  There is [ongoing work](https://github.com/bitsandbytes-foundation/bitsandbytes/blob/main/docs/source/installation.mdx#multi-backend) to support other hardware.
+    *   Inference speedup isn't guaranteed
+
+See the [bitsandbytes documentation](./bitsandbytes) for more details.
+
+### 2. GPTQ / GPTQModel
+
+*   **Description:** quantizes weights (commonly to 4-bit, but supports others) by processing calibration data. 
+*   **Pros:**
+    *   Often achieves high accuracy
+    *   Can lead to inference speedups
+    *   There are many pre-quantized GPTQ models on [Hugging Face Hub](https://huggingface.co/models?other=gptq). This means you can often skip doing the quantization yourself and directly download a GPTQ-quantized model.
+*   **Cons:**
+    *   Requires a calibration dataset and a separate calibration step before inference. This takes time and compute resources.
+    *   Accuracy depends on the quality and relevance of the calibration data (possible to overfit on calibration data).
+
+See the [GPTQ documentation](./gptq) for more details.
+
+### 3. AWQ (Activation-aware Weight Quantization)
+
+*   **Description:** Similar to GPTQ in workflow, that also typically targets 4-bit quantization.
+*   **Pros:**
+    *   Often achieves high accuracy at 4-bit, sometimes surpassing GPTQ for specific tasks.
+    *   Can lead to inference speedups
+    *   Shorter calibration time than GPTQ.
+    *   There are many pre-quantized AWQ models on [Hugging Face Hub](https://huggingface.co/models?other=awq). This means you can often skip doing the quantization yourself and directly download a AWQ-quantized model.
+*   **Cons:**
+    *   Requires calibration if quantizing yourself.
+
+See the [AWQ documentation](./awq) for more details.
+
+### 4. torchao
+
+*   **Description:** Library from PyTorch focused on architecture optimization. It's designed for strong integration with `torch.compile`.
+*   **Pros:**
+    *   Strong integration with `torch.compile`, potentially leading to significant inference speedups when compiled.
+    *   Offers decent CPU quantization support.
+    *   Flexibility in quantization schemes (int8, int4, fp8).
+*   **Cons:**
+    *   Newer compared to BnB or GPTQ, so the ecosystem and best practices are still evolving.
+    *   Performance gains often rely heavily on `torch.compile` working effectively for the model and hardware.
+    *   4-bit quantized model via torchao might not reach the same accuracy as GPTQ/AWQ’s 4-bit, because torchao might just do straightforward rounding.
+
+See the [torchao documentation](./torchao) for more details.
+
+### 5. HQQ (Half-Quadratic Quantization)
+
+*   **Description:** A fast, on-the-fly quantization method (like bitsandbytes) that doesn't require calibration data.
+*   **Pros:**
+    *   Fast quantization process and no calibration data required.
+    *   Multiple backends for fast inference.
+    *   Compatible with `torch.compile` for potential inference speedups.
+    *   It supports a wide range of bit depths (8, 4, 3, 2, 1-bit)
+*   **Cons:**
+    *   Accuracy at very low bit depths (<4-bit) can degrade significantly and requires careful evaluation for the specific task.
+    *   While quantization is fast, inference speed might not be faster than other methods unless using specific backends or `torch.compile`.
+See the [HQQ documentation](./hqq) for more details.
+
+### 6. compressed-tensors
+
+*   **Description:** Allows loading models saved in specific compressed formats (like FP8, INT8/4, potentially with sparsity).
+*   **Pros:** 
+    *   Supports flexible formats including FP8 and sparsity.
+*   **Cons:** 
+    *   Primarily for loading pre-quantized models, not performing the quantization within Transformers itself.
+
+See the [compressed-tensors documentation](./compressed_tensors) for more details.
+
+## Fine-Tuning
+
+Quantization can also be used during fine-tuning to save memory, most commonly with PEFT's QLoRA.
+
+### bitsandbytes (4-bit NF4 for QLoRA)
+
+*   **Description:** The standard method for QLoRA fine-tuning via PEFT.
+*   **Pros:** Enables fine-tuning large models on consumer GPUs; widely supported and documented for PEFT.
+*   **Cons:** Primarily for NVIDIA GPUs.
+
+Other methods offer PEFT compatibility, though bitsandbytes is the most established and straightforward path for QLoRA.
+
+See the [bitsandbytes documentation](./bitsandbytes#qlora) and [PEFT Docs](https://huggingface.co/docs/peft/developer_guides/quantization#aqlm-quantization) for more details. 
+
+## Advanced & Research-Oriented Methods
+
+Methods like [AQLM](./aqlm), [SpQR](./spqr), [VPTQ](./vptq), [HIGGS](./higgs), etc., often push the boundaries of compression (e.g., < 2-bit) or explore novel techniques.
path: docs/source/en/quantization/selecting.md
user: login: stevhliu
body: ```suggestion
Methods like [AQLM](./aqlm), [SpQR](./spqr), [VPTQ](./vptq), [HIGGS](./higgs), etc., push the boundaries of compression (< 2-bit) or explore novel techniques.
```
created_at: 2025-04-01T21:20:34Z
author_association: MEMBER
diff_hunk: @@ -0,0 +1,136 @@
+<!--Copyright 2024 The HuggingFace Team. All rights reserved.
+
+Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with
+the License. You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on
+an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the
+specific language governing permissions and limitations under the License.
+
+⚠️ Note that this file is in Markdown but contain specific syntax for our doc-builder (similar to MDX) that may not be
+rendered properly in your Markdown viewer.
+
+-->
+
+# Selecting a Quantization Method
+
+This guide helps you navigate the most common and production-ready quantization techniques available in Transformers and associated libraries. For a comprehensive overview of all supported methods and their features, refer back to the [Quantization Overview](./overview) table.
+
+## Inference
+
+**TLDR (Inference):**
+
+*   For ease of use and QLoRA fine-tuning on NVIDIA GPUs: **bitsandbytes**
+*   For good 4-bit accuracy with upfront calibration: **GPTQModel** or **AWQ**
+*   For speed via `torch.compile` and flexibility: **torchao**
+*   For fast on-the-fly quantization without calibration: **HQQ** is worth considering.
+*   For loading specific formats (FP8, Sparse) and nice quantized models: **compressed-tensors** 
+
+
+## Production-Ready & Widely Used Inference Methods:
+These methods are generally well-tested, widely adopted, and suitable for many production scenarios.
+
+### 1. bitsandbytes (`load_in_8bit` / `load_in_4bit`)
+
+*   **Description:** Enables 4-bit and 8-bit inference.
+*   **Pros:**
+    *   Very simple, no calibration dataset required for inference
+    *   Good community support and widely adopted.
+*   **Cons:**
+    *   Primarily optimized for NVIDIA GPUs (CUDA).  There is [ongoing work](https://github.com/bitsandbytes-foundation/bitsandbytes/blob/main/docs/source/installation.mdx#multi-backend) to support other hardware.
+    *   Inference speedup isn't guaranteed
+
+See the [bitsandbytes documentation](./bitsandbytes) for more details.
+
+### 2. GPTQ / GPTQModel
+
+*   **Description:** quantizes weights (commonly to 4-bit, but supports others) by processing calibration data. 
+*   **Pros:**
+    *   Often achieves high accuracy
+    *   Can lead to inference speedups
+    *   There are many pre-quantized GPTQ models on [Hugging Face Hub](https://huggingface.co/models?other=gptq). This means you can often skip doing the quantization yourself and directly download a GPTQ-quantized model.
+*   **Cons:**
+    *   Requires a calibration dataset and a separate calibration step before inference. This takes time and compute resources.
+    *   Accuracy depends on the quality and relevance of the calibration data (possible to overfit on calibration data).
+
+See the [GPTQ documentation](./gptq) for more details.
+
+### 3. AWQ (Activation-aware Weight Quantization)
+
+*   **Description:** Similar to GPTQ in workflow, that also typically targets 4-bit quantization.
+*   **Pros:**
+    *   Often achieves high accuracy at 4-bit, sometimes surpassing GPTQ for specific tasks.
+    *   Can lead to inference speedups
+    *   Shorter calibration time than GPTQ.
+    *   There are many pre-quantized AWQ models on [Hugging Face Hub](https://huggingface.co/models?other=awq). This means you can often skip doing the quantization yourself and directly download a AWQ-quantized model.
+*   **Cons:**
+    *   Requires calibration if quantizing yourself.
+
+See the [AWQ documentation](./awq) for more details.
+
+### 4. torchao
+
+*   **Description:** Library from PyTorch focused on architecture optimization. It's designed for strong integration with `torch.compile`.
+*   **Pros:**
+    *   Strong integration with `torch.compile`, potentially leading to significant inference speedups when compiled.
+    *   Offers decent CPU quantization support.
+    *   Flexibility in quantization schemes (int8, int4, fp8).
+*   **Cons:**
+    *   Newer compared to BnB or GPTQ, so the ecosystem and best practices are still evolving.
+    *   Performance gains often rely heavily on `torch.compile` working effectively for the model and hardware.
+    *   4-bit quantized model via torchao might not reach the same accuracy as GPTQ/AWQ’s 4-bit, because torchao might just do straightforward rounding.
+
+See the [torchao documentation](./torchao) for more details.
+
+### 5. HQQ (Half-Quadratic Quantization)
+
+*   **Description:** A fast, on-the-fly quantization method (like bitsandbytes) that doesn't require calibration data.
+*   **Pros:**
+    *   Fast quantization process and no calibration data required.
+    *   Multiple backends for fast inference.
+    *   Compatible with `torch.compile` for potential inference speedups.
+    *   It supports a wide range of bit depths (8, 4, 3, 2, 1-bit)
+*   **Cons:**
+    *   Accuracy at very low bit depths (<4-bit) can degrade significantly and requires careful evaluation for the specific task.
+    *   While quantization is fast, inference speed might not be faster than other methods unless using specific backends or `torch.compile`.
+See the [HQQ documentation](./hqq) for more details.
+
+### 6. compressed-tensors
+
+*   **Description:** Allows loading models saved in specific compressed formats (like FP8, INT8/4, potentially with sparsity).
+*   **Pros:** 
+    *   Supports flexible formats including FP8 and sparsity.
+*   **Cons:** 
+    *   Primarily for loading pre-quantized models, not performing the quantization within Transformers itself.
+
+See the [compressed-tensors documentation](./compressed_tensors) for more details.
+
+## Fine-Tuning
+
+Quantization can also be used during fine-tuning to save memory, most commonly with PEFT's QLoRA.
+
+### bitsandbytes (4-bit NF4 for QLoRA)
+
+*   **Description:** The standard method for QLoRA fine-tuning via PEFT.
+*   **Pros:** Enables fine-tuning large models on consumer GPUs; widely supported and documented for PEFT.
+*   **Cons:** Primarily for NVIDIA GPUs.
+
+Other methods offer PEFT compatibility, though bitsandbytes is the most established and straightforward path for QLoRA.
+
+See the [bitsandbytes documentation](./bitsandbytes#qlora) and [PEFT Docs](https://huggingface.co/docs/peft/developer_guides/quantization#aqlm-quantization) for more details. 
+
+## Advanced & Research-Oriented Methods
+
+Methods like [AQLM](./aqlm), [SpQR](./spqr), [VPTQ](./vptq), [HIGGS](./higgs), etc., often push the boundaries of compression (e.g., < 2-bit) or explore novel techniques.
+
+*   Consider these if:
+    *   You need extreme compression (sub-4-bit).
+    *   You are conducting research or require state-of-the-art results shown in their respective papers.
path: docs/source/en/quantization/selecting.md
user: login: stevhliu
body: ```suggestion
    *   You are conducting research or require state-of-the-art results from their respective papers.
```
created_at: 2025-04-01T21:21:10Z
author_association: MEMBER
diff_hunk: @@ -0,0 +1,136 @@
+<!--Copyright 2024 The HuggingFace Team. All rights reserved.
+
+Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with
+the License. You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on
+an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the
+specific language governing permissions and limitations under the License.
+
+⚠️ Note that this file is in Markdown but contain specific syntax for our doc-builder (similar to MDX) that may not be
+rendered properly in your Markdown viewer.
+
+-->
+
+# Selecting a Quantization Method
+
+This guide helps you navigate the most common and production-ready quantization techniques available in Transformers and associated libraries. For a comprehensive overview of all supported methods and their features, refer back to the [Quantization Overview](./overview) table.
+
+## Inference
+
+**TLDR (Inference):**
+
+*   For ease of use and QLoRA fine-tuning on NVIDIA GPUs: **bitsandbytes**
+*   For good 4-bit accuracy with upfront calibration: **GPTQModel** or **AWQ**
+*   For speed via `torch.compile` and flexibility: **torchao**
+*   For fast on-the-fly quantization without calibration: **HQQ** is worth considering.
+*   For loading specific formats (FP8, Sparse) and nice quantized models: **compressed-tensors** 
+
+
+## Production-Ready & Widely Used Inference Methods:
+These methods are generally well-tested, widely adopted, and suitable for many production scenarios.
+
+### 1. bitsandbytes (`load_in_8bit` / `load_in_4bit`)
+
+*   **Description:** Enables 4-bit and 8-bit inference.
+*   **Pros:**
+    *   Very simple, no calibration dataset required for inference
+    *   Good community support and widely adopted.
+*   **Cons:**
+    *   Primarily optimized for NVIDIA GPUs (CUDA).  There is [ongoing work](https://github.com/bitsandbytes-foundation/bitsandbytes/blob/main/docs/source/installation.mdx#multi-backend) to support other hardware.
+    *   Inference speedup isn't guaranteed
+
+See the [bitsandbytes documentation](./bitsandbytes) for more details.
+
+### 2. GPTQ / GPTQModel
+
+*   **Description:** quantizes weights (commonly to 4-bit, but supports others) by processing calibration data. 
+*   **Pros:**
+    *   Often achieves high accuracy
+    *   Can lead to inference speedups
+    *   There are many pre-quantized GPTQ models on [Hugging Face Hub](https://huggingface.co/models?other=gptq). This means you can often skip doing the quantization yourself and directly download a GPTQ-quantized model.
+*   **Cons:**
+    *   Requires a calibration dataset and a separate calibration step before inference. This takes time and compute resources.
+    *   Accuracy depends on the quality and relevance of the calibration data (possible to overfit on calibration data).
+
+See the [GPTQ documentation](./gptq) for more details.
+
+### 3. AWQ (Activation-aware Weight Quantization)
+
+*   **Description:** Similar to GPTQ in workflow, that also typically targets 4-bit quantization.
+*   **Pros:**
+    *   Often achieves high accuracy at 4-bit, sometimes surpassing GPTQ for specific tasks.
+    *   Can lead to inference speedups
+    *   Shorter calibration time than GPTQ.
+    *   There are many pre-quantized AWQ models on [Hugging Face Hub](https://huggingface.co/models?other=awq). This means you can often skip doing the quantization yourself and directly download a AWQ-quantized model.
+*   **Cons:**
+    *   Requires calibration if quantizing yourself.
+
+See the [AWQ documentation](./awq) for more details.
+
+### 4. torchao
+
+*   **Description:** Library from PyTorch focused on architecture optimization. It's designed for strong integration with `torch.compile`.
+*   **Pros:**
+    *   Strong integration with `torch.compile`, potentially leading to significant inference speedups when compiled.
+    *   Offers decent CPU quantization support.
+    *   Flexibility in quantization schemes (int8, int4, fp8).
+*   **Cons:**
+    *   Newer compared to BnB or GPTQ, so the ecosystem and best practices are still evolving.
+    *   Performance gains often rely heavily on `torch.compile` working effectively for the model and hardware.
+    *   4-bit quantized model via torchao might not reach the same accuracy as GPTQ/AWQ’s 4-bit, because torchao might just do straightforward rounding.
+
+See the [torchao documentation](./torchao) for more details.
+
+### 5. HQQ (Half-Quadratic Quantization)
+
+*   **Description:** A fast, on-the-fly quantization method (like bitsandbytes) that doesn't require calibration data.
+*   **Pros:**
+    *   Fast quantization process and no calibration data required.
+    *   Multiple backends for fast inference.
+    *   Compatible with `torch.compile` for potential inference speedups.
+    *   It supports a wide range of bit depths (8, 4, 3, 2, 1-bit)
+*   **Cons:**
+    *   Accuracy at very low bit depths (<4-bit) can degrade significantly and requires careful evaluation for the specific task.
+    *   While quantization is fast, inference speed might not be faster than other methods unless using specific backends or `torch.compile`.
+See the [HQQ documentation](./hqq) for more details.
+
+### 6. compressed-tensors
+
+*   **Description:** Allows loading models saved in specific compressed formats (like FP8, INT8/4, potentially with sparsity).
+*   **Pros:** 
+    *   Supports flexible formats including FP8 and sparsity.
+*   **Cons:** 
+    *   Primarily for loading pre-quantized models, not performing the quantization within Transformers itself.
+
+See the [compressed-tensors documentation](./compressed_tensors) for more details.
+
+## Fine-Tuning
+
+Quantization can also be used during fine-tuning to save memory, most commonly with PEFT's QLoRA.
+
+### bitsandbytes (4-bit NF4 for QLoRA)
+
+*   **Description:** The standard method for QLoRA fine-tuning via PEFT.
+*   **Pros:** Enables fine-tuning large models on consumer GPUs; widely supported and documented for PEFT.
+*   **Cons:** Primarily for NVIDIA GPUs.
+
+Other methods offer PEFT compatibility, though bitsandbytes is the most established and straightforward path for QLoRA.
+
+See the [bitsandbytes documentation](./bitsandbytes#qlora) and [PEFT Docs](https://huggingface.co/docs/peft/developer_guides/quantization#aqlm-quantization) for more details. 
+
+## Advanced & Research-Oriented Methods
+
+Methods like [AQLM](./aqlm), [SpQR](./spqr), [VPTQ](./vptq), [HIGGS](./higgs), etc., often push the boundaries of compression (e.g., < 2-bit) or explore novel techniques.
+
+*   Consider these if:
+    *   You need extreme compression (sub-4-bit).
+    *   You are conducting research or require state-of-the-art results shown in their respective papers.
+    *   You have significant compute resources available for potentially complex quantization procedures.
+*   Recommendation: Consult their specific documentation and associated papers carefully before choosing them for production use.
path: docs/source/en/quantization/selecting.md
user: login: stevhliu
body: ```suggestion
We recommend consulting each methods documentation and associated papers carefully before choosing one for use in production.
```
created_at: 2025-04-01T21:21:51Z
author_association: MEMBER
diff_hunk: @@ -0,0 +1,136 @@
+<!--Copyright 2024 The HuggingFace Team. All rights reserved.
+
+Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with
+the License. You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on
+an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the
+specific language governing permissions and limitations under the License.
+
+⚠️ Note that this file is in Markdown but contain specific syntax for our doc-builder (similar to MDX) that may not be
+rendered properly in your Markdown viewer.
+
+-->
+
+# Selecting a Quantization Method
+
+This guide helps you navigate the most common and production-ready quantization techniques available in Transformers and associated libraries. For a comprehensive overview of all supported methods and their features, refer back to the [Quantization Overview](./overview) table.
+
+## Inference
+
+**TLDR (Inference):**
+
+*   For ease of use and QLoRA fine-tuning on NVIDIA GPUs: **bitsandbytes**
+*   For good 4-bit accuracy with upfront calibration: **GPTQModel** or **AWQ**
+*   For speed via `torch.compile` and flexibility: **torchao**
+*   For fast on-the-fly quantization without calibration: **HQQ** is worth considering.
+*   For loading specific formats (FP8, Sparse) and nice quantized models: **compressed-tensors** 
+
+
+## Production-Ready & Widely Used Inference Methods:
+These methods are generally well-tested, widely adopted, and suitable for many production scenarios.
+
+### 1. bitsandbytes (`load_in_8bit` / `load_in_4bit`)
+
+*   **Description:** Enables 4-bit and 8-bit inference.
+*   **Pros:**
+    *   Very simple, no calibration dataset required for inference
+    *   Good community support and widely adopted.
+*   **Cons:**
+    *   Primarily optimized for NVIDIA GPUs (CUDA).  There is [ongoing work](https://github.com/bitsandbytes-foundation/bitsandbytes/blob/main/docs/source/installation.mdx#multi-backend) to support other hardware.
+    *   Inference speedup isn't guaranteed
+
+See the [bitsandbytes documentation](./bitsandbytes) for more details.
+
+### 2. GPTQ / GPTQModel
+
+*   **Description:** quantizes weights (commonly to 4-bit, but supports others) by processing calibration data. 
+*   **Pros:**
+    *   Often achieves high accuracy
+    *   Can lead to inference speedups
+    *   There are many pre-quantized GPTQ models on [Hugging Face Hub](https://huggingface.co/models?other=gptq). This means you can often skip doing the quantization yourself and directly download a GPTQ-quantized model.
+*   **Cons:**
+    *   Requires a calibration dataset and a separate calibration step before inference. This takes time and compute resources.
+    *   Accuracy depends on the quality and relevance of the calibration data (possible to overfit on calibration data).
+
+See the [GPTQ documentation](./gptq) for more details.
+
+### 3. AWQ (Activation-aware Weight Quantization)
+
+*   **Description:** Similar to GPTQ in workflow, that also typically targets 4-bit quantization.
+*   **Pros:**
+    *   Often achieves high accuracy at 4-bit, sometimes surpassing GPTQ for specific tasks.
+    *   Can lead to inference speedups
+    *   Shorter calibration time than GPTQ.
+    *   There are many pre-quantized AWQ models on [Hugging Face Hub](https://huggingface.co/models?other=awq). This means you can often skip doing the quantization yourself and directly download a AWQ-quantized model.
+*   **Cons:**
+    *   Requires calibration if quantizing yourself.
+
+See the [AWQ documentation](./awq) for more details.
+
+### 4. torchao
+
+*   **Description:** Library from PyTorch focused on architecture optimization. It's designed for strong integration with `torch.compile`.
+*   **Pros:**
+    *   Strong integration with `torch.compile`, potentially leading to significant inference speedups when compiled.
+    *   Offers decent CPU quantization support.
+    *   Flexibility in quantization schemes (int8, int4, fp8).
+*   **Cons:**
+    *   Newer compared to BnB or GPTQ, so the ecosystem and best practices are still evolving.
+    *   Performance gains often rely heavily on `torch.compile` working effectively for the model and hardware.
+    *   4-bit quantized model via torchao might not reach the same accuracy as GPTQ/AWQ’s 4-bit, because torchao might just do straightforward rounding.
+
+See the [torchao documentation](./torchao) for more details.
+
+### 5. HQQ (Half-Quadratic Quantization)
+
+*   **Description:** A fast, on-the-fly quantization method (like bitsandbytes) that doesn't require calibration data.
+*   **Pros:**
+    *   Fast quantization process and no calibration data required.
+    *   Multiple backends for fast inference.
+    *   Compatible with `torch.compile` for potential inference speedups.
+    *   It supports a wide range of bit depths (8, 4, 3, 2, 1-bit)
+*   **Cons:**
+    *   Accuracy at very low bit depths (<4-bit) can degrade significantly and requires careful evaluation for the specific task.
+    *   While quantization is fast, inference speed might not be faster than other methods unless using specific backends or `torch.compile`.
+See the [HQQ documentation](./hqq) for more details.
+
+### 6. compressed-tensors
+
+*   **Description:** Allows loading models saved in specific compressed formats (like FP8, INT8/4, potentially with sparsity).
+*   **Pros:** 
+    *   Supports flexible formats including FP8 and sparsity.
+*   **Cons:** 
+    *   Primarily for loading pre-quantized models, not performing the quantization within Transformers itself.
+
+See the [compressed-tensors documentation](./compressed_tensors) for more details.
+
+## Fine-Tuning
+
+Quantization can also be used during fine-tuning to save memory, most commonly with PEFT's QLoRA.
+
+### bitsandbytes (4-bit NF4 for QLoRA)
+
+*   **Description:** The standard method for QLoRA fine-tuning via PEFT.
+*   **Pros:** Enables fine-tuning large models on consumer GPUs; widely supported and documented for PEFT.
+*   **Cons:** Primarily for NVIDIA GPUs.
+
+Other methods offer PEFT compatibility, though bitsandbytes is the most established and straightforward path for QLoRA.
+
+See the [bitsandbytes documentation](./bitsandbytes#qlora) and [PEFT Docs](https://huggingface.co/docs/peft/developer_guides/quantization#aqlm-quantization) for more details. 
+
+## Advanced & Research-Oriented Methods
+
+Methods like [AQLM](./aqlm), [SpQR](./spqr), [VPTQ](./vptq), [HIGGS](./higgs), etc., often push the boundaries of compression (e.g., < 2-bit) or explore novel techniques.
+
+*   Consider these if:
+    *   You need extreme compression (sub-4-bit).
+    *   You are conducting research or require state-of-the-art results shown in their respective papers.
+    *   You have significant compute resources available for potentially complex quantization procedures.
+*   Recommendation: Consult their specific documentation and associated papers carefully before choosing them for production use.
+
+---
+
+Always benchmark the performance (accuracy and speed) of the quantized model on your specific task and hardware to ensure it meets your requirements. Refer to the individual documentation pages linked above for detailed usage instructions.
path: docs/source/en/quantization/selecting.md
user: login: stevhliu
body: ```suggestion

> [!TIP]
> Always benchmark the performance (accuracy and speed) of the quantized model on your specific task and hardware to ensure it meets your requirements. Refer to the individual documentation pages linked above for detailed usage instructions.
```
created_at: 2025-04-01T21:22:20Z
author_association: MEMBER
diff_hunk: @@ -0,0 +1,114 @@
+<!--Copyright 2024 The HuggingFace Team. All rights reserved.
+
+Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with
+the License. You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on
+an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the
+specific language governing permissions and limitations under the License.
+
+⚠️ Note that this file is in Markdown but contain specific syntax for our doc-builder (similar to MDX) that may not be
+rendered properly in your Markdown viewer.
+
+-->
+
+# Selecting a Quantization Method
+
+This guide helps you navigate the most common and production-ready quantization techniques available in Transformers and associated libraries. For a comprehensive overview of all supported methods and their features, refer back to the [Quantization Overview](./overview) table.
+
+## TLDR
+
+*   For ease of use and QLoRA fine-tuning on NVIDIA GPUs: **bitsandbytes**
+*   For good 4-bit accuracy with upfront calibration: **GPTQModel** or **AWQ**
path: docs/source/en/quantization/selecting.md
user: login: stevhliu
body: Let me see if I can find it!
created_at: 2025-04-01T21:32:30Z
author_association: MEMBER
diff_hunk: @@ -0,0 +1,136 @@
+<!--Copyright 2024 The HuggingFace Team. All rights reserved.
+
+Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with
+the License. You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on
+an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the
+specific language governing permissions and limitations under the License.
+
+⚠️ Note that this file is in Markdown but contain specific syntax for our doc-builder (similar to MDX) that may not be
+rendered properly in your Markdown viewer.
+
+-->
+
+# Selecting a Quantization Method
+
+This guide helps you navigate the most common and production-ready quantization techniques available in Transformers and associated libraries. For a comprehensive overview of all supported methods and their features, refer back to the [Quantization Overview](./overview) table.
+
+## Inference
+
+**TLDR (Inference):**
+
+*   For ease of use and QLoRA fine-tuning on NVIDIA GPUs: **bitsandbytes**
+*   For good 4-bit accuracy with upfront calibration: **GPTQModel** or **AWQ**
+*   For speed via `torch.compile` and flexibility: **torchao**
+*   For fast on-the-fly quantization without calibration: **HQQ** is worth considering.
+*   For loading specific formats (FP8, Sparse) and nice quantized models: **compressed-tensors** 
+
+
+## Production-Ready & Widely Used Inference Methods:
+These methods are generally well-tested, widely adopted, and suitable for many production scenarios.
+
+### 1. bitsandbytes (`load_in_8bit` / `load_in_4bit`)
path: docs/source/en/quantization/selecting.md
user: login: DerekLiu35
body: I'm not sure what the best way to this is. Maybe like this?
| Method                                   | Calibration Needed?    | Speed Focus                             | Hardware Focus                 | Notes / Considerations                                                                 |
|------------------------------------------|------------------------|-----------------------------------------|--------------------------------|----------------------------------------------------------------------------------------|
| [bitsandbytes](./bitsandbytes)           | No                     | Speedup not guaranteed                  | NVIDIA GPUs (primarily)        | Standard for QLoRA (see Fine-tuning). Multi-backend work ongoing.                      |
| [GPTQ](./gptq) / GPTQModel               | Yes                    | Optimized Kernels (ExLlama/Marlin)      | Broader (GPTQModel)            | Many pre-quantized models on [Hub](https://huggingface.co/models?other=gptq).          |
| [AWQ](./awq)                             | Yes                    | Optimized Kernels / Fused Modules       | Kernels: NVIDIA                | Many pre-quantized models on [Hub](https://huggingface.co/models?other=awq)            |
| [torchao](./torchao)                     | No                     | Speeds up from  `torch.compile`         | Broad                          | Newer library. int4wo in torchao might not reach the same accuracy as GPTQ/AWQ’s 4-bit |
| [HQQ](./hqq)                             | No                     | Needs backends or `torch.compile`       | Broad                          | Accuracy at very low bit depths (<4-bit) can degrade                                   |
| [compressed-tensors](./compressed_tensors)| N/A (Not quantizing)  | Depends on saved format/impl.           | Broad                          | Primarily for loading models quantized elsewhere, not quantizing within Transformers.  |
created_at: 2025-04-02T03:17:39Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,136 @@
+<!--Copyright 2024 The HuggingFace Team. All rights reserved.
+
+Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with
+the License. You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on
+an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the
+specific language governing permissions and limitations under the License.
+
+⚠️ Note that this file is in Markdown but contain specific syntax for our doc-builder (similar to MDX) that may not be
+rendered properly in your Markdown viewer.
+
+-->
+
+# Selecting a Quantization Method
+
+This guide helps you navigate the most common and production-ready quantization techniques available in Transformers and associated libraries. For a comprehensive overview of all supported methods and their features, refer back to the [Quantization Overview](./overview) table.
+
+## Inference
+
+**TLDR (Inference):**
+
+*   For ease of use and QLoRA fine-tuning on NVIDIA GPUs: **bitsandbytes**
+*   For good 4-bit accuracy with upfront calibration: **GPTQModel** or **AWQ**
+*   For speed via `torch.compile` and flexibility: **torchao**
+*   For fast on-the-fly quantization without calibration: **HQQ** is worth considering.
+*   For loading specific formats (FP8, Sparse) and nice quantized models: **compressed-tensors** 
+
+
+## Production-Ready & Widely Used Inference Methods:
+These methods are generally well-tested, widely adopted, and suitable for many production scenarios.
+
+### 1. bitsandbytes (`load_in_8bit` / `load_in_4bit`)
path: docs/source/en/quantization/selecting.md
user: login: stevhliu
body: Sorry I wasn't clear before, I was thinking more along the lines of this for each quantization method. For example, bitsandbytes would have:

| pros | cons |
|---|---|
| very simple, no calibration dataset required for inference | primarily optimized for NVIDIA GPUs (CUDA) |
| good community support and widely adopted | inference speedup isn't guaranteed |
created_at: 2025-04-02T15:44:38Z
author_association: MEMBER
diff_hunk: @@ -0,0 +1,109 @@
+<!--Copyright 2024 The HuggingFace Team. All rights reserved.
+
+Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with
+the License. You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on
+an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the
+specific language governing permissions and limitations under the License.
+
+⚠️ Note that this file is in Markdown but contain specific syntax for our doc-builder (similar to MDX) that may not be
+rendered properly in your Markdown viewer.
+
+-->
+
+# Selecting a quantization method
+
+There are many quantization methods available in Transformers for inference and fine-tuning. This guide helps you choose the most common and production-ready quantization techniques depending on your use case, and presents the advantages and disadvantages of each technique.
+
+For a comprehensive overview of all supported methods and their features, refer back to the table in the [Overview](./overview).
+
+## Inference
+
+Consider the quantization methods below for inference.
+
+| quantization method | use case |
+|---|---|
+| bitsandbytes | ease of use and QLoRA fine-tuning on NVIDIA GPUs |
+| compressed-tensors | loading specific quantized formats (FP8, Sparse) |
+| GPTQModel or AWQ | good 4-bit accuracy with upfront calibration |
+| HQQ | fast on the fly quantization without calibration |
+| torchao | flexibility and fast inference with torch.compile |
+
+### [bitsandbytes](./bitsandbytes.md)
path: docs/source/en/quantization/selecting.md
user: login: stevhliu
body: You can leave these headers just like this :)

```suggestion
### bitsandbytes
```
created_at: 2025-04-02T20:03:18Z
author_association: MEMBER
diff_hunk: @@ -0,0 +1,109 @@
+<!--Copyright 2024 The HuggingFace Team. All rights reserved.
+
+Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with
+the License. You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on
+an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the
+specific language governing permissions and limitations under the License.
+
+⚠️ Note that this file is in Markdown but contain specific syntax for our doc-builder (similar to MDX) that may not be
+rendered properly in your Markdown viewer.
+
+-->
+
+# Selecting a quantization method
+
+There are many quantization methods available in Transformers for inference and fine-tuning. This guide helps you choose the most common and production-ready quantization techniques depending on your use case, and presents the advantages and disadvantages of each technique.
+
+For a comprehensive overview of all supported methods and their features, refer back to the table in the [Overview](./overview).
+
+## Inference
+
+Consider the quantization methods below for inference.
+
+| quantization method | use case |
+|---|---|
+| bitsandbytes | ease of use and QLoRA fine-tuning on NVIDIA GPUs |
+| compressed-tensors | loading specific quantized formats (FP8, Sparse) |
+| GPTQModel or AWQ | good 4-bit accuracy with upfront calibration |
+| HQQ | fast on the fly quantization without calibration |
+| torchao | flexibility and fast inference with torch.compile |
+
+### [bitsandbytes](./bitsandbytes.md)
+
+| Pros                                                         | Cons                                                    |
+|--------------------------------------------------------------|---------------------------------------------------------|
+| Very simple, no calibration dataset required for inference.  | Primarily optimized for NVIDIA GPUs (CUDA).             |
+| Good community support and widely adopted.                   | Inference speedup isn't guaranteed.                     |
+
+### [GPTQ / GPTQModel](./gptq)
path: docs/source/en/quantization/selecting.md
user: login: stevhliu
body: ```suggestion
### GPTQ/GPTQModel
```
created_at: 2025-04-02T20:03:28Z
author_association: MEMBER
diff_hunk: @@ -0,0 +1,109 @@
+<!--Copyright 2024 The HuggingFace Team. All rights reserved.
+
+Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with
+the License. You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on
+an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the
+specific language governing permissions and limitations under the License.
+
+⚠️ Note that this file is in Markdown but contain specific syntax for our doc-builder (similar to MDX) that may not be
+rendered properly in your Markdown viewer.
+
+-->
+
+# Selecting a quantization method
+
+There are many quantization methods available in Transformers for inference and fine-tuning. This guide helps you choose the most common and production-ready quantization techniques depending on your use case, and presents the advantages and disadvantages of each technique.
+
+For a comprehensive overview of all supported methods and their features, refer back to the table in the [Overview](./overview).
+
+## Inference
+
+Consider the quantization methods below for inference.
+
+| quantization method | use case |
+|---|---|
+| bitsandbytes | ease of use and QLoRA fine-tuning on NVIDIA GPUs |
+| compressed-tensors | loading specific quantized formats (FP8, Sparse) |
+| GPTQModel or AWQ | good 4-bit accuracy with upfront calibration |
+| HQQ | fast on the fly quantization without calibration |
+| torchao | flexibility and fast inference with torch.compile |
+
+### [bitsandbytes](./bitsandbytes.md)
+
+| Pros                                                         | Cons                                                    |
+|--------------------------------------------------------------|---------------------------------------------------------|
+| Very simple, no calibration dataset required for inference.  | Primarily optimized for NVIDIA GPUs (CUDA).             |
+| Good community support and widely adopted.                   | Inference speedup isn't guaranteed.                     |
+
+### [GPTQ / GPTQModel](./gptq)
+
+| Pros                                                                 | Cons                                                                 |
+|----------------------------------------------------------------------|----------------------------------------------------------------------|
+| Often achieves high accuracy.                                        | Requires a calibration dataset and a separate calibration step.      |
+| Can lead to inference speedups.                                      | Possible to overfit on calibration data.                             |
+| Many pre-quantized GPTQ models on [Hugging Face Hub](https://huggingface.co/models?other=gptq). |                                           |
+
+### [AWQ (Activation-aware Weight Quantization)](./awq)
path: docs/source/en/quantization/selecting.md
user: login: stevhliu
body: ```suggestion
### AWQ (Activation-aware Weight Quantization)
```
created_at: 2025-04-02T20:03:37Z
author_association: MEMBER
diff_hunk: @@ -0,0 +1,109 @@
+<!--Copyright 2024 The HuggingFace Team. All rights reserved.
+
+Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with
+the License. You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on
+an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the
+specific language governing permissions and limitations under the License.
+
+⚠️ Note that this file is in Markdown but contain specific syntax for our doc-builder (similar to MDX) that may not be
+rendered properly in your Markdown viewer.
+
+-->
+
+# Selecting a quantization method
+
+There are many quantization methods available in Transformers for inference and fine-tuning. This guide helps you choose the most common and production-ready quantization techniques depending on your use case, and presents the advantages and disadvantages of each technique.
+
+For a comprehensive overview of all supported methods and their features, refer back to the table in the [Overview](./overview).
+
+## Inference
+
+Consider the quantization methods below for inference.
+
+| quantization method | use case |
+|---|---|
+| bitsandbytes | ease of use and QLoRA fine-tuning on NVIDIA GPUs |
+| compressed-tensors | loading specific quantized formats (FP8, Sparse) |
+| GPTQModel or AWQ | good 4-bit accuracy with upfront calibration |
+| HQQ | fast on the fly quantization without calibration |
+| torchao | flexibility and fast inference with torch.compile |
+
+### [bitsandbytes](./bitsandbytes.md)
+
+| Pros                                                         | Cons                                                    |
+|--------------------------------------------------------------|---------------------------------------------------------|
+| Very simple, no calibration dataset required for inference.  | Primarily optimized for NVIDIA GPUs (CUDA).             |
+| Good community support and widely adopted.                   | Inference speedup isn't guaranteed.                     |
+
+### [GPTQ / GPTQModel](./gptq)
+
+| Pros                                                                 | Cons                                                                 |
+|----------------------------------------------------------------------|----------------------------------------------------------------------|
+| Often achieves high accuracy.                                        | Requires a calibration dataset and a separate calibration step.      |
+| Can lead to inference speedups.                                      | Possible to overfit on calibration data.                             |
+| Many pre-quantized GPTQ models on [Hugging Face Hub](https://huggingface.co/models?other=gptq). |                                           |
+
+### [AWQ (Activation-aware Weight Quantization)](./awq)
+
+| Pros                                                                 | Cons                                                |
+|----------------------------------------------------------------------|-----------------------------------------------------|
+| Often achieves high accuracy at 4-bit. (Sometimes surpasses GPTQ on specific tasks.) | Requires calibration if quantizing yourself.        |
+| Can lead to inference speedups.                                      |                                                     |
+| Shorter calibration time than GPTQ.                                  |                                                     |
+| Many pre-quantized AWQ models on [Hugging Face Hub](https://huggingface.co/models?other=awq). |                                                     |
+
+### [torchao](./torchao)
path: docs/source/en/quantization/selecting.md
user: login: stevhliu
body: ```suggestion
### torchao
```
created_at: 2025-04-02T20:03:44Z
author_association: MEMBER
diff_hunk: @@ -0,0 +1,109 @@
+<!--Copyright 2024 The HuggingFace Team. All rights reserved.
+
+Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with
+the License. You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on
+an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the
+specific language governing permissions and limitations under the License.
+
+⚠️ Note that this file is in Markdown but contain specific syntax for our doc-builder (similar to MDX) that may not be
+rendered properly in your Markdown viewer.
+
+-->
+
+# Selecting a quantization method
+
+There are many quantization methods available in Transformers for inference and fine-tuning. This guide helps you choose the most common and production-ready quantization techniques depending on your use case, and presents the advantages and disadvantages of each technique.
+
+For a comprehensive overview of all supported methods and their features, refer back to the table in the [Overview](./overview).
+
+## Inference
+
+Consider the quantization methods below for inference.
+
+| quantization method | use case |
+|---|---|
+| bitsandbytes | ease of use and QLoRA fine-tuning on NVIDIA GPUs |
+| compressed-tensors | loading specific quantized formats (FP8, Sparse) |
+| GPTQModel or AWQ | good 4-bit accuracy with upfront calibration |
+| HQQ | fast on the fly quantization without calibration |
+| torchao | flexibility and fast inference with torch.compile |
+
+### [bitsandbytes](./bitsandbytes.md)
+
+| Pros                                                         | Cons                                                    |
+|--------------------------------------------------------------|---------------------------------------------------------|
+| Very simple, no calibration dataset required for inference.  | Primarily optimized for NVIDIA GPUs (CUDA).             |
+| Good community support and widely adopted.                   | Inference speedup isn't guaranteed.                     |
+
+### [GPTQ / GPTQModel](./gptq)
+
+| Pros                                                                 | Cons                                                                 |
+|----------------------------------------------------------------------|----------------------------------------------------------------------|
+| Often achieves high accuracy.                                        | Requires a calibration dataset and a separate calibration step.      |
+| Can lead to inference speedups.                                      | Possible to overfit on calibration data.                             |
+| Many pre-quantized GPTQ models on [Hugging Face Hub](https://huggingface.co/models?other=gptq). |                                           |
+
+### [AWQ (Activation-aware Weight Quantization)](./awq)
+
+| Pros                                                                 | Cons                                                |
+|----------------------------------------------------------------------|-----------------------------------------------------|
+| Often achieves high accuracy at 4-bit. (Sometimes surpasses GPTQ on specific tasks.) | Requires calibration if quantizing yourself.        |
+| Can lead to inference speedups.                                      |                                                     |
+| Shorter calibration time than GPTQ.                                  |                                                     |
+| Many pre-quantized AWQ models on [Hugging Face Hub](https://huggingface.co/models?other=awq). |                                                     |
+
+### [torchao](./torchao)
+
+| Pros                                                                 | Cons                                                                 |
+|----------------------------------------------------------------------|----------------------------------------------------------------------|
+| Strong integration with `torch.compile` for potential speedups.     | Newer library, ecosystem still evolving.                             |
+| Offers decent CPU quantization support.                              | Performance depends on `torch.compile` working well.                 |
+| Flexibility in quantization schemes (int8, int4, fp8).           | 4-bit quantization may not match GPTQ/AWQ in accuracy.              |
+
+### [HQQ (Half-Quadratic Quantization)](./hqq)
path: docs/source/en/quantization/selecting.md
user: login: stevhliu
body: ```suggestion
### HQQ (Half-Quadratic Quantization)
```
created_at: 2025-04-02T20:03:53Z
author_association: MEMBER
diff_hunk: @@ -0,0 +1,109 @@
+<!--Copyright 2024 The HuggingFace Team. All rights reserved.
+
+Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with
+the License. You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on
+an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the
+specific language governing permissions and limitations under the License.
+
+⚠️ Note that this file is in Markdown but contain specific syntax for our doc-builder (similar to MDX) that may not be
+rendered properly in your Markdown viewer.
+
+-->
+
+# Selecting a quantization method
+
+There are many quantization methods available in Transformers for inference and fine-tuning. This guide helps you choose the most common and production-ready quantization techniques depending on your use case, and presents the advantages and disadvantages of each technique.
+
+For a comprehensive overview of all supported methods and their features, refer back to the table in the [Overview](./overview).
+
+## Inference
+
+Consider the quantization methods below for inference.
+
+| quantization method | use case |
+|---|---|
+| bitsandbytes | ease of use and QLoRA fine-tuning on NVIDIA GPUs |
+| compressed-tensors | loading specific quantized formats (FP8, Sparse) |
+| GPTQModel or AWQ | good 4-bit accuracy with upfront calibration |
+| HQQ | fast on the fly quantization without calibration |
+| torchao | flexibility and fast inference with torch.compile |
+
+### [bitsandbytes](./bitsandbytes.md)
+
+| Pros                                                         | Cons                                                    |
+|--------------------------------------------------------------|---------------------------------------------------------|
+| Very simple, no calibration dataset required for inference.  | Primarily optimized for NVIDIA GPUs (CUDA).             |
+| Good community support and widely adopted.                   | Inference speedup isn't guaranteed.                     |
+
+### [GPTQ / GPTQModel](./gptq)
+
+| Pros                                                                 | Cons                                                                 |
+|----------------------------------------------------------------------|----------------------------------------------------------------------|
+| Often achieves high accuracy.                                        | Requires a calibration dataset and a separate calibration step.      |
+| Can lead to inference speedups.                                      | Possible to overfit on calibration data.                             |
+| Many pre-quantized GPTQ models on [Hugging Face Hub](https://huggingface.co/models?other=gptq). |                                           |
+
+### [AWQ (Activation-aware Weight Quantization)](./awq)
+
+| Pros                                                                 | Cons                                                |
+|----------------------------------------------------------------------|-----------------------------------------------------|
+| Often achieves high accuracy at 4-bit. (Sometimes surpasses GPTQ on specific tasks.) | Requires calibration if quantizing yourself.        |
+| Can lead to inference speedups.                                      |                                                     |
+| Shorter calibration time than GPTQ.                                  |                                                     |
+| Many pre-quantized AWQ models on [Hugging Face Hub](https://huggingface.co/models?other=awq). |                                                     |
+
+### [torchao](./torchao)
+
+| Pros                                                                 | Cons                                                                 |
+|----------------------------------------------------------------------|----------------------------------------------------------------------|
+| Strong integration with `torch.compile` for potential speedups.     | Newer library, ecosystem still evolving.                             |
+| Offers decent CPU quantization support.                              | Performance depends on `torch.compile` working well.                 |
+| Flexibility in quantization schemes (int8, int4, fp8).           | 4-bit quantization may not match GPTQ/AWQ in accuracy.              |
+
+### [HQQ (Half-Quadratic Quantization)](./hqq)
+
+| Pros                                                                 | Cons                                                                       |
+|----------------------------------------------------------------------|----------------------------------------------------------------------------|
+| Fast quantization process, no calibration data needed.              | Accuracy can degrade significantly at bit depths <4-bit.                     |
+| Multiple backends for fast inference.                                | Inference speed may not match others unless using `torch.compile` or backends. |
+| Compatible with `torch.compile`.                                     |                                                                            |
+| Supports wide range of bit depths (8, 4, 3, 2, 1-bit).              |                                                                            |
+
+### [compressed-tensors](./compressed_tensors)
path: docs/source/en/quantization/selecting.md
user: login: stevhliu
body: ```suggestion
### compressed-tensors
```
created_at: 2025-04-02T20:04:19Z
author_association: MEMBER
diff_hunk: @@ -0,0 +1,131 @@
+<!--Copyright 2024 The HuggingFace Team. All rights reserved.
+
+Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with
+the License. You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on
+an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the
+specific language governing permissions and limitations under the License.
+
+⚠️ Note that this file is in Markdown but contain specific syntax for our doc-builder (similar to MDX) that may not be
+rendered properly in your Markdown viewer.
+
+-->
+
+# Selecting a quantization method
+
+There are many quantization methods available in Transformers for inference and fine-tuning. This guide helps you choose the most common and production-ready quantization techniques depending on your use case, and presents the advantages and disadvantages of each technique.
+
+For a comprehensive overview of all supported methods and their features, refer back to the table in the [Overview](./overview).
+
+## Inference
+
+Consider the quantization methods below for inference.
+
+| quantization method | use case |
+|---|---|
+| bitsandbytes | ease of use and QLoRA fine-tuning on NVIDIA GPUs |
+| compressed-tensors | loading specific quantized formats (FP8, Sparse) |
+| GPTQModel or AWQ | good 4-bit accuracy with upfront calibration |
+| HQQ | fast on the fly quantization without calibration |
+| torchao | flexibility and fast inference with torch.compile |
+
+### No Calibration Required (On-the-fly Quantization)
+
+These methods are generally easier to use as they don't need a separate calibration dataset or step.
+
+#### bitsandbytes
+
+| Pros                                                         | Cons                                                    |
+|--------------------------------------------------------------|---------------------------------------------------------|
+| Very simple, no calibration dataset required for inference.  | Primarily optimized for NVIDIA GPUs (CUDA).             |
+| Good community support and widely adopted.                   | Inference speedup isn't guaranteed.                     |
+
+See the [bitsandbytes documentation](./bitsandbytes) for more details.
+
+#### HQQ (Half-Quadratic Quantization)
+
+| Pros                                                                 | Cons                                                                       |
+|----------------------------------------------------------------------|----------------------------------------------------------------------------|
+| Fast quantization process, no calibration data needed.              | Accuracy can degrade significantly at bit depths <4-bit.                     |
+| Multiple backends for fast inference.                                | Inference speed may not match others unless using `torch.compile` or backends. |
+| Compatible with `torch.compile`.                                     |                                                                            |
+| Supports wide range of bit depths (8, 4, 3, 2, 1-bit).              |                                                                            |
+
+See the [HQQ documentation](./hqq) for more details.
+
+#### torchao
+
+| Pros                                                                 | Cons                                                                 |
+|----------------------------------------------------------------------|----------------------------------------------------------------------|
+| Strong integration with `torch.compile` for potential speedups.     | Newer library, ecosystem still evolving.                             |
+| Offers decent CPU quantization support.                              | Performance depends on `torch.compile` working well.                 |
+| Flexibility in quantization schemes (int8, int4, fp8).           | 4-bit quantization (int4wo) may not match GPTQ/AWQ in accuracy.              |
+
+See the [torchao documentation](./torchao) for more details.
+
+### Calibration-based Quantization
+
+These methods require an upfront calibration step using a dataset to potentially achieve higher accuracy.
+
+#### GPTQ/GPTQModel
+
path: docs/source/en/quantization/selecting.md
user: login: stevhliu
body: You can include the estimation time in a sentence here after the section header and before the table!
created_at: 2025-04-07T16:16:56Z
author_association: MEMBER

PR comments:
user: login: HuggingFaceDocBuilderDev
created_at: 2025-04-01T02:14:01Z
body: The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_37159). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
author_association: NONE
user: login: DerekLiu35
created_at: 2025-04-07T15:57:49Z
body: Sounds good, but not sure where's the best place to include estimate of the calibration time.
author_association: CONTRIBUTOR