RFC: Add quantization capability to the Transformers Trainer API
Author: ftian1
Pull request: no

### Feature request

Add quantization capability to the Transformers Trainer API

### Motivation

Quantization is one of popular model compression technologies and widely used in the industry. At Intel(R) Xeon platforms and Nvidia GPU platforms it could bring significant performance speedup with slight accuray loss. Having a easy-of-use quantization interface in Transfomers will benifit customers.

Comparing with vanilla PyTorch quantization feature, the proposed interface supports accuracy-aware tuning capability to solve common accuracy loss issue when applying quantization technology.

**Design**

The proposed interface is like below:

```
  class AccuracyAwareTuningConf:
      def __init__(self, accuracy_criterion='relative', accuracy_loss=0.01, metric_name='F1', timeout=0):
          # The tuning configuration used to define accuracy goal.
          # Args:
          #     accuracy_criterion: String. relative loss or absolute loss.
          #     accuracy_loss: Float. The tolerated accuracy loss value.
          #     metric_name: String. The metric user cares about.
          #     timeout: Integer. 0 means early stop. non-zero means returns within defined time scope. unit is minute.

  class Trainer:
    ...

    def quantize(self, model=None, approach='static', calib_dataset=None, tuning_config=None)
        # The interface used to quantize model
        #
        # Args:
        #     model: Optional. if None, the model in Trainer initialization will be used.
        #     approach: String. "auto", "static" and "dynamic" are three supported quantization approaches.
        #     calib_dataset: Optional. if None, the train_dataset in Trainer initialization will be used.
        #     tuning_config: Optional. if none, it means doing quantization without tuning.

```

Note this interface is device agnostic. user could specify the device running calibration and quantization by training_args.device.

**Use Case**

Take Transformers Text-Classification task as an example, user could add below code to quantize model.

***Quantization without tuning***

```
    # Initialize our Trainer [original code]
    trainer = Trainer(
        model=model,
        args=training_args,
        train_dataset=train_dataset if training_args.do_train else None,
        eval_dataset=eval_dataset if training_args.do_eval else None,
        compute_metrics=compute_metrics,
        tokenizer=tokenizer,
        data_collator=data_collator,
    )
    
    # one line code change to do quantization without accuracy-aware tuning
    q_model = trainer.quantize()

```

***Quantization with tuning***

```
    # Initialize our Trainer [original code]
    trainer = Trainer(
        model=model,
        args=training_args,
        train_dataset=train_dataset if training_args.do_train else None,
        eval_dataset=eval_dataset if training_args.do_eval else None,
        compute_metrics=compute_metrics,
        tokenizer=tokenizer,
        data_collator=data_collator,
    )
    
    # two line code changes to do quantization with accuracy-aware tuning
    conf = AccuracyAwareTuningConf(accuracy_loss=0.005, metric='F1')
    q_model = trainer.quantize(tuning_conf=conf)
    
```

**Performance**

below are the accuracy and performance data of some quantized NLP models tuned on Intel(R) Xeon(R) Platinum 8380H CPU @ 2.90GHz with 4 cores per instance, batch size 1. 

<table class="tg">
<thead>
  <tr>
    <th class="tg-9wq8" rowspan="3">&nbsp;&nbsp;&nbsp;<br><span style="font-weight:bold;font-style:normal;color:#24292F">Model</span>&nbsp;&nbsp;&nbsp;</th>
    <th class="tg-9wq8" colspan="3" rowspan="2">&nbsp;&nbsp;&nbsp;<br><span style="font-weight:bold;font-style:normal;color:#24292F">Accuracy</span>&nbsp;&nbsp;&nbsp;</th>
    <th class="tg-9wq8" colspan="3" rowspan="2">   <br>     <span style="font-weight:bold;color:#24292F">Throughput (samples/sec)</span>   </th>
  </tr>
  <tr>
  </tr>
  <tr>
    <th class="tg-9wq8">&nbsp;&nbsp;&nbsp;<br><span style="font-weight:bold;font-style:normal;color:#24292F">INT8</span>&nbsp;&nbsp;&nbsp;</th>
    <th class="tg-9wq8">&nbsp;&nbsp;&nbsp;<br><span style="font-weight:bold;font-style:normal;color:#24292F">FP32</span>&nbsp;&nbsp;&nbsp;</th>
    <th class="tg-9wq8">&nbsp;&nbsp;&nbsp;<br><span style="font-weight:bold;font-style:normal;color:#24292F">Acc Ratio[(INT8-FP32)/FP32]</span>&nbsp;&nbsp;&nbsp;</th>
    <th class="tg-9wq8">&nbsp;&nbsp;&nbsp;<br><span style="font-weight:bold;font-style:normal;color:#24292F">INT8</span>&nbsp;&nbsp;&nbsp;</th>
    <th class="tg-9wq8">&nbsp;&nbsp;&nbsp;<br><span style="font-weight:bold;font-style:normal;color:#24292F">FP32</span>&nbsp;&nbsp;&nbsp;</th>
    <th class="tg-9wq8">&nbsp;&nbsp;&nbsp;<br><span style="font-weight:bold;font-style:normal;color:#24292F">Performance Ratio[INT8/FP32]</span>&nbsp;&nbsp;&nbsp;</th>
  </tr>
</thead>
<tbody>
  <tr>
    <td class="tg-lboi">&nbsp;&nbsp;&nbsp;<br><span style="font-weight:normal;font-style:normal;color:#24292F">Barthez MRPC</span>&nbsp;&nbsp;&nbsp;</td>
    <td class="tg-lboi">&nbsp;&nbsp;&nbsp;<br><span style="font-weight:normal;font-style:normal;color:#24292F">83.92%</span>&nbsp;&nbsp;&nbsp;</td>
    <td class="tg-lboi">&nbsp;&nbsp;&nbsp;<br><span style="font-weight:normal;font-style:normal;color:#24292F">83.81%</span>&nbsp;&nbsp;&nbsp;</td>
    <td class="tg-lboi">&nbsp;&nbsp;&nbsp;<br><span style="font-weight:normal;font-style:normal;color:#24292F">0.14%</span>&nbsp;&nbsp;&nbsp;</td>
    <td class="tg-lboi">&nbsp;&nbsp;&nbsp;<br><span style="font-weight:normal;font-style:normal;color:#24292F">161.06</span>&nbsp;&nbsp;&nbsp;</td>
    <td class="tg-lboi">&nbsp;&nbsp;&nbsp;<br><span style="font-weight:normal;font-style:normal;color:#24292F">89.61</span>&nbsp;&nbsp;&nbsp;</td>
    <td class="tg-lboi">&nbsp;&nbsp;&nbsp;<br><span style="font-weight:normal;font-style:normal;color:#24292F">1.80x</span>&nbsp;&nbsp;&nbsp;</td>
  </tr>
  <tr>
    <td class="tg-lboi">&nbsp;&nbsp;&nbsp;<br><span style="font-weight:normal;font-style:normal;color:#24292F">BERT base MRPC</span>&nbsp;&nbsp;&nbsp;</td>
    <td class="tg-lboi">&nbsp;&nbsp;&nbsp;<br><span style="font-weight:normal;font-style:normal;color:#24292F">89.90%</span>&nbsp;&nbsp;&nbsp;</td>
    <td class="tg-lboi">&nbsp;&nbsp;&nbsp;<br><span style="font-weight:normal;font-style:normal;color:#24292F">90.69%</span>&nbsp;&nbsp;&nbsp;</td>
    <td class="tg-lboi">&nbsp;&nbsp;&nbsp;<br><span style="font-weight:normal;font-style:normal;color:#24292F">-0.88%</span>&nbsp;&nbsp;&nbsp;</td>
    <td class="tg-lboi">&nbsp;&nbsp;&nbsp;<br><span style="font-weight:normal;font-style:normal;color:#24292F">244.27</span>&nbsp;&nbsp;&nbsp;</td>
    <td class="tg-lboi">&nbsp;&nbsp;&nbsp;<br><span style="font-weight:normal;font-style:normal;color:#24292F">125.28</span>&nbsp;&nbsp;&nbsp;</td>
    <td class="tg-lboi">&nbsp;&nbsp;&nbsp;<br><span style="font-weight:normal;font-style:normal;color:#24292F">1.95x</span>&nbsp;&nbsp;&nbsp;</td>
  </tr>
  <tr>
    <td class="tg-lboi">&nbsp;&nbsp;&nbsp;<br><span style="font-weight:normal;font-style:normal;color:#24292F">BERT base RTE</span>&nbsp;&nbsp;&nbsp;</td>
    <td class="tg-lboi">&nbsp;&nbsp;&nbsp;<br><span style="font-weight:normal;font-style:normal;color:#24292F">69.31%</span>&nbsp;&nbsp;&nbsp;</td>
    <td class="tg-lboi">&nbsp;&nbsp;&nbsp;<br><span style="font-weight:normal;font-style:normal;color:#24292F">69.68%</span>&nbsp;&nbsp;&nbsp;</td>
    <td class="tg-lboi">&nbsp;&nbsp;&nbsp;<br><span style="font-weight:normal;font-style:normal;color:#24292F">-0.52%</span>&nbsp;&nbsp;&nbsp;</td>
    <td class="tg-lboi">&nbsp;&nbsp;&nbsp;<br><span style="font-weight:normal;font-style:normal;color:#24292F">259.21</span>&nbsp;&nbsp;&nbsp;</td>
    <td class="tg-lboi">&nbsp;&nbsp;&nbsp;<br><span style="font-weight:normal;font-style:normal;color:#24292F">125.72</span>&nbsp;&nbsp;&nbsp;</td>
    <td class="tg-lboi">&nbsp;&nbsp;&nbsp;<br><span style="font-weight:normal;font-style:normal;color:#24292F">2.06x</span>&nbsp;&nbsp;&nbsp;</td>
  </tr>
  <tr>
    <td class="tg-lboi">&nbsp;&nbsp;&nbsp;<br><span style="font-weight:normal;font-style:normal;color:#24292F">BERT base SST2</span>&nbsp;&nbsp;&nbsp;</td>
    <td class="tg-lboi">&nbsp;&nbsp;&nbsp;<br><span style="font-weight:normal;font-style:normal;color:#24292F">91.06%</span>&nbsp;&nbsp;&nbsp;</td>
    <td class="tg-lboi">&nbsp;&nbsp;&nbsp;<br><span style="font-weight:normal;font-style:normal;color:#24292F">91.86%</span>&nbsp;&nbsp;&nbsp;</td>
    <td class="tg-lboi">&nbsp;&nbsp;&nbsp;<br><span style="font-weight:normal;font-style:normal;color:#24292F">-0.87%</span>&nbsp;&nbsp;&nbsp;</td>
    <td class="tg-lboi">&nbsp;&nbsp;&nbsp;<br><span style="font-weight:normal;font-style:normal;color:#24292F">262.73</span>&nbsp;&nbsp;&nbsp;</td>
    <td class="tg-lboi">&nbsp;&nbsp;&nbsp;<br><span style="font-weight:normal;font-style:normal;color:#24292F">125.69</span>&nbsp;&nbsp;&nbsp;</td>
    <td class="tg-lboi">&nbsp;&nbsp;&nbsp;<br><span style="font-weight:normal;font-style:normal;color:#24292F">2.09x</span>&nbsp;&nbsp;&nbsp;</td>
  </tr>
  <tr>
    <td class="tg-lboi">&nbsp;&nbsp;&nbsp;<br><span style="font-weight:normal;font-style:normal;color:#24292F">BERT large MRPC</span>&nbsp;&nbsp;&nbsp;</td>
    <td class="tg-lboi">&nbsp;&nbsp;&nbsp;<br><span style="font-weight:normal;font-style:normal;color:#24292F">89.50%</span>&nbsp;&nbsp;&nbsp;</td>
    <td class="tg-lboi">&nbsp;&nbsp;&nbsp;<br><span style="font-weight:normal;font-style:normal;color:#24292F">90.38%</span>&nbsp;&nbsp;&nbsp;</td>
    <td class="tg-lboi">&nbsp;&nbsp;&nbsp;<br><span style="font-weight:normal;font-style:normal;color:#24292F">-0.97%</span>&nbsp;&nbsp;&nbsp;</td>
    <td class="tg-lboi">&nbsp;&nbsp;&nbsp;<br><span style="font-weight:normal;font-style:normal;color:#24292F">88.92</span>&nbsp;&nbsp;&nbsp;</td>
    <td class="tg-lboi">&nbsp;&nbsp;&nbsp;<br><span style="font-weight:normal;font-style:normal;color:#24292F">36.55</span>&nbsp;&nbsp;&nbsp;</td>
    <td class="tg-lboi">&nbsp;&nbsp;&nbsp;<br><span style="font-weight:normal;font-style:normal;color:#24292F">2.43x</span>&nbsp;&nbsp;&nbsp;</td>
  </tr>
  <tr>
    <td class="tg-lboi">&nbsp;&nbsp;&nbsp;<br><span style="font-weight:normal;font-style:normal;color:#24292F">CamemBERT base MRPC</span>&nbsp;&nbsp;&nbsp;</td>
    <td class="tg-lboi">&nbsp;&nbsp;&nbsp;<br><span style="font-weight:normal;font-style:normal;color:#24292F">86.70%</span>&nbsp;&nbsp;&nbsp;</td>
    <td class="tg-lboi">&nbsp;&nbsp;&nbsp;<br><span style="font-weight:normal;font-style:normal;color:#24292F">86.82%</span>&nbsp;&nbsp;&nbsp;</td>
    <td class="tg-lboi">&nbsp;&nbsp;&nbsp;<br><span style="font-weight:normal;font-style:normal;color:#24292F">-0.14%</span>&nbsp;&nbsp;&nbsp;</td>
    <td class="tg-lboi">&nbsp;&nbsp;&nbsp;<br><span style="font-weight:normal;font-style:normal;color:#24292F">236.6</span>&nbsp;&nbsp;&nbsp;</td>
    <td class="tg-lboi">&nbsp;&nbsp;&nbsp;<br><span style="font-weight:normal;font-style:normal;color:#24292F">121.81</span>&nbsp;&nbsp;&nbsp;</td>
    <td class="tg-lboi">&nbsp;&nbsp;&nbsp;<br><span style="font-weight:normal;font-style:normal;color:#24292F">1.94x</span>&nbsp;&nbsp;&nbsp;</td>
  </tr>
  <tr>
    <td class="tg-lboi">&nbsp;&nbsp;&nbsp;<br><span style="font-weight:normal;font-style:normal;color:#24292F">Deberta MRPC</span>&nbsp;&nbsp;&nbsp;</td>
    <td class="tg-lboi">&nbsp;&nbsp;&nbsp;<br><span style="font-weight:normal;font-style:normal;color:#24292F">90.88%</span>&nbsp;&nbsp;&nbsp;</td>
    <td class="tg-lboi">&nbsp;&nbsp;&nbsp;<br><span style="font-weight:normal;font-style:normal;color:#24292F">90.91%</span>&nbsp;&nbsp;&nbsp;</td>
    <td class="tg-lboi">&nbsp;&nbsp;&nbsp;<br><span style="font-weight:normal;font-style:normal;color:#24292F">-0.04%</span>&nbsp;&nbsp;&nbsp;</td>
    <td class="tg-lboi">&nbsp;&nbsp;&nbsp;<br><span style="font-weight:normal;font-style:normal;color:#24292F">149.76</span>&nbsp;&nbsp;&nbsp;</td>
    <td class="tg-lboi">&nbsp;&nbsp;&nbsp;<br><span style="font-weight:normal;font-style:normal;color:#24292F">84.72</span>&nbsp;&nbsp;&nbsp;</td>
    <td class="tg-lboi">&nbsp;&nbsp;&nbsp;<br><span style="font-weight:normal;font-style:normal;color:#24292F">1.77x</span>&nbsp;&nbsp;&nbsp;</td>
  </tr>
  <tr>
    <td class="tg-lboi">&nbsp;&nbsp;&nbsp;<br><span style="font-weight:normal;font-style:normal;color:#24292F">DistilBERT base MRPC</span>&nbsp;&nbsp;&nbsp;</td>
    <td class="tg-lboi">&nbsp;&nbsp;&nbsp;<br><span style="font-weight:normal;font-style:normal;color:#24292F">88.23%</span>&nbsp;&nbsp;&nbsp;</td>
    <td class="tg-lboi">&nbsp;&nbsp;&nbsp;<br><span style="font-weight:normal;font-style:normal;color:#24292F">89.16%</span>&nbsp;&nbsp;&nbsp;</td>
    <td class="tg-lboi">&nbsp;&nbsp;&nbsp;<br><span style="font-weight:normal;font-style:normal;color:#24292F">-1.05%</span>&nbsp;&nbsp;&nbsp;</td>
    <td class="tg-lboi">&nbsp;&nbsp;&nbsp;<br><span style="font-weight:normal;font-style:normal;color:#24292F">426.4</span>&nbsp;&nbsp;&nbsp;</td>
    <td class="tg-lboi">&nbsp;&nbsp;&nbsp;<br><span style="font-weight:normal;font-style:normal;color:#24292F">246.13</span>&nbsp;&nbsp;&nbsp;</td>
    <td class="tg-lboi">&nbsp;&nbsp;&nbsp;<br><span style="font-weight:normal;font-style:normal;color:#24292F">1.73x</span>&nbsp;&nbsp;&nbsp;</td>
  </tr>
  <tr>
    <td class="tg-lboi">&nbsp;&nbsp;&nbsp;<br><span style="font-weight:normal;font-style:normal;color:#24292F">mBart WNLI</span>&nbsp;&nbsp;&nbsp;</td>
    <td class="tg-lboi">&nbsp;&nbsp;&nbsp;<br><span style="font-weight:normal;font-style:normal;color:#24292F">56.34%</span>&nbsp;&nbsp;&nbsp;</td>
    <td class="tg-lboi">&nbsp;&nbsp;&nbsp;<br><span style="font-weight:normal;font-style:normal;color:#24292F">56.34%</span>&nbsp;&nbsp;&nbsp;</td>
    <td class="tg-lboi">&nbsp;&nbsp;&nbsp;<br><span style="font-weight:normal;font-style:normal;color:#24292F">0.00%</span>&nbsp;&nbsp;&nbsp;</td>
    <td class="tg-lboi">&nbsp;&nbsp;&nbsp;<br><span style="font-weight:normal;font-style:normal;color:#24292F">66.23</span>&nbsp;&nbsp;&nbsp;</td>
    <td class="tg-lboi">&nbsp;&nbsp;&nbsp;<br><span style="font-weight:normal;font-style:normal;color:#24292F">30.86</span>&nbsp;&nbsp;&nbsp;</td>
    <td class="tg-lboi">&nbsp;&nbsp;&nbsp;<br><span style="font-weight:normal;font-style:normal;color:#24292F">2.15x</span>&nbsp;&nbsp;&nbsp;</td>
  </tr>
  <tr>
    <td class="tg-lboi">&nbsp;&nbsp;&nbsp;<br><span style="font-weight:normal;font-style:normal;color:#24292F">lvwerra/pegasus-samsum</span>&nbsp;&nbsp;&nbsp;</td>
    <td class="tg-lboi">&nbsp;&nbsp;&nbsp;<br><span style="font-weight:normal;font-style:normal;color:#24292F">42.39</span>&nbsp;&nbsp;&nbsp;</td>
    <td class="tg-lboi">&nbsp;&nbsp;&nbsp;<br><span style="font-weight:normal;font-style:normal;color:#24292F">42.67</span>&nbsp;&nbsp;&nbsp;</td>
    <td class="tg-lboi">&nbsp;&nbsp;&nbsp;<br><span style="font-weight:normal;font-style:normal;color:#24292F">-0.67%</span>&nbsp;&nbsp;&nbsp;</td>
    <td class="tg-lboi">&nbsp;&nbsp;&nbsp;<br><span style="font-weight:normal;font-style:normal;color:#24292F">3.86</span>&nbsp;&nbsp;&nbsp;</td>
    <td class="tg-lboi">&nbsp;&nbsp;&nbsp;<br><span style="font-weight:normal;font-style:normal;color:#24292F">1.14</span>&nbsp;&nbsp;&nbsp;</td>
    <td class="tg-lboi">&nbsp;&nbsp;&nbsp;<br><span style="font-weight:normal;font-style:normal;color:#24292F">3.38x</span>&nbsp;&nbsp;&nbsp;</td>
  </tr>
  <tr>
    <td class="tg-lboi">&nbsp;&nbsp;&nbsp;<br><span style="font-weight:normal;font-style:normal;color:#24292F">Roberta Base MRPC</span>&nbsp;&nbsp;&nbsp;</td>
    <td class="tg-lboi">&nbsp;&nbsp;&nbsp;<br><span style="font-weight:normal;font-style:normal;color:#24292F">88.25%</span>&nbsp;&nbsp;&nbsp;</td>
    <td class="tg-lboi">&nbsp;&nbsp;&nbsp;<br><span style="font-weight:normal;font-style:normal;color:#24292F">88.18%</span>&nbsp;&nbsp;&nbsp;</td>
    <td class="tg-lboi">&nbsp;&nbsp;&nbsp;<br><span style="font-weight:normal;font-style:normal;color:#24292F">0.08%</span>&nbsp;&nbsp;&nbsp;</td>
    <td class="tg-lboi">&nbsp;&nbsp;&nbsp;<br><span style="font-weight:normal;font-style:normal;color:#24292F">245.05</span>&nbsp;&nbsp;&nbsp;</td>
    <td class="tg-lboi">&nbsp;&nbsp;&nbsp;<br><span style="font-weight:normal;font-style:normal;color:#24292F">123.53</span>&nbsp;&nbsp;&nbsp;</td>
    <td class="tg-lboi">&nbsp;&nbsp;&nbsp;<br><span style="font-weight:normal;font-style:normal;color:#24292F">1.98x</span>&nbsp;&nbsp;&nbsp;</td>
  </tr>
</tbody>
</table>

### Your contribution

We are glad to contribute PR to HF community if this idea gets approved.

We would love to hear feedback from HF maintainer about this proposal.

Comments:
cc @sgugger 

This looks very exciting! The API suggested makes sense to me, I'd need to see the whole code to comment more :-)
By all means, please open a PR and tag me for review!

@ftian1 [Optimum](https://github.com/huggingface/optimum) has a quantization API which is device- and backend-agnostic. It also offers other kinds of optimization for accelerating inference so I think it makes more sense to keep all this in the same place. Do not hesitate to open a PR or create an issue there with your suggestions and we will be happy to discuss it!

@regisss @sgugger  thanks for the comments. why we think it's valuable to contribute to transformers is because it would bring better user experience with fewer line code changes comparing with Optimum.

we can have a PR at first and then do further discussion with your guidance. thanks

@ftian1 While I agree with you that it would provide a better UX for some users, there are a couple of points that make me think about it twice:

- I do not think that such a quantization API should be tailored for accuracy-aware quantization only or for a specific backend. Users should be able to use ONNX Runtime, Torch FX, Intel Neural Compressor or any other available backend. The ways these backends work and are configured are different from each other, so the API should be able to manage this. Optimum enables to do it so a possible solution would be to have a wrapper around Optimum's quantization API.
- I believe that having different places in the Hugging Face ecosystem where users can perform quantization will create quite a lot of confusion and will draw them away from other cool Optimum's optimization features, making it more difficult to deploy fast optimized models.

@regisss  thanks for the comments. I will invite Optimum owner @echarlaix to review this RFC and see what's her inputs.

Hi @ftian1,

We already support INC quantization aware training (as well as static and dynamic quantization) in `optimum` so it would be redundant to add this feature to `transformers` in my opinion and could create some confusion on which library to use to perform optimization as @regisss mentionned. As we already discussed, I think it makes sense to keep everything related to `neural-compressor` in `optimum` and to increase promotion around it. Also happy to discuss any modifications you would like us to apply on the `IncTrainer` ! (especially given our plan to refactorize the different `IncQuantizer` and `IncTrainer` classes after `neural-compressor` next big release)

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.