Qat: int4: first layer precision for int4 model

username: willdla
cooked: <p>workflow for the qat now is:<br>
using the same precision in each fake_quant for EVERY LAYER.<br>
fp32 → fake_quant → fp32</p>
<p><strong>problem i meet:</strong><br>
<strong>1st</strong><br>
input data may be 8bit in most common cases.<br>
when qat int4 model, first layer fake_quant “8bit data into 4bit” (or we call cut the data spread).<br>
in this process we lost too much(<strong>precision drop</strong> happens in the input data) …<br>
<strong>IF</strong> we can treat <strong>first layer</strong> with <strong>8bit</strong> qconfig, and treat <strong>other layer</strong> with <strong>4bit</strong> qconfig.<br>
we can keep some more necessary input data.</p>
<p><strong>2nd</strong><br>
is there any doc to use 2 or more qconfig in the same qat process.</p>
<p><strong>3rd</strong><br>
ive notice <a href="https://github.com/pytorch/pytorch/pull/67177" class="inline-onebox" rel="noopener nofollow ugc">Add MKLDNN quantization backend by Xia-Weiwen · Pull Request #67177 · pytorch/pytorch · GitHub</a> mkldnn<br>
but its a bit different, mkldnn just change internal compute logic,<br>
i just wanna to add a new backend.<br>
i find some reference here: <a href="https://github.com/pytorch/pytorch/wiki/Extending-PyTorch-Quantization-to-Custom-Backends" class="inline-onebox" rel="noopener nofollow ugc">Extending PyTorch Quantization to Custom Backends · pytorch/pytorch Wiki · GitHub</a><br>
Is there <strong>any suggestion</strong> on develop a new backend, especially <strong>for qat</strong>.</p>
<p><strong>Purpose for me:</strong></p>
<ol>
<li>im working to develop a new int4 qat qconfig(or we say a new int4 backend) for a specific dla,<br>
in my opinion using 4bit in all layers may cause precision drop, especially for the first layer.<br>
Im try to deal with the problem in the first layer to keep as more information as possible to prevent precision drop.</li>
<li>also, im try to find some use case/demo on how to use hybrid quant schemem, for example using 8bit qconfig and fp16 qconfig in the same qat process. any user interface.</li>
<li>im searching for some hybrid quant qat demo, do u have some?</li>
<li>any suggestion on develop a new int4 qat qconfig (or we say a new int4 backend).</li>
</ol>

username: jerryzh168
cooked: <p>Yeah, I would recommend using FX Graph Mode Quantization for this. We have post training quantization tutorial here: <a href="https://pytorch.org/tutorials/prototype/fx_graph_mode_ptq_static.html?highlight=quantization" class="inline-onebox" rel="noopener nofollow ugc">(prototype) FX Graph Mode Post Training Static Quantization — PyTorch Tutorials 1.10.0+cu102 documentation</a> (we might add a QAT tutorial later). You can use <code>prepare_qat_fx</code> and use the <code>qconfig_dict</code> api to do this.</p>
<p>We do have a <code>quint4x2</code> dtype currently: <a href="https://github.com/pytorch/pytorch/blob/master/test/quantization/fx/test_quantize_fx.py#L5068" class="inline-onebox" rel="noopener nofollow ugc">pytorch/test_quantize_fx.py at master · pytorch/pytorch · GitHub</a>, although I think this is mostly for weight. To support this with activation, I think you need:</p>
<ol>
<li>Add support for <code>quint4x2</code> in quantize_per_tensor <a href="https://github.com/pytorch/pytorch/blob/master/aten/src/ATen/native/quantized/QTensor.cpp#L14" class="inline-onebox" rel="noopener nofollow ugc">pytorch/QTensor.cpp at master · pytorch/pytorch · GitHub</a>
</li>
<li>Use the is_reference option during convert_fx, which will produce a model with (dequant - float_op - quant) patterns representing the model (you can take a look at <a href="https://github.com/pytorch/pytorch/wiki/Extending-PyTorch-Quantization-to-Custom-Backends" class="inline-onebox" rel="noopener nofollow ugc">Extending PyTorch Quantization to Custom Backends · pytorch/pytorch Wiki · GitHub</a> for reasons)</li>
<li>lower the model to the dla you are building, this can be through fx/torchscript or any ways you prefer</li>
</ol>

username: willdla
cooked: <p>Is there any guide for hybrid-quant for qat, mix int8 and int4 when training.</p>
<p>It would be better for use int8 in first and last layer, and use int4 in the inner layer.</p>
<p>first layer with int8 may prevent source data to be losted.<br>
last layer with int8 may help some other process after inference (like video output, other accelerator).</p>