Starting out with QAT

MrOCW: <p>Hi all, I’m fairly new to model optimization and I’ve tried ONNX PTQ methods. However, I am required to explore QAT for YOLO pytorch models and I’m not sure what to start with.</p>
<p>Should I use Eager Mode or FX Graph Mode Quantization?<br>
Which of them is easier and more general to different models?</p>
<p>Thanks in advance!</p>

MrOCW: <p>What is the difference between using <code>torch.quantization.quantize_qat</code> and following the tutorials available? i.e. fusing modules, adding quant and dequant etc</p>

Vasiliy_Kuznetsov: <p>Hi <a class="mention" href="/u/mrocw">@MrOCW</a> , eager mode quantization is manual as in you would have to change the modeling code do add quants/dequants and specify fusions.  FX graph mode quantization is automatic but it requires the model to be symbolically traceable.</p>
<p>Usually for new models I’d recommend trying FX graph mode quantization first.  If the model is not symbolically traceable, then you would have to either make it symbolically traceable (difficulty depends on the model), or use eager mode.</p>
<p>I remember looking at yolov3 a few months ago and that model was challenging to make symbolically traceable.</p>

MrOCW: <p>Guess I’ll work with eager for now since it seems to be more general? just that more work has to be done. However, I am not too sure about where to add the Quant and DeQuant modules? Currently working with a model that has Conv2d-BatchNorm2d-SiLU and since SiLU is not supported… do i fuse conv+bn then add a quant and dequant for every other instance of such module? for e.g. Quant → ConvBn → DeQuant → SiLU  ===&gt;   Quant → ConvBn → DeQuant → SiLU ===&gt; …</p>

Vasiliy_Kuznetsov: <p>yes, quant/dequant control which areas of the model you want to be in which dtype (torch.float vs torch.quint8).</p>
<blockquote>
<p>Quant → ConvBn → DeQuant → SiLU ===&gt; Quant → ConvBn → DeQuant → SiLU</p>
</blockquote>
<p>yep, that sounds right.  There is some example code here (<a href="https://pytorch.org/docs/stable/quantization.html" class="inline-onebox" rel="noopener nofollow ugc">Quantization — PyTorch 2.1 documentation</a>) with similar toy examples.</p>

MrOCW: <p>For my case, I just used Quant → model - &gt; Dequant with SiLUs being in the model and it still trains but with terrible accuracy. Is the SiLU being included in the Quant Dequant the main reason?</p>

jerryzh168: <p>QAT accuracy could depend on many things, including when do you turn on fake_quant and how many batches you train the model, are you training from scratch or from pretrained weights?</p>
<p>Also could you paste the modified model here?</p>

MrOCW: <p><a class="mention" href="/u/jerryzh168">@jerryzh168</a><br>
Heres a YOLOX_small model prepared for QAT.<br>
the YOLOX small pretrained weights were loaded first, then i applied fuse_modules with [“conv”,“bn”] and then prepare_qat</p>
<p>QAT model<br>
<a href="https://drive.google.com/file/d/1rUtNvHKzkR5mum3n9KHSwe11Ym-oQNR6/view?usp=sharing" class="onebox" target="_blank" rel="noopener nofollow ugc">https://drive.google.com/file/d/1rUtNvHKzkR5mum3n9KHSwe11Ym-oQNR6/view?usp=sharing</a></p>
<p>Converted model<br>
<a href="https://drive.google.com/file/d/1q_OOxc3jmdowLXXU9P7bANYHRbzVYJAi/view?usp=sharing" class="onebox" target="_blank" rel="noopener nofollow ugc">https://drive.google.com/file/d/1q_OOxc3jmdowLXXU9P7bANYHRbzVYJAi/view?usp=sharing</a></p>

jerryzh168: <p>the structure of prepared and converted model looks OK I think. how do you do QAT? are you turning on the fakequant in the very beginning? it’s typically suggested to turn on fakequant after a few batches so that the observer can be populated with correct stats first I think</p>

MrOCW: <p>Sorry how do i control that?<br>
Currently, I am doing</p>
<blockquote>
<p>Quant → fused model → Dequant</p>
</blockquote>
<p>and then just running the training loop without any edits.</p>
<p>Which way is recommended?<br>
1.Train without QAT, load the trained weights, fused and quant dequant, then repeat training<br>
2.Start QAT on my custom data right from the official pretrained weights</p>
<p>What are some hyperparameters I should take note of when performing QAT? (eg. epochs, learning rate, etc)</p>

MrOCW: <p>OK I tried by training without QAT, then load the pretrained model and train with QAT for 1 epoch and the mAP is close to the non QAT model. Is this workflow supposed to be the “correct” way?<br>
Am I not supposed to train the QAT from scratch since doing so doesnt increase my mAP from 0 at all</p>

jerryzh168: <p>the flow should be:</p>
<pre><code class="lang-auto">1. prepared = prepare_qat(model, ...)
2. disable fake_quant, but enable observation, prepared.apply(torch.ao.quantization.disable_fake_quant)
prepared.apply(torch.ao.quantization.enable_observer)
3. run a few epochs
4. enable fake_quant, and do QAT
prepared.apply(torch.ao.quantization.disable_fake_quant)
5. train a few epochs
6. convert to quantized model
</code></pre>
<p>how many epochs do we train before turning on fake_quant and after turning on fake_quant are the hyperparameters that we can experiment with.</p>

Alexey_Stolpovskiy: <p>How did you find this out? A great flow, thanks.</p>

HDCharles: <p>he’s on the ao team that developed those apis</p>