Unable to reproduce accuracy for quantized mobilenet_v2 model

Anirudh_Alameluvari: <p>As the title states, I’m trying to reproduce the results for the quantized mobilenet_v2 model trained with QAT. Pytorch reports an accuracy of 71.658% for the model (<a href="https://pytorch.org/vision/main/models/generated/torchvision.models.quantization.mobilenet_v2.html" class="inline-onebox" rel="noopener nofollow ugc">mobilenet_v2 — Torchvision main documentation</a>). I used pytorch’s source for quantization <a href="https://github.com/pytorch/vision/tree/release/0.13/references/classification#qat-mobilenetv3" class="inline-onebox" rel="noopener nofollow ugc">vision/references/classification at release/0.13 · pytorch/vision · GitHub</a> which states the mobilenetv2 model converges after 10 epochs. With the qnnpack backend in the first link, I get an accuracy of 69.2% and with an fbgemm model, I get an accuracy of 70.4%. My model converged after 6-7 epochs for both the backends but the accuracies are quite low compared to the reported accuracies.</p>
<p>Any ideas on what I should look at to bridge this gap?</p>
<p>Thanks!</p>

nkdatascientist: <p>Hi <a class="mention" href="/u/anirudh_alameluvari">@Anirudh_Alameluvari</a>,<br>
can you provide your training hyperparameter, especially the batch size and number of GPU used for QAT?</p>

Anirudh_Alameluvari: <p>Batch size used with qnnpack: 32, with fbgemm: 64. And number of GPUs is 1 in both the cases (A tesla K80).</p>

nkdatascientist: <p>Thanks for the information <a class="mention" href="/u/anirudh_alameluvari">@Anirudh_Alameluvari</a></p>
<p>This are all my observations,<br>
<strong>Quantization Parameters:</strong> I believe that you are using the same configurations.<br>
<strong>Hyper Parameter:</strong> You need to change the learning according to the batch size. In the <a href="https://arxiv.org/pdf/1801.04381.pdf" rel="noopener nofollow ugc">MobileNet-V2</a>  under Experiments topic, they mentioned about this, please look into that and change the learning rate for the QAT. I am user that, the batch size will impact the accuracy of the model and it will increase the number of iterations and epoch to achieve the expected accuracy.</p>
<p>other then this Nothing from my end and If you find a concrete problem, please tag and educate me.<br>
Thank you and Happy Learning!</p>

jcaip: <p>Hi Anirudh,</p>
<p>I don’t think this is the script used for the model in the first link, but we have a tutorial for QAT on mobilenet_v2 here:</p>
<p><a href="https://pytorch.org/tutorials/advanced/static_quantization_tutorial.html" class="onebox" target="_blank" rel="noopener nofollow ugc">https://pytorch.org/tutorials/advanced/static_quantization_tutorial.html</a></p>
<p>That I’ve run before and know should produce ~71.5% accuracy on FBGEMM. Could you try running that tutorial and seeing if you get expected results? It probably won’t be the same numbers as this <a href="https://pytorch.org/vision/main/models/generated/torchvision.models.quantization.mobilenet_v2.html" rel="noopener nofollow ugc">model</a> but it should be &gt; 70.4%.</p>

Anirudh_Alameluvari: <p>Hey <a class="mention" href="/u/jcaip">@jcaip</a>, thank you. Your suggestion worked. With fbgemm, I got an accuracy of 71.3%<br>
qq. Are there any suggestions for speeding up the model? I profiled my code using pytorch profiler and it turns out most of the time was spent in the kernel which I’m guessing refers to the I/O. My GPU is at a near 100% utilization</p>

Anirudh_Alameluvari: <p>Thank you <a class="mention" href="/u/nkdatascientist">@nkdatascientist</a>, I’ll tweak the batch size and try next. For now, changing the backend improved the accuracy quite a bit</p>

jcaip: <p>I think it’s the opposite actually. The kernel refers to GPU matmul ops, and if you were IO bound you would expect to see &lt; 100% GPU utilization.</p>
<p>You can try fusing_modules, see <a href="https://pytorch.org/docs/stable/quantization.html#introduction-to-quantization" rel="noopener nofollow ugc">here</a> under QAT API Example, but I think <code>qat_convert</code> may do this already.</p>