Significant Accuracy Drop After "Custom" Activation Quantization – Seeking Debugging Suggestions

satabios: <p>To deepen my understanding of Neural Network quantization, I’m re-implementing Post-Training Quantization (PTQ) from scratch with minimal reliance on PyTorch functions. The code can be found here: <a href="https://github.com/satabios/quantization/tree/master/quant" rel="noopener nofollow ugc">GitHub Repository</a>.</p>
<p>I followed these steps in my experiments:</p>
<ul>
<li>Developed a custom quantizer</li>
<li>Replaced Linear/Conv layers with custom quantized versions</li>
<li>Added input and output observers</li>
<li>Substituted the observers with quantized versions</li>
</ul>
<h3><a name="p-454134-weight-only-quantization-1" class="anchor" href="#p-454134-weight-only-quantization-1"></a>Weight-Only Quantization:</h3>
<p>I successfully built a quantization module that replaces traditional layers with quantized ones and observed how the weights are quantized layer-by-layer. This implementation is available here: <a href="https://github.com/satabios/quantization/tree/master/quant/layer_wise_weight_only" rel="noopener nofollow ugc">Layer-Wise Weight-Only Quantization</a>.</p>
<p>Following this, I moved on to apply quantization to activations. Although I understand that, in practice, scales and zero points are compressed into a single operation:<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/0/5/056935f2a47c7106bc822c161e42b2125b313e3b.jpeg" alt="image" data-base62-sha1="LRNCob0YhCnpFNt1tX3sMMYJAL" width="632" height="442"></p>
<p>I created a simplified version where scales are handled separately (assuming infinite compute resources). The diagram below illustrates my approach:<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/3/2/325f07abd5b94122dd031723af14637617d42de7.png" data-download-href="https://discuss.pytorch.org/uploads/default/325f07abd5b94122dd031723af14637617d42de7" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/3/2/325f07abd5b94122dd031723af14637617d42de7_2_345x499.png" alt="image" data-base62-sha1="7bBtp9pnhRSBfHYVgKBVJoMA1hB" width="345" height="499" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/3/2/325f07abd5b94122dd031723af14637617d42de7_2_345x499.png, https://discuss.pytorch.org/uploads/default/optimized/3X/3/2/325f07abd5b94122dd031723af14637617d42de7_2_517x748.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/3/2/325f07abd5b94122dd031723af14637617d42de7_2_690x998.png 2x" data-dominant-color="EEEEEE"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">802×1160 61.4 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>The code for this implementation can be found here: <a href="https://github.com/satabios/quantization/tree/master/quant/layer_wise_weights_activation" rel="noopener nofollow ugc">Layer-Wise Weight and Activation Quantization</a>.</p>
<p>Upon activation quantization, the accuracy of the model takes a dives from (~92.3% to 10%). Yet I’m unable to understand why this happens. Any suggestions on why my implementation fails would be really helpful. Thanks in advance.</p>
<p>Thanks,<br>
Sathya</p>

wilhelm: <p>Did you fix this issue?</p>