Misinterpretation of Moving Average Obeserver value

Thomas_CAMIER: <p>Hi,<br>
I am a newby with Pytorch quantization (and Python). I do not understand why my moving average value could be negative whereas my last module is a relu? This issue happen only for the first layer. See print.I believed that probe for moving average was the output of tdnn module.</p>
<p>see forwad of tdnn</p>
<blockquote>
<p>x = self.linear_layer(padded_input)<br>
x = self.relu(x.view(mb, -1, self.output_features))<br>
x= self.dequant(x)</p>
</blockquote>
<p>tdnn1 (after relu)</p>
<blockquote>
<p>(activation_post_process): MovingAverageMinMaxObserver(<strong>min_val=-2.8240833282470703</strong>, max_val=3.0684268474578857)</p>
</blockquote>
<p>tdnn2 (after relu)</p>
<blockquote>
<p>(activation_post_process): MovingAverageMinMaxObserver(<strong>min_val=0.0</strong>, max_val=3.1495800018310547)</p>
</blockquote>

Vasiliy_Kuznetsov: <p>Could you post more information about how you are calling the observer and where it is in your model?  From your printout we can see the min_val/max_val of the observer, but not how it gets the values.</p>
<p>If I had to guess, the observer is getting values from something other than a relu, but would be good to get more info to prove or disprove that.</p>

Thomas_CAMIER: <p>Hi Vassiliy,</p>
<p>Thanks for your consideration to my question.<br>
Let me rephrase my question more precisely.</p>
<p>model init<br>
self.quant = torch.quantization.QuantStub()<br>
self.dequant = torch.quantization.DeQuantStub()</p>
<p>model forward<br>
x = self.linear_layer(self.quant(padded_input))<br>
x = self.relu(x.view(mb, -1, self.output_features))<br>
x= self.dequant(x)</p>
<p>During the qat, when the MovingAverageMinMaxObserver is processed at torch.quantization.QuantStub() or at torch.quantization.DeQuantStub() or both? Regarding, the graphical view from “Netron”, I believed it was at DeQuantStub().<br>
FYI, I run qat thanks to this line : torch.quantization.prepare_qat(model, inplace=True).</p>
<p>See print.<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/8/9/89fd1f263992a0b60d088732f9becea37425f815.jpeg" data-download-href="https://discuss.pytorch.org/uploads/default/89fd1f263992a0b60d088732f9becea37425f815" title="tdnn1"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/8/9/89fd1f263992a0b60d088732f9becea37425f815_2_690x388.jpeg" alt="tdnn1" data-base62-sha1="jGHITBEqxpNrlOqtYA3ae7Qyu3j" width="690" height="388" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/8/9/89fd1f263992a0b60d088732f9becea37425f815_2_690x388.jpeg, https://discuss.pytorch.org/uploads/default/optimized/3X/8/9/89fd1f263992a0b60d088732f9becea37425f815_2_1035x582.jpeg 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/8/9/89fd1f263992a0b60d088732f9becea37425f815.jpeg 2x" data-dominant-color="E5E4E4"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">tdnn1</span><span class="informations">1280×720 109 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/6/5/652ef7eaadfd03af4755d172a753612d3f53bd6d.jpeg" data-download-href="https://discuss.pytorch.org/uploads/default/652ef7eaadfd03af4755d172a753612d3f53bd6d" title="tdnn2"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/6/5/652ef7eaadfd03af4755d172a753612d3f53bd6d_2_690x388.jpeg" alt="tdnn2" data-base62-sha1="er6QWoxY5qgO5KOR98dMV4CmT4F" width="690" height="388" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/6/5/652ef7eaadfd03af4755d172a753612d3f53bd6d_2_690x388.jpeg, https://discuss.pytorch.org/uploads/default/optimized/3X/6/5/652ef7eaadfd03af4755d172a753612d3f53bd6d_2_1035x582.jpeg 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/6/5/652ef7eaadfd03af4755d172a753612d3f53bd6d.jpeg 2x" data-dominant-color="E3E2E2"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">tdnn2</span><span class="informations">1280×720 113 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

Vasiliy_Kuznetsov: <p>Hi <a class="mention" href="/u/thomas_camier">@Thomas_CAMIER</a> , could you post the code for your full example (so that someone else could copy-paste your code and run it)?  That would enable us to help better, it’s still not 100% clear from what you’ve posted so far since it does not have the precise information of what your code is doing.</p>

Thomas_CAMIER: <p>Hi <a class="mention" href="/u/vasiliy_kuznetsov">@Vasiliy_Kuznetsov</a>,</p>
<p>I finally resolved my misunderstanding.<br>
Observer is processed at QuantStub (x = self.linear_layer(self.quant(padded_input))).<br>
This is why I have negative value since I quantize input of my “linear-relu” block and not the output. It was obvious but I was confused by the naming “activation_post_process” that is in fact the input in my case …<br>
Thanks for your help<br>
Regards,<br>
Thomas</p>