What do [De]QuantStub actually do?

username: vladium
cooked: <p>I have scoured all documentation I could locate and am still confused on certain accounts:</p>
<ul>
<li>class docstring (<a href="https://pytorch.org/docs/stable/quantization.html#torch.quantization.QuantStub" class="inline-onebox" rel="noopener nofollow ugc">Quantization — PyTorch 2.1 documentation</a>) says</li>
</ul>
<blockquote>
<p>Quantize stub module, before calibration, this is same as an observer, it will be swapped as nnq.Quantize in convert.</p>
</blockquote>
<p>which unfortunately isn’t very helpful at all (which “observer”?). That last part of that sentence seems to suggest that at inference time this module will be replaced with one that does actual float-to-int8 data conversion. But what does this module do at calibration time?</p>
<ul>
<li>furthermore, tutorials seem to suggest that QuantStub and DeQuantStub act as delimiters of parts of the model stack that will actually be subject to quantization; however, some other commentary (<a href="https://discuss.pytorch.org/t/quantization-how-to-quantize-model-which-include-not-support-to-quantize-layer/76528/2" class="inline-onebox">[quantization] how to quantize model which include not support to quantize layer - #2 by jerryzh168</a>) seems to suggest that these also “record tensor statistics” and hence unique instances of them are needed – what, one unique pair per each contiguous quantization region?</li>
</ul>
<p>Some more details would be very much appreciated.</p>

username: jerryzh168
cooked: <p>QuantStub is just a place holder for quantize op, it needs to be unique since it has state.<br>
DeQuantStub is a place holder for dequantize op, but it does not need to be unique since it’s stateless.</p>
<p>In eager mode quantization, users need to manually place <code>QuantStub</code> and <code>DeQuantStub</code> in the model whenever the activation in the code crosses the quantized and non-quantized boundary.</p>
<p>One thing to remember is for a quantized module, we always quantize the output of the module, but we don’t quantize the input of the module, so the quantization of the input Tensor should be taken care of by the previous module, that’s why we have QuantStub here, basically to quantize the input for the next quantized module in the sequence.</p>
<p>So in <code>prepare</code>, we’ll attach observer for the output of <code>QuantStub</code> to record the Tensor statistics of the output Tensor, just like for other modules like <code>nn.Conv2d</code>. observer is specified by the qconfig.</p>
<p>And in <code>convert</code> <code>QuantStub</code> will be swapped as <code>nnq.Quantize</code> module, and output of <code>nnq.Quantize</code>(input of <code>nnq.Conv2d</code>) will be quantized.</p>

username: vladium
cooked: <p>Thank you, Jerry, this helps.</p>
<p>To clarify</p>
<blockquote>
<p>In <em>eager</em> mode quantization</p>
</blockquote>
<p>– that is the only mode available at the moment, correct? The JIT does not quantize (yet?).</p>
<p>This might sounds like a nit, but I think the following actually reflects a fundamental difficulty in quantizing in eager mode:</p>
<blockquote>
<p>so the quantization of the input Tensor should be taken care of by the <em>previous</em> module,</p>
</blockquote>
<p>How does pytorch decide what is “previous”? The true sequence of layer invocations is determined by the procedural code in <code>forward()</code> and it can involve branching at runtime or just data flow merging like with skip connections.</p>
<p>(I am <a href="https://discuss.pytorch.org/t/am-i-correct-in-concluding-that-resnet-that-comes-with-pytorch-cant-be-quantized-by-pytorch/82405">yet to succeed in quantizing resnet</a> because of this, I suspect)</p>

username: jerryzh168
cooked: <blockquote>
<p>– that is the only mode available at the moment, correct? The JIT does not quantize (yet?).</p>
</blockquote>
<p>yeah, eager mode is the only mode that’s supported in public release, but graph mode is coming up in 1.6 as well.</p>
<blockquote>
<p>How does pytorch decide what is “previous”?</p>
</blockquote>
<p>PyTorch don’t do this in eager mode. that’s why in eager mode users need to manually place QuantStub and DeQuantStub themselves. this is done automatically in graph mode quantization.<br>
eager mode will just swap all modules that has a qconfig, so user need to make sure the swap makes sense and set qconfig and place QuantStub/DeQuantStub correctly.</p>

username: sumagic
cooked: <p>when i do qat(quant awaring training), the tutorials said I should put QuantStub and DeQuantStub between self.modules within forward. so what did they do at this time, just meaningful at inference and valid time?</p>

username: jerryzh168
cooked: <p>yeah in training it’s a no-op, but after convert we replace it with a dequantize op</p>

username: sumagic
cooked: <p>thank you very much.<br>
so they just work during module convert, and do not do anything during qat training even after model.eval()? I change the quant config and then valid the module, the mAP is different</p>

username: sumagic
cooked: <p>The purpose is to quant part of the module during qat, but when i execute prepare_qat, i see that all of the modules are inserted FakeQuantize. QuantStub and DeQuantStub do not do anything. Is there any way to insert FakeQuantize to part of the module?</p>

username: jerryzh168
cooked: <aside class="quote no-group" data-username="sumagic" data-post="7" data-topic="82402">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/sumagic/48/64034_2.png" class="avatar"> sumagic:</div>
<blockquote>
<p>so they just work during module convert, and do not do anything during qat training even after model.eval()? I change the quant config and then valid the module, the mAP is different</p>
</blockquote>
</aside>
<p>yeah you’ll need to configure <code>qconfig</code> correctly.</p>
<p>eager mode is hard to use in general, if your model works with torch.export it might be easier to try out our new flows: <a href="https://pytorch.org/tutorials/prototype/pt2e_quant_ptq_static.html" class="inline-onebox" rel="noopener nofollow ugc">(prototype) PyTorch 2.0 Export Post Training Static Quantization — PyTorch Tutorials 2.0.1+cu117 documentation</a> and <a href="https://pytorch.org/tutorials/prototype/pt2e_quantizer.html" class="inline-onebox" rel="noopener nofollow ugc">How to Write a Quantizer for PyTorch 2 Export Quantization — PyTorch Tutorials 2.1.1+cu121 documentation</a></p>