Do I really need two separate model definition for a quantized and an "unquantized" model?

username: Yuseq
cooked: <p>Hi all,<br>
I successfully quantized parts of a visual transformer model I found online.</p>
<pre><code class="lang-auto">    backend = 'qnnpack'
    model.qconfig = torch.quantization.get_default_qconfig(backend)
    torch.backends.quantized.engine = backend

    model = model.to(device='cpu')
    quantized_model = torch.quantization.quantize_dynamic(model, qconfig_spec={torch.nn.Linear},
                                                          dtype=torch.qint8)
    torch.save(quantized_model.state_dict(), ...)
</code></pre>
<p>Note: I do not save my model in the torchscript format, because as far as I can tell, it would require me to significantly rewrite the repo code (see the error below).</p>
<blockquote>
<p>torch.jit.frontend.NotSupportedError: Compiled functions can’t take variable number of arguments or use keyword-only arguments with defaults:</p>
</blockquote>
<p>One of the layers of the model contains the following line:</p>
<pre><code class="lang-auto">q = F.linear(input=x, weight=self.q_proj.weight, bias=self.q_bias)
</code></pre>
<p>By default, <code>x</code>, <code>self.q_proj</code> and <code>self.q_bias</code> are <code>torch.float32</code> variables. After quantization <code>x</code> is still <code>torch.float32</code> and <code>self.q_proj</code>, <code>self.q_bias</code> become <code>torch.qint8</code> variables. If I try to run inference on the quantized model this line fails with</p>
<blockquote>
<p>TypeError: linear(): argument ‘weight’ must be Tensor, not method</p>
</blockquote>
<p>If I modify it like so</p>
<pre><code class="lang-auto">q = F.linear(input=x, weight=self.q_proj.weight(), bias=self.q_bias)
</code></pre>
<p>it fails with</p>
<blockquote>
<p>q = F.linear(input=x, weight=self.q_proj.weight(), bias=self.q_bias)<br>
RuntimeError: self and mat2 must have the same dtype, but got Float and QInt8</p>
</blockquote>
<p>And if I modify it further like so</p>
<pre><code class="lang-auto">q = F.linear(input=x, weight=torch.dequantize(self.q_proj.weight()), bias=torch.dequantize(self.q_bias))
</code></pre>
<p>The code passes through.</p>
<p>Here’s the problem: as it stands I need two model definitions: one for quantized models with the added changes and one for “unquantized” model (there are other modifications like this one whenever the behaviour of float32 tensor differs from qint8 tensor). Here are some scenarios I’m considering:</p>
<ol>
<li>I have to serialize the model. This is not ideal and it requires significant changes to the original code. I’m also not sure if it would resolve the problem.</li>
<li>Keep two separate model definitions. Very annoying to maintain.</li>
<li>Configure quantization script - this is what I’m looking for, but I don’t if it’s possible.</li>
<li>The original code does not use torch appropriately - I need to rewrite failing parts so that the behaviour is identical no matter the dtype of the tensor.</li>
</ol>

username: jerryzh168
cooked: <p>so here is the new tool: <a href="https://pytorch.org/docs/main/quantization.html#prototype-pytorch-2-export-quantization" class="inline-onebox" rel="noopener nofollow ugc">Quantization — PyTorch main documentation</a> can you use this one instead?</p>

username: jerryzh168
cooked: <aside class="quote no-group" data-username="Yuseq" data-post="1" data-topic="199698">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/y/c67d28/48.png" class="avatar"> Yuseq:</div>
<blockquote>
<pre><code class="lang-auto">    quantized_model = torch.quantization.quantize_dynamic(model, qconfig_spec={torch.nn.Linear},
</code></pre>
</blockquote>
</aside>
<p>some example code for dynamic quantization can be found here: <a href="https://github.com/pytorch/pytorch/blob/main/test/quantization/pt2e/test_xnnpack_quantizer.py#L542" class="inline-onebox" rel="noopener nofollow ugc">pytorch/test/quantization/pt2e/test_xnnpack_quantizer.py at main · pytorch/pytorch · GitHub</a></p>