Supported quantized tensor operations

username: kwojcicki
cooked: <p>Is there a list of currently supported operations for quantized tensors?</p>
<p>I run into issues quantizing a network requiring tensor additions:</p>
<pre><code class="lang-auto">RuntimeError: Could not run 'aten::add.Tensor' with arguments from the 'QuantizedCPUTensorId' backend. 'aten::add.Tensor' is only available for these backends: [SparseCPUTensorId, CPUTensorId, VariableTensorId, MkldnnCPUTensorId].
</code></pre>
<p>Others have reported running into issues with div and cat operations - I presume these are also not supported atm.</p>

username: kwojcicki
cooked: <p>In case this is helpful to anyone, there are:</p>
<p><a href="https://pytorch.org/docs/stable/quantization.html#torch.nn.quantized.FloatFunctional" class="onebox" target="_blank" rel="nofollow noopener">https://pytorch.org/docs/stable/quantization.html#torch.nn.quantized.FloatFunctional</a></p>
<p>and</p>
<p><a href="https://pytorch.org/docs/stable/quantization.html#torch.nn.quantized.QFunctional" class="onebox" target="_blank" rel="nofollow noopener">https://pytorch.org/docs/stable/quantization.html#torch.nn.quantized.QFunctional</a></p>
<p>that support add and other operations.</p>

username: Zafar
cooked: <p>Add is supported, but not as a <code>at::add.Tensor</code>. The reason is that addition (or any arithmetic) requires output scale/zero_point. Also, often times quantized ops need to be stateful. Hence, there are stateful <code>FloatFunctional</code> and <code>QFunctional</code></p>

username: Lennart_Muller-Wirts
cooked: <p>Hey there <img src="https://discuss.pytorch.org/images/emoji/apple/wave.png?v=9" title=":wave:" class="emoji" alt=":wave:"><br>
I don’t see, how I can apply this solution <img src="https://discuss.pytorch.org/images/emoji/apple/frowning.png?v=9" title=":frowning:" class="emoji" alt=":frowning:"><br>
Do I have to replace the my <code>out += residual</code> operation in my model with QFunctional?<br>
Regards LMW</p>

username: Zafar
cooked: <p>No, you need to follow the static quantization flow, and replace all occurances of the addition with a layer <code>FloatFunctional</code>. For example, if you have a model that looks like that:</p>
<pre><code class="lang-auto">class Foo(nn.Module):
  def __init__(self):
    super().__init__()
  def forward(self, x):
    y = x + x
    z = y + 2 * x
    return x
</code></pre>
<p>it is not “quantizable”. The reason is that the additions and multiplication are in the forward, and don’t store the state of the scale and zero_point needed for the quantization.</p>
<p>Your quantizable model will looks something like that:</p>
<pre><code class="lang-auto">class Foo(nn.Module):
  def __init__(self):
    super().__init__()
    self.first_functional = nn.quantized.FloatFunctional()
    self.second_functional = nn.quantized.FloatFunctional()
    self.third_functional = nn.quantized.FloatFunctional()

  def forward(self, x):
    y = self.first_functional.add(x, x)
    x2 = self.second_functional.mul_static(x, 2)
    z = self.third_functional.add(y, x2)
    return z
</code></pre>
<p>Notice that we don’t reuse the float functional modules, and use a different one for every arithmetic operator. This is done so that the quantization parameters for each operation would be computed differently.</p>
<p>Once you rewrite your models, you can run the static quantization (<code>prepare</code>, <code>calibrate</code>, <code>convert</code> steps): <a href="https://pytorch.org/docs/stable/quantization.html#static-quantization" class="inline-onebox" rel="noopener nofollow ugc">Quantization — PyTorch 1.7.1 documentation</a></p>