Quantization scales and bias

username: shas19
cooked: <p>I was using Pytorch for post-training quantization for my resnet18 model. Following is part of the code.</p>
<pre><code class="lang-auto">net.qconfig = torch.quantization.QConfig(
    activation=torch.quantization.MinMaxObserver.with_args(dtype=torch.quint8, qscheme=torch.per_tensor_symmetric), 
    weight=torch.quantization.MinMaxObserver.with_args(dtype=torch.qint8, qscheme=torch.per_tensor_symmetric))
</code></pre>
<p>I wanted to print bias and scale for each tensor that is being used internally for each Tensor.<br>
Can someone please help me do it the right way?</p>
<p>Thanks.</p>

username: Vasiliy_Kuznetsov
cooked: <p>Hi <a class="mention" href="/u/shas19">@shas19</a> , if you print out the quantized network it should show the scale and zero_points of various layers.  Is there something else you are looking for?  Could you be more specific?</p>

username: ken012git
cooked: <p>Hi,</p>
<p>I have the same question for printing quantized weights.  Is there a way to see the quantized int values which are used in the convolution operators, like the example shown below:</p>
<pre><code class="lang-auto">class M(torch.nn.Module):
    def __init__(self):
        super(M, self).__init__()
        # QuantStub converts tensors from floating point to quantized
        self.quant = torch.quantization.QuantStub()
        self.conv = torch.nn.Conv2d(1, 1, 1)
        self.bn = torch.nn.BatchNorm2d(1)
        self.relu = torch.nn.ReLU()
        # DeQuantStub converts tensors from quantized to floating point
        self.dequant = torch.quantization.DeQuantStub()

    def forward(self, x):
        x = self.quant(x)
        x = self.conv(x)
        print(x)  # &lt;- input is quantized
        print(self.conv.weight)   # &lt;- It prints: bound method Conv2d.weight of QuantizedConv2d(1, 1, kernel_size=(1, 1), stride=(1, 1), scale=1.0, zero_point=0)
        print(self.conv.bias)   # &lt;- It prints: &lt;bound method Conv2d.bias of QuantizedConv2d(1, 1, kernel_size=(1, 1), stride=(1, 1), scale=1.0, zero_point=0)&gt;
        x = self.bn(x)
        x = self.relu(x)
        x = self.dequant(x)
        return x

# conver the model to qat
qconfig = QConfig(
    activation = FakeQuantize.with_args(observer=MovingAverageMinMaxObserver),
    weight = FakeQuantize.with_args(
        observer=MovingAverageMinMaxObserver,
        quant_min=-128, quant_max=127,
        dtype=torch.qint8)
)
model_fp32 = M()
model_fp32.train()
model_fp32.qconfig = qconfig
model_fp32_prepared = prepare_qat(model_fp32)
model_fp32_prepared.eval()
model_int8 = convert(model_fp32_prepared)
# run the model
input_fp32 = torch.randn(4, 1, 4, 4)
res = model_int8(input_fp32)
</code></pre>

username: ken012git
cooked: <p>Also, I did look into the code. I found the qat conv module calling fake quant before doing convolution.</p>
<aside class="onebox githubblob">
  <header class="source">

      <a href="https://github.com/pytorch/pytorch/blob/master/torch/nn/qat/modules/conv.py#L36" target="_blank" rel="noopener nofollow ugc">github.com</a>
  </header>

  <article class="onebox-body">
    <h4><a href="https://github.com/pytorch/pytorch/blob/master/torch/nn/qat/modules/conv.py#L36" target="_blank" rel="noopener nofollow ugc">pytorch/pytorch/blob/master/torch/nn/qat/modules/conv.py#L36</a></h4>



  <pre class="onebox">    <code class="lang-py">
      <ol class="start lines" start="26" style="counter-reset: li-counter 25 ;">
          <li>    factory_kwargs = {'device': device, 'dtype': dtype}</li>
          <li>    super().__init__(in_channels, out_channels, kernel_size,</li>
          <li>                     stride=stride, padding=padding, dilation=dilation,</li>
          <li>                     groups=groups, bias=bias, padding_mode=padding_mode,</li>
          <li>                     **factory_kwargs)</li>
          <li>    assert qconfig, 'qconfig must be provided for QAT module'</li>
          <li>    self.qconfig = qconfig</li>
          <li>    self.weight_fake_quant = qconfig.weight(factory_kwargs=factory_kwargs)</li>
          <li>
          </li>
<li>def forward(self, input):</li>
          <li class="selected">    return self._conv_forward(input, self.weight_fake_quant(self.weight), self.bias)</li>
          <li>
          </li>
<li>@classmethod</li>
          <li>def from_float(cls, mod):</li>
          <li>    r"""Create a qat module from a float module or qparams_dict</li>
          <li>
          </li>
<li>        Args: `mod` a float module, either produced by torch.quantization utilities</li>
          <li>        or directly from user</li>
          <li>    """</li>
          <li>    assert type(mod) == cls._FLOAT_MODULE, 'qat.' + cls.__name__ + '.from_float only works for ' + \</li>
          <li>        cls._FLOAT_MODULE.__name__</li>
      </ol>
    </code>
  </pre>


  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>

<aside class="onebox githubblob">
  <header class="source">

      <a href="https://github.com/pytorch/pytorch/blob/master/torch/quantization/fake_quantize.py#L141" target="_blank" rel="noopener nofollow ugc">github.com</a>
  </header>

  <article class="onebox-body">
    <h4><a href="https://github.com/pytorch/pytorch/blob/master/torch/quantization/fake_quantize.py#L141" target="_blank" rel="noopener nofollow ugc">pytorch/pytorch/blob/master/torch/quantization/fake_quantize.py#L141</a></h4>



  <pre class="onebox">    <code class="lang-py">
      <ol class="start lines" start="131" style="counter-reset: li-counter 130 ;">
          <li>        _scale, _zero_point = self.calculate_qparams()</li>
          <li>        _scale, _zero_point = _scale.to(self.scale.device), _zero_point.to(self.zero_point.device)</li>
          <li>        if self.scale.shape != _scale.shape:</li>
          <li>            self.scale.resize_(_scale.shape)</li>
          <li>            self.zero_point.resize_(_zero_point.shape)</li>
          <li>        self.scale.copy_(_scale)</li>
          <li>        self.zero_point.copy_(_zero_point)</li>
          <li>
          </li>
<li>    if self.fake_quant_enabled[0] == 1:</li>
          <li>        if self.is_per_channel:</li>
          <li class="selected">            X = torch.fake_quantize_per_channel_affine(</li>
          <li>                X, self.scale, self.zero_point,</li>
          <li>                self.ch_axis, self.quant_min, self.quant_max)</li>
          <li>        else:</li>
          <li>            X = torch.fake_quantize_per_tensor_affine(</li>
          <li>                X, float(self.scale), int(self.zero_point),</li>
          <li>                self.quant_min, self.quant_max)</li>
          <li>    return X</li>
          <li>
          </li>
<li>@torch.jit.export</li>
          <li>def extra_repr(self):</li>
      </ol>
    </code>
  </pre>


  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>

<p>And, the fake quant calls a fake quantize affine function, for example the <code>fake_quantize_per_tensor_affine</code> function in <code>pytorch/torch/onnx/symbolic_opset10.py</code></p>
<p>I print the output of <code>self.weight_fake_quant(self.weight)</code> and see all quanted and requanted floating point numbers. My question is that the convolution seems using the floating point <code>r</code> instead of the integers <code>q</code> from the formula in the QAT’s paper <code>r = S*(q - Z)</code>?</p>