Quantization parameters in QuantizedConv2d

APaul: <p>I would like to find where are the parameters quant_max, quant_min, min_val, max_val stored in QuantizedConv2d block. I was able to locate them using the following code in the observers</p>
<pre><code class="lang-auto">from torch.quantization.observer import MinMaxObserver, MovingAverageMinMaxObserver, HistogramObserver
C, L = 3, 4
normal = torch.distributions.normal.Normal(0,1)
inputs = [normal.sample((C, L)), normal.sample((C, L))]

observers = [MinMaxObserver(), MovingAverageMinMaxObserver(), HistogramObserver()]
for obs in observers:
  for x in inputs: 
      obs(x) 
  print(obs.__class__.__name__, obs.calculate_qparams(), obs.quant_min, obs.quant_max, obs.min_val, obs.max_val)
</code></pre>
<p>The output being</p>
<pre><code class="lang-auto">MinMaxObserver (tensor([0.0113]), tensor([136], dtype=torch.int32)) 0 255 tensor(-1.5460) tensor(1.3433)
MovingAverageMinMaxObserver (tensor([0.0111]), tensor([134], dtype=torch.int32)) 0 255 tensor(-1.4766) tensor(1.3414)
HistogramObserver (tensor([0.0082]), tensor([143], dtype=torch.int32)) 0 255 tensor(-1.5460) tensor(1.3612)
</code></pre>
<p>However I am not able to locate the same after converting a Conv2d block into QuantizedConv2d block</p>
<pre><code class="lang-auto">class M(nn.Module):

    def __init__(self):
            super(M, self).__init__()
            self.quant = torch.quantization.QuantStub()
            self.conv = torch.nn.Conv2d(in_channels=1, out_channels=1, kernel_size=3, stride=1, padding=1)            
            self.dequant = torch.quantization.DeQuantStub()
    def forward(self, x):
        x = self.quant(x)
        x = self.conv(x)  
        x = self.dequant(x)         
        return x

model_fp32 = M()
model_fp32.eval()
model_fp32.qconfig = torch.quantization.get_default_qconfig('fbgemm')
model_fp32_prepared = torch.quantization.prepare(model_fp32)
model_fp32_converted = torch.quantization.convert(model_fp32_prepared, inplace=True)
</code></pre>
<p>The weights of the model after passing the inputs through the model</p>
<pre><code class="lang-auto">model_fp32_converted(inputs)
model_fp32_converted.conv.weight()
</code></pre>
<pre><code class="lang-auto">tensor([[[[ 0.03294463083148002625, -0.13431271910667419434,
           -0.21540719270706176758],
          [ 0.14191532135009765625, -0.14191532135009765625,
           -0.24835182726383209229],
          [ 0.32184368371963500977, -0.14444953203201293945,
           -0.21287299692630767822]]]], size=(1, 1, 3, 3), dtype=torch.qint8,
       quantization_scheme=torch.per_channel_affine,
       scale=tensor([0.00253420230001211166], dtype=torch.float64),
       zero_point=tensor([0]), axis=0)
</code></pre>
<p>I would like to know</p>
<ul>
<li>
<p>How to get quant_max, quant_min, min_val, max_val from this model_fp32_converted.conv block?</p>
</li>
<li>
<p>Point the location in the github code where these parameters from the observers get stored in the QuantizedConv2d block</p>
</li>
</ul>

tom: <p>I don’t think the QuantConv2d needs these values (it needs the scale and zero point it should use for the various inputs and output but not the observed min/max), so I doubt that you can get them from it.</p>
<p>Best regards</p>
<p>Thomas</p>

APaul: <p>Hi Thomas, thanks for your response. If I understood the quantization process correctly I think there should be an observer block inside the QuantizeConv2D block which stores the min and max value from which the scale and zero_point is computed</p>
<p>Here is the link to where the observer is computing the scale and zero point</p><aside class="onebox githubblob" data-onebox-src="https://github.com/pytorch/pytorch/blob/748d9d24940cd17938df963456c90fa1a13f3932/torch/ao/quantization/observer.py#L258">
  <header class="source">

      <a href="https://github.com/pytorch/pytorch/blob/748d9d24940cd17938df963456c90fa1a13f3932/torch/ao/quantization/observer.py#L258" target="_blank" rel="noopener nofollow ugc">github.com</a>
  </header>

  <article class="onebox-body">
    <h4><a href="https://github.com/pytorch/pytorch/blob/748d9d24940cd17938df963456c90fa1a13f3932/torch/ao/quantization/observer.py#L258" target="_blank" rel="noopener nofollow ugc">pytorch/pytorch/blob/748d9d24940cd17938df963456c90fa1a13f3932/torch/ao/quantization/observer.py#L258</a></h4>



    <pre class="onebox">      <code class="lang-py">
        <ol class="start lines" start="248" style="counter-reset: li-counter 247 ;">
            <li>    # The variable names are prefixed with "initial" because their values (qmin and qmax) might be adjusted</li>
            <li>    # based on whether quantization range is reduced and the datatype (signed/unsigned) used by the observer.</li>
            <li>    assert (</li>
            <li>        quant_min &lt;= 0 &lt;= quant_max</li>
            <li>    ), "Used-specified quantization range must include 0."</li>
            <li>    assert (</li>
            <li>        quant_min &lt; quant_max</li>
            <li>    ), "qmin must be strictly less than qmax for user-specified quantization range."</li>
            <li>
            </li>
<li>@torch.jit.export</li>
            <li class="selected">def _calculate_qparams(</li>
            <li>    self, min_val: torch.Tensor, max_val: torch.Tensor</li>
            <li>) -&gt; Tuple[torch.Tensor, torch.Tensor]:</li>
            <li>    r"""Calculates the quantization parameters, given min and max</li>
            <li>    value tensors. Works for both per tensor and per channel cases</li>
            <li>
            </li>
<li>    Args:</li>
            <li>        min_val: Minimum values per channel</li>
            <li>        max_val: Maximum values per channel</li>
            <li>
            </li>
<li>    Returns:</li>
        </ol>
      </code>
    </pre>



  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>


atharvp-amb: <p>Hi Avishek,</p>
<p>Were you able figure out the answers to your questions? I would also like to understand how the Observer modules in Pytorch are being used to set the clipping range for the quantized blocks, especially in the convolution blocks. I want to experiment with different statistical observations if possible, so I would be grateful if you have any advice in that direction as well. Thank you.</p>
<p>Atharv</p>

TingfengTang: <p>I am also finding a method to extract the parameter in a quantized model containing QuantizedConv2d,especially the quant_min ans quant_max used for  requantization. After invoking</p>
<pre><code class="lang-auto">model_fp32_converted = torch.quantization.convert(model_fp32_prepared, inplace=True)
</code></pre>
<p>,the obervers are not exist in the model model_fp32_converted ,the process you described is happened only in training ,not in inference.<br>
Can <a class="mention" href="/u/hdcharles">@HDCharles</a> help us.</p>

HDCharles: <p>those values are generally hardcoded into the backend. see self._packed_params which holds the relevant information in an expected form which is then passed to the relevant op in the forward</p><aside class="onebox githubblob" data-onebox-src="https://github.com/pytorch/pytorch/blob/main/torch/ao/nn/quantized/modules/conv.py#L340-L342">
  <header class="source">

      <a href="https://github.com/pytorch/pytorch/blob/main/torch/ao/nn/quantized/modules/conv.py#L340-L342" target="_blank" rel="noopener nofollow ugc">github.com</a>
  </header>

  <article class="onebox-body">
    <h4><a href="https://github.com/pytorch/pytorch/blob/main/torch/ao/nn/quantized/modules/conv.py#L340-L342" target="_blank" rel="noopener nofollow ugc">pytorch/pytorch/blob/main/torch/ao/nn/quantized/modules/conv.py#L340-L342</a></h4>



    <pre class="onebox"><code class="lang-py">
      <ol class="start lines" start="340" style="counter-reset: li-counter 339 ;">
          <li>if self.padding_mode == 'zeros':</li>
          <li>    self._packed_params = torch.ops.quantized.conv1d_prepack(</li>
          <li>        w, b, self.stride, self.padding, self.dilation, self.groups)</li>
      </ol>
    </code></pre>



  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>

<p>these ops are understood to use int8 dtypes which has a pre-specified qmin/qmax and though you can set things up to use a differen qmin/qmax to get a different scale/zero_point, in the actual calculation, its going to clamp to whatever its set to use in the backend.</p>
<p>e.g.</p>
<aside class="onebox githubblob" data-onebox-src="https://github.com/pytorch/pytorch/blob/a1d0db1c60b746c1c9f2500bb231c528a461eda5/aten/src/ATen/native/quantized/cpu/qconv.cpp#L470-L485">
  <header class="source">

      <a href="https://github.com/pytorch/pytorch/blob/a1d0db1c60b746c1c9f2500bb231c528a461eda5/aten/src/ATen/native/quantized/cpu/qconv.cpp#L470-L485" target="_blank" rel="noopener nofollow ugc">github.com</a>
  </header>

  <article class="onebox-body">
    <h4><a href="https://github.com/pytorch/pytorch/blob/a1d0db1c60b746c1c9f2500bb231c528a461eda5/aten/src/ATen/native/quantized/cpu/qconv.cpp#L470-L485" target="_blank" rel="noopener nofollow ugc">pytorch/pytorch/blob/a1d0db1c60b746c1c9f2500bb231c528a461eda5/aten/src/ATen/native/quantized/cpu/qconv.cpp#L470-L485</a></h4>



    <pre class="onebox"><code class="lang-cpp">
      <ol class="start lines" start="470" style="counter-reset: li-counter 469 ;">
          <li>fbgemm::ReQuantizeOutput&lt;</li>
          <li>    kReluFused,</li>
          <li>    fbgemm::QuantizationGranularity::TENSOR,</li>
          <li>    float&gt;</li>
          <li>    output_proc_obj(</li>
          <li>        kNoOpObj,</li>
          <li>        output_multiplier_float.data(),</li>
          <li>        output_zero_point,</li>
          <li>        act_zero_point,</li>
          <li>        w_zp.data(),</li>
          <li>        nullptr, /* row offset buffer */</li>
          <li>        col_offsets.data(),</li>
          <li>        bias_data,</li>
          <li>        M,</li>
          <li>        groups_,</li>
          <li>        act_times_w_scale.data());</li>
      </ol>
    </code></pre>



  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>

<p>its not passing the qmin/qmax to the requant kernel, its assumed</p>

TingfengTang: <p>if i specify the activation configuration in qconfig (for example,[0~255] to [0,235]), how does the backend konw the quant_max and quant_min to clamp the activation durning inference,for examle, in quantized::conv2d or aten::quantize_per_tensor.</p>

TingfengTang: <p>oh,i see, the activation of output tensor’s range is always [0,255],or [-128,127] ,which is only controlled by dtype，and qscheme is always by per_tensor_affine. The weight’s range can be configured by qconfig(for example ,reduce_range) and it will be fixed  in the serialized model. So quant_max and quant_min are not associated with weight and activation. Those two parameters are implicitly represented by dtype.</p>

HDCharles: <p>Yep, there is however an option in eager mode quant to keep the qconfigs</p>
<p><a href="https://pytorch.org/docs/stable/generated/torch.ao.quantization.convert.html" rel="noopener nofollow ugc">https://pytorch.org/docs/stable/generated/torch.ao.quantization.convert.html</a></p>
<p>You could get the qmin/qmax from that</p>