What's the supported datatype for activation in torch.ao.nn.quantized.linear?

oleotiger: <h2><a name="supported-datatype-1" class="anchor" href="#supported-datatype-1"></a>supported datatype</h2>
<p>What is the supported datatype  for weight and activation in <code>torch.ao.nn.quantized.modules.linear.Linear</code>?<br>
I didn’t seen any comments on the datatype. Are uint8 and int8 both supported for  weight and activate?</p>
<h2><a name="relationship-between-layer-and-interface-2" class="anchor" href="#relationship-between-layer-and-interface-2"></a>Relationship between layer and interface</h2>
<p>What’is the relationship between <code>torch.ao.nn.quantized.modules.linear.Linear</code> and <code>torch.ao.nn.quantized.functional.linear</code>? Does <code>modules.linear.Linear</code> call <code>functional.linear</code>?</p>
<h2><a name="why-only-uint8-supported-for-activation-in-functional-interface-linear-3" class="anchor" href="#why-only-uint8-supported-for-activation-in-functional-interface-linear-3"></a>Why only uint8 supported for activation in functional interface linear?</h2>
<p>In <code>torch.ao.nn.quantized.functional.linear</code>, input is defined as tensor of type <code>torch.quint8</code>. Why cannot it support <code>torch.qint8</code>? Why is only <code>uint8</code> supported?</p>

tom: <p>I think the permissible quantized datatypes largely depend on backend support. What backend are you using?</p>
<p>I think both <code>torch.ao.nn.quantized.Linear</code> and torch.ao.nn.quantized.functional.linear<code>call</code>torch.ops.quantized.linear` under the hood (from using ?? in Jupyter to look at the source).</p>
<p>Best regards</p>
<p>Thomas</p>

HDCharles: <h2><a name="supported-datatype-1" class="anchor" href="#supported-datatype-1"></a>supported datatype</h2>
<p>well for weight its (qint8 or fp16) <a href="https://github.com/pytorch/pytorch/blob/main/torch/ao/nn/quantized/modules/linear.py#L30" rel="noopener nofollow ugc">https://github.com/pytorch/pytorch/blob/main/torch/ao/nn/quantized/modules/linear.py#L30</a></p>
<p>for activation it depends on the backend, the default qconfig has an activation observer looking for a quint8 for all the known backends though</p><aside class="onebox githubblob" data-onebox-src="https://github.com/pytorch/pytorch/blob/075cb6bab6ba72f4f8ceba09d627b1b70fbbede6/torch/ao/quantization/qconfig.py#L224">
  <header class="source">

      <a href="https://github.com/pytorch/pytorch/blob/075cb6bab6ba72f4f8ceba09d627b1b70fbbede6/torch/ao/quantization/qconfig.py#L224" target="_blank" rel="noopener nofollow ugc">github.com</a>
  </header>

  <article class="onebox-body">
    <h4><a href="https://github.com/pytorch/pytorch/blob/075cb6bab6ba72f4f8ceba09d627b1b70fbbede6/torch/ao/quantization/qconfig.py#L224" target="_blank" rel="noopener nofollow ugc">pytorch/pytorch/blob/075cb6bab6ba72f4f8ceba09d627b1b70fbbede6/torch/ao/quantization/qconfig.py#L224</a></h4>



    <pre class="onebox"><code class="lang-py">
      <ol class="start lines" start="214" style="counter-reset: li-counter 213 ;">
          <li>"""</li>
          <li>Fused version of `default_qat_config`, has performance benefits.</li>
          <li>"""</li>
          <li></li>
          <li>default_reuse_input_qconfig = QConfig(activation=default_reuse_input_observer,</li>
          <li>                                      weight=NoopObserver)</li>
          <li>"""</li>
          <li>Default qconfig for operators that reuse the observers from input Tensor, e.g. reshape</li>
          <li>"""</li>
          <li></li>
          <li class="selected">def get_default_qconfig(backend='x86', version=0):</li>
          <li>    """</li>
          <li>    Returns the default PTQ qconfig for the specified backend.</li>
          <li></li>
          <li>    Args:</li>
          <li>      * `backend` (str): a string representing the target backend. Currently supports</li>
          <li>        `x86` (default), `fbgemm`, `qnnpack` and `onednn`.</li>
          <li></li>
          <li>    Return:</li>
          <li>        qconfig</li>
          <li>    """</li>
      </ol>
    </code></pre>



  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>

<h2><a name="relationship-between-layer-and-interface-2" class="anchor" href="#relationship-between-layer-and-interface-2"></a>Relationship between layer and interface</h2>
<p>torch.ao.nn.quantized.modules.linear.Linear and torch.ao.nn.quantized.functional.linear are effectively the same thing, they’re just 2 ways to access the underlying kernel:</p><aside class="onebox githubblob" data-onebox-src="https://github.com/pytorch/pytorch/blob/main/torch/ao/nn/quantized/functional.py#L392C21-L392C21">
  <header class="source">

      <a href="https://github.com/pytorch/pytorch/blob/main/torch/ao/nn/quantized/functional.py#L392C21-L392C21" target="_blank" rel="noopener nofollow ugc">github.com</a>
  </header>

  <article class="onebox-body">
    <h4><a href="https://github.com/pytorch/pytorch/blob/main/torch/ao/nn/quantized/functional.py#L392C21-L392C21" target="_blank" rel="noopener nofollow ugc">pytorch/pytorch/blob/main/torch/ao/nn/quantized/functional.py#L392C21-L392C21</a></h4>



    <pre class="onebox"><code class="lang-py">
      <ol class="start lines" start="392" style="counter-reset: li-counter 391 ;">
          <li>return torch.ops.quantized.linear(input, _packed_params, scale, zero_point)</li>
      </ol>
    </code></pre>



  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>
<aside class="onebox githubblob" data-onebox-src="https://github.com/pytorch/pytorch/blob/main/torch/ao/nn/quantized/modules/linear.py#L168">
  <header class="source">

      <a href="https://github.com/pytorch/pytorch/blob/main/torch/ao/nn/quantized/modules/linear.py#L168" target="_blank" rel="noopener nofollow ugc">github.com</a>
  </header>

  <article class="onebox-body">
    <h4><a href="https://github.com/pytorch/pytorch/blob/main/torch/ao/nn/quantized/modules/linear.py#L168" target="_blank" rel="noopener nofollow ugc">pytorch/pytorch/blob/main/torch/ao/nn/quantized/modules/linear.py#L168</a></h4>



    <pre class="onebox"><code class="lang-py">
      <ol class="start lines" start="158" style="counter-reset: li-counter 157 ;">
          <li></li>
          <li>def extra_repr(self):</li>
          <li>    return 'in_features={}, out_features={}, scale={}, zero_point={}, qscheme={}'.format(</li>
          <li>        self.in_features, self.out_features, self.scale, self.zero_point, self.weight().qscheme()</li>
          <li>    )</li>
          <li></li>
          <li>def __repr__(self):</li>
          <li>    return _hide_packed_params_repr(self, LinearPackedParams)</li>
          <li></li>
          <li>def forward(self, x: torch.Tensor) -&gt; torch.Tensor:</li>
          <li class="selected">    return torch.ops.quantized.linear(</li>
          <li>        x, self._packed_params._packed_params, self.scale, self.zero_point)</li>
          <li></li>
          <li># ===== Serialization methods =====</li>
          <li># The special consideration here is that we have to unpack the weights into their</li>
          <li># regular QTensor form for serialization. Packed weights should not live</li>
          <li># outside the process in which they were created, rather they should be derived</li>
          <li># from the QTensor weight.</li>
          <li>#</li>
          <li># Version 1</li>
          <li>#   self</li>
      </ol>
    </code></pre>



  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>
<p>
fx quantization uses functionals while eager mode uses modules.</p>
<h2><a name="why-only-uint8-supported-for-activation-in-functional-interface-linear-3" class="anchor" href="#why-only-uint8-supported-for-activation-in-functional-interface-linear-3"></a>Why only uint8 supported for activation in functional interface linear?</h2>
<p>its actuall only quint8 support, i.e. a quantization wrapper on a uint8 tensor. But the reason we don’t support qint8 activation is generally simply just based on kernels, without a kernel to do the math, we can’t support it. the ao team doesn’t actually create fbgemm, xnnpack, qnnpack…etc we just create APIs to use those kernel’s.</p>
<p>As for why those team’s dont support qint8, i think its because quint8 tends to be used more for affine quantization and qint8 tends to be used more for symmetric quantization (since its symmetric about 0) and generally activations get affine quantization since they are often not centered around 0 (the output of a relu for example).</p>