How to use quantize_per_tensor

aprasad: <p>I have a model which is trained in Kaldi and I’m able to load the model parameters in PyTorch as tensors.<br>
I  am trying to perform post-quantization of the weight matrices and I’ve tried to use the quantize_per_tensor function.<br>
For. ex:</p>
<pre><code class="lang-auto">a = torch.rand(10)
b = torch.rand(10)
scale_a = (max_a - min_a) / (qmax - qmin)
zpt_a = qmin - min_a / scale_a
scale_b = (max_b - min_b) / (qmax - qmin)
zpt_b = qmin - min_b / scale_b
a_quant = torch.quantize_per_tensor(a, scale_a, -127, torch.qint8)
b_quant = torch.quantize_per_tensor(b, scale_b, -127, torch.qint8)
a_quant + b_quant
</code></pre>
<p>When I add the 2 quantized tensors, I get the below error</p>
<pre><code class="lang-auto">Traceback (most recent call last):
  File "&lt;stdin&gt;", line 1, in &lt;module&gt;
RuntimeError: Could not run 'aten::add.Tensor' with arguments from the 'QuantizedCPU' backend. 'aten::add.Tensor' is only available for these backends: [CPU, CUDA, MkldnnCPU, SparseCPU, SparseCUDA, Meta, Named, Autograd, Profiler, Tracer].
</code></pre>
<p>It seems that I can convert fp32 to int8 but not perform any integer arithmetic .<br>
Any help as to how to use this will be appreciated.</p>
<p>Thanks!</p>

dskhudia: <p><a class="mention" href="/u/aprasad">@aprasad</a>: If you are just looking to quantize weights only and want to keep activations in fp32, please look into dynamic quantization. It does exactly that and you it will calculate scale/zero_points automatically for you as well.</p>

dskhudia: <p>BTW if you want to use add (or other such operations) on quantized tensor you can use in the following way.</p>
<p>qfn = torch.nn.quantized.QFunctional()<br>
qfn.add(a_quant, b_quant)</p>
<p><a href="https://pytorch.org/docs/stable/quantization.html#qfunctional" class="onebox" target="_blank" rel="nofollow noopener">https://pytorch.org/docs/stable/quantization.html#qfunctional</a></p>

aprasad: <p><a class="mention" href="/u/dskhudia">@dskhudia</a>: Thank you for the reply. The quantized functional works.<br>
I want to quantize both the weights  and the activations and run inference in Pytorch with my custom class.</p>
<p>Is there a function that calculates the scale/zero_points automatically for that?</p>

aprasad: <p><a class="mention" href="/u/dskhudia">@dskhudia</a>: I tried using the qfn to multiply 2 matrices and I get the below error.</p>
<p>For <code>a</code> of shape (1,10) and <code>b</code> of shape (10,20)  if i do</p>
<pre><code class="lang-auto">qfn.mul(a_quant, b_quant)
</code></pre>
<p>I get,</p>
<pre><code class="lang-auto">r = ops.quantized.mul(x, y, scale=self.scale, zero_point=self.zero_point)
RuntimeError: The size of tensor a (10) must match the size of tensor b (20) at non-singleton dimension 1
</code></pre>
<p>Is there any function for matrix multiplication of quantized tensors?</p>

jerryzh168: <p>you can use linear for that I think.</p>

piupiuisland: <p>Hi, I have a question about the quantization scheme.</p>
<p>when I do <em>xq = torch.quantize_per_tensor(x, scale = 0.25, zero_point = 15, dtype=torch.quint8)</em>,<br>
the result is a “quantization_scheme=torch.per_tensor_affine” tensor. I wonder how can I change the scheme to <em>per_tensor_symmetric</em> tensor?</p>

jerryzh168: <p>We do not have per_tensor_symmetric tensor in the backend actually since per_tensor_symmetric can be represented by per_tensor_affine tensor, e.g. a torch.per_tensor_symmetric, torch.qint8 tensor with a scale would be the same as a torch.per_tensor_symmetric, torch.qint8 with the same scale and a zero_point of 0. You can use per_tensor_symmetric in observers though.</p>

piupiuisland: <p>Thanks for your reply. I’m checking histogram observer recently. But I wonder, is the norm computation correct? Why there are 3 norms computed ?</p>
<aside class="quote no-group" data-username="jerryzh168" data-post="8" data-topic="94303" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/jerryzh168/48/15217_2.png" class="avatar"> jerryzh168:</div>
<blockquote>
<p>We do not have per_tensor_symmetric tensor in the backend actually since per_tensor_symmetric can be represented by per_tensor_affine tensor, e.g. a torch.per_tensor_symmetric, torch.qint8 tensor with a scale would be the same as a torch.per_tensor_symmetric, torch.qint8 with the same scale and a zero_point of 0. You can use per_tensor_symmetric in observers though.</p>
</blockquote>
</aside>
<pre><code>def _compute_quantization_error(self, next_start_bin: int, next_end_bin: int):
    r"""
    Compute the quantization error if we use start_bin to end_bin as the
    min and max to do the quantization.
    """
    # print('at _compute_quantization_error, next_start, next_end', next_start_bin, next_end_bin)

    bin_width = (self.max_val.item() - self.min_val.item()) / self.bins

    # compute new bin_width
    dst_bin_width = bin_width * (next_end_bin - next_start_bin + 1) / self.dst_nbins      # divided by 256, quantized range is [-128, 127]
    if dst_bin_width == 0.0:
        return 0.0

    src_bin = torch.arange(self.bins).to(self.device)
    # distances from the beginning of first dst_bin to the beginning and
    # end of src_bin
    src_bin_begin = (src_bin - next_start_bin) * bin_width
    src_bin_end = src_bin_begin + bin_width

    # which dst_bins the beginning and end of src_bin belong to?
    dst_bin_of_begin = torch.clamp(src_bin_begin // dst_bin_width, 0, self.dst_nbins - 1)
    dst_bin_of_begin_center = (dst_bin_of_begin + 0.5) * dst_bin_width

    dst_bin_of_end = torch.clamp(src_bin_end // dst_bin_width, 0, self.dst_nbins - 1)
    dst_bin_of_end_center = (dst_bin_of_end + 0.5) * dst_bin_width

    density = self.histogram / bin_width

    norm = torch.zeros(self.bins)

    delta_begin = src_bin_begin - dst_bin_of_begin_center
    delta_end = dst_bin_width / 2
    # print('type of delta_begin, delta_end', type(delta_begin), type(delta_end))

    delta_begin= delta_begin.to(self.device)

    ## compute norm from 3 parts: begin of each new bin, center of each new bin, end of each new bin
    norm += self._get_norm(delta_begin, torch.ones(self.bins) * delta_end, density)

    norm += (dst_bin_of_end - dst_bin_of_begin - 1) * self._get_norm(
        torch.tensor(-dst_bin_width / 2), torch.tensor(dst_bin_width / 2), density
    )

    dst_bin_of_end_center = (
        dst_bin_of_end * dst_bin_width + dst_bin_width / 2
    )

    delta_begin = -dst_bin_width / 2
    delta_end = src_bin_end - dst_bin_of_end_center
    norm += self._get_norm(torch.tensor(delta_begin), delta_end, density)

    return norm.sum().item()
</code></pre>

jerryzh168: <p>cc <a class="mention" href="/u/hx89">@hx89</a> who implmeneted histogram observer</p>