How to extract the quantized weight of quantized NN model

username: 111179
cooked: <p>I am using Post Training Quantization and try to extract the quantized weight for inference phase, but I failed.<br>
I try to directly use</p>
<p>for weight in quantized_model.state_dict():<br>
np.set_printoptions(suppress=True)<br>
print(weight, “\n”, quantized_model.state_dict()[weight].detach().cpu().clone().numpy())</p>
<p>get “TypeError: NumPy conversion for QuantizedCPUQInt8Type is not supported”</p>
<p>Could give me any advice for extracting the quantized weight from the quantized model?<br>
Thank you very much!!</p>

username: Vasiliy_Kuznetsov
cooked: <p>check out <a href="https://pytorch.org/docs/stable/quantization.html#quantized-torch-tensor-operations" rel="nofollow noopener">https://pytorch.org/docs/stable/quantization.html#quantized-torch-tensor-operations</a>.  Some options:</p>
<ol>
<li>convert your quantized tensor to floating point with <code>x.dequantize()</code>
</li>
<li>get the raw integer values with <code>x.int_repr()</code>, this should be used together with <code>x.q_scale()</code> and <code>x.q_zero_point</code>.</li>
</ol>

username: jerryzh168
cooked: <p>the error message shows that the weight is already a quantized weight, it’s just <code>numpy</code> operation is not supported in quantized weight, so if you remove the <code>numpy()</code> call it should work.</p>

username: Hsilinh
cooked: <p>If I have quantized weight, how can I extract in each layer. How can I separate tensor, scale and zero_point into array or numpy array?</p>
<p>(Pdb) quantized_model.state_dict()[‘model_fp32.conv1.weight’]<br>
tensor([[[[ 6.6680e-03,  5.3344e-03,  2.0004e-02,  …, -2.8006e-02,<br>
-6.8014e-02, -1.4670e-02],<br>
[ 1.3336e-02,  9.3353e-03,  2.1338e-02,  …, -1.1602e-01,<br>
-3.7341e-02,  5.0677e-02],<br>
[ 2.0004e-02, -1.8671e-02, -5.3344e-02,  …, -7.3348e-02,<br>
9.8687e-02,  8.4017e-02],<br>
…,<br>
[-4.9344e-02, -2.9339e-02,  1.3336e-03,  …,  1.3203e-01,<br>
9.3353e-03, -6.0012e-02],<br>
[-4.0008e-03,  2.1338e-02, -4.0008e-03,  …,  5.3344e-03,<br>
-6.6680e-02,  3.0673e-02],<br>
[ 6.6680e-03,  6.6680e-03,  6.6680e-03,  …, -3.0673e-02,<br>
1.2002e-02,  8.4017e-02]],</p>
<pre><code>     [[ 1.0669e-02,  2.0004e-02,  3.2007e-02,  ...,  4.0008e-03,
       -3.2007e-02,  5.3344e-03],
      [ 2.9339e-02,  3.6007e-02,  5.6012e-02,  ..., -8.9352e-02,
       -2.4005e-02,  2.9339e-02],
      [ 4.8010e-02,  2.9339e-02, -6.6680e-03,  ..., -5.6012e-02,
        7.4682e-02,  1.6003e-02],
      ...,
      [-1.8671e-02,  1.0669e-02,  2.8006e-02,  ...,  8.5351e-02,
       -8.0017e-02, -1.6803e-01],
      [ 1.8671e-02,  4.2675e-02, -1.3336e-03,  ..., -6.9348e-02,
       -1.5070e-01, -4.1342e-02],
      [ 2.1338e-02,  0.0000e+00, -2.8006e-02,  ..., -1.0936e-01,
       -4.0008e-02,  5.3344e-02]],

      ...,
      [ 3.7336e-02,  3.6070e-02,  3.9867e-02,  ...,  5.6953e-02,
        5.9484e-02,  3.8602e-02],
      [ 2.3414e-02,  5.0625e-02,  4.7461e-02,  ...,  6.9610e-02,
        5.8219e-02,  4.4930e-02],
      [ 6.9610e-03,  4.0500e-02,  5.0625e-02,  ...,  5.7586e-02,
        6.7078e-02,  5.1891e-02]]]], size=(64, 3, 7, 7), dtype=torch.qint8,
   quantization_scheme=torch.per_channel_affine,
   scale=tensor([1.3336e-03, 1.6211e-03, 1.5573e-03, 1.2130e-03, 4.1713e-04, 1.1682e-03,
    3.4338e-04, 1.4215e-03, 1.3556e-03, 1.1205e-03, 1.8750e-03, 8.0779e-07,
    7.6834e-04, 1.2095e-03, 4.1902e-04, 1.2486e-03, 9.4629e-04, 2.1297e-03,
    3.7952e-07, 4.7981e-04, 1.3093e-04, 1.4277e-03, 2.5236e-03, 1.3021e-03,
    2.5295e-03, 3.1860e-04, 3.6392e-04, 2.9317e-03, 1.9204e-03, 3.7749e-07,
    8.5399e-04, 9.0634e-04, 2.1589e-03, 1.1871e-03, 2.4718e-03, 3.7885e-07,
    2.1255e-06, 1.6729e-03, 8.8754e-07, 1.0482e-05, 1.2293e-03, 2.2856e-04,
    4.7812e-04, 3.0442e-03, 2.9528e-03, 1.5815e-03, 6.9944e-04, 1.3110e-03,
    2.4429e-03, 3.9633e-04, 1.4372e-03, 2.6385e-04, 2.5296e-04, 6.3496e-07,
    3.4408e-04, 3.4195e-04, 1.0956e-06, 6.7232e-04, 6.8186e-04, 2.4606e-03,
    3.9087e-04, 5.1462e-04, 8.7948e-07, 6.3281e-04], dtype=torch.float64),
   zero_point=tensor([0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
    0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
    0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]),
   axis=0)
</code></pre>
<p>Thanks,</p>
<p>Stephen</p>

username: Hsilinh
cooked: <p>I type quantized_model.state_dict()[‘model_fp32.conv1.weight’].q_scale()</p>
<p>*** RuntimeError: Expected quantizer-&gt;qscheme() == kPerTensorAffine to be true, but got false.  (Could this error message be improved?  If so, please report an enhancement request to PyTorch.)</p>
<p>The data_structure shows: Tensor quantization_scheme=torch.per_channel_affine<br>
What’s wrong with that?</p>
<p>I can print out the int_repr() with dtype = torch.int8</p>
<p>Thanks,</p>
<p>Stephen</p>

username: Hsilinh
cooked: <p>I tried to narrow down to a simple question.</p>
<blockquote>
<blockquote>
<blockquote>
<p>x = torch.tensor([[-1.0, 0.0], [1.0, 2.0]])<br>
y = torch.quantize_per_channel(x, torch.tensor([0.1, 0.01]), torch.tensor([10, 0]), 0, torch.quint8)<br>
y<br>
tensor([[-1.,  0.],<br>
[ 1.,  2.]], size=(2, 2), dtype=torch.quint8,<br>
quantization_scheme=torch.per_channel_affine,<br>
scale=tensor([0.1000, 0.0100], dtype=torch.float64),<br>
zero_point=tensor([10,  0]), axis=0)<br>
y.int_repr()<br>
tensor([[  0,  10],<br>
[100, 200]], dtype=torch.uint8)<br>
y.q_scale()<br>
*** RuntimeError: Expected quantizer-&gt;qscheme() == kPerTensorAffine to be true, but got false.  (Could this error message be improved?  If so, please report an enhancement request to PyTorch.)</p>
</blockquote>
</blockquote>
</blockquote>
<p>I installed pytorch version : 1.10.0</p>
<p>Thanks,<br>
Stephen</p>

username: Karthik_Ganesan
cooked: <p>Hi, not sure if you have already solved this but this is because torch supports two different quantization schemes: <code>per tensor affine</code> and <code>per channel affine</code>. In <code>per tensor affine</code>, a single <code>scale</code> and <code>zero point</code> are saved per tensor. So you can use <code>.q_scale()</code> as you did to get that single value.</p>
<p>However, in your case, PyTorch is using <code>per channel affine</code> which means there are N <code>scale</code> and <code>zero point</code> values, where N = number of channels. In this case you have to use `.q_per_channel_scales()’ to return a tensor of all the scale values.</p>