Changing Quantized Weights

gakadam: <p>I have a post-training statically quantized NN. I want to change a couple of weight values of one of the convolution layer before the inference. The weight change should be based on int8 values and not on the save-format (which is torch.qint8 with corresponding scales and zero points). So far I have done the following:</p>
<pre><code class="lang-auto"># instantiate the quantized net (not shown here).

# get one of the conv layers
tmp = model_int8.state_dict()['features.0.weight']

# get int repr
tmp_int8 = tmp.int_repr()

# change value (value change is dependent on the int8 value)
tmp_int8[0][0][0][0] = new_value

# TODO: how to convert tmp_int8 to torch.qint8 type?
new_tmp = convert_int8_to_qint8(tmp_int8) # how to do this

# TODO: based on the above step:
model_int8.state_dict()['features.0.weight'] = new_tmp
</code></pre>
<p>My question is how to change the int8 tensor to torch.qint8 based on the scales and zero_points of the original weight tensor (something similar to <code>torch.quantized_per_channel()</code> but for int8 to qint8)? OR is there another way to do this?</p>
<p>Thank you.</p>

jerryzh168: <p>if this is a per channel quantized tensor? you can call <a href="https://github.com/pytorch/pytorch/blob/master/aten/src/ATen/native/native_functions.yaml#L4645" class="inline-onebox" rel="noopener nofollow ugc">pytorch/native_functions.yaml at master · pytorch/pytorch · GitHub</a> to assemble a per channel quantized tensor with int_repr, scales and zero_points etc.<br>
however, this api might be deprecated in future pytorch releases</p>

gakadam: <p><a class="mention" href="/u/jerryzh168">@jerryzh168</a> thanks for the reply. yes, it is per channel quantization.</p>
<p>So, I can now generate <code>torch.qint8</code> tensor from <code>tmp_int8</code> tensor.<br>
I verified by printing the <code>new_tmp</code> tensor to see the new values are changed.</p>
<p>However, the following line is not updating the model weights:</p>
<pre><code class="lang-auto">model_int8.state_dict()['features.0.weight'] = new_tmp
</code></pre>
<p>When I print the <code>model_int8.state_dict()['features.0.weight']</code> it still shows the old values. How can I fix this?</p>
<p>Thank you.</p>

jerryzh168: <p>I think you probably need to do <code>model_int8.features[0].weight = new_tmp</code></p>

gakadam: <p>Hi <a class="mention" href="/u/jerryzh168">@jerryzh168</a> , thank you, it worked.</p>
<p>Can you please tell me the difference between <code>model_int8.state_dict()['features.0.weight']</code> &amp; <code>model_int8.features[0].weight</code>?</p>
<p>Because after changing the code you suggested above, when I tried printing with <code>model_int8.features[0].weight</code> it shows <strong>updated</strong> values. But <code>model_int8.state_dict()['features.0.weight']</code> shows <strong>old</strong> vales.</p>

jerryzh168: <p>oh really? model_int8.state_dict() is probably a read only copy and changing that won’t change the original weight. I’m not sure why model_int8.state_dict() is not updated after you modify the weight though, that is not expected I think, are you sure that it did not change?</p>

gakadam: <p>I reran to confirm it just now and <code>model_int8.features[0].weight</code> shows <strong>updated</strong> values, but <code>model_int8.state_dict()['features.0.weight']</code> shows <strong>old</strong> vales.</p>
<p>Do I need to save and reload the model after a manual weight update – I do not see why?<br>
Also, in the above case of mismatch, which weight values will be used for inference:  <code>model_int8.features[0].weight</code> or <code>model_int8.state_dict()['features.0.weight']</code>?</p>

gakadam: <p><a class="mention" href="/u/jerryzh168">@jerryzh168</a> Hey, My problem persists. Please check: <a href="https://discuss.pytorch.org/t/changed-quantized-weights-not-reflecting-in-state-dict/109741" class="inline-onebox">Changed quantized weights not reflecting in state_dict()</a></p>
<p>Thanks for help.</p>