How to change a quant tensor

username: Forceless
cooked: <p>Hi,<br>
I’m trying to analyze the reliability of a quantized model<br>
But I have a question:<br>
How can I change the respondent value to the model<br>
I change the value directly like what I do in normal models  that will not affect the value of the model’s value<br>
like:</p>
<pre><code class="lang-python">number.int_repr()[0]=number.int_repr()[0]*2
number.int_repr()[0]
# Don't have any change
number.dequantize()=number.dequantize()*2
number.dequantize()
# Don't change too
</code></pre>
<p>Thank you</p>

username: jerryzh168
cooked: <p>what kind of changes you want? if you want multiplications by constant then you can just to <code>quantized_tensor = quantized_tensor * 2</code> I think. we also have a list of tensor methods defined here: <a href="https://pytorch.org/docs/master/quantization-support.html#torch-tensor-quantization-related-methods" class="inline-onebox" rel="noopener nofollow ugc">Quantization API Reference — PyTorch master documentation</a></p>

username: Forceless
cooked: <p>My fault,<br>
Thank you for your patience<br>
I have read this API Reference, but it can’t help my work<br>
I want to change a number at specified position of matrix<br>
First I get a value from the original matrix<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/3/b/3b9d4b22b36f4a73122b738b3520c036d1a09a76.png" alt="image" data-base62-sha1="8vnazue8VGspPqGu1zQILVT0InA" width="595" height="205"><br>
Then I change this value<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/5/4/54717bcf6f005331012d840f7a2f6ce64d3412b6.png" alt="image" data-base62-sha1="c31fc5ViaC6ohBu5FLB3wGWanMG" width="562" height="187"><br>
I tried to assign this value to the respondent number<br>
But, it doesn’t affect this matrix at all.<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/c/1/c1cd1b1bd448c85f38adaf3469a714593ab69973.png" alt="image" data-base62-sha1="rErAwG80CkossVPD8UFr5BwiKK7" width="580" height="175"></p>

username: Forceless
cooked: <p>Here’s how I get a quant tensor from model<br>
<a href="https://colab.research.google.com/drive/1MJ2QTY_8o6GgKQDh9z0qyBELluwLvx29?usp=sharing" rel="noopener nofollow ugc">Colab</a></p>

username: jerryzh168
cooked: <p>I see, thanks for the clarification, I think what you need is <a href="https://github.com/pytorch/pytorch/blob/master/aten/src/ATen/native/native_functions.yaml#L5375" class="inline-onebox" rel="noopener nofollow ugc">pytorch/native_functions.yaml at master · pytorch/pytorch · GitHub</a> and <a href="https://github.com/pytorch/pytorch/blob/master/aten/src/ATen/native/native_functions.yaml#L5380" class="inline-onebox" rel="noopener nofollow ugc">pytorch/native_functions.yaml at master · pytorch/pytorch · GitHub</a> which re-assembles quantized Tensor from int_repr, please let me know if it works, thanks</p>

username: Forceless
cooked: <p>Thank you,<br>
I successfully changed weight in quant models.<br>
But when I use _make_per_tensor_quantized_tensor, it’s changeable.<br>
However, when I use _make_per_channel_quantized_tensor, it’s not changeable</p>

username: jerryzh168
cooked: <p>ah, maybe it’s a bug, would you like to file an issue and attach a small repro for it?</p>

username: Forceless
cooked: <p>Sure, I’m glad to do that.<br>
Here’s issue <a href="https://github.com/pytorch/pytorch/issues/68322" rel="noopener nofollow ugc">Quantization: torch._make_per_channel_quantized_tensor doesn’t work well · Issue #68322 · pytorch/pytorch (github.com)</a><br>
And a few days ago you give me a prototype of FX Graph Mode <a href="https://github.com/pytorch/pytorch/blob/master/torch/fx/experimental/fx2trt/example/quantized_resnet_test.py" rel="noopener nofollow ugc">pytorch/quantized_resnet_test.py at master · pytorch/pytorch (github.com)</a><br>
But I still don’t know how to inference with CUDA<br>
Does this implementation have some examples</p>

username: HDCharles
cooked: <p>You pretty much can’t do quantized inference with cuda, there are no native quantized cuda kernels atm, our team is working to support lowering to custom backends using Fx to TRT but its not complete yet.</p>
<p>also maybe <a class="mention" href="/u/jerryzh168">@jerryzh168</a> can confirm, but I believe the intended solution was to do:</p>
<p>goal: set int_repr of a quantized tensor x to 3.</p>
<pre><code class="lang-auto"># x is per channel quantized tensor
x_int = x.int_repr()
x_int[0][0][0][0]=3
x_new = torch._make_per_tensor_quantized_tensor(x_int, x.per_channel_scales(), ... )
</code></pre>
<p>not</p>
<p><code>x.int_repr()=3</code></p>
<p>which i’m fairly sure is not intended to work</p>

username: jerryzh168
cooked: <p>here is the example that you can run int8 model in TensorRT: <a href="https://github.com/pytorch/pytorch/blob/master/torch/fx/experimental/fx2trt/example/quantized_resnet_test.py#L22" class="inline-onebox" rel="noopener nofollow ugc">pytorch/quantized_resnet_test.py at master · pytorch/pytorch · GitHub</a></p>

username: Forceless
cooked: <p>You are right,<br>
I should change its int_repr() tensor<br>
before _make_per_tensor_quantized_tensor<br>
Thank you</p>