How to quantize register_parameters or nn.parameters?

username: hwlee
cooked: <p>I quantized the convolution model with a state tensor.<br>
The state tensor is intended to be used like a queue.</p>
<p>Firstly, I tried that make a qint8 tensor for register_parameter.<br>
But, I got a type error, when running the quantized model in PyTorch and libtorch.</p>
<p>I’m sorry that some of the code below was omitted because i couldn’t copy the entire text dut to some reason.</p>
<pre><code class="lang-auto">class conv_with_state(nn.Module):
    def __init__():
	    scale,zero_point = 1e-4,2
	    dtype= torch.qint8
	    float_state_tensor = torch.zeros(state_size,in_channel,1)
	    q_tensor = torch.quantize_per_tensor(float_state_tensor,scale,zero_point_dtype)
	    self.shift_size = 2
	    self.f_cat = nn.quantized.FloatFunctional()
	    self.register_parameter('internal_state',nn.Parameter(int_state_tensor,requires_grad=False))
	    self.conv = nn.Conv2d(--)

    def forward(self,x):
	    self.internal_state[-self.shift_size:].data.copy_(self.internal_state[self.shift_size:])
	    x = self.f_cat.cat(self.internal_state[-self.shift_size:].clone(),x)
	    self.internal_state.data.copy_(x)
	    self.conv(x)

</code></pre>
<pre><code class="lang-auto">Traceback of TorchScript, original code (most recent call last):
	File "torch/nn/quantized/modules/functional_modules.py", line 174 in forward
	def cat(self,x,dim=0):
	r = ops.quantized.cat(x, scale=self.scale, zero_point=slef.zero_point,dim=dim)
	    ~~~~~~~~~~~~~~~~~ &lt;--- HERE
	r = self.activation_post_process(r)
	return r
RuntimeError: All dtypes must be the same.
Aborted (core dumped)
</code></pre>
<hr>
<p>So I tried to wrap register_parameter with QuantStub.<br>
Similar to the code above, I had a type problem.</p>
<pre><code class="lang-auto">class conv_with_state(nn.Module):
    def __init__():
	    self.quant = torch.quantization.QuantStub()
	    float_state_tensor = torch.zeros(state_size,in_channel,1)
	    int_state_tensor = self.quant(float_state_tensor)
        self.register_parameter('internal_state',nn.Parameter(int_state_tensor,requires_grad=False))
	    self.conv = nn.Conv2d(--)

    def forward(self,x):
	    self.internal_state.data.copy_(torch.roll(self.internal_state,-self.shift_size,0))
	    self.internal_state[-self.shift_size:].data.copy_(x)
	    x = self.internal_state
	    self.conv(x)
</code></pre>
<pre><code class="lang-auto">Traceback (most recent call last):
        self.internal_state[-self.shift_size:].data.copy_(x)
RuntimeError: Copyting from quantized Tensor to non-quantized Tensor is not allowed, please use dequantize to get a float Tensor from a quantized Tensor
</code></pre>
<hr>
<p>How i got quantized tensor parameters?</p>

username: Vasiliy_Kuznetsov
cooked: <p>hi <a class="mention" href="/u/hwlee">@hwlee</a> , could you post an e2e reproducible example which includes both your model as well as how you are calling the quantization APIs?</p>
<p>High level, the first error means that when you do <code>cat</code> all the tensors have to be of the same dtype (otherwise <code>cat</code> is not defined), and the second error means that you are trying to copy a quantized tensor to a floating point tensor, which is also not defined.  It’s hard to recommend a fix without some more context, i.e. and e2e example.</p>

username: hwlee
cooked: <p>Hi <a class="mention" href="/u/vasiliy_kuznetsov">@Vasiliy_Kuznetsov</a> , thank you for your reply</p>
<p>I add e2e reproducibel example below.</p>
<h2>
<a name="my-first-try-1" class="anchor" href="#my-first-try-1"></a>My first try</h2>
<p>This code is based on non-quantize code version. I just ran the model in torch mobile. It is perfectly running in native app. But, when module was adopted quantization, return below error message.That’s a reasonable error.</p>
<pre><code class="lang-auto">import torch
import torch.nn as nn
from torch.utils.mobile_optimizer import optimize_for_mobile

class conv_with_conv(nn.Module):

    def __init__(self,input_dim,in_ch,out_ch,kernel_size,stride):
        super().__init__()
        self.shift_size = input_dim
        self.register_buffer('internal_state',torch.zeros(kernel_size-stride+input_dim,in_ch))
        self.conv = nn.Conv2d(in_ch,out_ch,(kernel_size,1),stride)

    def forward(self,x):

        self.internal_state.data.copy_(torch.roll(self.internal_state,-self.shift_size))
        self.internal_state[-self.shift_size:].data.copy_(x.squeeze(-1).squeeze(0))
        x = self.internal_state.unsqueeze(0).transpose(1,2).unsqueeze(-1)
        x = self.conv(x)

        return x

class test_model(nn.Module):

    def __init__(self,):
        super().__init__()

        self.quant = torch.quantization.QuantStub()
        self.dequant = torch.quantization.DeQuantStub()

        self.conv = conv_with_conv(2,40,1,3,1)

    def forward(self,x):
        x = self.quant(x)
        x = self.conv(x)
        x = self.dequant(x)
        return x

model = test_model()
model.eval()
model.to('cpu')
dumy_input = torch.rand(1,2,40,1)
out = model(dumy_input)
print('float out',out)

model.qconfig = torch.quantization.get_default_qat_qconfig('fbgemm')
torch.quantization.prepare(model,inplace=True)
torch.quantization.convert(model,inplace=True)
jit_model = torch.jit.script(model)
torchscript_model_optimized = optimize_for_mobile(jit_model)
torchscript_model_optimized._save_for_lite_interpreter("test.ptl")

out = model(dumy_input)
print('quant out',out)
</code></pre>
<p>Error message of frist code</p>
<pre><code class="lang-auto">float out tensor([[[[-0.0496],
          [-0.1337]]]], grad_fn=&lt;ThnnConv2DBackward&gt;)
/home/hwlee/.local/lib/python3.8/site-packages/torch/quantization/observer.py:121: UserWarning: Please use quant_min and quant_max to specify the range for observers.                     reduce_range will be deprecated in a future release of PyTorch.
  warnings.warn(
/home/hwlee/.local/lib/python3.8/site-packages/torch/quantization/observer.py:243: UserWarning: must run observer before calling calculate_qparams.                                        Returning default scale and zero point
  warnings.warn(
Traceback (most recent call last):
  File "conv_with_state.py", line 57, in &lt;module&gt;
    out = model(dumy_input)
  File "/home/hwlee/.local/lib/python3.8/site-packages/torch/nn/modules/module.py", line 889, in _call_impl
    result = self.forward(*input, **kwargs)
  File "conv_with_state.py", line 35, in forward
    x = self.conv(x)
  File "/home/hwlee/.local/lib/python3.8/site-packages/torch/nn/modules/module.py", line 889, in _call_impl
    result = self.forward(*input, **kwargs)
  File "conv_with_state.py", line 16, in forward
    self.internal_state[-self.shift_size:].data.copy_(x.squeeze(-1).squeeze(0))
RuntimeError: Copying from quantized Tensor to non-quantized Tensor is not allowed, please use dequantize to get a float Tensor from a quantized Tensor
</code></pre>
<h2>
<a name="second-2" class="anchor" href="#second-2"></a>Second</h2>
<p>So, I apply QuanStub to tensor of register_buffer. But, I got type error about copying from quantized tensor to non-quan tensor.</p>
<pre><code class="lang-auto">import torch
import torch.nn as nn
from torch.utils.mobile_optimizer import optimize_for_mobile

class conv_with_conv(nn.Module):

    def __init__(self,input_dim,in_ch,out_ch,kernel_size,stride):
        super().__init__()
        self.shift_size = input_dim
        self.quant = torch.quantization.QuantStub()
        self.register_buffer('internal_state',self.quant(torch.zeros(kernel_size-stride+input_dim,in_ch)))
        self.conv = nn.Conv2d(in_ch,out_ch,(kernel_size,1),stride)

    def forward(self,x):

        self.internal_state.data.copy_(torch.roll(self.internal_state,-self.shift_size))
        self.internal_state[-self.shift_size:].data.copy_(x.squeeze(-1).squeeze(0))
        x = self.internal_state.unsqueeze(0).transpose(1,2).unsqueeze(-1)
        x = self.conv(x)

        return x

class test_model(nn.Module):

    def __init__(self,):
        super().__init__()

        self.quant = torch.quantization.QuantStub()
        self.dequant = torch.quantization.DeQuantStub()

        self.conv = conv_with_conv(2,40,1,3,1)

    def forward(self,x):
        x = self.quant(x)
        x = self.conv(x)
        x = self.dequant(x)
        return x

model = test_model()
model.eval()
model.to('cpu')
dumy_input = torch.rand(1,2,40,1)
out = model(dumy_input)
print('float out',out)

model.qconfig = torch.quantization.get_default_qat_qconfig('fbgemm')
torch.quantization.prepare(model,inplace=True)
torch.quantization.convert(model,inplace=True)
jit_model = torch.jit.script(model)
torchscript_model_optimized = optimize_for_mobile(jit_model)
torchscript_model_optimized._save_for_lite_interpreter("test.ptl")

out = model(dumy_input)
print('quant out',out)
</code></pre>
<p>Error message of second code</p>
<pre><code class="lang-auto">float out tensor([[[[-0.1726],
          [-0.2154]]]], grad_fn=&lt;ThnnConv2DBackward&gt;)
/home/hwlee/.local/lib/python3.8/site-packages/torch/quantization/observer.py:121: UserWarning: Please use quant_min and quant_max to specify the range for observers.                     reduce_range will be deprecated in a future release of PyTorch.
  warnings.warn(
/home/hwlee/.local/lib/python3.8/site-packages/torch/quantization/observer.py:243: UserWarning: must run observer before calling calculate_qparams.                                        Returning default scale and zero point
  warnings.warn(
Traceback (most recent call last):
  File "conv_with_state2.py", line 55, in &lt;module&gt;
    out = model(dumy_input)
  File "/home/hwlee/.local/lib/python3.8/site-packages/torch/nn/modules/module.py", line 889, in _call_impl
    result = self.forward(*input, **kwargs)
  File "conv_with_state2.py", line 36, in forward
    x = self.conv(x)
  File "/home/hwlee/.local/lib/python3.8/site-packages/torch/nn/modules/module.py", line 889, in _call_impl
    result = self.forward(*input, **kwargs)
  File "conv_with_state2.py", line 17, in forward
    self.internal_state[-self.shift_size:].data.copy_(x.squeeze(-1).squeeze(0))
RuntimeError: Copying from quantized Tensor to non-quantized Tensor is not allowed, please use dequantize to get a float Tensor from a quantized Tensor
</code></pre>
<h2>
<a name="last-try-3" class="anchor" href="#last-try-3"></a>Last try</h2>
<p>I made qint8 tensor for register_buffer tensor. And replace torch.roll() to just copy_() and clone(). Because torch.roll() can’t operate in quantization model. Addtionaly, I didn’t call forward() before quantization.convert(). It must occur type error of cat() for qint8 tensor and float tensor.<br>
Despite these change, Cat() fucntion is still return “dtype isn’t same”.  Also, I tried print dtype of tensor after quantized convert. But is print nothing.</p>
<pre><code class="lang-auto">import torch
import torch.nn as nn
from torch.utils.mobile_optimizer import optimize_for_mobile

class conv_with_conv(nn.Module):

    def __init__(self,input_dim,in_ch,out_ch,kernel_size,stride):
        super().__init__()
        self.shift_size = input_dim

        scale,zero_point = 1e-4,2
        dtype = torch.qint8
        float_state = torch.zeros(kernel_size-stride+input_dim,in_ch)
        int_state = torch.quantize_per_tensor(float_state,scale,zero_point,dtype)

        self.f_cat = nn.quantized.FloatFunctional()

        self.register_buffer('internal_state',int_state)
        self.conv = nn.Conv2d(in_ch,out_ch,(kernel_size,1),stride)

    def forward(self,x):

        self.internal_state[:self.shift_size].data.copy_(self.internal_state[self.shift_size:].clone())
        x = self.f_cat.cat((self.internal_state[:-self.shift_size].unsqueeze(0).unsqueeze(-1).clone(),x))

        x = self.internal_state.unsqueeze(0).transpose(1,2).unsqueeze(-1)
        x = self.conv(x)

        return x


class test_model(nn.Module):

    def __init__(self,):
        super().__init__()

        self.quant = torch.quantization.QuantStub()
        self.dequant = torch.quantization.DeQuantStub()

        self.conv = conv_with_conv(2,40,1,3,1)

    def forward(self,x):
        x = self.quant(x)
        x = self.conv(x)
        x = self.dequant(x)
        return x


model = test_model()
model.eval()
model.to('cpu')
dumy_input = torch.rand(1,2,40,1)

model.qconfig = torch.quantization.get_default_qat_qconfig('fbgemm')
torch.quantization.prepare(model,inplace=True)
torch.quantization.convert(model,inplace=True)
jit_model = torch.jit.script(model)
torchscript_model_optimized = optimize_for_mobile(jit_model)
torchscript_model_optimized._save_for_lite_interpreter("test.ptl")

out = model(dumy_input)
print('quant out',out)
</code></pre>
<p>Error message of last code</p>
<pre><code class="lang-auto">/home/hwlee/.local/lib/python3.8/site-packages/torch/quantization/observer.py:121: UserWarning: Please use quant_min and quant_max to specify the range for observers.                     reduce_range will be deprecated in a future release of PyTorch.
  warnings.warn(
/home/hwlee/.local/lib/python3.8/site-packages/torch/quantization/observer.py:243: UserWarning: must run observer before calling calculate_qparams.                                        Returning default scale and zero point
  warnings.warn(
Traceback (most recent call last):
  File "conv_with_state3.py", line 61, in &lt;module&gt;
    out = model(dumy_input)
  File "/home/hwlee/.local/lib/python3.8/site-packages/torch/nn/modules/module.py", line 889, in _call_impl
    result = self.forward(*input, **kwargs)
  File "conv_with_state3.py", line 44, in forward
    x = self.conv(x)
  File "/home/hwlee/.local/lib/python3.8/site-packages/torch/nn/modules/module.py", line 889, in _call_impl
    result = self.forward(*input, **kwargs)
  File "conv_with_state3.py", line 24, in forward
    x = self.f_cat.cat((self.internal_state[:-self.shift_size].unsqueeze(0).unsqueeze(-1).clone(),x))
  File "/home/hwlee/.local/lib/python3.8/site-packages/torch/nn/quantized/modules/functional_modules.py", line 212, in cat
    r = ops.quantized.cat(x, scale=self.scale, zero_point=self.zero_point, dim=dim)
RuntimeError: All dtypes must be the same.
</code></pre>
<p>Please let me know if I’m doing something wrong or if there are other possible attempts.</p>

username: Vasiliy_Kuznetsov
cooked: <p>Here is a code snippet of your example modified with print statements of the dtypes of inputs to <code>torch.cat</code>: <a href="https://gist.github.com/vkuzo/a9dbaf4d2e69d8289aa7d21d6d7e4e3c" class="inline-onebox" rel="noopener nofollow ugc">gist:a9dbaf4d2e69d8289aa7d21d6d7e4e3c · GitHub</a></p>
<p>Looks like currently you are trying to concatenate a tensor with dtype <code>torch.qint8</code> with a tensor with dtype <code>torch.quint8</code>.  If you make those dtypes match, it will work. Hope this helps.</p>