Post Quantizing conv1d, PReLU & layerNorm layers can be done?

username: BOLLOJU_ARAVIND
cooked: <p>I build a pytorch model based on conv1d. I gone through quantization and implemented some cases as well but all those are working on conv2d, bn,relu but In my case, my model is built on conv1d and PReLU. Does this quatization valid for these network layers? Because when I did quantization only the layers which are included in mapping is only quantized. Let me show you those layers for which quantization is valid(i.e which are included in mapping)<br>
Please find the list of modules it supports:(according to source code in went through)<br>
(Actual layer : quantized layer)<br>
nn.Linear: nnq.Linear,<br>
nn.ReLU: nnq.ReLU,<br>
nn.ReLU6: nnq.ReLU6,<br>
nn.Conv2d: nnq.Conv2d,<br>
nn.Conv3d: nnq.Conv3d,<br>
nn.BatchNorm2d: nnq.BatchNorm2d,<br>
nn.BatchNorm3d: nnq.BatchNorm3d,<br>
QuantStub: nnq.Quantize,<br>
DeQuantStub: nnq.DeQuantize,</p>
<h1>Wrapper Modules:</h1>
<p>nnq.FloatFunctional: nnq.QFunctional,</p>
<h1>Intrinsic modules:</h1>
<p>nni.ConvReLU2d: nniq.ConvReLU2d,<br>
nni.ConvReLU3d: nniq.ConvReLU3d,<br>
nni.LinearReLU: nniq.LinearReLU,<br>
nniqat.ConvReLU2d: nniq.ConvReLU2d,<br>
nniqat.LinearReLU: nniq.LinearReLU,<br>
nniqat.ConvBn2d: nnq.Conv2d,<br>
nniqat.ConvBnReLU2d: nniq.ConvReLU2d,</p>
<h1>QAT modules:</h1>
<p>nnqat.Linear: nnq.Linear,<br>
nnqat.Conv2d: nnq.Conv2d,</p>
<p>Is it, it means that quantization can’t be done on conv1d and PReLU?</p>

username: supriyar
cooked: <p>We are in the process of implementing the Conv1d module and ConvReLU1d fused module. The PR list is here <a href="https://github.com/pytorch/pytorch/pull/38438" rel="nofollow noopener">https://github.com/pytorch/pytorch/pull/38438</a>. Feel free to try out your model with the changes in this PR. The quantization flow should convert it.<br>
We don’t currently support fusion with PReLU and LayerNorm, so they will have to be executed separately.</p>

username: BOLLOJU_ARAVIND
cooked: <p>Hi <a class="mention" href="/u/supriyar">@supriyar</a></p>
<p>Fusing is optional one in quantization if I’m not wrong. We need our modules to be quantized i.e., each layer we implemented, in order to get our quantized parameters to pass through it. Only the quantized model will work if all the layers were quantized is it right? Or else we need to dequantize the parameters again before it passes through not quantized layer is it so?</p>
<p>Thank you <a class="mention" href="/u/supriyar">@supriyar</a></p>

username: BOLLOJU_ARAVIND
cooked: <p>Hi <a class="mention" href="/u/supriyar">@supriyar</a></p>
<p>Hey, you gave me that PR link above in the last comment for support of quantized conv1d. I decided to give a try with that but the torch module is not importing and it showing the error ‘torch.version’ is not there. So i copied ‘version.py’ from my earlier version after this too its not importing torch throwing another error ‘torch._c import default_generators’ failed to import.</p>
<p>May I know that the changes in that PR is applicable for torch CPU version or not?</p>
<p>Thanks <a class="mention" href="/u/supriyar">@supriyar</a> ,<br>
Aravind</p>

username: supriyar
cooked: <blockquote>
<p>Only the quantized model will work if all the layers were quantized is it right? Or else we need to dequantize the parameters again before it passes through not quantized layer is it so?</p>
</blockquote>
<p>You can insert <code>QuantStub</code>, <code>DequantStub</code> blocks around the code that can be quantized. Please see <a href="https://pytorch.org/tutorials/advanced/static_quantization_tutorial.html" class="inline-onebox" rel="noopener nofollow ugc">(beta) Static Quantization with Eager Mode in PyTorch — PyTorch Tutorials 2.1.1+cu121 documentation</a> for an example of this.</p>
<blockquote>
<p>May I know that the changes in that PR is applicable for torch CPU version or not?</p>
</blockquote>
<p>The change is applicable for CPU, to get these changes you can either build pytorch from source or install from pytorch nightly.</p>

username: BOLLOJU_ARAVIND
cooked: <p>Hi <a class="mention" href="/u/supriyar">@supriyar</a>,</p>
<p>Thank you, Now I’m able to work with the changes in new PR as you suggested I tried with nightly.</p>
<p>Thanks <a class="mention" href="/u/supriyar">@supriyar</a>,<br>
Aravind.</p>

username: BOLLOJU_ARAVIND
cooked: <p>Hi <a class="mention" href="/u/supriyar">@supriyar</a></p>
<p>I have done Quantizing my model, now I tried to save it with ‘jit.save(jit.script(model))’ but it pops the error as ‘‘aten::slice.t(t[] l, int start, int end=9223372036854775807, int step=1) -&gt; (t[]): could not match type tensor to list[t] in argument ‘l’: cannot match list[t] to tensor.’’ very similar to this 2 more pops also arised. I googled about this error and in some discussions I found that it is the error regarding slicing(: , : , : ) operator says that ‘jit.script will not support scripting for directly used slicing operator’. Is it the error actually pointed to that? Because I too used slicing operator in the middle of my model and error throws at the same line.</p>
<p>This is one thing and now I go with another alternative to save quantized model i.e., with state_dict(). I’m able to save model with this but when I want to perform inference I have to initialize the model with the params which I have saved earlier with state_dict(). Now the params are quantized one but our model is defined for float. So error popped up as ‘exception occured : (‘copying from quantized tensor to non-quantized tensor is not allowed, please use dequantize to get a float tensor from a quantized tensor’’. This is the issue that I can’t save my model with jit and if I do so with state_dict() here I can’t initialize my model to go with inference.</p>
<p>Can you suggest any alternative?</p>
<p>Thanks <a class="mention" href="/u/supriyar">@supriyar</a><br>
Aravind.</p>

username: supriyar
cooked: <p>Could you give a small repro for the error with <code>aten::slice</code> not working with jit?</p>
<p>Regarding loading the quantized state_dict - you will first have to convert the model to quantized model before loading the state dict as well. You can call prepare and convert APIs to do this (no need to calibrate). This way the model state dict will match the saved quantized state dict and you should be able to load it.</p>

username: BOLLOJU_ARAVIND
cooked: <p>Hi <a class="mention" href="/u/supriyar">@supriyar</a></p>
<p>Thanks for the suggestion.</p>
<aside class="quote no-group" data-username="supriyar" data-post="8" data-topic="80954">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/s/db5fbb/48.png" class="avatar"> supriyar:</div>
<blockquote>
<p>call prepare and convert APIs</p>
</blockquote>
</aside>
<p>I tried with this one and its done.</p>
<p>Hey regarding</p>
<aside class="quote no-group" data-username="supriyar" data-post="8" data-topic="80954">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/s/db5fbb/48.png" class="avatar"> supriyar:</div>
<blockquote>
<p><code>aten::slice</code> not working with jit</p>
</blockquote>
</aside>
<p>This has done successfully because JIT might accepts only direct representation of Int for slicing as “<strong>w[:,:,:-16]</strong>” where as I initially represented as  “w = w[:,:,:-2*2**3]”. So tried a chance and it worked.</p>
<p>Something I would like to say is JIT is not able to find attribute “.new_tensor” where as “.clone().detach()” is identified</p>
<p>Thanks <a class="mention" href="/u/supriyar">@supriyar</a>,<br>
Aravind</p>

username: BOLLOJU_ARAVIND
cooked: <p>Hi <a class="mention" href="/u/supriyar">@supriyar</a>,</p>
<p>I have some doubt about operations on quantized variables. Here I’m quoting it</p>
<blockquote>
<pre><code>    z = self.dequant_tcnet(z)
    w = self.dequant_tcnet(w)
    v = self.dequant_tcnet(v)
    x = self.dequant_tcnet(x)
    z = z + w
    x = x + v
    x = self.quant_tcnet2(x)
</code></pre>
</blockquote>
<p>Here inorder to perform these 2 operations  z = z + w; x = x + v I need to dequantize the variables involved in that operations and then perform those operations. Can we able to perform those operations without dequantizing i.e., as quantized variables because if I do run this without dequantizing variables “<em>RuntimeError: Could not run ‘aten::add.Tensor’ with arguments from the ‘QuantizedCPU’ backend. ‘aten::add.Tensor’ is only available for these backends: [CPU, MkldnnCPU, SparseCPU, Autograd, Profile]</em>.” error is interrupting.</p>
<p>Can we perform “addition” on 2 quantized tensor variables? without dequantizing them in any other alternative.</p>
<p>Thanks <a class="mention" href="/u/supriyar">@supriyar</a>,<br>
Aravind.</p>

username: BOLLOJU_ARAVIND
cooked: <p>Hi <a class="mention" href="/u/supriyar">@supriyar</a>,</p>
<blockquote>
<pre><code>z = self.dequant_tcnet(z)
w = self.dequant_tcnet(w)
v = self.dequant_tcnet(v)
x = self.dequant_tcnet(x)
z = z + w
x = x + v
x = self.quant_tcnet2(x)
</code></pre>
</blockquote>
<blockquote>
<p>Can we perform “addition” on 2 quantized tensor variables? without dequantizing them in any other alternative.</p>
</blockquote>
<p>For this i tried QFunctional &amp; FloatFunctional but output is not up to the mark by using this. where as placing quantstubs works good.</p>
<p>I have some concern here. My float model which takes 0.2 - 0.3 second (~300ms) to process single input whereas after i quantizing my model with Int8 Quantization the time taken is increased from 0.2-0.3(float precision) to 0.4 to 0.5(Int8).</p>
<p>Here i show you the exact float model block and quantized model block</p>
<blockquote>
<pre><code>  ***** float block ******
   *****Round 1********
    y = self.conv1x12(x)
    y = self.prelu2(y)
    y = self.norm2(y)
    w = self.depthwise_conv12(y)
    #w = w[:,:,:-2*2**2]
    w = w[:,:,:-8]
    y = self.depthwise_conv2(y)
    y = y[:,:,:-8]
    y = self.prelu22(y)
    y = self.norm22(y)
    v = self.pointwise_conv2(y)
    z = z + w
    x = x + v
</code></pre>
</blockquote>
<p>This is float model block. This block/computation will be repeated 13 more times (total 14 blocks). This is taking 0.2 - 0.3 seconds</p>
<blockquote>
<p><em><strong><strong>Quantized block</strong></strong></em><br>
<strong><strong>round 1</strong></strong>**<br>
y = self.conv1x12(x)<br>
y = self.prelu2(y)<br>
y = self.norm2(y)<br>
w = self.depthwise_conv12(y)<br>
w = w[:,:,:-8]<br>
y = self.depthwise_conv2(y)<br>
y = y[:,:,:-8]<br>
y = self.prelu22(y)<br>
y = self.norm22(y)<br>
v = self.pointwise_conv2(y)<br>
w = self.dequant_tcnet(w)<br>
z = self.dequant_tcnet(z)<br>
v = self.dequant_tcnet(v)<br>
x = self.dequant_tcnet(x)<br>
z = z + w<br>
x = x + v<br>
x = self.quant_tcnet3(x)</p>
</blockquote>
<p>This is quantized block model where is placed quantstubs for those arthematic operations &amp; remaining all layers are quantized. This quantized model is taking 0.4 - 0.5 seconds</p>
<p>So after quantizing my model, the size of model is optimized but computation time is not optimized. Could you tell me is there any flaw? I crosschecked and the output is also good but computation is not reduced</p>
<p>Thanks <a class="mention" href="/u/supriyar">@supriyar</a>,<br>
Aravind.</p>

username: BOLLOJU_ARAVIND
cooked: <p>Hi <a class="mention" href="/u/supriyar">@supriyar</a> <a class="mention" href="/u/raghuramank100">@raghuramank100</a>,</p>
<p>Referring to above mentioned issue I want to make it clear about for which layers i have done quantization.</p>
<ol>
<li>Conv1d (from nightly)</li>
<li>LayerNorm (from nightly)</li>
<li>ReLU</li>
<li>Linear</li>
</ol>
<p>additional layers:</p>
<ol>
<li>quantstub / dequantstubs</li>
<li>QFunctional / FloatFunctional</li>
</ol>
<p>All these layers are quantized and I fused Relu and Conv1d as well ( since beginning Im referring to this documentation <a href="https://pytorch.org/tutorials/advanced/static_quantization_tutorial.html" rel="nofollow noopener">Static Quantization with eager mode in pytorch</a></p>
<p>If I use FloatFunctional, I’m not using Quant/Dequantstubs in my model where arithmetic operations are triggered between quantized layers.</p>
<p>After successfully done quantizing, still my model CPU computation is not reduced instead computation has increased after quantization!</p>
<p>Could you tell me in which cases this might happen?</p>
<p>Thanks,<br>
Aravind</p>

username: supriyar
cooked: <p>For add you could use <code>torch.nn.FloatFunctional</code>, the extra dequant and quant ops in the network could be slowing things down.</p>
<p>Regarding performance you can try running the torch.autograd.profiler on your model for some iterations to see which ops take up most time. It will give you an op level breakdown with runtime so you can compare float vs quantized model.</p>

username: BOLLOJU_ARAVIND
cooked: <p>Hi <a class="mention" href="/u/supriyar">@supriyar</a>,</p>
<p>Yah, I did that replacement of quantstubs/dequantstubs with floatfunctional. I’ll drop it here</p>
<pre><code>    &gt; y = self.conv1x11(x)
</code></pre>
<blockquote>
<pre><code>    y = self.prelu1(y)
    y = self.norm1(y)
    w = self.depthwise_conv11(y)
    #w = w[:,:,:-2*2**1]
    w = w[:,:,:-4]
    y = self.depthwise_conv1(y)
    #y = y[:,:,:-2*2**1]
    y = y[:,:,:-4]
    y = self.prelu11(y)
    y = self.norm11(y)
    v = self.pointwise_conv1(y)
    #w = self.pointwise_conv_skp1(y)
    #z = self.dequant_tcnet(z)
    #w = self.dequant_tcnet(w)
    #v = self.dequant_tcnet(v)
    #x = self.dequant_tcnet(x)
    #z = z + w
    #x = x + v
    z = self.Qf_s.add(z,w)
    x = self.Qf_s.add(x,v)
    #x = self.quant_tcnet2(x)
    #z = self.quant(z)
</code></pre>
</blockquote>
<p>as you can see now I removed stubs and using FloatFunctional but yet usage is not reduced</p>
<p>Thanks <a class="mention" href="/u/supriyar">@supriyar</a>,<br>
Aravind</p>

username: BOLLOJU_ARAVIND
cooked: <p>Hi <a class="mention" href="/u/supriyar">@supriyar</a> <a class="mention" href="/u/raghuramank100">@raghuramank100</a>,</p>
<p>Regarding CPU usage i debug the usage with</p>
<aside class="quote no-group" data-username="supriyar" data-post="13" data-topic="80954">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/s/db5fbb/48.png" class="avatar"> supriyar:</div>
<blockquote>
<p>torch.autograd.profiler</p>
</blockquote>
</aside>
<p>Here I’m listing the output:</p>
<blockquote>
<p>Usage of Float model:</p>
<hr>
<p>Name                      Self CPU total %  Self CPU total   CPU total %      CPU total        CPU time avg     Number of Calls</p>
<hr>
<p><strong>slow_conv_dilated2d</strong>       25.34%           <strong>127.183ms</strong>        98.72%           495.549ms        42.081us         11776<br>
size                      15.79%           79.242ms         15.79%           79.242ms         0.463us          171097<br>
_cat                      9.76%            49.007ms         11.61%           58.285ms         2.534ms          23<br>
mkldnn_convolution        9.67%            48.534ms         19.48%           97.764ms         1.397ms          70<br>
threshold                 6.45%            32.364ms         6.52%            32.726ms         1.091ms          30<br>
slice                     5.26%            26.391ms         9.76%            49.012ms         2.065us          23737<br>
native_layer_norm         3.79%            19.017ms         7.74%            38.849ms         669.817us        58<br>
<em>convolution              3.71%            18.632ms         86.18%           432.610ms        7.459ms          58<br>
empty                     2.81%            14.117ms         2.82%            14.143ms         1.179us          11991<br>
select                    2.65%            13.278ms         9.45%            47.425ms         4.027us          11778<br>
as_strided                2.52%            12.638ms         2.52%            12.638ms         0.530us          23837<br>
fill</em>                     2.37%            11.903ms         2.38%            11.933ms         2.026us          5889<br>
add                       2.23%            11.208ms         4.53%            22.760ms         421.489us        54</p>
<p>total time : 502 ms</p>
</blockquote>
<blockquote>
<p>Cpu usage of Quantized model without fusing:</p>
<hr>
<p>Name                         Self CPU total %  Self CPU total   CPU total %      CPU total        CPU time avg     Number of Calls</p>
<hr>
<p><strong>quantized::conv1d</strong>            81.39%           <strong>454.615ms</strong>        81.39%           454.615ms        7.838ms          58<br>
quantized::add               9.25%            51.657ms         9.25%            51.657ms         1.987ms          26<br>
quantized::layer_norm        7.62%            42.582ms         7.62%            42.582ms         1.468ms          29<br>
relu                         0.63%            3.513ms          1.31%            7.296ms          121.595us        60<br>
quantized::mul               0.59%            3.322ms          0.59%            3.322ms          3.322ms          1<br>
quantized::linear            0.18%            1.019ms          0.18%            1.019ms          1.019ms          1</p>
<p>total time : 558ms</p>
</blockquote>
<blockquote>
<p>CPU usage of quantized model after fusing:</p>
<hr>
<p>Name                         Self CPU total %  Self CPU total   CPU total %      CPU total        CPU time avg     Number of Calls</p>
<hr>
<p><strong>quantized::conv1d</strong>            41.77%           <strong>239.086ms</strong>        41.77%           239.086ms        7.970ms          30<br>
<strong>quantized::conv1d_relu</strong>       39.94%           <strong>228.633ms</strong>        39.94%           228.633ms        8.165ms          28<br>
quantized::add               9.35%            53.523ms         9.35%            53.523ms         2.059ms          26<br>
quantized::layer_norm        7.67%            43.932ms         7.67%            43.932ms         1.515ms          29<br>
quantized::mul               0.62%            3.564ms          0.62%            3.564ms          3.564ms          1<br>
index_add_                   0.27%            1.542ms          0.54%            3.100ms          1.550ms          2<br>
quantized::linear            0.18%            1.017ms          0.18%            1.017ms          1.017ms          1<br>
relu                         0.06%            370.631us        0.13%            762.027us        190.507us        4</p>
<p>total time : 572 ms</p>
</blockquote>
<p>If you see these three usages conv1d is taking high usage after quantization</p>
<blockquote>
<p>quantized::conv1d            <strong>239.086ms</strong><br>
quantized::conv1d_relu         <strong>228.633ms</strong><br>
slow_conv_dilated2d         <strong>127.183ms</strong></p>
</blockquote>
<p>Could anyone have any view why this was happened i.e., quantized conv1d increased CPU usage?</p>
<p>Thanks <a class="mention" href="/u/supriyar">@supriyar</a> <a class="mention" href="/u/raghuramank100">@raghuramank100</a>,<br>
Aravind.</p>

username: BOLLOJU_ARAVIND
cooked: <p>Hi <a class="mention" href="/u/raghuramank100">@raghuramank100</a> <a class="mention" href="/u/supriyar">@supriyar</a></p>
<aside class="quote no-group" data-username="BOLLOJU_ARAVIND" data-post="15" data-topic="80954">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/bolloju_aravind/48/24107_2.png" class="avatar"> BOLLOJU_ARAVIND:</div>
<blockquote>
<p>quantized::conv1d <strong>239.086ms</strong><br>
quantized::conv1d_relu <strong>228.633ms</strong><br>
slow_conv_dilated2d <strong>127.183ms</strong></p>
</blockquote>
</aside>
<p>Do you have any idea why this quantized Conv1d takes more time?</p>
<p>Thanks <a class="mention" href="/u/raghuramank100">@raghuramank100</a> <a class="mention" href="/u/supriyar">@supriyar</a>,<br>
Aravind</p>

username: supriyar
cooked: <p>We are currently implementing the operator using quantized::conv2d under the hood after unsqueezing the activation and weight tensors. That might be proving sub-optimal for certain input shapes.</p>
<p>Could you give us details about the input and weight dimensions and the parameters (kernel, stride, pad, dilation) to conv1d in your case?</p>

username: BOLLOJU_ARAVIND
cooked: <p>Hi <a class="mention" href="/u/supriyar">@supriyar</a>,</p>
<p>Thanks for the reply.</p>
<p>Here I’m attaching some layers in my model</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/e/8/e816d030ae0c4dcd6893f993ed0169f27b337f1d.png" data-download-href="https://discuss.pytorch.org/uploads/default/e816d030ae0c4dcd6893f993ed0169f27b337f1d" title="Screenshot_2020-06-09-07-25-38-14"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/e/8/e816d030ae0c4dcd6893f993ed0169f27b337f1d_2_690x279.png" alt="Screenshot_2020-06-09-07-25-38-14" data-base62-sha1="x79EjIEZb1jmzZ4dMTsxSmvuK4J" width="690" height="279" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/e/8/e816d030ae0c4dcd6893f993ed0169f27b337f1d_2_690x279.png, https://discuss.pytorch.org/uploads/default/optimized/3X/e/8/e816d030ae0c4dcd6893f993ed0169f27b337f1d_2_1035x418.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/e/8/e816d030ae0c4dcd6893f993ed0169f27b337f1d.png 2x" data-dominant-color="F2F2F2"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Screenshot_2020-06-09-07-25-38-14</span><span class="informations">1261×511 214 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>These are the  layers in float model</p>
<p>Thanks <a class="mention" href="/u/supriyar">@supriyar</a>,<br>
Aravind.</p>