Current status of automatic quantization support

username: masahi
cooked: <p>Hi, I remember from one of the talks at the last dev con or elsewhere that PyTorch v1.4 would ship with support for automatic quantization of jitted models. v1.4 is out now, but it is not clear from looking at the release note whether this feature is official.</p>
<p>Is automatic quantization ready for experimental use? Can it already quantize an entire network from torchvision for example?</p>
<p>Thanks<br>
masa</p>

username: raghuramank100
cooked: <p>Yes, we are very close to having graph mode ready, there are still a few changes that havent been completed yet. (cc <a class="mention" href="/u/jerryzh168">@jerryzh168</a> for updates)</p>

username: jerryzh168
cooked: <p>Yes, graph mode quantization will be ready for testing after <a href="https://github.com/pytorch/pytorch/pull/32303" rel="nofollow noopener">https://github.com/pytorch/pytorch/pull/32303</a> lands.<br>
Next we are going to test more models and fix the issues we see in the process, this might take a few more weeks to a month or so, but in the meantime, feel free to checkout the master and test the graph mode quantization on your model as well.<br>
I’ll reply here when this is ready for trying out.</p>

username: masahi
cooked: <p>Thanks for the replies. I’m already using the master build so I’ll try it out. I have an image segmentation model to test, it always uncovers issues that folks working with imagenet models do not see.</p>
<p><a class="mention" href="/u/jerryzh168">@jerryzh168</a> Are you open for contribution? I’ve never contributed to such a large project as PyTorch, but I’m interested in the opportunity.</p>

username: jerryzh168
cooked: <p>Yes contributions are very welcome, especially expanding the coverage of the graph mode quantization to more models. Currently we are still trying to make the backbone support work, so a good point might be after <a href="https://github.com/pytorch/pytorch/pull/32816" rel="nofollow noopener">https://github.com/pytorch/pytorch/pull/32816</a>. After this PR we are already pretty close to eager mode result, there might be more PRs coming up but shouldn’t be too many.</p>

username: masahi
cooked: <p>Hi <a class="mention" href="/u/jerryzh168">@jerryzh168</a>, I tried graph mode quantization test cases in test_quantization.py. In <code>test_conv</code>, I dumped the quantized IR like so:</p>
<pre><code class="lang-auto">model_quantized = quantize_script(
    model_under_test,
    qconfig_dict,
    default_eval_fn,
    [self.img_data],
    inplace=False)
self.assertEqual(model_quantized(self.img_data[0][0]), result_eager)

torch._C._jit_pass_inline(model_quantized.graph)
print(model_quantized.graph)
</code></pre>
<p>But the output is confusing. There are some quantization related operators inserted, but they are just doing quantize/dequantize round trip and the actual convolution is done in fp32.</p>
<pre><code class="lang-auto">graph(%self.1 : __torch__.torch.nn.modules.module.___torch_mangle_7.Module,
      %input : Float(2, 3, 10, 10)):
  %2 : __torch__.torch.nn.modules.module.___torch_mangle_8.Module = prim::GetAttr[name="conv"](%self.1)
  %4 : float = prim::GetAttr[name="input.1_scale"](%2)
  %5 : int = prim::GetAttr[name="input.1_zero_point"](%2)
  %6 : int = prim::GetAttr[name="input.1_scalar_type"](%2)
  %input.1.quant : Tensor = aten::quantize_per_tensor(%input, %4, %5, %6)
  %input.1.dequant.0 : Tensor = aten::dequantize(%input.1.quant)
  %9 : bool = prim::Constant[value=1](), scope: __module.conv # /home/masa/work/pytorch/pytorch/torch/nn/modules/conv.py:345:0
  %10 : bool = prim::Constant[value=0](), scope: __module.conv # /home/masa/work/pytorch/pytorch/torch/nn/modules/conv.py:345:0
  %11 : int = prim::Constant[value=1](), scope: __module.conv # /home/masa/work/pytorch/pytorch/torch/nn/modules/conv.py:345:0
  %12 : None = prim::Constant(), scope: __module.conv
  %13 : Tensor = prim::GetAttr[name="weight"](%2)
  %14 : float = prim::GetAttr[name="8_scale"](%2)
  %15 : int = prim::GetAttr[name="8_zero_point"](%2)
  %16 : int = prim::GetAttr[name="8_scalar_type"](%2)
  %8.quant : Tensor = aten::quantize_per_tensor(%13, %14, %15, %16)
  %18 : int[] = prim::Constant[value=[1, 1]]()
  %19 : int[] = prim::Constant[value=[0, 0]]()
  %20 : int[] = prim::Constant[value=[1, 1]]()
  %21 : int[] = prim::Constant[value=[0, 0]]()
  %22 : Tensor = quantized::conv2d_prepack(%8.quant, %12, %18, %19, %20, %11)
  %23 : Tensor, %24 : Tensor? = quantized::conv2d_unpack(%22)
  %25 : Tensor = aten::dequantize(%23)
  %26 : Tensor = aten::conv2d(%input.1.dequant.0, %25, %24, %18, %19, %20, %11)
  %27 : float = prim::GetAttr[name="15_scale"](%2)
  %28 : int = prim::GetAttr[name="15_zero_point"](%2)
  %29 : int = prim::GetAttr[name="15_scalar_type"](%2)
  %15.quant : Tensor = aten::quantize_per_tensor(%26, %27, %28, %29)
  %15.dequant.0 : Tensor = aten::dequantize(%15.quant)
  return (%15.dequant.0)
</code></pre>
<p>Is this expected? I was hoping to see <code>quantized::conv2d</code> op there.</p>
<p>I’m using torch built from source and the commit is at:</p>
<pre><code class="lang-auto">In [2]: torch.__version__                                                                                                                                                                     
Out[2]: '1.5.0a0+c75d06d'
</code></pre>

username: jerryzh168
cooked: <aside class="quote no-group" data-username="masahi" data-post="6" data-topic="66905">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/masahi/48/19725_2.png" class="avatar"> masahi:</div>
<blockquote>
<p>I’m using torch built from source and the commit is at:</p>
<pre><code class="lang-auto">
</code></pre>
</blockquote>
</aside>
<p>Yes, graph mode quantization is still in active development, we should be getting something working very soon though, please stay tuned.</p>