Int16 precision support on quantization

iviarcio: <p>Hello everyone,<br>
I’m using the quantization-schema=symmetric_with_power2_scale and quantization-precision=Int8 for a new device. Except for a few operators like LRN, this schema works fine. But I’d like to have Int16 precision also. I’ve been studying the “base.h”, “quantization.h” &amp; “quantization.cpp” modules but I’m a little confused on how to progress. Any help or guidance is welcome. Thanks</p>

jfix: <p>Hi <a class="mention" href="/u/iviarcio">@iviarcio</a>, we have an <code>ElemKind::Int16QTy</code>, though I don’t think it’s used too extensively. Have you read through our <a href="https://github.com/pytorch/glow/blob/master/docs/Quantization.md" rel="noopener nofollow ugc">doc on quantization</a>? Not sure exactly where your confusion lies so not sure what info you need.</p>

iviarcio: <p>Sorry <a class="mention" href="/u/jfix">@jfix</a>, but I started studying Glow recently so my questions are pretty basic. What I’ve done so far to support a new device is the following: I created a new backend based on the CPU backend, but removing code, basically from CPULLVMIRGen, and registered it. Then I rewrote transformPostLowering based on the OpenCL backend. I can compile and validate Low-level IR representation for new device requirements in Int8. Example:<br>
…/…/glow/build/bin/image-classifier images/*.png <br>
-image-layout=NCHW -image-mode=0to1 <br>
-dump-profile=profile.yaml <br>
-model=resnet50 -model-input-name=“gpu_0/data”<br>
…/…/glow/build/bin/model-compiler -load-profile=profile.yaml <br>
-model=resnet50 -model-input=“gpu_0/data”,float,[1,3,224,224] <br>
-backend=NMP -emit-bundle=bundle <br>
-quantization-schema=symmetric_with_power2_scale <br>
-quantization-precision=Int8 -quantization-precision-bias=Int8 <br>
-dump-ir &gt; resnet50.lir<br>
As you can see in the gist <a href="https://gist.github.com/iviarcio/33c6d83aba68fa8b06c49906f0920eb0" rel="noopener nofollow ugc">resnet50.lir</a> both the NCHW layout and the quantization are ok. But when I try to use Int16 for quantization, the file <a href="https://gist.github.com/iviarcio/a53cf5feef166885d998b6dee93b64ef" rel="noopener nofollow ugc">resnet50-16.lir</a> do not show the expected result. Any help or guidance is welcome. Thanks again, Jordan.</p>

jfix: <p>So it looks like nothing is being quantized. One note here is that the default CPU backend doesn’t have any Int16 quantization support AFAIK. So unless you have added that support, this is probably why you aren’t seeing any int16 in the model – the quantization flow will skip it if <code>Backend::isOpSupported()</code> returns false for it. You will need to add support for those kernels in int16 to libjit, and update your backend’s <code>isOpSupported()</code> to return true for these newly supported ops w/ int16. And there may be other issues in LLVM IR gen, not sure.</p>

iviarcio: <p>I got it. Thanks, <a class="mention" href="/u/jfix">@jfix</a>. Now I have Int16 quantization working <img src="https://discuss.pytorch.org/images/emoji/apple/grinning.png?v=9" title=":grinning:" class="emoji" alt=":grinning:"> <a href="https://gist.github.com/iviarcio/a53cf5feef166885d998b6dee93b64ef" rel="noopener nofollow ugc">resnet50-16.lir</a></p>