Quantization example resnet50

weiwei_lee: <p>Hi,<br>
I am a beginner of glow. When trying quantization example for resnet50</p>
<pre><code class="lang-auto">./bin/image-classifier tests/images/imagenet/*.png -image-mode=0to1 -m=resnet50 -model-input-name=gpu_0/data -dump-profile="profile.yaml"
</code></pre>
<p>what is the file “resnet50” I should prepare for this example?    resnet50.pth file?<br>
cuz it shows “ONNXModelLoader.cpp line161 error”</p>
<p>Thx!</p>

jfix: <p>Hi <a class="mention" href="/u/weiwei_lee">@weiwei_lee</a> – resnet50 here represents the directory containing Caffe2 or ONNX protobufs. We have a script to download some from <code>utils/download_caffe2_models.sh</code> and <code>utils/download_onnx_models.sh</code>.</p>

weiwei_lee: <p><a class="mention" href="/u/jfix">@jfix</a>  Thx for ur answer!</p>
<p>another question, how to quantize pytorch model instead?</p>

jfix: <p><a class="mention" href="/u/weiwei_lee">@weiwei_lee</a> I’m not exactly sure what the question is – do you have your own model defined in PyTorch that you want to run with Glow, and then use quantization for it?</p>

weiwei_lee: <p><a class="mention" href="/u/jfix">@jfix</a><br>
Yes, that’s my question!<br>
I wanna quantize my pytorch model directly, instead of ONNX or Caffe2 model.<br>
Does that work?</p>

jfix: <p><a class="mention" href="/u/weiwei_lee">@weiwei_lee</a> You can do this e.g. by exporting from PyTorch into ONNX, and then load the ONNX proto representation of the model into Glow – see <a href="https://pytorch.org/tutorials/advanced/super_resolution_with_caffe2.html" rel="nofollow noopener">this tutorial</a>. This of course is dependent on what we support in our ONNX importer and in Glow, so not everything you export will work right out of the box.</p>

weiwei_lee: <p>Ok, Thx <a class="mention" href="/u/jfix">@jfix</a></p>
<p>I successfully get profile.yaml and use it to quantize model<br>
however, how to save “quantized model” ?</p>

jfix: <p><a class="mention" href="/u/weiwei_lee">@weiwei_lee</a> By save, do you mean you want to save the compiled quantized model as a standalone executable for a CPU? We have directions on that <a href="https://github.com/pytorch/glow/blob/master/docs/AOT.md#creating-standalone-executable-bundles" rel="nofollow noopener">here</a>. If not, please clarify what you’re looking for.</p>

weiwei_lee: <p><a class="mention" href="/u/jfix">@jfix</a></p>
<p>Yes, that is what I want.<br>
Is there any document explain more detail about quantization flow and rule in inference stage?<br>
Because I am curious that in glow,</p>
<ol>
<li>it load model and yaml to transform weight of model into int8?</li>
<li>Input to each layer is float or int8?</li>
<li>is the multiplication rule like tflite using gemmlowp to do int8*int8?</li>
</ol>

jfix: <p><a class="mention" href="/u/weiwei_lee">@weiwei_lee</a></p>
<p>Yes, please take a look at all of our docs – I think they mostly seem that they will answer your questions. <a href="https://github.com/pytorch/glow/blob/master/docs/Quantization.md#quantization-in-glow" rel="nofollow noopener">Here is the doc on quantization</a>. You gather a profile on the floating point graph with whatever inputs you want, it dumps the yaml file, then you load it back in and it will quantize the graph given the profile you gathered. Assuming the CPU backend supports the node in quantized form, all inputs will be int8. Otherwise conversions to/from int8 will be performed. You can look at the graph that is generated to see what is and is not quantized, via the command line option <code>-dump-graph-DAG="graph.dot"</code>.</p>

weiwei_lee: <p><a class="mention" href="/u/jfix">@jfix</a></p>
<p>Thanks for your clearly explanation!</p>

Stella: <p>I am trying to quantize another pytorch model with the approach described in this post. I have glow build and model converted to .onnx , what should I put as the -model-input-name? when i do<br>
./bin/image-classifier tests/images/imagenet/*.png -image-mode=0to1 -m=‘path/to/my/model’ -model-input-name=gpu_0/data -dump-profile=“profile.yaml”<br>
I got error file: ~/glow/lib/Importer/ProtobufLoader.cpp line: 33 message: There is no tensor registered with name 0.<br>
I assume it’s the input name was incorrect?</p>
<p>Thanks!</p>

jfix: <p>Yes, it’s probably due to <code>-model-input-name</code>. You’ll need to take a look at the proto itself to find out the name of the external input that is the image itself, which is the input to the first operator of the model. In most of our image classification models, this is <code>data</code> or <code>gpu_0/data</code>. You can see some examples in our image classifier run script <a href="https://github.com/pytorch/glow/blob/master/tests/images/run.sh" rel="nofollow noopener">here</a>.</p>

Stella: <p><a class="mention" href="/u/jfix">@jfix</a> Thank you for the quick respond! How do I get the input name of the model?  I have tried print(model) and for name, param in model.named_parameters(): print (name, param.data.shape) , both start from the actual layers, not including the input.  sorry it’s a basic question.</p>

jfix: <p>You could use something like <a href="https://electronjs.org/apps/netron" rel="noopener nofollow ugc">Netron</a> to view your protobuf, and view what the very first operator’s input is (see the image below, for the very start of a Caffe2 Resnet50 model – you’d use <code>gpu_0/data</code>). Otherwise you should be able to just inspect the protobuf text of the model to see what the input name of the first operator is – it should be external input.</p>
<p><img src="https://discuss.pytorch.org/uploads/default/original/2X/5/55cd761bbb47b05d963597581fee30830047d639.png" alt="08%20PM" data-base62-sha1="cf2MBoQoY7CHxI001nv0QMe2hh7" width="241" height="217"></p>

Stella: <p>Thank you for your help <a class="mention" href="/u/jfix">@jfix</a>! That works wonders!<br>
However got an error eventually glow/lib/Importer/ONNXModelLoader.cpp line: 896 error code: MODEL_LOADER_UNSUPPORTED_OPERATOR message: Failed to load operator. I guess it means some of the operations in my model are supported for quantization at this point?</p>

jfix: <p>Yeah, some op is unsupported – what options were you using for the <code>image-classifier</code>? Was this when trying to get the profile (<code>-dump-profile</code>), or load the profile for quantization (<code>-load-profile</code>)? Have you tried just running it in fp32 (i.e. without the mentioned options)?</p>
<p>For <code>MODEL_LOADER_UNSUPPORTED_OPERATOR</code>, the operator is unsupported in fp32 too, and we’d need to add support to the importer for it. We need to improve our error messages here – in the meantime could you just add a simple print statement just before the <code>RETURN_ERR()</code> in <code>ONNXModelLoader::LoadOperator()</code> to print the <code>typeName</code> to see what it is reporting as unsupported.</p>
<p>Also, feel free to open an issue for supporting whatever op it is on Github!</p>

Stella: <p>Thanks <a class="mention" href="/u/jfix">@jfix</a>  for clarifying it! I got the error when doing ./bin/image-classifier tests/images/imagenet/*.png -image-mode=0to1 -m=‘path/to/my/model’ -model-input-name=gpu_0/data -dump-profile=“profile.yaml” .<br>
The model I am trying to quantize with is mobilefacenet, made with grouped convolutions and dense blocks.</p>

jfix: <p>Got it, cool. Well like I said, if you find out what op is being reported as unsupported please open an issue with it on GH.</p>

AlexKoff88: <p>Hi,</p>
<p>Look what is available here: <a href="https://github.com/opencv/openvino_training_extensions/tree/develop/pytorch_toolkit/nncf" rel="nofollow noopener">https://github.com/opencv/openvino_training_extensions/tree/develop/pytorch_toolkit/nncf</a>.</p>
<p>This is a Quantization Aware Training in PyTorch with ability to export the quantized model to ONNX.</p>
<p>There are many results there including ResNet-50 ready to use config for quantization.</p>