Steps/ideas for layer-wise quantization implementation

whiteking64: <p>Hi,<br>
I am interested in implementing layer-wise quantization in glow.<br>
I have looked through the issue just about this topic (<a href="https://github.com/pytorch/glow/issues/4982" class="inline-onebox" rel="noopener nofollow ugc">[Quantization] layer-wise quantization · Issue #4982 · pytorch/glow · GitHub</a>), but it seems to be left untouched.</p>
<p>My question is, given node list of type <code>glow::Node</code> (or node name list e.g. <code>Conv_conv_1__2</code>) in a model, and each node has its quantization options respectively (for example, these options can be <code>enableChannelwiseOpt</code>, <code>quantizationCalibrationOpt</code>, <code>keepOriginalPrecisionForNodesOpt</code>, …), how do we incorporate these info to the code? Below I will explain a little more backgrounds.</p>
<p>Model profiling or model compiling, whichever it is, glow takes roughly three steps for the model loader:</p>
<ol>
<li>load the model (<code>loader.loadModel()</code>)</li>
<li>get compilation settings (<code>loader.getCompilationContext(QuantizationMode)</code>)</li>
<li>compile the model using the <code>CompilationContext</code> config (<code>loader.compile(cctx)</code>)</li>
</ol>
<p>For simplicity, I think the first thing to do is load and add quantization options to all of the nodes in a model, in <code>2.</code>. To contain these info, we can add a map to <code>precConfig</code> of type <code>PrecisionConfiguration</code>. For example, the map’s key is the node names, and the value is also a map, with the key quantization options and the value the actual preferences. This is done by modifying <code>Loader::getCompilationContext</code> function in <code>Loader.cpp</code>.</p>
<p>Then, modify <code>transformForPrecisionMode</code> function in <code>GraphOptimizer.cpp</code>, as mentioned at the issue link above. This function is called in <code>optimizeFunction</code> in the same file, which is called in <code>Loader::compile</code> function in step <code>3.</code> (I suppose).</p>
<p>In <code>transformForPrecisionMode</code>, if we conduct profiling, <code>profileQuantization</code> function in <code>GraphOptimizer/Quantization.cpp</code> will be called. Or if we want to execute quantization, <code>quantizeFunction</code> function in <code>quantization/Quantization.cpp</code> will be called.</p>
<p>Either way, I am not sure exactly where to relate the above layer-wise config to these functions. I don’t see modification points in the former case (profiling), because profiling nodes themselves being inserted are not related to configs like calibration method and precision selection, and these are for executing quantization. The latter, however, since the <code>quantizeFunction</code> function creates an <code>FunctionQuantizer</code> instance inside it, I know there are some member functions to be modified in that class. Although I looked at the class, it"s a little complex and I didn’t manage to find where to implement.</p>
<p>Also, <code>glow::lower</code> function in <code>Lower.cpp</code> needs some expansion, since <code>KindSet</code> <code>doNotLowerKinds</code> is included there. What do you think?</p>
<p>To sum up, it is not clear to me that what functions called in/after <code>transformForPrecisionMode</code> should be modified, and how. Additionally if any functions other than <code>transformForPrecisionMode</code> needs to be changed, I would like to know what they are.</p>
<p>Any hints or thoughts?</p>

jfix: <p>We actually have <a href="https://github.com/pytorch/glow/pull/5145" rel="noopener nofollow ugc">https://github.com/pytorch/glow/pull/5145</a> which is pretty similar and I should land soon. Does it help solve what you’re looking for?</p>

whiteking64: <p>Thank you! I didn’t know that PR.<br>
I looked at that PR and, what it is doing is, as I understand it, basically insert a <code>createConvertTo</code> node  into <code>NodeList nodes_</code> during loading the protobuf and optimizing the graph.</p>
<p>The <code>createConvertTo</code> node traces back to <code>Node::addResult</code> function in <code>Node.cpp</code>. It does not seem to actually convert the type of the tensor in a node into the desired type (like FP16). Where does this converting process undertaken ? In some quantization nodes? Maybe I don’t understand the difference between <code>ConvertToNode</code> and <code>QuantizeNode</code>.</p>
<p>Also, since what I ultimately want to do here is to quantize each node with each (different) quantization preference (e.g. calibration technique, quantization precision, etc.), should we additionally modify <code>quantization::quantizeFunction</code>? I assume the above code you shared is not enough for executing quantization on compilation.</p>

jfix: <p>You are correct – that PR needs an extension to include quantization specification. Basically we would want to include some spec in the yaml file there to further specify the dtype and scale/bias to use etc.</p>

whiteking64: <p>Thank you very much for the explanation!</p>