Unpacking Quantized model in C++ with torchscript format

k.osama: <p>Hi I am working on a quantized model in C++. I have trained and quantized the model in Python and loaded to C++ (post training quantization). I wonder if I can parse the jitted model parameters (torchscript format) in C++ ? I could not find any layer-unpacking modules in  torch::jit::script::Module .<br>
After loading the model, I can dump the scriptModule modules and parameters using (torch::jit::script::Module) m-&gt;dump():</p>
<pre><code>--------------------------------------------------------------------------------------------------------------------
dumping module module __torch__.Net {
	  parameters {
	  }
	  attributes {
		training = False
	(Here-&gt;)  fc1 = &lt;__torch__.torch.nn.intrinsic.quantized.modules.linear_relu.LinearReLU object at 0x5555566b75f0&gt;
		Relu1 = &lt;__torch__.torch.nn.modules.linear.Identity object at 0x5555566b3d40&gt;
		fc2 = &lt;__torch__.torch.nn.intrinsic.quantized.modules.linear_relu.LinearReLU object at 0x5555566a7030&gt;
		Relu2 = &lt;__torch__.torch.nn.modules.linear.Identity object at 0x5555567b1440&gt;
		droput2 = &lt;__torch__.torch.nn.modules.dropout.Dropout object at 0x5555567b1640&gt;
		fc3 = &lt;__torch__.torch.nn.quantized.modules.linear.Linear object at 0x5555567b22c0&gt;
		quant = &lt;__torch__.torch.nn.quantized.modules.Quantize object at 0x5555567b2ad0&gt;
		dequant = &lt;__torch__.torch.nn.quantized.modules.DeQuantize object at 0x5555567b8350&gt;
		logMax = &lt;__torch__.torch.nn.modules.activation.LogSoftmax object at 0x5555567b87c0&gt;
	  }
	  methods {
		method forward {
		  graph(%self.1 : __torch__.Net,
				%x.1 : Tensor):
			%7 : int = prim::Constant[value=-1]() # ~//MNIST_PyTorch_Quantize.py:50:19
			%8 : int = prim::Constant[value=784]() # ~//MNIST_PyTorch_Quantize.py:50:23
			%3 : __torch__.torch.nn.quantized.modules.Quantize = prim::GetAttr[name="quant"](%self.1)
			%x0.1 : Tensor = prim::CallMethod[name="forward"](%3, %x.1) # :0:0
			%9 : int[] = prim::ListConstruct(%7, %8)
			%x1.1 : Tensor = aten::view(%x0.1, %9) # ~//MNIST_PyTorch_Quantize.py:50:12
			%12 : __torch__.torch.nn.intrinsic.quantized.modules.linear_relu.LinearReLU = prim::GetAttr[name="fc1"](%self.1)
			%x2.1 : Tensor = prim::CallMethod[name="forward"](%12, %x1.1) # :0:0
			%16 : __torch__.torch.nn.modules.linear.Identity = prim::GetAttr[name="Relu1"](%self.1)
			%x3.1 : Tensor = prim::CallMethod[name="forward"](%16, %x2.1) # :0:0
			%20 : __torch__.torch.nn.intrinsic.quantized.modules.linear_relu.LinearReLU = prim::GetAttr[name="fc2"](%self.1)
			%x4.1 : Tensor = prim::CallMethod[name="forward"](%20, %x3.1) # :0:0
			%24 : __torch__.torch.nn.modules.linear.Identity = prim::GetAttr[name="Relu2"](%self.1)
			%x5.1 : Tensor = prim::CallMethod[name="forward"](%24, %x4.1) # :0:0
			%28 : __torch__.torch.nn.modules.dropout.Dropout = prim::GetAttr[name="droput2"](%self.1)
			%x6.1 : Tensor = prim::CallMethod[name="forward"](%28, %x5.1) # :0:0
			%32 : __torch__.torch.nn.quantized.modules.linear.Linear = prim::GetAttr[name="fc3"](%self.1)
			%x7.1 : Tensor = prim::CallMethod[name="forward"](%32, %x6.1) # :0:0
			%36 : __torch__.torch.nn.quantized.modules.DeQuantize = prim::GetAttr[name="dequant"](%self.1)
			%x8.1 : Tensor = prim::CallMethod[name="forward"](%36, %x7.1) # :0:0
			%40 : __torch__.torch.nn.modules.activation.LogSoftmax = prim::GetAttr[name="logMax"](%self.1)
			%42 : Tensor = prim::CallMethod[name="forward"](%40, %x8.1) # :0:0
			return (%42)
		}
	  }
	  submodules {
		module __torch__.torch.nn.intrinsic.quantized.modules.linear_relu.LinearReLU {
		  parameters {
		  }
		  attributes {
			training = False
			in_features = 784
			out_features = 512
			scale = 0.048926487565040588
			zero_point = 0
			_packed_params = &lt;__torch__.torch.nn.quantized.modules.linear.LinearPackedParams object at 0x5555566bcc40&gt;
		  }
		  methods {
			method forward {
			  graph(%self.1 : __torch__.torch.nn.intrinsic.quantized.modules.linear_relu.LinearReLU,
					%input.1 : Tensor):
				%4 : __torch__.torch.nn.quantized.modules.linear.LinearPackedParams = prim::GetAttr[name="_packed_params"](%self.1)
				%5 : Tensor = prim::GetAttr[name="_packed_params"](%4)
				%7 : float = prim::GetAttr[name="scale"](%self.1)
				%9 : int = prim::GetAttr[name="zero_point"](%self.1)
				%Y_q.1 : Tensor = quantized::linear_relu(%input.1, %5, %7, %9) # ~/python3.7/site-packages/torch/nn/intrinsic/quantized/modules/linear_relu.py:29:14
					return (%Y_q.1)
			}
		  }
		  submodules {
		   (+)        module __torch__.torch.nn.quantized.modules.linear.LinearPackedParams {
		  }
		}
		 module __torch__.torch.nn.modules.linear.Identity {} (+)
		 module __torch__.torch.nn.intrinsic.quantized.modules.linear_relu.LinearReLU {} (+)
		 module __torch__.torch.nn.modules.linear.Identity {} (+)
		 module __torch__.torch.nn.modules.dropout.Dropout {} (+)
		 module __torch__.torch.nn.quantized.modules.linear.Linear {} (+) 
		 module __torch__.torch.nn.quantized.modules.Quantize {} (+) 
		 module __torch__.torch.nn.quantized.modules.DeQuantize {}  (+)
		 module __torch__.torch.nn.modules.activation.LogSoftmax {} (+)
		} //   end of submodules 
		}  //   end of  dumping module module __torch__.Net
	--------------------------------------------------------------------------------------------------------
</code></pre>
<p>Notes: (+) means there are collapsed lines omitted to save space.<br>
Torch version 1.6.0+.<br>
My Questions:<br>
1- During model load, are the module layers packed again to a format like in torch::nn::Linear and torch::nn::Conv1d … etc? how can I access them?<br>
2- Are the pointers printed in the dump (like the line marked with “(Here-&gt;)”  ) for Python objects and methods or they are C++ objects and methods? are they cast-able to the formats in torch::nn:* ?<br>
3- What is the recommended procedure to structure the model again in terms of the number of layers , the attributes\configuration of each layer and the corresponding trained weights from the jitted\torchscript format ?</p>

jerryzh168: <blockquote>
<p>1- During model load, are the module layers packed again to a format like in torch::nn::Linear and torch::nn::Conv1d … etc? how can I access them?</p>
</blockquote>
<p>what is <code>torch::nn::Linear</code>? are you talking about the pytorch c++ API? the weights are packed in linear modules, you can use <a href="https://github.com/pytorch/pytorch/blob/master/torch/nn/quantized/modules/linear.py#L34" rel="noopener nofollow ugc">https://github.com/pytorch/pytorch/blob/master/torch/nn/quantized/modules/linear.py#L34</a> to get the unpacked weights.</p>
<blockquote>
<p>Are the pointers printed in the dump (like the line marked with “(Here-&gt;)” ) for Python objects and methods or they are C++ objects and methods? are they cast-able to the formats in torch::nn:* ?</p>
</blockquote>
<p>These are TorchScript objects I think. I’m not sure about the relationship between the pytorch c++ API and TorchScript, cc <a class="mention" href="/u/michael_suo">@Michael_Suo</a> could you comment on this?</p>
<blockquote>
<p>What is the recommended procedure to structure the model again in terms of the number of layers , the attributes\configuration of each layer and the corresponding trained weights from the jitted\torchscript format ?</p>
</blockquote>
<p>Not sure I understand the question, could you be more concrete?</p>

k.osama: <aside class="quote no-group" data-username="jerryzh168" data-post="2" data-topic="90188">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/jerryzh168/48/15217_2.png" class="avatar"> jerryzh168:</div>
<blockquote>
<p>what is <code>torch::nn::Linear</code> ? are you talking about the pytorch c++ API? the weights are packed in linear modules, you can use <a href="https://github.com/pytorch/pytorch/blob/master/torch/nn/quantized/modules/linear.py#L34" rel="noopener nofollow ugc">https://github.com/pytorch/pytorch/blob/master/torch/nn/quantized/modules/linear.py#L34 </a> to get the unpacked weights.</p>
</blockquote>
</aside>
<p>Yes, I mean C++ API. As mentioned in description, I have trained the model in Python and exported it as torchscript to C++. Is the function “_weight_bias” accessible in torchscript format () (I can see that it is has the decorator <span class="mention">@torch.jit.export</span>) ? If yes, can you show please how to use it with loaded torchscript models ?<br>
Rephrase 3-: Given a model that is trained in python, saved as “torchscript”  and loaded into C++ front end, I want to extract the number of layers, the type of each layer (Linear,conv1d…), the size of each layer and weights associated with each layer… How can I do that ?</p>

jerryzh168: <aside class="quote no-group" data-username="k.osama" data-post="3" data-topic="90188">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/k.osama/48/27010_2.png" class="avatar"> k.osama:</div>
<blockquote>
<p>torchscript format () (I can see that it is has the decorator <span class="mention">@torch.jit.export</span>) ? If yes, can you show please how to use</p>
</blockquote>
</aside>
<p>here is an example calling method in a TorchScript Moudule: <a href="https://codebrowser.bddppq.com/pytorch/pytorch/torch/csrc/jit/passes/quantization/insert_quant_dequant.cpp.html#978" class="inline-onebox" rel="noopener nofollow ugc">insert_quant_dequant.cpp source code [pytorch/torch/csrc/jit/passes/quantization/insert_quant_dequant.cpp] - Woboq Code Browser</a></p>
<p>if by C++ API you mean TorchScript then this should work. There is another C++ API that is authoring models in C++(<a href="https://pytorch.org/docs/stable/cpp_index.html#authoring-models-in-c" class="inline-onebox" rel="noopener nofollow ugc">C++ — PyTorch 2.1 documentation</a>) which I’m not familiar with.</p>
<p>for question 3: API of torch::jit::Module can be found in <a href="https://codebrowser.bddppq.com/pytorch/pytorch/torch/csrc/jit/api/module.h.html#torch::jit::Module" class="inline-onebox" rel="noopener nofollow ugc">module.h source code [pytorch/torch/csrc/jit/api/module.h] - Woboq Code Browser</a></p>