How to export the weight after quantization

username: Nvcenter
cooked: <p>Hi, I have defined a neural network with a fully connected layer and applied Post Training Static Quantization for quantization. The version I use for pytorch is 2.0.0+cu118.<br>
Here is the network architecture and the quantization process:</p>
<pre><code class="lang-auto">class HPC(nn.Module):
    def __init__(self, input_features, out_features): 
        super(HPC, self).__init__()
        self.linear1 = nn.Linear(input_features, 512) 
        self.linear2 = nn.Linear(512, 256)
        self.linear3 = nn.Linear(256, 64)
        self.linear4 = nn.Linear(64, out_features)
        self.sigmoid = nn.Sigmoid()
        self.quant = torch.ao.quantization.QuantStub()
        self.dequant = torch.ao.quantization.DeQuantStub()

    def forward(self, n_input): 
        out = self.quant(n_input)
        out = (self.linear1(out))
        out = (self.linear2(out))
        out = (self.linear3(out)) 
        out = self.sigmoid(self.linear4(out))
        out = self.quant(out)
        return out

</code></pre>
<pre><code class="lang-auto">import copy
#copy the trained model
myModel = copy.deepcopy(model)
myModel.eval()
myModel.qconfig = torch.ao.quantization.default_qconfig
print(myModel.qconfig)
torch.ao.quantization.prepare(myModel, inplace=True)
for i in range(200):
  X_train_arr_tensor = torch.Tensor(X_train_arr[i].flatten()).cuda()
  X_train_arr_tensors = torch.unsqueeze(X_train_arr_tensor, 0)
  myModel(X_train_arr_tensors)
torch.ao.quantization.convert(myModel, inplace=True)
</code></pre>
<p>When I completed the above operation, I tried to print out the quantized network parameters and the results are shown below.</p>
<pre><code class="lang-auto">for i in myModel.state_dict():
    print(i)
</code></pre>
<blockquote>
<p>linear1.scale<br>
linear1.zero_point<br>
linear1._packed_params.dtype<br>
linear1._packed_params._packed_params<br>
linear2.scale<br>
linear2.zero_point<br>
linear2._packed_params.dtype<br>
linear2._packed_params._packed_params<br>
linear3.scale<br>
linear3.zero_point<br>
linear3._packed_params.dtype<br>
linear3._packed_params._packed_params<br>
linear4.scale<br>
linear4.zero_point<br>
linear4._packed_params.dtype<br>
linear4._packed_params._packed_params<br>
leaky.scale<br>
leaky.zero_point<br>
quant.scale<br>
quant.zero_point</p>
</blockquote>
<p>When I printed out myModel.linear1._packed_params, I found that the scale and zero_point values in _packed_params were not equal to myModel.linear1.scale and myModel.linear1.zero_point.</p>
<pre><code class="lang-auto">print(myModel.linear1._packed_params)
</code></pre>
<blockquote>
<p>(tensor([[-0.0381, -0.0122,  0.0366,  …, -0.0059,  0.0067, -0.0362],<br>
[ 0.0196,  0.0037, -0.0277,  …,  0.0074, -0.0362,  0.0126],<br>
…,<br>
[ 0.0126,  0.0403, -0.0104,  …,  0.0359, -0.0107,  0.0263]],<br>
device=‘cuda:0’, size=(512, 504), dtype=torch.qint8,<br>
quantization_scheme=torch.per_tensor_affine, <strong>scale=0.00036982630263082683,</strong><br>
<strong>zero_point=0</strong>), Parameter containing:<br>
tensor([-2.7267e-02, -4.1525e-02,  3.3713e-02,…, -3.9943e-02], device=‘cuda:0’, requires_grad=True))</p>
</blockquote>
<pre><code class="lang-auto">print("this is scale: ",myModel.linear1.scale)
print("this is zero_point:",myModel.linear1.zero_point)
</code></pre>
<blockquote>
<p>this is scale:  0.030707480385899544<br>
this is zero_point: 63</p>
</blockquote>
<p>There are no quantized weights and biases for the fully connected layers here. May I know how can I obtain the weights and parameters for each layer?</p>

username: AbdulsalamBande
cooked: <p>After quantization, the weights are stored in a packed format to enable efficient execution. The weights in the packed format are not directly accessible as regular tensors. Try this function.</p>
<pre><code class="lang-auto">def print_quantized_weights(model):
    for name, module in model.named_modules():
        if isinstance(module, nn.quantized.Linear):
            # Unpack the weight and bias
            weight, bias = module._packed_params.unpack()
            
            # Dequantize the weights to get the float values
            weight = weight.dequantize()
            
            print(f"{name}.weight:", weight)
            print(f"{name}.bias:", bias)
            print(f"{name}.scale:", module.scale)
            print(f"{name}.zero_point:", module.zero_point)
            print()

print_quantized_weights(myModel)

</code></pre>

username: Nvcenter
cooked: <p>Thanks bro, i will try it</p>

username: Nvcenter
cooked: <p>Thank you very much, you have solved a problem that has been troubling me for a long time! <img src="https://discuss.pytorch.org/images/emoji/apple/smiley.png?v=12" title=":smiley:" class="emoji" alt=":smiley:" loading="lazy" width="20" height="20"></p>