Loading state to a quantized Inception_v3 model

Nimrod_Daniel: <p>Hello,</p>
<p>I’ve read the excellent static quantization tutorial, which worked really well with my MobileNet_v2  pretrained weights<br>
<a href="https://pytorch.org/tutorials/advanced/static_quantization_tutorial.html?highlight=quantization" class="onebox" target="_blank" rel="noopener nofollow ugc">https://pytorch.org/tutorials/advanced/static_quantization_tutorial.html?highlight=quantization</a></p>
<p>Now I want to do the same with Inception_v3 model. Thing is,  when I try to load the state dictionary I get the following error (full trace bellow), why and how do I fix that? Do I need to create a new class as they did in the tutorial?</p>
<blockquote>
<p>per_channel_quantized_model = torchvision.models.quantization.inception_v3(pretrained=True, aux_logits=False, quantize=True)<br>
per_channel_quantized_model.fc = nn.Linear(2048, 2)<br>
state_dict = torch.load(float_model_file, map_location=torch.device(‘cpu’))<br>
per_channel_quantized_model.load_state_dict(state_dict)</p>
</blockquote>
<blockquote>
<p>File “”, line 5, in <br>
per_channel_quantized_model.load_state_dict(state_dict)<br>
File “/home/nimrod/anaconda3/lib/python3.7/site-packages/torch/nn/modules/module.py”, line 1030, in load_state_dict<br>
load(self)<br>
File “/home/nimrod/anaconda3/lib/python3.7/site-packages/torch/nn/modules/module.py”, line 1028, in load<br>
load(child, prefix + name + ‘.’)<br>
File “/home/nimrod/anaconda3/lib/python3.7/site-packages/torch/nn/modules/module.py”, line 1028, in load<br>
load(child, prefix + name + ‘.’)<br>
File “/home/nimrod/anaconda3/lib/python3.7/site-packages/torch/nn/modules/module.py”, line 1025, in load<br>
state_dict, prefix, local_metadata, True, missing_keys, unexpected_keys, error_msgs)<br>
File “/home/nimrod/anaconda3/lib/python3.7/site-packages/torch/nn/quantized/modules/conv.py”, line 120, in _load_from_state_dict<br>
state_dict[prefix + ‘weight’], state_dict[prefix + ‘bias’])<br>
KeyError: ‘Conv2d_1a_3x3.conv.bias’</p>
</blockquote>

Vasiliy_Kuznetsov: <p>Using eager mode quantization often changes the model hierarchy of the model, so it’s possible that the module hierarchy of your quantized model no longer matches your fp32 state_dict. What is the origin of the fp32 state_dict, is it also from torchvision?</p>
<p>In practice, people usually fix this by either loading the weights before fusing the model, or by writing custom state dict mappers which modify the state keys according to the module hierarchy changes.</p>

Nimrod_Daniel: <p>Yes, the original state dict is a fp32 state_dict, a torchvision model.</p>
<p>I loaded the state_dict before fusing the model, I fuse it only afterward. So writing a state_dict mappers (or creating the model class “from scratch” as they did in the tutorial) is the only solution?<br>
Can you give an example of how I can create such a mapper?</p>

Vasiliy_Kuznetsov: <aside class="quote no-group" data-username="Nimrod_Daniel" data-post="1" data-topic="108187">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/nimrod_daniel/48/10282_2.png" class="avatar"> Nimrod_Daniel:</div>
<blockquote>
<p>per_channel_quantized_model = torchvision.models.quantization.inception_v3(pretrained=True, aux_logits=False, quantize=True)</p>
</blockquote>
</aside>
<blockquote>
<p>per_channel_quantized_model = torchvision.models.quantization.inception_v3(pretrained=True, aux_logits=False, quantize=True)</p>
</blockquote>
<p>this line should already load a pretrained model.  To clarify, is the reason you are loading weights again is to populate the new fc layer, or something else?</p>

Nimrod_Daniel: <p>It loads imagenet weights, but I need a different representation, not just the fc layer, meaning the weights of the whole net. You can ignore the pretrained value.</p>

Vasiliy_Kuznetsov: <p>Makes sense.  The <code>torchvision.models.quantization.inception_v3(pretrained=True, aux_logits=False, quantize=True)</code> line is torchvision’s best effort to provide a pretrained model ready for quantization for use cases where the default fp32 pretrained weights are fine.  Unfortunately, if you need to load a different version of floating point weights, a mapping of the state dict is required.</p>
<p>Here is a code snippet which does this for an unrelated model, but the principle is the same:</p>
<pre><code class="lang-auto">        def get_new_bn_key(old_bn_key):
            # tries to adjust the key for conv-bn fusion, where
            # root
            #   - conv
            #   - bn
            #
            # becomes
            #
            # root
            #   - conv
            #     - bn
            return old_bn_key.replace(".bn.", ".conv.bn.")

        non_qat_to_qat_state_dict_map = {}
        for key in original_state_dict.keys():
            if key in new_state_dict.keys():
                non_qat_to_qat_state_dict_map[key] = key
            else:
                maybe_new_bn_key = get_new_bn_key(key)
                if maybe_new_bn_key in new_state_dict.keys():
                    non_qat_to_qat_state_dict_map[key] = maybe_new_bn_key
        ...
        # when loading the state dict, use the mapping created above
</code></pre>
<p>We are planning to release a tool soon (hopefully v1.8) to automate all of this, so this should get easier in the near future.</p>

Nimrod_Daniel: <p>Yeah, I get the idea. You copy the keys if they with the same name, or change the name and then copy the keys in the new map. Afterward, you load the weights to the quantized model according to the new mapping.</p>
<p>If I look at the Inception and Quantized Inception’s state_dict  (before fusing, of course) then they have completely different names, and also a different length of state_dict. How can I tackle that ?</p>

Vasiliy_Kuznetsov: <p>Here is a high level of the differences:</p>
<pre><code class="lang-auto"># fp32
# torchvision.models.inception_v3(pretrained=False, aux_logits=False)
# state_dict keys of Conv2d_1a_3x3
# conv
Conv2d_1a_3x3.conv.weight           
# bn
Conv2d_1a_3x3.bn.weight             
Conv2d_1a_3x3.bn.bias               
Conv2d_1a_3x3.bn.running_mean       
Conv2d_1a_3x3.bn.running_var        
Conv2d_1a_3x3.bn.num_batches_tracked

# ready for quantization but not quantized
# mq = torchvision.models.quantization.inception_v3(pretrained=False, aux_logits=False, quantize=False)
# state_dict keys of Conv2d_1a_3x3
# conv
Conv2d_1a_3x3.conv.weight   
# bn        
Conv2d_1a_3x3.bn.weight             
Conv2d_1a_3x3.bn.bias               
Conv2d_1a_3x3.bn.running_mean       
Conv2d_1a_3x3.bn.running_var        
Conv2d_1a_3x3.bn.num_batches_tracked

# quantized and fused
# mq = torchvision.models.quantization.inception_v3(pretrained=False, aux_logits=False, quantize=True)
# state_dict keys of Conv2d_1a_3x3
# conv, including quantization-specific scale+zp
Conv2d_1a_3x3.conv.weight    
Conv2d_1a_3x3.conv.bias      
Conv2d_1a_3x3.conv.scale     
Conv2d_1a_3x3.conv.zero_point
# no bn, it was fused into the conv
</code></pre>
<p>You could use the ready for quantization but not quantized model if you are running quantization yourself, the state_dict will be closer to fp32 version since it is before fusion.  Then you’d have to go block by block and see if there are additional differences.</p>

Vasiliy_Kuznetsov: <aside class="quote no-group" data-username="Nimrod_Daniel" data-post="7" data-topic="108187">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/nimrod_daniel/48/10282_2.png" class="avatar"> Nimrod_Daniel:</div>
<blockquote>
<p>they have completely different names</p>
</blockquote>
</aside>
<p>which names in particular are different?  It should be pretty similar.</p>

Nimrod_Daniel: <p>The state_dicts have the same lengths, my mistake. That was quite odd and confusing, though the lengths make sense now. The naming differences shouldn’t be much of a problem.<br>
I’ll deal with that and then perform the fusion.</p>
<p>It could have been great if I could perform a QAT using CUDA instead of dealing with the conversion, hopefully a CUDA support will available soon.<br>
So far the PTQ with optimized calibration works really well with my data,  I get almost no drop in AP:) , but maybe in slightly different cases it would be useful in order to avoid drop in AP.</p>
<p>Thanks.</p>

Vasiliy_Kuznetsov: <aside class="quote no-group" data-username="Nimrod_Daniel" data-post="10" data-topic="108187">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/nimrod_daniel/48/10282_2.png" class="avatar"> Nimrod_Daniel:</div>
<blockquote>
<p>It could have been great if I could perform a QAT using CUDA instead of dealing with the conversion, hopefully a CUDA support will available soon.</p>
</blockquote>
</aside>
<p>calling <code>prepare</code> and running the QAT fine-tuning is supported on CUDA.  The only thing not supported is calling <code>convert</code> and running the quantized kernels.  Not sure if that is what you were referring to.</p>

Nimrod_Daniel: <p>Good to know. I thought that there’s no CUDA support also for QAT. Thanks.</p>