Dynamic Quantization not reducing model size

username: Raghav_Gurbaxani
cooked: <p>Hi</p>
<p>I am trying to use the quantize_dynamic module, but the size of the quantized model is the same as the original model in my case.</p>
<pre><code class="lang-auto">import os
import torch
from craft import CRAFT
trained_model='craft_mlt_25k.pth'

def print_size_of_model(model):
    torch.save(model.state_dict(), "temp.p")
    print('Size (MB):', os.path.getsize("temp.p")/1e6)
    os.remove('temp.p')
    
from collections import OrderedDict
def copyStateDict(state_dict):
    if list(state_dict.keys())[0].startswith("module"):
        start_idx = 1
    else:
        start_idx = 0
    new_state_dict = OrderedDict()
    for k, v in state_dict.items():
        name = ".".join(k.split(".")[start_idx:])
        new_state_dict[name] = v
    return new_state_dict

net = CRAFT()     # initialize
net.load_state_dict(copyStateDict(torch.load(trained_model, map_location='cpu')))
net.eval()
print_size_of_model(net)
quantized = torch.quantization.quantize_dynamic(net, dtype=torch.qint8)
print_size_of_model(quantized)
</code></pre>
<p>Size of both original and quantized model is 83.14 MB. Why doesn’t the model size change ?<br>
Any suggestions would be appreciated</p>
<p>Similarily when I tried to quantize the pretrained model ResNet18 model from torchvision, size only changed from 46 MB to 45 MB.</p>

username: raghuramank100
cooked: <p>Hi,<br>
Dynamic quantization only helps in reducing the model size for models that use Linear and LSTM  modules. For the case of resnet18, the model consists of conv layers which do not have dynamic quantization support yet. For your model, can you check if it has linear layers?</p>

username: Raghav_Gurbaxani
cooked: <p><a class="mention" href="/u/raghuramank100">@raghuramank100</a><br>
Thanks for your response.<br>
But dynamic quantization is able to reduce model size from 553 MB to 182 MB, while VGG16 is mostly convolution layers, why such a drastic change then ?</p>

username: jerryzh168
cooked: <p>it’s probably because most of the weights are in the last few Linear layers?</p>

username: raghuramank100
cooked: <p>Yes, for the case of VGG16, the last two fc layers contain the bulk of the weights.</p>

username: Sagar_Gupta
cooked: <p>Hi,<br>
I am trying to quantize a BERT but the size is not reducing, I wonder why, Can someone help me?</p>
<p>Here is the code snippet:-</p>
<p>import torch as torch</p>
<p>import os</p>
<p>from transformers.modeling_bert import BertConfig, BertForPreTraining, load_tf_weights_in_bert, BertModel</p>
<p>tf_checkpoint_path="./distlang/"<br>
bert_config_file = “./config.json”<br>
pytorch_dump_path="./distlangpytorch/"</p>
<p>device = “cpu”</p>
<p>torch.backends.quantized.engine = ‘qnnpack’</p>
<p>qconfig = torch.quantization.get_default_qconfig(‘fbgemm’)</p>
<p>print(qconfig)<br>
config = BertConfig.from_json_file(bert_config_file)<br>
print(“Building PyTorch model from configuration: {}”.format(str(config)))<br>
model = BertModel.from_pretrained("./distlangpytorch/")</p>
<p>model.to(device)</p>
<p>torch.quantization.prepare(model)</p>
<p>quantized_model=torch.quantization.convert(model)</p>
<p>def print_size_of_model(model):<br>
torch.save(model.state_dict(), “temp.p”)<br>
print(‘Size (MB):’, os.path.getsize(“temp.p”)/1e6)<br>
os.remove(‘temp.p’)</p>
<p>print_size_of_model(model)<br>
print_size_of_model(quantized_model)</p>
<p>quantized_output_dir = “./quantized_model”<br>
if not os.path.exists(quantized_output_dir):<br>
os.makedirs(quantized_output_dir)</p>

username: BOLLOJU_ARAVIND
cooked: <p>Hi <a class="mention" href="/u/sagar_gupta">@Sagar_Gupta</a>,</p>
<p>In this mode of quantization the model has to be calibrated(evaluate your model after prepare()) to capture the qparams (zeropoint&amp;scale)<br>
Which are needed to quantize the model i.e weights and all</p>

username: deepak_mangla
cooked: <p>Hi <a class="mention" href="/u/bolloju_aravind">@BOLLOJU_ARAVIND</a>,  Can you refer some guide for  doing this?</p>

username: Shisho_Sama
cooked: <p>Does Graph Mode quantization suffers from this issue as well? I recently tried to quantize a jit saved model and did not see any difference, the model size is nearly the same, but the forward pass has gotten worse (nearly two times slower)<br>
What could be the underlying issue here?</p>