How to quantize Mobilenet_V2 after training

hukudo: <p>I trained mobilenet v2 with output is 16 classes and save model. After I load model and quantize that model, then get error when predict. Can you help me fix this bug. thanks a lot</p>
<p>Code load model after training</p>
<pre><code class="lang-auto">def load_model(model_path, num_classes=16, device=torch.device('cpu')):
    model = models.mobilenet_v2()
    n_input_features = model.classifier[1].in_features
    model.classifier[1] = nn.Linear(n_input_features, num_classes)
    model.load_state_dict(torch.load(model_path, map_location=device))
    model.eval().to(device)
    return model
</code></pre>
<p>Code quantize model</p>
<pre><code class="lang-auto">model_fp32 = load_model("mobilenetv2_16_classes.pt").to("cpu")
model_fp32.eval()
print(model_fp32)
backend = "fbgemm"
model_fp32.qconfig = torch.quantization.get_default_qconfig(backend)
torch.backends.quantized.engine = backend
model_static_quantized = torch.quantization.prepare(model_fp32, inplace=False)
model_static_quantized = torch.quantization.convert(model_static_quantized, inplace=False)
</code></pre>
<p>Quantize success</p>
<pre><code class="lang-auto">def print_model_size(mdl):
    torch.save(mdl.state_dict(), "tmp.pt")
    print("%.2f MB" %(os.path.getsize("tmp.pt")/1e6))
    os.remove('tmp.pt')

print_model_size(model_fp32) 
print_model_size(model_static_quantized)
</code></pre>
<pre><code class="lang-auto">9.6MB
2.7MB
</code></pre>
<p>Then I run and get Error</p>
<pre><code class="lang-auto">model_static_quantized(norm_img)
</code></pre>
<pre><code class="lang-auto">return ops.quantized.conv2d(
NotImplementedError: Could not run 'quantized::conv2d.new' with arguments from the 'CPU' backend. This could be because the operator doesn't exist for this backend, or was omitted during the selective/custom build process (if using custom build). If you are a Facebook employee using PyTorch on mobile, please visit https://fburl.com/ptmfixes for possible resolutions. 'quantized::conv2d.new' is only available for these backends: [QuantizedCPU, BackendSelect, Python, Named, Conjugate, Negative, ADInplaceOrView, AutogradOther, AutogradCPU, AutogradCUDA, AutogradXLA, AutogradLazy, AutogradXPU, AutogradMLC, Tracer, UNKNOWN_TENSOR_TYPE_ID, Autocast, Batched, VmapMode].```</code></pre>

tom: <p>The error message means that the input needs to be quantized.<br>
The easiest way to achieve this might be to wrap the model into a larger model and add <code>torch.quantization.QuantStub/DeQuantStub</code> layers before calling into the model.</p>
<p>The <a href="https://pytorch.org/tutorials/advanced/static_quantization_tutorial.html#model-architecture" rel="noopener nofollow ugc">quantization tutorial</a> has these steps, though they are easy to miss.</p>
<p>Best regards</p>
<p>Thomas</p>

hukudo: <p>Thanks <a class="mention" href="/u/tom">@tom</a> . But I don’t know how to quantize my model Mobilenetv2 finetuned with output is 16 classes. Can you help me!</p>

tom: <p>Something like</p>
<pre><code class="lang-python">
import torch
import torchvision
model = torchvision.models.mobilenet_v2()
# load you weights here into model
qmodel = torch.quantization.QuantWrapper(model)

qmodel.eval()

backend = "fbgemm"
qmodel.qconfig = torch.quantization.get_default_qconfig(backend)
torch.backends.quantized.engine = backend
model_static_quantized = torch.quantization.prepare(qmodel, inplace=False)

# feed real inputs here
model_static_quantized(torch.randn(1, 3, 224, 224))

model_static_quantized = torch.quantization.convert(model_static_quantized, inplace=False)

model_static_quantized
</code></pre>
<p>seems to work for me.</p>
<p>Best regards</p>
<p>Thomas</p>