Int 8 quantization vision transformer

Akshansh_Yadav_D23CS: <p>i have tried various method to convert the model into int8 or int16, but none of the technique is working properly, i need help to resolve this problem.  import torch<br>
import timm<br>
import torch.quantization</p>
<p>class QuantizedModel(torch.nn.Module):<br>
def <strong>init</strong>(self, model):<br>
super().<strong>init</strong>()<br>
self.model_fp32 = model<br>
self.quant = torch.quantization.QuantStub()<br>
self.dequant = torch.quantization.DeQuantStub()</p>
<pre><code>def forward(self, x):
    x = self.quant(x)
    x = self.model_fp32(x)
    x = self.dequant(x)
    return x
</code></pre>
<h1><a name="load-and-prepare-vision-transformer-model-1" class="anchor" href="#load-and-prepare-vision-transformer-model-1"></a>Load and prepare Vision Transformer model</h1>
<p>model = timm.create_model(‘vit_base_patch16_224’, pretrained=True)<br>
model.eval()</p>
<h1><a name="wrap-model-with-quantization-stubs-2" class="anchor" href="#wrap-model-with-quantization-stubs-2"></a>Wrap model with quantization stubs</h1>
<p>quantized_model = QuantizedModel(model)</p>
<h1><a name="quantization-configuration-3" class="anchor" href="#quantization-configuration-3"></a>Quantization configuration</h1>
<p>backend = “fbgemm”  # or “qnnpack” for mobile<br>
model.qconfig = torch.quantization.get_default_qconfig(backend)<br>
torch.backends.quantized.engine = backend</p>
<h1><a name="prepare-and-convert-model-4" class="anchor" href="#prepare-and-convert-model-4"></a>Prepare and convert model</h1>
<p>model_static_quantized = torch.quantization.prepare(quantized_model, inplace=False)</p>
<h1><a name="insert-calibration-steps-here-with-representative-data-5" class="anchor" href="#insert-calibration-steps-here-with-representative-data-5"></a>Insert calibration steps here with representative data</h1>
<p>model_static_quantized = torch.quantization.convert(model_static_quantized, inplace=False)</p>
<h1><a name="save-and-load-quantized-model-6" class="anchor" href="#save-and-load-quantized-model-6"></a>Save and load quantized model</h1>
<p>torch.jit.save(torch.jit.script(model_static_quantized), ‘vit_base_patch16_224_quantized.pt’)<br>
module = torch.jit.load(‘vit_base_patch16_224_quantized.pt’)</p>
<h1><a name="run-inference-7" class="anchor" href="#run-inference-7"></a>Run inference</h1>
<p>inputs = torch.randn((1, 3, 224, 224))<br>
outputs = module(inputs)</p>
<p>error is like this,</p>
<p>Could not run ‘quantized::conv2d.new’ with arguments from the ‘CPU’ backend. This could be because the operator doesn’t exist for this backend, or was omitted during the selective/custom build process (if using custom build). If you are a Facebook employee using PyTorch on mobile, please visit <a href="https://fburl.com/ptmfixes" class="inline-onebox" rel="noopener nofollow ugc">Internal Login</a> for possible resolutions. ‘quantized::conv2d.new’ is only available for these backends: [Meta, QuantizedCPU, QuantizedCUDA, BackendSelect, Python, FuncTorchDynamicLayerBackMode, Functionalize, Named, Conjugate, Negative, ZeroTensor, ADInplaceOrView, AutogradOther, AutogradCPU, AutogradCUDA, AutogradXLA, AutogradMPS, AutogradXPU, AutogradHPU, AutogradLazy, AutogradMeta, Tracer, AutocastCPU, AutocastCUDA, FuncTorchBatched, BatchedNestedTensor, FuncTorchVmapMode, Batched, VmapMode, FuncTorchGradWrapper, PythonTLSSnapshot, FuncTorchDynamicLayerFrontMode, PreDispatch, PythonDispatcher].</p>