Could not run 'aten::_log_softmax.out' with arguments from the 'QuantizedCPU' backend. This could be because the operator doesn't exist for this backend, or was omitted during the selective/custom build process (if using custom build)

Siva6233: <p>I am trying to quantize my Alexnet model imported from torchvision.I am not able to get an inference from the quantized model.</p>
<p>I used the following code to quantize the model:</p>
<pre><code class="lang-auto">model_int8 = torch.quantization.QuantWrapper(model_fp32)
model_int8.qconfig = torch.quantization.default_qconfig
torch.quantization.prepare(model_int8, inplace=True)
with torch.no_grad():
    model_int8(input_data)
torch.quantization.convert(model_int8, inplace=True)
</code></pre>
<p>This is the model after quantization and the weights are of type qint8.I am running the whole process in the cpu.</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/8/6/867e6cf45f8e681bc2a2f0ca8bf357004afc4f51.png" data-download-href="https://discuss.pytorch.org/uploads/default/867e6cf45f8e681bc2a2f0ca8bf357004afc4f51" title="image"><img src="https://discuss.pytorch.org/uploads/default/original/3X/8/6/867e6cf45f8e681bc2a2f0ca8bf357004afc4f51.png" alt="image" data-base62-sha1="jbMQuvJem12v3Cj1mpeBFmefNBL" width="690" height="300" data-dominant-color="F4F4F4"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">885×386 18 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>When I try to get the inference i get the following error:</p>
<blockquote>
<p>Could not run ‘aten::_log_softmax.out’ with arguments from the ‘QuantizedCPU’ backend. This could be because the operator doesn’t exist for this backend, or was omitted during the selective/custom build process (if using custom build). If you are a Facebook employee using PyTorch on mobile, please visit <a href="https://fburl.com/ptmfixes" class="inline-onebox" rel="noopener nofollow ugc">Internal Login</a> for possible resolutions. ‘aten::_log_softmax.out’ is only available for these backends: [CPU, CUDA, Meta, BackendSelect, Python, FuncTorchDynamicLayerBackMode, Functionalize, Named, Conjugate, Negative, ZeroTensor, ADInplaceOrView, AutogradOther, AutogradCPU, AutogradCUDA, AutogradHIP, AutogradXLA, AutogradMPS, AutogradIPU, AutogradXPU, AutogradHPU, AutogradVE, AutogradLazy, AutogradMeta, AutogradMTIA, AutogradPrivateUse1, AutogradPrivateUse2, AutogradPrivateUse3, AutogradNestedTensor, Tracer, AutocastCPU, AutocastCUDA, FuncTorchBatched, FuncTorchVmapMode, Batched, VmapMode, FuncTorchGradWrapper, PythonTLSSnapshot, FuncTorchDynamicLayerFrontMode, PythonDispatcher].</p>
</blockquote>

HDCharles: <p>you can’t just apply a quantization wrapper around hte whole model since many ops cannot be quantized. This error occurs because you tried passing a quantized datatype into a non-quantized operator due to trying to treat the whole model as if its quantized.</p>
<p>see <a href="https://pytorch.org/docs/stable/quantization.html" class="inline-onebox" rel="noopener nofollow ugc">Quantization — PyTorch 2.1 documentation</a> for more information or check out one of the tutorials.</p>

Siva6233: <p>Hello <a class="mention" href="/u/hdcharles">@HDCharles</a> ,</p>
<p>Thank you for the reply. I checked out the post training quantization section in the documentation and it worked after I followed the procedure provided there.</p>