RuntimeError: Could not run 'aten::empty.memory_format' with arguments from the 'QuantizedCPU' backend

Andrew_Yang: <p>I am trying to quantize the model of image network. and I faced the error.</p>
<p>For the call the function below</p>
<pre><code class="lang-auto">dummy_input = torch.randn(1, 3, 320, 320).cpu()
script_model = torch.jit.trace(net, dummy_input)
</code></pre>
<p>the tensor operation of sum shows the Exception</p>
<pre><code class="lang-auto">norm = x.sum(dim=1, keepdim=True).sqrt()
</code></pre>
<p>RuntimeError: Could not run ‘aten::empty.memory_format’ with arguments from the ‘QuantizedCPU’ backend. ‘aten::empty.memory_format’ is only available for these backends: [CPU, CUDA, MkldnnCPU, SparseCPU, SparseCUDA, BackendSelect, Autograd, Profiler, Tracer].</p>
<p>Is there any way bypassing this error?</p>

jerryzh168: <p>it’s because quantized::sum is not supported, can you put dequant/quant around the sum op?</p>

zoeyyes: <aside class="quote no-group" data-username="jerryzh168" data-post="2" data-topic="98560">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/jerryzh168/48/15217_2.png" class="avatar"> jerryzh168:</div>
<blockquote>
<p>put dequant/quant around the sum op</p>
</blockquote>
</aside>
<p>hi, how to put dequant/quant around the sum op?? sorry,i am new</p>

Vasiliy_Kuznetsov: <p>There is an example here: <a href="https://pytorch.org/docs/stable/quantization.html" rel="noopener nofollow ugc">https://pytorch.org/docs/stable/quantization.html</a> - if you search that page for <code>torch.quantization.QuantStub()</code> and <code>torch.quantization.DeQuantStub()</code>, that should help.</p>

escorciav: <p>I did the dequant/quant biz, and it works (as it stopped complaining).</p>
<p>However, I’m not sure that’s the proper way. I believe that the PTQ tutorial (InvertedResiual, sum) has the “proper” way of doing it. Let me know <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=12" title=":slight_smile:" class="emoji" alt=":slight_smile:" loading="lazy" width="20" height="20"></p>
<p><a class="mention" href="/u/ptrblck">@ptrblck</a> do you know about quantization?<br>
Kindly let me know if I should create a new post. Answering here as I fall onto this trap <img src="https://discuss.pytorch.org/images/emoji/apple/mouse_trap.png?v=12" title=":mouse_trap:" class="emoji" alt=":mouse_trap:" loading="lazy" width="20" height="20"> <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=12" title=":slight_smile:" class="emoji" alt=":slight_smile:" loading="lazy" width="20" height="20"></p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/7/5/753a09c63b3a689f58277a67a93aa43e26f3e9dc.png" data-download-href="https://discuss.pytorch.org/uploads/default/753a09c63b3a689f58277a67a93aa43e26f3e9dc" title="2023-02-20_pytorch-forum"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/7/5/753a09c63b3a689f58277a67a93aa43e26f3e9dc_2_470x500.png" alt="2023-02-20_pytorch-forum" data-base62-sha1="gJ2d2hGcsqu4NK3Zdy2f9FNTJhi" width="470" height="500" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/7/5/753a09c63b3a689f58277a67a93aa43e26f3e9dc_2_470x500.png, https://discuss.pytorch.org/uploads/default/optimized/3X/7/5/753a09c63b3a689f58277a67a93aa43e26f3e9dc_2_705x750.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/7/5/753a09c63b3a689f58277a67a93aa43e26f3e9dc_2_940x1000.png 2x" data-dominant-color="242625"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">2023-02-20_pytorch-forum</span><span class="informations">990×1051 137 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

jcaip: <p>Hi Victor,</p>
<p>Can you link the tutorial that you are referring to?</p>

escorciav: <p>Here you go <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=12" title=":slight_smile:" class="emoji" alt=":slight_smile:" loading="lazy" width="20" height="20"></p>
<p><a href="https://pytorch.org/tutorials/advanced/static_quantization_tutorial.html#post-training-static-quantization" class="onebox" target="_blank" rel="noopener nofollow ugc">https://pytorch.org/tutorials/advanced/static_quantization_tutorial.html#post-training-static-quantization</a></p>
<p>Out of curiosity what kind of model are you trying to quantize?<br>
I’m struggling to quantize YOLOv5 with Pytorch Quantization API <img src="https://discuss.pytorch.org/images/emoji/apple/sweat_smile.png?v=12" title=":sweat_smile:" class="emoji" alt=":sweat_smile:" loading="lazy" width="20" height="20"></p>

jcaip: <p>I am the current pytorch quantization oncall <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=12" title=":slight_smile:" class="emoji" alt=":slight_smile:" loading="lazy" width="20" height="20"></p>
<p><code>torch.ao.quantization.convert</code> will add in Quant/Dequant pairs for you, but not all ops are supported for quantized inputs, hence the need to manually add in dequant to run your op in fp32.</p>
<p>Also, if you have additional questions it would be better if you created a new question.</p>