RuntimeError: Could not run 'aten::add_.Tensor' with arguments from the 'QuantizedCPU' backend

aidanaranjo: <p>I am trying to run quantization on a model. The model I am using is the pretrained wide_resnet101_2. The code is running on CPU. Before quantization, the model is 510MB and after quantization it is down to 129MB. It seems like the quantization is working. The problem arises when the quantized model is called later in the code to run the tester.</p>
<p><img src="https://discuss.pytorch.org/uploads/default/original/3X/8/e/8e441339185bac0843114bb651edfd80766f6ca5.png" alt="image" data-base62-sha1="kixEmueesQb3mkuhCjQfEKHBdWt" width="648" height="314"></p>
<p>The error is in the line 70: RuntimeError: Could not run ‘aten::add_.Tensor’ with arguments from the ‘QuantizedCPU’ backend. This could be because the operator doesn’t exist for this backend, or was omitted during the selective/custom build process (if using custom build). If you are a Facebook employee using PyTorch on mobile, please visit <a href="https://fburl.com/ptmfixes" class="inline-onebox" rel="noopener nofollow ugc">Internal Login</a> for possible resolutions. ‘aten::add_.Tensor’ is only available for these backends: [CPU, MkldnnCPU, SparseCPU, Meta, BackendSelect, Named, AutogradOther, AutogradCPU, AutogradCUDA, AutogradXLA, AutogradNestedTensor, UNKNOWN_TENSOR_TYPE_ID, AutogradPrivateUse1, AutogradPrivateUse2, AutogradPrivateUse3, Tracer, Autocast, Batched, VmapMode].</p>
<p>Any help with this?</p>

jerryzh168: <aside class="quote no-group quote-modified" data-username="aidanaranjo" data-post="1" data-topic="110039">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/a/b5ac83/48.png" class="avatar"> aidanaranjo:</div>
<blockquote>
<p>The error is in the line 70: RuntimeError: Could not run ‘aten::add_.Tensor’ with arguments from the ‘QuantizedCPU’ backend. This could be because the operator doesn’t exist for this backend, or was omitted during the selective/custom build process (if using custom build). If you are a Facebook employee using PyTorch on mobile, please visit <a href="https://fburl.com/ptmfixes" class="inline-onebox" rel="noopener nofollow ugc">Internal Login</a> for possible resolutions. ‘aten::add_.Tensor’ is only available for these backends: [CPU, MkldnnCPU, SparseCPU, Meta, BackendSelect, Named, AutogradOther, AutogradCPU, AutogradCUDA, AutogradXLA, AutogradNestedTensor, UNKNOWN_TENSOR_TYPE_ID, AutogradPrivateUse1, AutogradPrivateUse2, AutogradPrivateUse3, Tracer, Autocast, Batched, VmapMode].</p>
</blockquote>
</aside>
<p>This means the input of aten::add_ is a quantized Tensor. To address the problem you can either<br>
(1). place a DequantStub and QuantStub around the aten::add_ op.<br>
e.g.</p>
<pre data-code-wrap="python"><code class="lang-python">def __init__(...):
    ...
    self.quant = torch.quantization.QuantStub()
    self.dequant = torch.quantization.DeQuantStub()
   ...

def forward(...):
     ...
     self.dequant(x)
     x += ...
     self.quant(x)
     ...
</code></pre>
<p>or<br>
(2). quantize aten::add_ by replacing it with FloatFunctional(<a href="https://github.com/pytorch/pytorch/blob/master/torch/nn/quantized/modules/functional_modules.py" rel="noopener nofollow ugc">https://github.com/pytorch/pytorch/blob/master/torch/nn/quantized/modules/functional_modules.py</a>)</p>

addisonklinke: <p>Providing an example code snippet for approach 2</p>
<pre><code class="lang-python">def __init__(...):
    ...
    self.quant_x = torch.quantization.QuantStub()
    self.quant_y = torch.quantization.QuantStub()
    self.dequant = torch.quantization.DeQuantStub()
    self.ff = torch.nn.quantized.FloatFunctional()
    ...

def forward(x, y):
    ...
    x = self.quant_x(x)
    y = self.quant_y(y)
    out = self.ff.add(x, y)
    ...
</code></pre>