Error while post training quantizing the model

username: jinooooo
cooked: <p>I have been getting this error constantly and I need some help on how to solve it.<br>
‘NotImplementedError: Could not run ‘aten::add.out’ with arguments from the ‘QuantizedCPU’ backend’</p>
<p>I’m attaching screenshots of my code + error.</p>
<p>Thanks</p>
<p>Inference script</p>
<pre><code> predicted_labels = None
 models_dir = 'xxxxxx'
for model_name in glob.glob(models_dir + '/*.pth'):
     model = CustomResnet(model_name = CFG.model_name, pretrained = False)
     state = torch.load(model_name, 
                   map_location=torch.device('cpu'))['model']
model.load_state_dict(state)
model = model.to(device)
model.eval()


model = torch.quantization.fuse_modules(model, modules_to_fuse)
model.qconfig = torch.quantization.default_qconfig
#print(model.qconfig)
torch.quantization.prepare(model, inplace = True)
model.eval()
torch.quantization.convert(model, inplace=True)

print_size_of_model(model)

temp_preds = None

with torch.no_grad():
    for images in tqdm(test_loader, desc=f'Predicting. '):
        images = images.to(device)
        st = time.time()
        predictions = model(images)
        et = time.time()
        predictions = predictions.softmax(1).to('cpu').numpy()
        
        if temp_preds is None:
            temp_preds = predictions
        else:
            temp_preds = np.vstack((temp_preds, predictions))

if predicted_labels is None:
    predicted_labels = temp_preds
else:
    predicted_labels += temp_preds
</code></pre>
<p>predicted_labels /= (len(glob.glob(models_dir + ‘/*.pth’)))<br>
test_set[‘predicted’] = predicted_labels.argmax(1)<br>
final_test_accuracy = get_score(test_set[‘target’], test_set[‘predicted’])</p>
<p>print(“========================================= PERFORMANCE =============================================”)<br>
print_size_of_model(model)<br>
print(f’Final test accuracy = {final_test_accuracy} ')<br>
print(‘Elapsed time = {:0.4f} milliseconds’.format((et - st) * 1000))<br>
print(“====================================================================================================”)</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/2/a/2a342c7dc1fc8270a7fa8bebc55cc3efda255150.png" data-download-href="https://discuss.pytorch.org/uploads/default/2a342c7dc1fc8270a7fa8bebc55cc3efda255150" title="one"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/2/a/2a342c7dc1fc8270a7fa8bebc55cc3efda255150_2_690x269.png" alt="one" data-base62-sha1="61lPO26OKR9xU6KQhOenhCCN1io" width="690" height="269" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/2/a/2a342c7dc1fc8270a7fa8bebc55cc3efda255150_2_690x269.png, https://discuss.pytorch.org/uploads/default/optimized/3X/2/a/2a342c7dc1fc8270a7fa8bebc55cc3efda255150_2_1035x403.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/2/a/2a342c7dc1fc8270a7fa8bebc55cc3efda255150.png 2x" data-dominant-color="EBEEF0"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">one</span><span class="informations">1285×502 46.6 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/5/a/5af92e784538cda64f7ddb98320a4a482fa255b8.png" data-download-href="https://discuss.pytorch.org/uploads/default/5af92e784538cda64f7ddb98320a4a482fa255b8" title="two"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/5/a/5af92e784538cda64f7ddb98320a4a482fa255b8_2_689x396.png" alt="two" data-base62-sha1="cYMQ9V6imjPXP4OB0RcZxb61PpC" width="689" height="396" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/5/a/5af92e784538cda64f7ddb98320a4a482fa255b8_2_689x396.png, https://discuss.pytorch.org/uploads/default/optimized/3X/5/a/5af92e784538cda64f7ddb98320a4a482fa255b8_2_1033x594.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/5/a/5af92e784538cda64f7ddb98320a4a482fa255b8.png 2x" data-dominant-color="F6DFDF"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">two</span><span class="informations">1290×741 58.8 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

username: andrewor
cooked: <p>Hi Jino,</p>
<p>What does your model look like? It would appear that you are passing a quantized tensor into a kernel expecting floating point values. Usually the way to fix this is to add a <code>torch.quantization.DeQuantStub</code> before the offending op. <a href="https://pytorch.org/docs/stable/quantization.html#passing-a-quantized-tensor-into-a-non-quantized-kernel" rel="noopener nofollow ugc">This link</a> provides an example of how to fix it.</p>
<p>Best,<br>
-Andrew</p>

username: jinooooo
cooked: <p>hey andrew, thanks for writing back, my architecture has been defined the way i posted in the screenshot and dequantstub doesnt work? could you please have a look at the attached?</p>

username: APaul
cooked: <p>Hi Jino,</p>
<p>Looking at the output message it seems like the issue is happening maybe because of an addition operation in the pretrained resnet model. That needs to be replaced by FloatFunctional(). The error message has been conveniently truncated so can’t say where is it happening.</p>
<p>Best,<br>
Paul</p>

username: jinooooo
cooked: <p>Ill give it a try and keep you updated, thanks <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=12" title=":slight_smile:" class="emoji" alt=":slight_smile:" loading="lazy" width="20" height="20"></p>