Quantized model inference error related to Mish activation function

Anouar_LAOUICHI: <p>I’m tinkering with post-training static quantization in PyTorch by trying out different activation functions on the same model, then I try to quantize it and run inference ( I want to see what are the activations that are supported). For example, I replaced ReLU with leakyReLU on ResNet50 then applied quantization. The inference ran just fine ( it was a bit slower with a 3% accuracy drop but this does not matter as I’m only experimenting). After that, I tried the Mish activation function, the conversion was successful, however, I got the following error during inference:</p>
<pre><code class="lang-auto">---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
 in 
      7 helper.print_size_of_model(resnet)
      8 
----&gt; 9 top1, top5, time_elapsed= helper.evaluate(resnet, criterion, testloader, neval_batches=num_eval_batches)
     10 print('Evaluation accuracy on %d images, top5: %2.2f, top1: %2.2f'%(num_eval_batches * eval_batch_size, top5.avg,top1.avg))
     11 print('time slapsed: %s' % str(datetime.timedelta(seconds=time_elapsed)))

d:\github\PyTorch_CIFAR10\helper.py in evaluate(model, criterion, data_loader, neval_batches, device)
     30         for image, target in data_loader:
     31             image.to(device)
---&gt; 32             output = model(image)
     33             loss = criterion(output, target)
     34             cnt += 1

~\anaconda3\envs\PFE_env\lib\site-packages\torch\nn\modules\module.py in __call__(self, *input, **kwargs)
    548             result = self._slow_forward(*input, **kwargs)
    549         else:
--&gt; 550             result = self.forward(*input, **kwargs)
    551         for hook in self._forward_hooks.values():
    552             hook_result = hook(self, input, result)

d:\github\PyTorch_CIFAR10\cifar10_models\resnetQ.py in forward(self, x)
    227         x = self.conv1(x)
    228         x = self.bn1(x)
--&gt; 229         x = self.relu(x)
    230         x = self.maxpool(x)
    231         x = self.layer1(x)

~\anaconda3\envs\PFE_env\lib\site-packages\torch\nn\modules\module.py in __call__(self, *input, **kwargs)
    548             result = self._slow_forward(*input, **kwargs)
    549         else:
--&gt; 550             result = self.forward(*input, **kwargs)
    551         for hook in self._forward_hooks.values():
    552             hook_result = hook(self, input, result)

d:\github\PyTorch_CIFAR10\cifar10_models\resnetQ.py in forward(self, x)
     24 
     25     def forward(self, x):
---&gt; 26         x = x * (torch.tanh(torch.nn.functional.softplus(x)))
     27         return x
     28 

RuntimeError: Could not run 'aten::empty.memory_format' with arguments from the 'QuantizedCPUTensorId' backend. 'aten::empty.memory_format' is only available for these backends: [CPUTensorId, CUDATensorId, MkldnnCPUTensorId, SparseCPUTensorId, SparseCUDATensorId, BackendSelect, VariableTensorId].
</code></pre>
<p>the mish layer is defined by:</p>
<pre><code class="lang-auto">class Mish(torch.nn.Module):
    def __init__(self):
        super().__init__()

    def forward(self, x):
        x = x * (torch.tanh(torch.nn.functional.softplus(x)))
        return x
</code></pre>
<p>Any help in this matter would be greatly appreciated because ultimately, I want to apply quantization on YOLOv4 which relies on Mish as an activation function.</p>

Vasiliy_Kuznetsov: <p>Softplus currently does not have a quantized implementation.  This error also usually means that a quantized tensor is being passed to a non-quantized function.</p>
<p>For a quick fix, you could add a <code>torch.quantization.DeQuantStub()</code> and <code>torch.quantization.QuantStub()</code> around the areas of the network which cannot be quantized such as Softplus.</p>
<p>The longer term fix would be to add quantization support for Softplus to PyTorch.</p>

Anouar_LAOUICHI: <p>The quick fix seems reasonable enough. Hopefully it won’t have a significant impact on inference time. I guess I’ll just have to try it out and see. Thank you for your help.</p>