ConvBnReLU quantized performance

username: Dorozhko-Anton
cooked: <p>I was trying to quantize FBNet model in PyTorch.<br>
Quantized version has several times bigger latency than fp32.<br>
but on raspberry pi it gives some gain in latency but still slow.</p>
<p>here is the result of small benchmark (just one Conv+Bn+ReLU)<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/f/2/f2ba1128ddba148bcdad381aad25b6a9db1eb19c.png" alt="qperf_bug_1" data-base62-sha1="yDgc47KUhxppWFh0q9WhCCV6Ae8" width="654" height="415"></p>
<p>Is this an expected behavior ?</p>
<pre><code class="lang-auto">import torch
import torch.nn as nn
import torch.nn.quantized as nnq
from torch.quantization import QuantStub, DeQuantStub

class ConvBNRelu(nn.Module):
    def __init__(self, cfg):
        super(ConvBNRelu, self).__init__()
        self.conv = nn.Conv2d(in_channels=cfg['in_channels'],
                              out_channels=cfg['out_channels'],
                              kernel_size=cfg['kernel_size'],
                              stride=cfg['stride'],
                              padding=cfg['padding']) 
        self.bn = nn.BatchNorm2d(num_features=cfg['out_channels'])
        self.relu = nn.ReLU()
   
    def forward(self, x):
        x = self.conv(x)
        x = self.bn(x)
        x = self.relu(x)
        return x
        
class Backbone(nn.Module):
    def __init__(self):
        super(Backbone, self).__init__()
        cfg = {
            'stem' : {
                'in_channels' : 3,
                'out_channels' : 32,
                'kernel_size' : (3, 3),
                'stride' : (2, 2),
                'padding' : (1, 1),
            }
        }
        self.stem = ConvBNRelu(cfg['stem'])
        self.quant = QuantStub()
        self.dequant = DeQuantStub()
        
    def forward(self, x):
        x = self.quant(x)
        x = self.stem(x)
        x = self.dequant(x)
        return x
    
    def fuse_model(self):
        torch.quantization.fuse_modules(self.stem, ['conv', 'bn', 'relu'], inplace=True)

model = Backbone()
torch.manual_seed(123)
x = torch.randn([1, 3, 34, 320], dtype=torch.float)*10

print(model)
%timeit model(x) 

model.qconfig = torch.quantization.get_default_qconfig('qnnpack')
torch.backends.quantized.engine = 'qnnpack'
torch.quantization.prepare(model, inplace=True);
with torch.no_grad():
    for i in range(5):
        model(x)    
torch.quantization.convert(model, inplace=True);

print(model)
%timeit model(x) 
</code></pre>

username: lly-zero-one
cooked: <p>Thanks for the post. It seems to be some perf issue of the int8 kernel implementation for this specific shape. We will be investigating it.</p>

username: dskhudia
cooked: <p>Are you just trying this shape (in_channels  = 3, out_channels=32, kernel_size=3, stride=2 and padding=1) or is this actually used in a network? The reason I ask this is because usually the first conv is in_channels  = 3, out_channels=64, kernel_size=7, stride=2 and padding=3 and we do have optimized fast path for this in fbgemm.</p>

username: Dorozhko-Anton
cooked: <p>This conv2 params are actually used in FBNet like architectures from maskrrcnn_benchmark in “first” block. (to be more precise Chamnet-Mobile like arch )</p>

username: Dorozhko-Anton
cooked: <p>Oh, this is a typo</p>
<pre><code class="lang-auto">x = torch.randn([1, 3, 34, 320], dtype=torch.float)*10  
</code></pre>
<p>It should be [1, 3, 320, 320], but this doesn’t change the ratios of time mesurements</p>