List of operations supported by QuantizedCPU backend

username: SpiridonSunRotator
cooked: <p>Hi, I was trying to quantize Vision Transformer from <code>timm</code> as follows:</p>
<pre><code class="lang-python">class QuantAttention(nn.Module):

    def __init__(
        self,
        dim,
        num_heads=8,
        qkv_bias=False,
        qk_norm=False,
        attn_drop=0.,
        proj_drop=0.,
        norm_layer=nn.LayerNorm,
    ):
        super().__init__()
        assert dim % num_heads == 0, 'dim should be divisible by num_heads'
        self.num_heads = num_heads
        self.head_dim = dim // num_heads
        self.scale = self.head_dim ** -0.5

        self.qkv = nn.Linear(dim, dim * 3, bias=qkv_bias)
        self.q_norm = norm_layer(self.head_dim) if qk_norm else nn.Identity()
        self.k_norm = norm_layer(self.head_dim) if qk_norm else nn.Identity()
        self.attn_drop = nn.Dropout(attn_drop)
        self.proj = nn.Linear(dim, dim)
        self.proj_drop = nn.Dropout(proj_drop)

        # quantization stubs
        self.quant = QuantStub()
        self.dequant = DeQuantStub()

    def forward(self, x):
        B, N, C = x.shape
        x = self.quant(x)
        qkv = self.qkv(x).reshape(B, N, 3, self.num_heads, self.head_dim).permute(2, 0, 3, 1, 4)
        q, k, v = qkv.unbind(0)
        q, k = self.q_norm(q), self.k_norm(k)
        q = q * self.scale

        attn = q @ k.transpose(-2, -1)
        attn = attn.softmax(dim=-1)
        attn = self.attn_drop(attn)
        x = attn @ v

        x = x.transpose(1, 2).reshape(B, N, C)
        x = self.proj(x)
        x = self.dequant(x)
        x = self.proj_drop(x)
        return x
</code></pre>
<p>However, some operations, in particular:</p>
<ul>
<li><code>q = q * self.scale</code> - multiplication by scalar</li>
<li><code>attn = q @ k.transpose(-2, -1)</code> batched matmul<br>
are not supported by <code>QuantizedCPU</code> and the error message claims the following:<br>
<code>NotImplementedError: Could not run 'aten::bmm.out' with arguments from the 'QuantizedCPU' backend. This could be because the operator doesn't exist for this backend, or was omitted during the selective/custom build process (if using custom build).</code></li>
</ul>
<p>The quantization pipeline is the following:</p>
<pre><code class="lang-python">quant_attn.qconfig = torch.ao.quantization.get_default_qconfig('x86')
quant_attn = torch.ao.quantization.prepare(quant_attn)
for _ in range(10):
    inputs = torch.randn(1, 16, 128, device=device)
    quant_attn(inputs)
quant_attn = torch.ao.quantization.convert(quant_attn)
</code></pre>
<p>The message appears on the call:</p>
<pre><code class="lang-python">inputs = torch.randn(1, 16, 128, device=device)
outputs = quant_attn(inputs)
</code></pre>
<p>Where can one find a list of supported operations to better understand the limitations of the current Quantization API?</p>
<p><strong>Environment</strong></p>
<ul>
<li><code>torch                            2.1.0+cu118</code></li>
<li><code>torchvision                      0.16.0+cu118</code></li>
</ul>

username: jerryzh168
cooked: <aside class="quote no-group" data-username="SpiridonSunRotator" data-post="1" data-topic="191323">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/s/8491ac/48.png" class="avatar"> SpiridonSunRotator:</div>
<blockquote>
<p>Where can one find a list of supported operations to better understand the limitations of the current Quantization API?</p>
</blockquote>
</aside>
<p>here is a somewhat outdated support matrix: <a href="https://pytorch.org/docs/master/quantization.html#quantization-support-matrix" class="inline-onebox" rel="noopener nofollow ugc">Quantization — PyTorch master documentation</a></p>
<p>but yes, quantized bmm is not supported. but you can produce a representation for quantized bmm: (dq → bmm → q) in <a href="https://pytorch.org/docs/main/quantization.html#prototype-pytorch-2-export-quantization" rel="noopener nofollow ugc">the new quantization flow</a> and lower it to some backend that does support this op.</p>

username: SpiridonSunRotator
cooked: <p>Thanks for your answer and help!</p>