Could not run 'quantized::conv2d.new' with arguments from the 'QuantizedCUDA' backend

MrOCW: <p>x=self.quant(x)<br>
x=self.conv(x)<br>
x=self.bn(x)<br>
x=self.act(x)<br>
x=self.dequant(x)</p>
<p>I trained a QAT model and when i tried evaluating the model, i got the error.</p>
<blockquote>
<p>Could not run ‘quantized::conv2d.new’ with arguments from the ‘QuantizedCUDA’ backend … … ‘quantized::conv2d.new’ is only available for these backends: [QuantizedCPU, …].</p>
</blockquote>
<p>when i added x = x.to(‘cpu’) before x = self.quant(x), to make it a QuantizedCPU backend (note that doing so, i am unable to train the model again as i will get:</p>
<blockquote>
<p>RuntimeError: Input type (torch.FloatTensor) and weight type (torch.cuda.FloatTensor) should be the same or input should be a MKLDNN tensor and weight is a dense tensor</p>
</blockquote>
<p>which is another problem…?),  i will then get:</p>
<blockquote>
<p>Could not run ‘aten::silu.out’ with arguments from the ‘QuantizedCPU’ backend. This could be because the operator doesn’t exist for this backend, or was omitted during the selective/custom build process (if using custom build). If you are a Facebook employee using PyTorch on mobile, please visit <a href="https://fburl.com/ptmfixes" class="inline-onebox" rel="noopener nofollow ugc">Internal Login</a> for possible resolutions. ‘aten::silu.out’ is only available for these backends: [CPU, CUDA,…</p>
</blockquote>
<p>so i changed position of dequant to<br>
x=x.to(‘cpu’)<br>
x=self.quant(x)<br>
x=self.conv(x)<br>
x=self.bn(x)<br>
x=self.dequant(x)<br>
x=self.act(x)</p>
<p>I get the error pointing to x=self.quant(x) :</p>
<blockquote>
<p>Could not run ‘aten::quantize_per_tensor’ with arguments from the ‘QuantizedCPU’ backend. This could be because the operator doesn’t exist for this backend, or was omitted during the selective/custom build process (if using custom build). If you are a Facebook employee using PyTorch on mobile, please visit <a href="https://fburl.com/ptmfixes" class="inline-onebox" rel="noopener nofollow ugc">Internal Login</a> for possible resolutions. ‘aten::quantize_per_tensor’ is only available for these backends: [CPU, CUDA,</p>
</blockquote>
<p>and if i remove x = self.quant(x), i get back:</p>
<blockquote>
<p>Could not run ‘quantized::conv2d.new’ with arguments from the ‘CPU’ backend. This could be because the operator doesn’t exist for this backend, or was omitted during the selective/custom build process (if using custom build). If you are a Facebook employee using PyTorch on mobile, please visit <a href="https://fburl.com/ptmfixes" class="inline-onebox" rel="noopener nofollow ugc">Internal Login</a> for possible resolutions. ‘quantized::conv2d.new’ is only available for these backends: [QuantizedCPU,</p>
</blockquote>
<p>Please help as i’ve been encountering errors after errors even after searching online for solutions.</p>

minhnvl: <p>Hi, I’m also getting same your problem. If possible, Could you give me some solutions to fix this error?</p>

MrOCW: <p>Hi, I have not found the solution<br>
cc <a class="mention" href="/u/jerryzh168">@jerryzh168</a></p>

HDCharles: <aside class="quote no-group" data-username="MrOCW" data-post="1" data-topic="133516">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/m/90db22/48.png" class="avatar"> MrOCW:</div>
<blockquote>
<p>Could not run ‘quantized::conv2d.new’ with arguments from the ‘QuantizedCUDA’ backend … … ‘quantized::conv2d.new’ is only available for these backends: [QuantizedCPU, …].</p>
</blockquote>
</aside>
<p>this error occurs when you try to run a quantized op with weight or input on cuda. For example if you were to take a correctly quantized model and then do .to(‘cuda’) and then run the model, you’d get this error.</p>
<p>based on the second error message, your weight is on cuda. Note: changing where x.to(‘cpu’) is located will not fix this problem if the actual op weight is on cuda.</p>
<p>my guess is that somewhere in your code you have model.to(‘cuda’) (likely during training) and you are not converting it back to cpu i.e. model.to(‘cpu’) before trying to do quantization.</p>
<p>addtionally, it looks like your self.act op is aten::silu which isn’t being converted to a quantized op (looks like it doesn’t have a quantized implementation <a href="https://github.com/pytorch/pytorch/blob/master/torch/nn/quantized/modules/activation.py" rel="noopener nofollow ugc">https://github.com/pytorch/pytorch/blob/master/torch/nn/quantized/modules/activation.py</a>). You can either implement it yourself or change to something along the lines of</p>
<p>y = sigmoid(x)<br>
x = y * x</p>
<p>I would also maybe start with a less weird model and make sure the flow works for you before iterating on that. Something like: <a href="https://pytorch.org/tutorials/advanced/static_quantization_tutorial.html" class="inline-onebox" rel="noopener nofollow ugc">(beta) Static Quantization with Eager Mode in PyTorch — PyTorch Tutorials 2.1.1+cu121 documentation</a> could be a good starting point.</p>

MrOCW: <blockquote>
<p>my guess is that somewhere in your code you have model.to(‘cuda’) (likely during training) and you are not converting it back to cpu i.e. model.to(‘cpu’) before trying to do quantization.</p>
</blockquote>
<p>Strange because I have done model.to(‘cpu’) before torch.quantization.convert(model)</p>

HDCharles: <p>you can inspect the model and identify whether the weight is stored correctly, its possible its not transfering over or something, though usually modules move over their attributes by default.</p>

sarramrg: <p>Hi <a class="mention" href="/u/hdcharles">@HDCharles</a> and <a class="mention" href="/u/mrocw">@MrOCW</a> <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=12" title=":slight_smile:" class="emoji" alt=":slight_smile:" loading="lazy" width="20" height="20"> !<br>
Did you find any solutions to this issue?<br>
I am facing the same error and could not find any solution yet.</p>
<p>Also, I checked in my code, I always have <code>model_static_quantized.to(torch.device('cpu'))</code> or <code>model.to(torch.device('cpu'))</code>. So everything is on CPU and not CUDA.<br>
How can I identify if the weights are stored correctly or not please? <a class="mention" href="/u/hdcharles">@HDCharles</a></p>
<p>Thanks in advance for your help!<br>
any additional inputs <a class="mention" href="/u/jerryzh168">@jerryzh168</a>?</p>

jerryzh168: <p>is your model input a cuda tensor?</p>
<p>This is a user error, it means quantization is not used correctly, in eager mode, user are expected to reason about the quantized model and is expected to place QuantStub, DeQuantStub and set qconfigs correctly. We may be able to provide more help if you post your modified model for quantization and qconfig settings.</p>

sarramrg: <p>Hi <a class="mention" href="/u/jerryzh168">@jerryzh168</a>!<br>
The error I have is <code>NotImplementedError: Could not run 'quantized::conv2d.new' with arguments from the 'CPU' backend</code></p>
<p>The inference is done on the following input:<br>
<code>dummy_input = torch.rand((1, 3, 512, 512), dtype=torch.float32, device="cpu")</code></p>
<p>The modified model for quantization is:</p>
<pre><code class="lang-auto"># Define the Model
class ResNet(nn.Module):
    def __init__(self, config, output_dim):
        super().__init__()

        # QuantStub converts tensors from floating point to quantized
        # self.quant = torch.quantization.QuantStub()

        block, n_blocks, channels = config
        self.in_channels = channels[0]

        assert len(n_blocks) == len(channels) == 4

        self.conv1 = nn.Conv2d(
            3,
            self.in_channels,
            kernel_size=(7, 7),
            stride=(2, 2),
            padding=3,
            bias=False,
        )
        self.bn1 = nn.BatchNorm2d(self.in_channels)
        self.relu = nn.ReLU(inplace=True)
        self.maxpool = nn.MaxPool2d(kernel_size=3, stride=2, padding=1)

        self.layer1 = self.get_resnet_layer(block, n_blocks[0], channels[0])
        self.layer2 = self.get_resnet_layer(block, n_blocks[1], channels[1], stride=2)
        self.layer3 = self.get_resnet_layer(block, n_blocks[2], channels[2], stride=2)
        self.layer4 = self.get_resnet_layer(block, n_blocks[3], channels[3], stride=2)

        self.avgpool = nn.AdaptiveAvgPool2d((1, 1))
        self.fc = nn.Linear(self.in_channels, output_dim)

        # DeQuantStub converts tensors from quantized to floating point
        # self.dequant = torch.quantization.DeQuantStub()




    def get_resnet_layer(self, block, n_blocks, channels, stride=1):

        layers = []

        if self.in_channels != block.expansion * channels:
            downsample = True
        else:
            downsample = False

        layers.append(block(self.in_channels, channels, stride, downsample))

        for i in range(1, n_blocks):
            layers.append(block(block.expansion * channels, channels))

        self.in_channels = block.expansion * channels

        return nn.Sequential(*layers)

    def forward(self, x):
        # manually specify where tensors will be converted from floating
        # point to quantized in the quantized model
        # x = self.quant(x)
        x = self.conv1(x)
        x = self.bn1(x)
        x = self.relu(x)
        x = self.maxpool(x)

        x = self.layer1(x)
        x = self.layer2(x)
        x = self.layer3(x)
        x = self.layer4(x)

        x = self.avgpool(x)

        h = x.view(x.shape[0], -1)

        x = self.fc(h)
        # manually specify where tensors will be converted from quantized
        # to floating point in the quantized model
        # x = self.dequant(x)

        return x, h


class BasicBlock(nn.Module):

    expansion = 1

    def __init__(self, in_channels, out_channels, stride=1, downsample=False):
        super().__init__()
        # QuantStub converts tensors from floating point to quantized
        self.quant = torch.quantization.QuantStub()

        self.conv1 = nn.Conv2d(
            in_channels,
            out_channels,
            kernel_size=3,
            stride=stride,
            padding=1,
            bias=False,
        )
        self.bn1 = nn.BatchNorm2d(out_channels)

        self.conv2 = nn.Conv2d(
            out_channels, out_channels, kernel_size=3, stride=1, padding=1, bias=False
        )
        self.bn2 = nn.BatchNorm2d(out_channels)

        self.relu = nn.ReLU(inplace=True)


        if downsample:
            conv = nn.Conv2d(
                in_channels, out_channels, kernel_size=1, stride=stride, bias=False
            )
            bn = nn.BatchNorm2d(out_channels)
            downsample = nn.Sequential(conv, bn)
        else:
            downsample = None

        self.downsample = downsample
        # DeQuantStub converts tensors from quantized to floating point
        self.dequant = torch.quantization.DeQuantStub()

    def forward(self, x):
        # manually specify where tensors will be converted from floating
        # point to quantized in the quantized model
        x = self.quant(x)

        i = x

        x = self.conv1(x)
        x = self.bn1(x)
        x = self.relu(x)

        x = self.conv2(x)
        x = self.bn2(x)

        if self.downsample is not None:
            i = self.quant(i)
            i = self.downsample(i)
            i = self.dequant(i)

        x += i
        x = self.relu(x)
        # manually specify where tensors will be converted from quantized
        # to floating point in the quantized model
        x = self.dequant(x)
        return x


ResNetConfig = namedtuple("ResNetConfig", ["block", "n_blocks", "channels"])

resnet18_config = ResNetConfig(
    block=BasicBlock, n_blocks=[2, 2, 2, 2], channels=[64, 128, 256, 512]
)

</code></pre>
<p>About the qconfig settings:</p>
<pre><code class="lang-auto">backend = "fbgemm"  # x86 machine
torch.backends.quantized.engine = backend
model_q.qconfig = torch.quantization.get_default_qconfig(backend)
</code></pre>
<p>Depending on where I place QuantStub, DeQuantStub, I obtain this error or the one mentioned above.<br>
<a href="https://discuss.pytorch.org/t/could-not-run-aten-quantize-per-tensor-with-arguments-from-the-quantizedcpu-backend/151382/4">this error</a></p>
<p>Thanks a lot for your help <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=12" title=":slight_smile:" class="emoji" alt=":slight_smile:" loading="lazy" width="20" height="20"><br>
Please let me know if you need any additional inputs</p>

Vasiliy_Kuznetsov: <p>hi <a class="mention" href="/u/sarramrg">@sarramrg</a> , have you checked out <a href="https://pytorch.org/docs/master/quantization.html#passing-a-non-quantized-tensor-into-a-quantized-kernel" class="inline-onebox" rel="noopener nofollow ugc">Quantization — PyTorch master documentation</a> ? It has some example code on what this error means and how to resolve it.</p>
<p>Also, have you considered using FX graph mode quantization (<a href="https://pytorch.org/docs/master/quantization.html#prototype-fx-graph-mode-quantization" class="inline-onebox" rel="noopener nofollow ugc">Quantization — PyTorch master documentation</a> )?  As long as your model is symbolically traceable, this workflow will place the quant/dequant for you automatically.</p>

Vikram_M: <pre><code class="lang-auto">from torchvision.io import read_image
from torchvision.models import resnet50, ResNet50_Weights
import torch
weights = ResNet50_Weights.DEFAULT
model = resnet50(weights=weights)
model.eval()
model.qconfig = torch.quantization.get_default_qconfig('fbgemm')
qmodel = torch.quantization.prepare(model,inplace=False)
qmodel.eval()
qmodel = torch.quantization.convert(qmodel)
</code></pre>
<p>This is the model which I used, later got the above error, can you pls help?</p>
<pre><code class="lang-auto">qmodel(batch)
</code></pre>

jerryzh168: <p>I think you’ll need to get resnet50 from <a href="https://github.com/pytorch/vision/tree/main/torchvision/models/quantization" class="inline-onebox" rel="noopener nofollow ugc">vision/torchvision/models/quantization at main · pytorch/vision · GitHub</a> instead</p>