Error when converting quantization aware trained model for evaluation

username: Mxbonn
cooked: <p>I get the error:<br>
<code>Didn't find kernel to dispatch to for operator 'aten::fake_quantize_per_tensor_affine'.  Tried to look up kernel for dispatch key 'QuantizedCPUTensorId'. Registered dispatch keys are: [CUDATensorId, CPUTensorId, VariableTensorId].</code><br>
when trying to evaluate the model after QAT.<br>
My workflow is the same as in the tutorial/documentation:</p>
<pre><code class="lang-python">model.train()
prepare_qat(model, inplace=True)
train(model, run_args)
eval_model = convert(model, inplace=False)
eval(eval_model)
</code></pre>

username: jerryzh168
cooked: <p>This means you are feeding a quantized tensor to fake_quantize op, which only supports unquantized tensor, probably because the op with FakeQuantize is not swapped out somehow after the convert, can you paste the model?</p>

username: Mxbonn
cooked: <p>My original model looks like:</p>
<pre><code class="lang-auto">ResNetCifar10(
  (conv1): Conv2d(3, 16, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
  (bn1): BatchNorm2d(16, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
  (relu): ReLU()
  (layer1): Sequential(
    (0): BasicBlockCifar10(
      (conv1): Conv2d(16, 16, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
      (bn1): BatchNorm2d(16, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
      (relu1): ReLU()
      (conv2): Conv2d(16, 16, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
      (bn2): BatchNorm2d(16, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
      (relu2): ReLU()
      (shortcut): Sequential()
    )
    (1): BasicBlockCifar10(
      (conv1): Conv2d(16, 16, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
      (bn1): BatchNorm2d(16, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
      (relu1): ReLU()
      (conv2): Conv2d(16, 16, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
      (bn2): BatchNorm2d(16, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
      (relu2): ReLU()
      (shortcut): Sequential()
    )
    (2): BasicBlockCifar10(
      (conv1): Conv2d(16, 16, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
      (bn1): BatchNorm2d(16, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
      (relu1): ReLU()
      (conv2): Conv2d(16, 16, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
      (bn2): BatchNorm2d(16, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
      (relu2): ReLU()
      (shortcut): Sequential()
    )
  )
  (layer2): Sequential(
    (0): BasicBlockCifar10(
      (conv1): Conv2d(16, 32, kernel_size=(3, 3), stride=(2, 2), padding=(1, 1), bias=False)
      (bn1): BatchNorm2d(32, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
      (relu1): ReLU()
      (conv2): Conv2d(32, 32, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
      (bn2): BatchNorm2d(32, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
      (relu2): ReLU()
      (shortcut): LambdaLayer()
    )
    (1): BasicBlockCifar10(
      (conv1): Conv2d(32, 32, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
      (bn1): BatchNorm2d(32, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
      (relu1): ReLU()
      (conv2): Conv2d(32, 32, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
      (bn2): BatchNorm2d(32, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
      (relu2): ReLU()
      (shortcut): Sequential()
    )
    (2): BasicBlockCifar10(
      (conv1): Conv2d(32, 32, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
      (bn1): BatchNorm2d(32, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
      (relu1): ReLU()
      (conv2): Conv2d(32, 32, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
      (bn2): BatchNorm2d(32, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
      (relu2): ReLU()
      (shortcut): Sequential()
    )
  )
  (layer3): Sequential(
    (0): BasicBlockCifar10(
      (conv1): Conv2d(32, 64, kernel_size=(3, 3), stride=(2, 2), padding=(1, 1), bias=False)
      (bn1): BatchNorm2d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
      (relu1): ReLU()
      (conv2): Conv2d(64, 64, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
      (bn2): BatchNorm2d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
      (relu2): ReLU()
      (shortcut): LambdaLayer()
    )
    (1): BasicBlockCifar10(
      (conv1): Conv2d(64, 64, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
      (bn1): BatchNorm2d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
      (relu1): ReLU()
      (conv2): Conv2d(64, 64, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
      (bn2): BatchNorm2d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
      (relu2): ReLU()
      (shortcut): Sequential()
    )
    (2): BasicBlockCifar10(
      (conv1): Conv2d(64, 64, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
      (bn1): BatchNorm2d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
      (relu1): ReLU()
      (conv2): Conv2d(64, 64, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
      (bn2): BatchNorm2d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
      (relu2): ReLU()
      (shortcut): Sequential()
    )
  )
  (linear): Linear(in_features=64, out_features=10, bias=True)
)
</code></pre>
<p>I then convert it to a model that can be trained using QAT:<br>
(quant and dequant, fuse layers and torch.quantization.prepare_qat)</p>
<pre><code class="lang-auto">QuantWrapper(
  (quant): QuantStub(
    (observer): FakeQuantize(
      fake_quant_enabled=True, observer_enabled=True,            scale=None, zero_point=None
      (observer): MovingAverageMinMaxObserver(min_val=None, max_val=None)
    )
  )
  (dequant): DeQuantStub(
    (observer): FakeQuantize(
      fake_quant_enabled=True, observer_enabled=True,            scale=None, zero_point=None
      (observer): MovingAverageMinMaxObserver(min_val=None, max_val=None)
    )
  )
  (module): ResNetCifar10(
    (conv1): ConvBnReLU2d(
      3, 16, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False
      (observer): FakeQuantize(
        fake_quant_enabled=True, observer_enabled=True,            scale=None, zero_point=None
        (observer): MovingAverageMinMaxObserver(min_val=None, max_val=None)
      )
      (weight_fake_quant): FakeQuantize(
        fake_quant_enabled=True, observer_enabled=True,            scale=None, zero_point=None
        (observer): MovingAverageMinMaxObserver(min_val=None, max_val=None)
      )
    )
    (bn1): Identity()
    (relu): Identity()
    (layer1): Sequential(
      (0): BasicBlockCifar10(
        (conv1): ConvBnReLU2d(
          16, 16, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False
          (observer): FakeQuantize(
            fake_quant_enabled=True, observer_enabled=True,            scale=None, zero_point=None
            (observer): MovingAverageMinMaxObserver(min_val=None, max_val=None)
          )
          (weight_fake_quant): FakeQuantize(
            fake_quant_enabled=True, observer_enabled=True,            scale=None, zero_point=None
            (observer): MovingAverageMinMaxObserver(min_val=None, max_val=None)
          )
        )
        (bn1): Identity()
        (relu1): Identity()
        (conv2): ConvBn2d(
          16, 16, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False
          (observer): FakeQuantize(
            fake_quant_enabled=True, observer_enabled=True,            scale=None, zero_point=None
            (observer): MovingAverageMinMaxObserver(min_val=None, max_val=None)
          )
          (weight_fake_quant): FakeQuantize(
            fake_quant_enabled=True, observer_enabled=True,            scale=None, zero_point=None
            (observer): MovingAverageMinMaxObserver(min_val=None, max_val=None)
          )
        )
        (bn2): Identity()
        (relu2): ReLU(
          (observer): FakeQuantize(
            fake_quant_enabled=True, observer_enabled=True,            scale=None, zero_point=None
            (observer): MovingAverageMinMaxObserver(min_val=None, max_val=None)
          )
        )
        (shortcut): Sequential(
          (observer): FakeQuantize(
            fake_quant_enabled=True, observer_enabled=True,            scale=None, zero_point=None
            (observer): MovingAverageMinMaxObserver(min_val=None, max_val=None)
          )
        )
      )
      (1): BasicBlockCifar10(
        (conv1): ConvBnReLU2d(
          16, 16, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False
          (observer): FakeQuantize(
            fake_quant_enabled=True, observer_enabled=True,            scale=None, zero_point=None
            (observer): MovingAverageMinMaxObserver(min_val=None, max_val=None)
          )
          (weight_fake_quant): FakeQuantize(
            fake_quant_enabled=True, observer_enabled=True,            scale=None, zero_point=None
            (observer): MovingAverageMinMaxObserver(min_val=None, max_val=None)
          )
        )
        (bn1): Identity()
        (relu1): Identity()
        (conv2): ConvBn2d(
          16, 16, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False
          (observer): FakeQuantize(
            fake_quant_enabled=True, observer_enabled=True,            scale=None, zero_point=None
            (observer): MovingAverageMinMaxObserver(min_val=None, max_val=None)
          )
          (weight_fake_quant): FakeQuantize(
            fake_quant_enabled=True, observer_enabled=True,            scale=None, zero_point=None
            (observer): MovingAverageMinMaxObserver(min_val=None, max_val=None)
          )
        )
        (bn2): Identity()
        (relu2): ReLU(
          (observer): FakeQuantize(
            fake_quant_enabled=True, observer_enabled=True,            scale=None, zero_point=None
            (observer): MovingAverageMinMaxObserver(min_val=None, max_val=None)
          )
        )
        (shortcut): Sequential(
          (observer): FakeQuantize(
            fake_quant_enabled=True, observer_enabled=True,            scale=None, zero_point=None
            (observer): MovingAverageMinMaxObserver(min_val=None, max_val=None)
          )
        )
      )
      (2): BasicBlockCifar10(
        (conv1): ConvBnReLU2d(
          16, 16, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False
          (observer): FakeQuantize(
            fake_quant_enabled=True, observer_enabled=True,            scale=None, zero_point=None
            (observer): MovingAverageMinMaxObserver(min_val=None, max_val=None)
          )
          (weight_fake_quant): FakeQuantize(
            fake_quant_enabled=True, observer_enabled=True,            scale=None, zero_point=None
            (observer): MovingAverageMinMaxObserver(min_val=None, max_val=None)
          )
        )
        (bn1): Identity()
        (relu1): Identity()
        (conv2): ConvBn2d(
          16, 16, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False
          (observer): FakeQuantize(
            fake_quant_enabled=True, observer_enabled=True,            scale=None, zero_point=None
            (observer): MovingAverageMinMaxObserver(min_val=None, max_val=None)
          )
          (weight_fake_quant): FakeQuantize(
            fake_quant_enabled=True, observer_enabled=True,            scale=None, zero_point=None
            (observer): MovingAverageMinMaxObserver(min_val=None, max_val=None)
          )
        )
        (bn2): Identity()
        (relu2): ReLU(
          (observer): FakeQuantize(
            fake_quant_enabled=True, observer_enabled=True,            scale=None, zero_point=None
            (observer): MovingAverageMinMaxObserver(min_val=None, max_val=None)
          )
        )
        (shortcut): Sequential(
          (observer): FakeQuantize(
            fake_quant_enabled=True, observer_enabled=True,            scale=None, zero_point=None
            (observer): MovingAverageMinMaxObserver(min_val=None, max_val=None)
          )
        )
      )
    )
    (layer2): Sequential(
      (0): BasicBlockCifar10(
        (conv1): ConvBnReLU2d(
          16, 32, kernel_size=(3, 3), stride=(2, 2), padding=(1, 1), bias=False
          (observer): FakeQuantize(
            fake_quant_enabled=True, observer_enabled=True,            scale=None, zero_point=None
            (observer): MovingAverageMinMaxObserver(min_val=None, max_val=None)
          )
          (weight_fake_quant): FakeQuantize(
            fake_quant_enabled=True, observer_enabled=True,            scale=None, zero_point=None
            (observer): MovingAverageMinMaxObserver(min_val=None, max_val=None)
          )
        )
        (bn1): Identity()
        (relu1): Identity()
        (conv2): ConvBn2d(
          32, 32, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False
          (observer): FakeQuantize(
            fake_quant_enabled=True, observer_enabled=True,            scale=None, zero_point=None
            (observer): MovingAverageMinMaxObserver(min_val=None, max_val=None)
          )
          (weight_fake_quant): FakeQuantize(
            fake_quant_enabled=True, observer_enabled=True,            scale=None, zero_point=None
            (observer): MovingAverageMinMaxObserver(min_val=None, max_val=None)
          )
        )
        (bn2): Identity()
        (relu2): ReLU(
          (observer): FakeQuantize(
            fake_quant_enabled=True, observer_enabled=True,            scale=None, zero_point=None
            (observer): MovingAverageMinMaxObserver(min_val=None, max_val=None)
          )
        )
        (shortcut): LambdaLayer()
      )
      (1): BasicBlockCifar10(
        (conv1): ConvBnReLU2d(
          32, 32, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False
          (observer): FakeQuantize(
            fake_quant_enabled=True, observer_enabled=True,            scale=None, zero_point=None
            (observer): MovingAverageMinMaxObserver(min_val=None, max_val=None)
          )
          (weight_fake_quant): FakeQuantize(
            fake_quant_enabled=True, observer_enabled=True,            scale=None, zero_point=None
            (observer): MovingAverageMinMaxObserver(min_val=None, max_val=None)
          )
        )
        (bn1): Identity()
        (relu1): Identity()
        (conv2): ConvBn2d(
          32, 32, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False
          (observer): FakeQuantize(
            fake_quant_enabled=True, observer_enabled=True,            scale=None, zero_point=None
            (observer): MovingAverageMinMaxObserver(min_val=None, max_val=None)
          )
          (weight_fake_quant): FakeQuantize(
            fake_quant_enabled=True, observer_enabled=True,            scale=None, zero_point=None
            (observer): MovingAverageMinMaxObserver(min_val=None, max_val=None)
          )
        )
        (bn2): Identity()
        (relu2): ReLU(
          (observer): FakeQuantize(
            fake_quant_enabled=True, observer_enabled=True,            scale=None, zero_point=None
            (observer): MovingAverageMinMaxObserver(min_val=None, max_val=None)
          )
        )
        (shortcut): Sequential(
          (observer): FakeQuantize(
            fake_quant_enabled=True, observer_enabled=True,            scale=None, zero_point=None
            (observer): MovingAverageMinMaxObserver(min_val=None, max_val=None)
          )
        )
      )
      (2): BasicBlockCifar10(
        (conv1): ConvBnReLU2d(
          32, 32, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False
          (observer): FakeQuantize(
            fake_quant_enabled=True, observer_enabled=True,            scale=None, zero_point=None
            (observer): MovingAverageMinMaxObserver(min_val=None, max_val=None)
          )
          (weight_fake_quant): FakeQuantize(
            fake_quant_enabled=True, observer_enabled=True,            scale=None, zero_point=None
            (observer): MovingAverageMinMaxObserver(min_val=None, max_val=None)
          )
        )
        (bn1): Identity()
        (relu1): Identity()
        (conv2): ConvBn2d(
          32, 32, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False
          (observer): FakeQuantize(
            fake_quant_enabled=True, observer_enabled=True,            scale=None, zero_point=None
            (observer): MovingAverageMinMaxObserver(min_val=None, max_val=None)
          )
          (weight_fake_quant): FakeQuantize(
            fake_quant_enabled=True, observer_enabled=True,            scale=None, zero_point=None
            (observer): MovingAverageMinMaxObserver(min_val=None, max_val=None)
          )
        )
        (bn2): Identity()
        (relu2): ReLU(
          (observer): FakeQuantize(
            fake_quant_enabled=True, observer_enabled=True,            scale=None, zero_point=None
            (observer): MovingAverageMinMaxObserver(min_val=None, max_val=None)
          )
        )
        (shortcut): Sequential(
          (observer): FakeQuantize(
            fake_quant_enabled=True, observer_enabled=True,            scale=None, zero_point=None
            (observer): MovingAverageMinMaxObserver(min_val=None, max_val=None)
          )
        )
      )
    )
    (layer3): Sequential(
      (0): BasicBlockCifar10(
        (conv1): ConvBnReLU2d(
          32, 64, kernel_size=(3, 3), stride=(2, 2), padding=(1, 1), bias=False
          (observer): FakeQuantize(
            fake_quant_enabled=True, observer_enabled=True,            scale=None, zero_point=None
            (observer): MovingAverageMinMaxObserver(min_val=None, max_val=None)
          )
          (weight_fake_quant): FakeQuantize(
            fake_quant_enabled=True, observer_enabled=True,            scale=None, zero_point=None
            (observer): MovingAverageMinMaxObserver(min_val=None, max_val=None)
          )
        )
        (bn1): Identity()
        (relu1): Identity()
        (conv2): ConvBn2d(
          64, 64, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False
          (observer): FakeQuantize(
            fake_quant_enabled=True, observer_enabled=True,            scale=None, zero_point=None
            (observer): MovingAverageMinMaxObserver(min_val=None, max_val=None)
          )
          (weight_fake_quant): FakeQuantize(
            fake_quant_enabled=True, observer_enabled=True,            scale=None, zero_point=None
            (observer): MovingAverageMinMaxObserver(min_val=None, max_val=None)
          )
        )
        (bn2): Identity()
        (relu2): ReLU(
          (observer): FakeQuantize(
            fake_quant_enabled=True, observer_enabled=True,            scale=None, zero_point=None
            (observer): MovingAverageMinMaxObserver(min_val=None, max_val=None)
          )
        )
        (shortcut): LambdaLayer()
      )
      (1): BasicBlockCifar10(
        (conv1): ConvBnReLU2d(
          64, 64, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False
          (observer): FakeQuantize(
            fake_quant_enabled=True, observer_enabled=True,            scale=None, zero_point=None
            (observer): MovingAverageMinMaxObserver(min_val=None, max_val=None)
          )
          (weight_fake_quant): FakeQuantize(
            fake_quant_enabled=True, observer_enabled=True,            scale=None, zero_point=None
            (observer): MovingAverageMinMaxObserver(min_val=None, max_val=None)
          )
        )
        (bn1): Identity()
        (relu1): Identity()
        (conv2): ConvBn2d(
          64, 64, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False
          (observer): FakeQuantize(
            fake_quant_enabled=True, observer_enabled=True,            scale=None, zero_point=None
            (observer): MovingAverageMinMaxObserver(min_val=None, max_val=None)
          )
          (weight_fake_quant): FakeQuantize(
            fake_quant_enabled=True, observer_enabled=True,            scale=None, zero_point=None
            (observer): MovingAverageMinMaxObserver(min_val=None, max_val=None)
          )
        )
        (bn2): Identity()
        (relu2): ReLU(
          (observer): FakeQuantize(
            fake_quant_enabled=True, observer_enabled=True,            scale=None, zero_point=None
            (observer): MovingAverageMinMaxObserver(min_val=None, max_val=None)
          )
        )
        (shortcut): Sequential(
          (observer): FakeQuantize(
            fake_quant_enabled=True, observer_enabled=True,            scale=None, zero_point=None
            (observer): MovingAverageMinMaxObserver(min_val=None, max_val=None)
          )
        )
      )
      (2): BasicBlockCifar10(
        (conv1): ConvBnReLU2d(
          64, 64, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False
          (observer): FakeQuantize(
            fake_quant_enabled=True, observer_enabled=True,            scale=None, zero_point=None
            (observer): MovingAverageMinMaxObserver(min_val=None, max_val=None)
          )
          (weight_fake_quant): FakeQuantize(
            fake_quant_enabled=True, observer_enabled=True,            scale=None, zero_point=None
            (observer): MovingAverageMinMaxObserver(min_val=None, max_val=None)
          )
        )
        (bn1): Identity()
        (relu1): Identity()
        (conv2): ConvBn2d(
          64, 64, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False
          (observer): FakeQuantize(
            fake_quant_enabled=True, observer_enabled=True,            scale=None, zero_point=None
            (observer): MovingAverageMinMaxObserver(min_val=None, max_val=None)
          )
          (weight_fake_quant): FakeQuantize(
            fake_quant_enabled=True, observer_enabled=True,            scale=None, zero_point=None
            (observer): MovingAverageMinMaxObserver(min_val=None, max_val=None)
          )
        )
        (bn2): Identity()
        (relu2): ReLU(
          (observer): FakeQuantize(
            fake_quant_enabled=True, observer_enabled=True,            scale=None, zero_point=None
            (observer): MovingAverageMinMaxObserver(min_val=None, max_val=None)
          )
        )
        (shortcut): Sequential(
          (observer): FakeQuantize(
            fake_quant_enabled=True, observer_enabled=True,            scale=None, zero_point=None
            (observer): MovingAverageMinMaxObserver(min_val=None, max_val=None)
          )
        )
      )
    )
    (linear): Linear(
      in_features=64, out_features=10, bias=True
      (observer): FakeQuantize(
        fake_quant_enabled=True, observer_enabled=True,            scale=None, zero_point=None
        (observer): MovingAverageMinMaxObserver(min_val=None, max_val=None)
      )
      (weight_fake_quant): FakeQuantize(
        fake_quant_enabled=True, observer_enabled=True,            scale=None, zero_point=None
        (observer): MovingAverageMinMaxObserver(min_val=None, max_val=None)
      )
    )
  )
)
</code></pre>
<p>Which can be trained, but I then try to convert it to a evaluation model using<br>
<code>torch.quantization.convert</code> like in the tutorial:<br>
The converted model looks like:</p>
<pre><code class="lang-auto">QuantWrapper(
  (quant): Quantize(scale=tensor([0.0203]), zero_point=tensor([120]), dtype=torch.quint8)
  (dequant): DeQuantize()
  (module): ResNetCifar10(
    (conv1): QuantizedConvReLU2d(3, 16, kernel_size=(3, 3), stride=(1, 1), scale=0.00903782807290554, zero_point=0, padding=(1, 1))
    (bn1): Identity()
    (relu): Identity()
    (layer1): Sequential(
      (0): BasicBlockCifar10(
        (conv1): QuantizedConvReLU2d(16, 16, kernel_size=(3, 3), stride=(1, 1), scale=0.009634326212108135, zero_point=0, padding=(1, 1))
        (bn1): Identity()
        (relu1): Identity()
        (conv2): QuantizedConv2d(16, 16, kernel_size=(3, 3), stride=(1, 1), scale=0.018731188029050827, zero_point=130, padding=(1, 1))
        (bn2): Identity()
        (relu2): QuantizedReLU()
        (shortcut): Sequential(
          (observer): FakeQuantize(
            fake_quant_enabled=True, observer_enabled=True,            scale=tensor([0.0086]), zero_point=tensor([0])
            (observer): MovingAverageMinMaxObserver(min_val=0.0, max_val=2.1906778812408447)
          )
        )
      )
      (1): BasicBlockCifar10(
        (conv1): QuantizedConvReLU2d(16, 16, kernel_size=(3, 3), stride=(1, 1), scale=0.010925675742328167, zero_point=0, padding=(1, 1))
        (bn1): Identity()
        (relu1): Identity()
        (conv2): QuantizedConv2d(16, 16, kernel_size=(3, 3), stride=(1, 1), scale=0.01969429850578308, zero_point=141, padding=(1, 1))
        (bn2): Identity()
        (relu2): QuantizedReLU()
        (shortcut): Sequential(
          (observer): FakeQuantize(
            fake_quant_enabled=True, observer_enabled=True,            scale=tensor([0.0112]), zero_point=tensor([0])
            (observer): MovingAverageMinMaxObserver(min_val=0.0, max_val=2.8511974811553955)
          )
        )
      )
      (2): BasicBlockCifar10(
        (conv1): QuantizedConvReLU2d(16, 16, kernel_size=(3, 3), stride=(1, 1), scale=0.007963746786117554, zero_point=0, padding=(1, 1))
        (bn1): Identity()
        (relu1): Identity()
        (conv2): QuantizedConv2d(16, 16, kernel_size=(3, 3), stride=(1, 1), scale=0.016999518498778343, zero_point=136, padding=(1, 1))
        (bn2): Identity()
        (relu2): QuantizedReLU()
        (shortcut): Sequential(
          (observer): FakeQuantize(
            fake_quant_enabled=True, observer_enabled=True,            scale=tensor([0.0144]), zero_point=tensor([0])
            (observer): MovingAverageMinMaxObserver(min_val=0.0, max_val=3.661449670791626)
          )
        )
      )
    )
    (layer2): Sequential(
      (0): BasicBlockCifar10(
        (conv1): QuantizedConvReLU2d(16, 32, kernel_size=(3, 3), stride=(2, 2), scale=0.009311596862971783, zero_point=0, padding=(1, 1))
        (bn1): Identity()
        (relu1): Identity()
        (conv2): QuantizedConv2d(32, 32, kernel_size=(3, 3), stride=(1, 1), scale=0.02033022604882717, zero_point=115, padding=(1, 1))
        (bn2): Identity()
        (relu2): QuantizedReLU()
        (shortcut): LambdaLayer()
      )
      (1): BasicBlockCifar10(
        (conv1): QuantizedConvReLU2d(32, 32, kernel_size=(3, 3), stride=(1, 1), scale=0.006638957187533379, zero_point=0, padding=(1, 1))
        (bn1): Identity()
        (relu1): Identity()
        (conv2): QuantizedConv2d(32, 32, kernel_size=(3, 3), stride=(1, 1), scale=0.018797850236296654, zero_point=136, padding=(1, 1))
        (bn2): Identity()
        (relu2): QuantizedReLU()
        (shortcut): Sequential(
          (observer): FakeQuantize(
            fake_quant_enabled=True, observer_enabled=True,            scale=tensor([0.0147]), zero_point=tensor([0])
            (observer): MovingAverageMinMaxObserver(min_val=0.0, max_val=3.7611701488494873)
          )
        )
      )
      (2): BasicBlockCifar10(
        (conv1): QuantizedConvReLU2d(32, 32, kernel_size=(3, 3), stride=(1, 1), scale=0.007839899510145187, zero_point=0, padding=(1, 1))
        (bn1): Identity()
        (relu1): Identity()
        (conv2): QuantizedConv2d(32, 32, kernel_size=(3, 3), stride=(1, 1), scale=0.026490747928619385, zero_point=85, padding=(1, 1))
        (bn2): Identity()
        (relu2): QuantizedReLU()
        (shortcut): Sequential(
          (observer): FakeQuantize(
            fake_quant_enabled=True, observer_enabled=True,            scale=tensor([0.0144]), zero_point=tensor([0])
            (observer): MovingAverageMinMaxObserver(min_val=0.0, max_val=3.678955316543579)
          )
        )
      )
    )
    (layer3): Sequential(
      (0): BasicBlockCifar10(
        (conv1): QuantizedConvReLU2d(32, 64, kernel_size=(3, 3), stride=(2, 2), scale=0.009626154787838459, zero_point=0, padding=(1, 1))
        (bn1): Identity()
        (relu1): Identity()
        (conv2): QuantizedConv2d(64, 64, kernel_size=(3, 3), stride=(1, 1), scale=0.02358696237206459, zero_point=109, padding=(1, 1))
        (bn2): Identity()
        (relu2): QuantizedReLU()
        (shortcut): LambdaLayer()
      )
      (1): BasicBlockCifar10(
        (conv1): QuantizedConvReLU2d(64, 64, kernel_size=(3, 3), stride=(1, 1), scale=0.0076929363422095776, zero_point=0, padding=(1, 1))
        (bn1): Identity()
        (relu1): Identity()
        (conv2): QuantizedConv2d(64, 64, kernel_size=(3, 3), stride=(1, 1), scale=0.02710540033876896, zero_point=124, padding=(1, 1))
        (bn2): Identity()
        (relu2): QuantizedReLU()
        (shortcut): Sequential(
          (observer): FakeQuantize(
            fake_quant_enabled=True, observer_enabled=True,            scale=tensor([0.0241]), zero_point=tensor([0])
            (observer): MovingAverageMinMaxObserver(min_val=0.0, max_val=6.156522750854492)
          )
        )
      )
      (2): BasicBlockCifar10(
        (conv1): QuantizedConvReLU2d(64, 64, kernel_size=(3, 3), stride=(1, 1), scale=0.008539369329810143, zero_point=0, padding=(1, 1))
        (bn1): Identity()
        (relu1): Identity()
        (conv2): QuantizedConv2d(64, 64, kernel_size=(3, 3), stride=(1, 1), scale=0.06625250726938248, zero_point=92, padding=(1, 1))
        (bn2): Identity()
        (relu2): QuantizedReLU()
        (shortcut): Sequential(
          (observer): FakeQuantize(
            fake_quant_enabled=True, observer_enabled=True,            scale=tensor([0.0236]), zero_point=tensor([0])
            (observer): MovingAverageMinMaxObserver(min_val=0.0, max_val=6.006941795349121)
          )
        )
      )
    )
    (linear): QuantizedLinear(in_features=64, out_features=10, scale=0.10857795923948288, zero_point=69)
  )
)
</code></pre>
<p>It appears that some fakequantization nodes remain after converting.<br>
shortcut is an empty <code>nn.Sequential()</code> layer (depending on the network arguments this gets overwritten to handle different strides)</p>
<p>But it running the model results in an error:</p>
<pre><code class="lang-auto">RuntimeError: Didn't find kernel to dispatch to for operator 'aten::sub'. Tried to look up kernel for dispatch key 'QuantizedCPUTensorId'. Registered dispatch keys are: [CUDATensorId, SparseCPUTensorId, VariableTensorId, CPUTensorId, SparseCUDATensorId]
</code></pre>

username: Mxbonn
cooked: <p>Just found the related pytorch issue, so I assume its a known bug?<br>
<aside class="onebox githubissue">
  <header class="source">
      <a href="https://github.com/pytorch/pytorch/issues/28375" target="_blank" rel="nofollow noopener">github.com/pytorch/pytorch</a>
  </header>
  <article class="onebox-body">
    <div class="github-row">
  <div class="github-icon-container" title="Issue">
	  <svg width="60" height="60" class="github-icon" viewbox="0 0 14 16" aria-hidden="true"><path d="M7 2.3c3.14 0 5.7 2.56 5.7 5.7s-2.56 5.7-5.7 5.7A5.71 5.71 0 0 1 1.3 8c0-3.14 2.56-5.7 5.7-5.7zM7 1C3.14 1 0 4.14 0 8s3.14 7 7 7 7-3.14 7-7-3.14-7-7-7zm1 3H6v5h2V4zm0 6H6v2h2v-2z"></path></svg>
  </div>

  <div class="github-info-container">
    <h4>
      <a href="https://github.com/pytorch/pytorch/issues/28375" target="_blank" rel="nofollow noopener">torch.quantization.convert() doesn't remove observer from empty Sequential() module, causing error when scripting</a>
    </h4>

    <div class="github-info">
      <div class="date">
        opened <span class="discourse-local-date" data-format="ll" data-date="2019-10-21" data-time="19:17:42" data-timezone="UTC">07:17PM - 21 Oct 19 UTC</span>
      </div>


      <div class="user">
        <a href="https://github.com/hx89" target="_blank" rel="nofollow noopener">
          <img alt="hx89" src="https://avatars1.githubusercontent.com/u/43588773?v=4" class="onebox-avatar-inline" width="20" height="20">
          hx89
        </a>
      </div>
    </div>
  </div>
</div>

<div class="github-row">
  <p class="github-content">🐛 Bug
torch.quantization.convert() doesn't remove observer from empty Sequential() module, causing errors when scripting the model since observers are not scriptable.
Steps to...</p>
</div>

<div class="labels">
    <span style="display:inline-block;margin-top:2px;background-color: #B8B8B8;padding: 2px;border-radius: 4px;color: #fff;margin-left: 3px;">jit</span>
    <span style="display:inline-block;margin-top:2px;background-color: #B8B8B8;padding: 2px;border-radius: 4px;color: #fff;margin-left: 3px;">quantization</span>
    <span style="display:inline-block;margin-top:2px;background-color: #B8B8B8;padding: 2px;border-radius: 4px;color: #fff;margin-left: 3px;">triaged</span>
</div>

  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>
</p>

username: Zafar
cooked: <p>The issue is that there is a dangling <code>Sequential</code> module. When seeing an empty <code>Sequential</code>, the logic is  (erroneously) inserting the observer in there, which causes it to have the error you are seeing. This is fixed in <a href="https://github.com/pytorch/pytorch/pull/28384" rel="nofollow noopener">https://github.com/pytorch/pytorch/pull/28384</a> – if you rebase to <code>master</code>, it will fix it. Alternatively, you can modify the model to drop the shortcuts.</p>
<p>If you are still seeing the error after rebasing, please, let us know <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>