Error during QAT training of ResNet50

JohnnY_Sirorezka: <p>Hello,</p>
<p>Could smn please help me to understand how can I fix my code. I’m doing a QAT training of ResNet50 and encountering an error in one of the convoluational layers:</p>
<blockquote>
<p>RuntimeError: <code>zero_point</code> must be between <code>quant_min</code> and <code>quant_max</code></p>
</blockquote>
<p>Code is relatively simple.</p>
<pre><code class="lang-auto">from torchvision.models import resnet50, ResNet50_Weights

from torch.ao.quantization import (
    fuse_modules, 
    QuantWrapper, 
    get_default_qat_qconfig)

qconfig  = get_default_qat_qconfig('fbgemm')


def apply_recursive(model, qconfig):
  model.qconfig = qconfig
  # qconfig should be applied to all children
  for name, module in model.named_children():
      apply_recursive(module, qconfig)
  return model


def get_qat_fused_manual():

  model_fp32 = resnet50(weights=ResNet50_Weights.IMAGENET1K_V2)
  model_fp32.eval()

  model_fp32 = apply_recursive(model_fp32, qconfig)
  
  # fuse layers
  fuse_modules(model_fp32,[['conv1', 'bn1', 'relu']], inplace=True)

  for layer in ['layer1','layer2','layer3', 'layer4']:
    module = model_fp32.get_submodule(layer)
    n_iter = len(module)
    for i in range(n_iter):
      submodule = module.get_submodule(str(i))
      fuse_modules(submodule,[['conv1', 'bn1'], ['conv2', 'bn2'], ['conv3', 'bn3', 'relu']], inplace=True)

      # replace in downsample
      for name, m in submodule.named_children():
        if name == 'downsample':
          fuse_modules(submodule,[['downsample.0', 'downsample.1']], inplace=True)

  model_fused = model_fp32

  model_fused = QuantWrapper(model_fp32)
  model_fused = torch.ao.quantization.prepare_qat(model_fused.train())
  return model_fused

qat_model = get_qat_fused_manual()

data_loader_train = torch.utils.data.DataLoader(
    dataset_train,
    batch_size = 32,
    sampler=val_sampler,
    num_workers=args.workers
)


device = 'cuda'
parameters = qat_model.parameters()
criterion = torch.nn.CrossEntropyLoss(label_smoothing=0)
optimizer = torch.optim.SGD(
            parameters,
            lr=1e-5,
            momentum=0.9,
            # weight_decay=args.weight_decay,
            nesterov="nesterov" 
        )

qat_model.to(device)
qat_model.train()
for iter, (image, target) in enumerate(data_loader_train):
    print (iter)
    image, target = image.to(device), target.to(device)
    output = qat_model(image)
    loss = criterion(output, target)
    optimizer.zero_grad()
    loss.backward()
    optimizer.step()
</code></pre>
<p>From what I see the training is happening for several iterations and then the <code>model.forward</code> method triggers an error</p>
<pre><code class="lang-auto">
/usr/local/lib/python3.10/dist-packages/torch/ao/quantization/stubs.py in forward(self, X)
     61     def forward(self, X):
     62         X = self.quant(X)
---&gt; 63         X = self.module(X)
     64         return self.dequant(X)

....

/usr/local/lib/python3.10/dist-packages/torchvision/models/resnet.py in _forward_impl(self, x)
    266     def _forward_impl(self, x: Tensor) -&gt; Tensor:
    267         # See note [TorchScript super()]
--&gt; 268         x = self.conv1(x)
    269         x = self.bn1(x)
    270         x = self.relu(x)


...

/usr/local/lib/python3.10/dist-packages/torch/ao/quantization/fake_quantize.py in forward(self, X)
    351 
    352     def forward(self, X: torch.Tensor) -&gt; torch.Tensor:
--&gt; 353         return torch.fused_moving_avg_obs_fake_quant(
    354             X,
    355             self.observer_enabled,

RuntimeError: `zero_point` must be between `quant_min` and `quant_max`.

</code></pre>
<p>So could anyone point me what is the problem here? Unable to find anything related on the internet.<br>
Thank you.</p>

HDCharles: <p>can you print what the zero_point is? i’d probably edit fake_quantize.py above the return line that’s throwing the error to print the zero_point.</p>
<p>if i were to guess, i dont see any quant stubs being inserted so i would guess that it has something to do with that.</p>
<p>Otherwise, does it work if you do qat <a href="https://pytorch.org/docs/stable/quantization.html#quantization-aware-training-for-static-quantization" rel="noopener nofollow ugc">normally</a>?<br>
without fusion?</p>

JohnnY_Sirorezka: <p>Ok, so before the last forward pass it was looking like this<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/7/0/70cd1b73b94a9e7c4b5aabb38f8099a988d7c6ed.png" data-download-href="https://discuss.pytorch.org/uploads/default/70cd1b73b94a9e7c4b5aabb38f8099a988d7c6ed" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/7/0/70cd1b73b94a9e7c4b5aabb38f8099a988d7c6ed_2_690x363.png" alt="image" data-base62-sha1="g5SUQCcc9n95ku7Gase0pYC337D" width="690" height="363" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/7/0/70cd1b73b94a9e7c4b5aabb38f8099a988d7c6ed_2_690x363.png, https://discuss.pytorch.org/uploads/default/optimized/3X/7/0/70cd1b73b94a9e7c4b5aabb38f8099a988d7c6ed_2_1035x544.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/7/0/70cd1b73b94a9e7c4b5aabb38f8099a988d7c6ed.png 2x" data-dominant-color="F2F2F2"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1077×568 101 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

JohnnY_Sirorezka: <p>After making one more forward pass:<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/6/b/6b7905a042772dacf90ced4cc38388709c3ae8cc.png" data-download-href="https://discuss.pytorch.org/uploads/default/6b7905a042772dacf90ced4cc38388709c3ae8cc" title="image"><img src="https://discuss.pytorch.org/uploads/default/original/3X/6/b/6b7905a042772dacf90ced4cc38388709c3ae8cc.png" alt="image" data-base62-sha1="fkKnpYWWR7BoKgbadtJIMdfDHt2" width="690" height="309" data-dominant-color="F0F0F0"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1097×492 26.3 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

JohnnY_Sirorezka: <p>After playing a bit more with this I found out that replacing:</p>
<p><code>fuse_module   -&gt;  fuse_modules_qat</code></p>
<p>fixes my issue with Observers. Unfortunately this is not mentioned in the link that you’ve mentioned (basically the main manual on qat) or in any other manuals…<br>
<a href="https://pytorch.org/docs/stable/quantization.html#quantization-aware-training-for-static-quantization" class="onebox" target="_blank" rel="noopener nofollow ugc">https://pytorch.org/docs/stable/quantization.html#quantization-aware-training-for-static-quantization</a></p>
<p>In any case, appreciate such fast response, HDCharles.</p>
<p>Thanks.</p>