Inference accuracy mismatch between original, quantized, dequantized model

crystalbreezy: <p>I’m trying to do quantize-aware training on a VGG11 model for CIFAR10 dataset and finally want to do a dequantization on the quantized int8 weights to see the accuracy impact. The training should be successful and both the original fp32 model and the quantized model give reasonable inference accuracy, which is approximately 90%. However, when I was trying to do manual dequantization on the quantized model, the inference accuracy of the dequantized model dropped down to about 10%. Since I’m using the same int8 weight and the q_scale and zero_points during dequantization, I used to believe that the inference accuracy should be similar to the quantized model. Can anyone tell me what could be the potential reason?</p>
<p>Here is my code: (quantized_model and original_fp_model gives 90% accuracy and the other two gives 10% accuracy)<br>
import torch<br>
import torchvision<br>
import torch.quantization as quantization<br>
import torchvision.models as models<br>
import torch.nn as nn<br>
import torchvision.transforms as transforms</p>
<p>class QATVGG11(nn.Module):<br>
def <strong>init</strong>(self, feature_output_size):<br>
super(QATVGG11, self).<strong>init</strong>()<br>
self.features = models.vgg11(pretrained=True).features<br>
self.classifier = nn.Sequential(<br>
nn.Linear(feature_output_size, 4096),<br>
nn.ReLU(inplace=True),<br>
<span class="hashtag-raw">#nn</span>.Dropout(p=0.5),<br>
nn.Linear(4096, 10))<br>
self.quant = quantization.QuantStub()<br>
self.dequant = quantization.DeQuantStub()</p>
<pre><code>def forward(self, x):
    x = self.quant(x)
    x = self.features(x)
    x = torch.flatten(x, 1)
    x = self.classifier(x)
    x = self.dequant(x)
    return x
</code></pre>
<p>def load_quantized_model(model_save_path):<br>
# Recreate the model architecture<br>
model = QATVGG11(feature_output_size=25088)  # Provide the correct feature_output_size used in training<br>
model.to(‘cpu’)  # Load to CPU for simplicity<br>
model.qconfig = quantization.get_default_qat_qconfig(‘fbgemm’)<br>
model = quantization.prepare_qat(model, inplace=False)<br>
model.load_state_dict(torch.load(model_save_path))<br>
model.eval()  # Set the model to evaluation mode<br>
quantized_model = quantization.convert(model, inplace=False)<br>
return quantized_model</p>
<h1><a name="p-453518-path-to-the-saved-qat-model-1" class="anchor" href="#p-453518-path-to-the-saved-qat-model-1"></a>Path to the saved qat model</h1>
<p>model_save_path = ‘vgg11_trained_fp32_nodropout.pth’<br>
quantized_model = load_quantized_model(model_save_path)</p>
<p>def dequantize_modified_tensor(quantized_tensor, update_mode):<br>
if not quantized_tensor.is_quantized:<br>
raise ValueError(“The tensor is not quantized.”)</p>
<pre><code>scales = quantized_tensor.q_per_channel_scales()
zero_points = quantized_tensor.q_per_channel_zero_points()
int_repr = quantized_tensor.int_repr()

scales = scales.view(-1, 1, 1, 1)  # Shape for broadcasting
zero_points = zero_points.view(-1, 1, 1, 1)  # Shape for broadcasting

float_tensor_orig = (int_repr - zero_points) * scales
if update_mode == "origin":
    float_tensor = (int_repr - zero_points) * scales
elif update_mode == "dequantize":
    float_tensor = quantized_tensor.dequantize()

return float_tensor
</code></pre>
<p>def replace_weights_with_dequantized(model, update_mode):<br>
state_dict = model.state_dict()<br>
model_fp32 = QATVGG11(feature_output_size=25088)<br>
state_dict_fp32 = model_fp32.state_dict()<br>
for name, param in state_dict.items():<br>
if ‘weight’ in name and isinstance(param, torch.Tensor):<br>
quantized_tensor = param<br>
# Dequantize the tensor<br>
float_tensor = dequantize_modified_tensor(quantized_tensor, update_mode)<br>
# Replace the model’s weight with the dequantized tensor<br>
state_dict_fp32[name].copy_(float_tensor)<br>
model_fp32.load_state_dict(state_dict_fp32)<br>
return model_fp32</p>
<p>new_model_origin = replace_weights_with_dequantized(quantized_model, ‘origin’)<br>
new_model_origin_deq = replace_weights_with_dequantized(quantized_model, ‘dequantize’)</p>
<p>def evaluate_model(model, data_loader, device):<br>
model.to(device)<br>
model.eval()</p>
<pre><code>state_dict = model.state_dict()
for name, param in state_dict.items():
correct = 0
total = 0
for data, target in test_loader:
    data, target = data.to(device), target.to(device)
    output = model(data)
    _, predicted = torch.max(output, 1)
    total += target.size(0)
    correct += (predicted == target).sum().item()
print(f'Accuracy of the model on the test set: {100 * correct / total:.2f}%')
</code></pre>
<p>original_fp_model = QATVGG11(feature_output_size=25088)<br>
original_fp_model.qconfig = quantization.get_default_qat_qconfig(‘fbgemm’)<br>
original_fp_model = quantization.prepare_qat(original_fp_model, inplace=False)<br>
original_fp_model.load_state_dict(torch.load(model_save_path))</p>
<p>print(“Original model:”)<br>
evaluate_model(quantized_model, test_loader, ‘cpu’)<br>
print(“Original fp model:”)<br>
evaluate_model(original_fp_model, test_loader, ‘cpu’)<br>
print(“Original model after dequantization:”)<br>
evaluate_model(new_model_origin, test_loader, ‘cpu’)<br>
print(“Original model after pytorch dequantization:”)<br>
evaluate_model(new_model_origin_deq, test_loader, ‘cpu’)</p>

satabios: <p>Accuray is expected to change with quantization, the accuracy degrades upon quantization. This is quite common in model quantization.</p>

HDCharles: <p>so the model you get after QAT (before convert) is actually in fp32 but it has additional ops in order to mimic quantization, namely rounding and clamping i.e. <code>fake quantize</code>. Thats the difference between the QAT model and the dequantized model you’ve constructed. You can disable the fake quantize which would be the easier way to ‘dequantize’ the model.</p>
<p><a href="https://pytorch.org/docs/main/generated/torch.ao.quantization.fake_quantize.disable_fake_quant.html#torch.ao.quantization.fake_quantize.disable_fake_quant" class="onebox" target="_blank" rel="noopener nofollow ugc">https://pytorch.org/docs/main/generated/torch.ao.quantization.fake_quantize.disable_fake_quant.html#torch.ao.quantization.fake_quantize.disable_fake_quant</a></p>
<p>if possible, i would test the fp32 model without fake quant and see if that matches what you’re seeing with the dequantize stuff, it may be that something is not happening correctly in the dequant.</p>