Quantization of a vgg16 pretrained model

Siva6233: <p>I tried quantizing the weights of a vgg16 pretrained model from torchvision.models .<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/3/c/3c0b43b64f1d41227ed2de58ee409bacddf780a0.png" data-download-href="https://discuss.pytorch.org/uploads/default/3c0b43b64f1d41227ed2de58ee409bacddf780a0" title="image"><img src="https://discuss.pytorch.org/uploads/default/original/3X/3/c/3c0b43b64f1d41227ed2de58ee409bacddf780a0.png" alt="image" data-base62-sha1="8zaMvbVk8Hc0NMd07US9TbfaaVG" width="690" height="442" data-dominant-color="F2F2F2"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">905×580 25 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
The image shows what the models looks like after quantization.The feature weights of the model are in torch.qint8 format . I want to get inference using the quantized model but I am getting an runtime error.</p>
<p><strong>RuntimeError: Unable to find an engine to execute this computation in Quantized Conv2D Cudnn</strong>.</p>
<blockquote>
<p>all_labels_int8 = <span class="chcklst-box fa fa-square-o fa-fw"></span><br>
all_predictions_int8 = <span class="chcklst-box fa fa-square-o fa-fw"></span><br>
correct_int8 = 0<br>
total_int8 = 0<br>
start_time_int8 = time()<br>
with torch.no_grad():<br>
model_int8.eval()<br>
for images, labels in testloader:<br>
all_labels_int8.extend(labels.numpy())<br>
<span class="hashtag-raw">#images</span>, labels = images.to(device), labels.to(device)<br>
images = torch.quantize_per_tensor(images, scale=1.0, zero_point=0, dtype=torch.qint8)<br>
outputs = model_int8(images)<br>
_, predicted = torch.max(outputs.data, 1)<br>
total_int8 += labels.size(0)<br>
correct_int8 += (predicted == labels).sum().item()<br>
<span class="hashtag-raw">#predicted_tensor_cpu</span> = predicted.to(‘cpu’)<br>
all_predictions_int8.extend(predicted.numpy())<br>
end_time_int8 = time()<br>
print("Time: ",end_time_int8 - start_time_int8)</p>
<p>print(‘Accuracy achieved by the network on test images is: %d%%’ % (100 * correct_int8 / total_int8))</p>
</blockquote>
<p>This is the code i used for the inference.</p>

jcaip: <aside class="quote no-group" data-username="Siva6233" data-post="1" data-topic="192663">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/s/85e7bf/48.png" class="avatar"> Siva6233:</div>
<blockquote>
<p><strong>RuntimeError: Unable to find an engine to execute this computation in Quantized Conv2D Cudnn</strong>.</p>
</blockquote>
</aside>
<p>Hi <a class="mention" href="/u/siva6233">@Siva6233</a> can you share the full stack trace please?</p>

Siva6233: <p>Hello <a class="mention" href="/u/jcaip">@jcaip</a>,<br>
This is the complete stack trace.</p>
<blockquote>
<hr>
<p>RuntimeError                              Traceback (most recent call last)<br>
Cell In[43], line 12<br>
10 <span class="hashtag-raw">#images</span>, labels = images.to(device), labels.to(device)<br>
11 images = torch.quantize_per_tensor(images, scale=1.0, zero_point=0, dtype=torch.qint8)<br>
—&gt; 12 outputs = model_int8(images)<br>
13 _, predicted = torch.max(outputs.data, 1)<br>
14 total_int8 += labels.size(0)</p>
<p>File /opt/conda/lib/python3.10/site-packages/torch/nn/modules/module.py:1501, in Module._call_impl(self, *args, **kwargs)<br>
1496 # If we don’t have any hooks, we want to skip the rest of the logic in<br>
1497 # this function, and just call forward.<br>
1498 if not (self._backward_hooks or self._backward_pre_hooks or self._forward_hooks or self._forward_pre_hooks<br>
1499         or _global_backward_pre_hooks or _global_backward_hooks<br>
1500         or _global_forward_hooks or _global_forward_pre_hooks):<br>
 → 1501     return forward_call(*args, **kwargs)<br>
1502 # Do not call functions when jit is used<br>
1503 full_backward_hooks, non_full_backward_hooks = <span class="chcklst-box fa fa-square-o fa-fw"></span>, <span class="chcklst-box fa fa-square-o fa-fw"></span></p>
<p>File /opt/conda/lib/python3.10/site-packages/torchvision/models/vgg.py:66, in VGG.forward(self, x)<br>
65 def forward(self, x: torch.Tensor) → torch.Tensor:<br>
—&gt; 66     x = self.features(x)<br>
67     x = self.avgpool(x)<br>
68     x = torch.flatten(x, 1)</p>
<p>File /opt/conda/lib/python3.10/site-packages/torch/nn/modules/module.py:1501, in Module._call_impl(self, *args, **kwargs)<br>
1496 # If we don’t have any hooks, we want to skip the rest of the logic in<br>
1497 # this function, and just call forward.<br>
1498 if not (self._backward_hooks or self._backward_pre_hooks or self._forward_hooks or self._forward_pre_hooks<br>
1499         or _global_backward_pre_hooks or _global_backward_hooks<br>
1500         or _global_forward_hooks or _global_forward_pre_hooks):<br>
 → 1501     return forward_call(*args, **kwargs)<br>
1502 # Do not call functions when jit is used<br>
1503 full_backward_hooks, non_full_backward_hooks = <span class="chcklst-box fa fa-square-o fa-fw"></span>, <span class="chcklst-box fa fa-square-o fa-fw"></span></p>
<p>File /opt/conda/lib/python3.10/site-packages/torch/nn/modules/container.py:217, in Sequential.forward(self, input)<br>
215 def forward(self, input):<br>
216     for module in self:<br>
 → 217         input = module(input)<br>
218     return input</p>
<p>File /opt/conda/lib/python3.10/site-packages/torch/nn/modules/module.py:1501, in Module._call_impl(self, *args, **kwargs)<br>
1496 # If we don’t have any hooks, we want to skip the rest of the logic in<br>
1497 # this function, and just call forward.<br>
1498 if not (self._backward_hooks or self._backward_pre_hooks or self._forward_hooks or self._forward_pre_hooks<br>
1499         or _global_backward_pre_hooks or _global_backward_hooks<br>
1500         or _global_forward_hooks or _global_forward_pre_hooks):<br>
 → 1501     return forward_call(*args, **kwargs)<br>
1502 # Do not call functions when jit is used<br>
1503 full_backward_hooks, non_full_backward_hooks = <span class="chcklst-box fa fa-square-o fa-fw"></span>, <span class="chcklst-box fa fa-square-o fa-fw"></span></p>
<p>File /opt/conda/lib/python3.10/site-packages/torch/ao/nn/quantized/modules/conv.py:469, in Conv2d.forward(self, input)<br>
466     _reversed_padding_repeated_twice = _reverse_repeat_padding(self.padding)<br>
467     input = F.pad(input, _reversed_padding_repeated_twice,<br>
468                   mode=self.padding_mode)<br>
 → 469 return ops.quantized.conv2d(<br>
470     input, self._packed_params, self.scale, self.zero_point)</p>
<p>File /opt/conda/lib/python3.10/site-packages/torch/_ops.py:502, in OpOverloadPacket.<strong>call</strong>(self, *args, **kwargs)<br>
497 def <strong>call</strong>(self, *args, **kwargs):<br>
498     # overloading <strong>call</strong> to ensure torch.ops.foo.bar()<br>
499     # is still callable from JIT<br>
500     # We save the function ptr as the <code>op</code> attribute on<br>
501     # OpOverloadPacket to access it here.<br>
 → 502     return self._op(*args, **kwargs or {})</p>
<p>RuntimeError: Unable to find an engine to execute this computation in Quantized Conv2D Cudnn</p>
</blockquote>

jcaip: <aside class="quote no-group" data-username="Siva6233" data-post="3" data-topic="192663">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/s/85e7bf/48.png" class="avatar"> Siva6233:</div>
<blockquote>
<p>RuntimeError: Unable to find an engine to execute this computation</p>
</blockquote>
</aside>
<p>Thanks, this seems like a CUDA/CUDNN version error to me. Just making sure, can you run the model without quantization? If so can you share the result you get from running</p>
<pre><code class="lang-auto">wget https://raw.githubusercontent.com/pytorch/pytorch/main/torch/utils/collect_env.py
# For security purposes, please check the contents of collect_env.py before running it.
python collect_env.py
</code></pre>

Siva6233: <p>Hello <a class="mention" href="/u/jcaip">@jcaip</a> ,</p>
<p>Thank you for the reply.</p>
<p>I am able to run the model without quantization.</p>
<blockquote>
<p>VGG(<br>
(features): Sequential(<br>
(0): Conv2d(3, 64, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))<br>
(1): ReLU(inplace=True)<br>
(2): Conv2d(64, 64, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))<br>
(3): ReLU(inplace=True)<br>
(4): MaxPool2d(kernel_size=2, stride=2, padding=0, dilation=1, ceil_mode=False)<br>
(5): Conv2d(64, 128, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))<br>
(6): ReLU(inplace=True)<br>
(7): Conv2d(128, 128, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))<br>
(8): ReLU(inplace=True)<br>
(9): MaxPool2d(kernel_size=2, stride=2, padding=0, dilation=1, ceil_mode=False)<br>
(10): Conv2d(128, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))<br>
(11): ReLU(inplace=True)<br>
(12): Conv2d(256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))<br>
(13): ReLU(inplace=True)<br>
(14): Conv2d(256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))<br>
(15): ReLU(inplace=True)<br>
(16): MaxPool2d(kernel_size=2, stride=2, padding=0, dilation=1, ceil_mode=False)<br>
(17): Conv2d(256, 512, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))<br>
(18): ReLU(inplace=True)<br>
(19): Conv2d(512, 512, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))<br>
(20): ReLU(inplace=True)<br>
(21): Conv2d(512, 512, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))<br>
(22): ReLU(inplace=True)<br>
(23): MaxPool2d(kernel_size=2, stride=2, padding=0, dilation=1, ceil_mode=False)<br>
(24): Conv2d(512, 512, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))<br>
(25): ReLU(inplace=True)<br>
(26): Conv2d(512, 512, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))<br>
(27): ReLU(inplace=True)<br>
(28): Conv2d(512, 512, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))<br>
(29): ReLU(inplace=True)<br>
(30): MaxPool2d(kernel_size=2, stride=2, padding=0, dilation=1, ceil_mode=False)<br>
)<br>
(avgpool): AdaptiveAvgPool2d(output_size=(7, 7))<br>
(classifier): Sequential(<br>
(fc1): Linear(in_features=25088, out_features=4096, bias=True)<br>
(relu1): ReLU()<br>
(dropout1): Dropout(p=0.5, inplace=False)<br>
(fc2): Linear(in_features=4096, out_features=14, bias=True)<br>
(output): LogSoftmax(dim=1)<br>
)<br>
)<br>
This is the model I am using(without quantization).</p>
</blockquote>
<p>I am using the model for flower classification and i just want the time take and accuracy of the model.<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/7/7/77201ce8ea615229d0f8f891ffa7ee2de3c94ad5.png" data-download-href="https://discuss.pytorch.org/uploads/default/77201ce8ea615229d0f8f891ffa7ee2de3c94ad5" title="image"><img src="https://discuss.pytorch.org/uploads/default/original/3X/7/7/77201ce8ea615229d0f8f891ffa7ee2de3c94ad5.png" alt="image" data-base62-sha1="gZPCtEin1ABvTDd79ad7Fw5zSx7" width="690" height="371" data-dominant-color="EBEDEE"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">728×392 25.3 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>This is the output i get without quantization.</p>
<blockquote>
<p>Collecting environment information…<br>
PyTorch version: 2.0.0<br>
Is debug build: False<br>
CUDA used to build PyTorch: 11.8<br>
ROCM used to build PyTorch: N/A</p>
<p>OS: Ubuntu 22.04.3 LTS (x86_64)<br>
GCC version: (Ubuntu 11.4.0-1ubuntu1~22.04) 11.4.0<br>
Clang version: Could not collect<br>
CMake version: version 3.22.1<br>
Libc version: glibc-2.35</p>
<p>Python version: 3.10.12 | packaged by conda-forge | (main, Jun 23 2023, 22:40:32) [GCC 12.3.0] (64-bit runtime)<br>
Python platform: Linux-5.15.133±x86_64-with-glibc2.35<br>
Is CUDA available: True<br>
CUDA runtime version: 11.8.89<br>
CUDA_MODULE_LOADING set to: LAZY<br>
GPU models and configuration:<br>
GPU 0: Tesla T4<br>
GPU 1: Tesla T4</p>
<p>Nvidia driver version: 470.161.03<br>
cuDNN version: Probably one of the following:<br>
/usr/lib/x86_64-linux-gnu/libcudnn.so.8.9.0<br>
/usr/lib/x86_64-linux-gnu/libcudnn_adv_infer.so.8.9.0<br>
/usr/lib/x86_64-linux-gnu/libcudnn_adv_train.so.8.9.0<br>
/usr/lib/x86_64-linux-gnu/libcudnn_cnn_infer.so.8.9.0<br>
/usr/lib/x86_64-linux-gnu/libcudnn_cnn_train.so.8.9.0<br>
/usr/lib/x86_64-linux-gnu/libcudnn_ops_infer.so.8.9.0<br>
/usr/lib/x86_64-linux-gnu/libcudnn_ops_train.so.8.9.0<br>
HIP runtime version: N/A<br>
MIOpen runtime version: N/A<br>
Is XNNPACK available: True</p>
<p>CPU:<br>
Architecture:                       x86_64<br>
CPU op-mode(s):                     32-bit, 64-bit<br>
Address sizes:                      46 bits physical, 48 bits virtual<br>
Byte Order:                         Little Endian<br>
CPU(s):                             4<br>
On-line CPU(s) list:                0-3<br>
Vendor ID:                          GenuineIntel<br>
Model name:                         Intel(R) Xeon(R) CPU @ 2.00GHz<br>
CPU family:                         6<br>
Model:                              85<br>
Thread(s) per core:                 2<br>
Core(s) per socket:                 2<br>
Socket(s):                          1<br>
Stepping:                           3<br>
BogoMIPS:                           4000.24<br>
Flags:                              fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ss ht syscall nx pdpe1gb rdtscp lm constant_tsc rep_good nopl xtopology nonstop_tsc cpuid tsc_known_freq pni pclmulqdq ssse3 fma cx16 pcid sse4_1 sse4_2 x2apic movbe popcnt aes xsave avx f16c rdrand hypervisor lahf_lm abm 3dnowprefetch invpcid_single pti ssbd ibrs ibpb stibp fsgsbase tsc_adjust bmi1 hle avx2 smep bmi2 erms invpcid rtm mpx avx512f avx512dq rdseed adx smap clflushopt clwb avx512cd avx512bw avx512vl xsaveopt xsavec xgetbv1 xsaves arat md_clear arch_capabilities<br>
Hypervisor vendor:                  KVM<br>
Virtualization type:                full<br>
L1d cache:                          64 KiB (2 instances)<br>
L1i cache:                          64 KiB (2 instances)<br>
L2 cache:                           2 MiB (2 instances)<br>
L3 cache:                           38.5 MiB (1 instance)<br>
NUMA node(s):                       1<br>
NUMA node0 CPU(s):                  0-3<br>
Vulnerability Gather data sampling: Unknown: Dependent on hypervisor status<br>
Vulnerability Itlb multihit:        Not affected<br>
Vulnerability L1tf:                 Mitigation; PTE Inversion<br>
Vulnerability Mds:                  Mitigation; Clear CPU buffers; SMT Host state unknown<br>
Vulnerability Meltdown:             Mitigation; PTI<br>
Vulnerability Mmio stale data:      Vulnerable: Clear CPU buffers attempted, no microcode; SMT Host state unknown<br>
Vulnerability Retbleed:             Mitigation; IBRS<br>
Vulnerability Spec rstack overflow: Not affected<br>
Vulnerability Spec store bypass:    Mitigation; Speculative Store Bypass disabled via prctl and seccomp<br>
Vulnerability Spectre v1:           Mitigation; usercopy/swapgs barriers and __user pointer sanitization<br>
Vulnerability Spectre v2:           Mitigation; IBRS, IBPB conditional, STIBP conditional, RSB filling, PBRSB-eIBRS Not affected<br>
Vulnerability Srbds:                Not affected<br>
Vulnerability Tsx async abort:      Mitigation; Clear CPU buffers; SMT Host state unknown</p>
<p>Versions of relevant libraries:<br>
[pip3] flake8==6.1.0<br>
[pip3] msgpack-numpy==0.4.8<br>
[pip3] mypy-extensions==1.0.0<br>
[pip3] numpy==1.24.3<br>
[pip3] onnx==1.15.0<br>
[pip3] pytorch-ignite==0.4.13<br>
[pip3] pytorch-lightning==2.1.1<br>
[pip3] torch==2.0.0<br>
[pip3] torchaudio==2.0.1<br>
[pip3] torchdata==0.6.0<br>
[pip3] torchinfo==1.8.0<br>
[pip3] torchmetrics==1.2.0<br>
[pip3] torchtext==0.15.1<br>
[pip3] torchvision==0.15.1<br>
[conda] cudatoolkit               11.8.0              h4ba93d1_12    conda-forge<br>
[conda] magma-cuda118             2.6.1                         1    pytorch<br>
[conda] mkl                       2023.1.0         h213fc3f_46344<br>
[conda] msgpack-numpy             0.4.8                    pypi_0    pypi<br>
[conda] numpy                     1.26.1                   pypi_0    pypi<br>
[conda] pytorch-ignite            0.4.13                   pypi_0    pypi<br>
[conda] pytorch-lightning         2.1.1                    pypi_0    pypi<br>
[conda] torch                     2.0.0                    pypi_0    pypi<br>
[conda] torchaudio                2.0.1                    pypi_0    pypi<br>
[conda] torchdata                 0.6.0                    pypi_0    pypi<br>
[conda] torchinfo                 1.8.0                    pypi_0    pypi<br>
[conda] torchmetrics              1.2.0                    pypi_0    pypi<br>
[conda] torchtext                 0.15.1                   pypi_0    pypi<br>
[conda] torchvision               0.15.1                   pypi_0    pypi</p>
</blockquote>
<p>Thank you</p>