Quantization - RuntimeError: apply_dynamic is not implemented for this packed parameter type

username: Rayne91
cooked: <p>I have been trying to apply dynamic quantization to a model.<br>
After performing the quantization, I try to revaluate the model to check for any modification in the prediction power.<br>
The problem is that, after moving everything to device=‘cpu’ and running outputs = model(data), I receive RuntimeError: apply_dynamic is not implemented for this packed parameter type error.<br>
Do you know if there are any steps that I am missing with the quantization or is this a possible bug of the code?</p>
<p>torch 2.1.2+cu121<br>
torchaudio 2.1.2+cu121<br>
torchprofile 0.0.4<br>
torchvision 0.16.2+cu121</p>
<p>Here the error</p>
<pre><code class="lang-auto">Traceback (most recent call last):
  File "&lt;string&gt;", line 1, in &lt;module&gt;
  File "CONDA_ENV\lib\site-packages\torch\nn\modules\module.py", line 1518, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "CONDA_ENV\lib\site-packages\torch\nn\modules\module.py", line 1527, in _call_impl
    return forward_call(*args, **kwargs)
  File "CONDA_ENV\lib\site-packages\torch\ao\nn\quantized\dynamic\modules\linear.py", line 54, in forward
    Y = torch.ops.quantized.linear_dynamic(
  File "CONDA_ENV\lib\site-packages\torch\_ops.py", line 692, in __call__
    return self._op(*args, **kwargs or {})
RuntimeError: apply_dynamic is not implemented for this packed parameter type
</code></pre>
<p>Here my original and quantized models:</p>
<pre><code class="lang-auto">model
CNNSpec(
  (audio_conv1): Conv2d(1, 32, kernel_size=(3, 3), stride=(1, 1))
  (audio_pool1): MaxPool2d(kernel_size=(2, 2), stride=(2, 2), padding=0, dilation=1, ceil_mode=False)
  (fc1): Linear(in_features=92736, out_features=64, bias=True)
  (fc2): Linear(in_features=64, out_features=2, bias=True)
)

model_dynamic_quantization
CNNSpec(
  (audio_conv1): Conv2d(1, 32, kernel_size=(3, 3), stride=(1, 1))
  (audio_pool1): MaxPool2d(kernel_size=(2, 2), stride=(2, 2), padding=0, dilation=1, ceil_mode=False)
  (fc1): DynamicQuantizedLinear(in_features=92736, out_features=64, dtype=torch.qint8, qscheme=torch.per_tensor_affine)
  (fc2): DynamicQuantizedLinear(in_features=64, out_features=2, dtype=torch.qint8, qscheme=torch.per_tensor_affine)
) 
</code></pre>
<p>Here the code:</p>
<pre><code class="lang-auto">import torch
import torch.nn as nn
import torch.nn.functional as F

def dynamic_quantization(model):
    model_quantized = torch.quantization.quantize_dynamic(model, {torch.nn.Linear}, dtype=torch.qint8)
    return model_quantized

model_dynamic_quantization = dynamic_quantization(model)

device = 'cpu'
model_dynamic_quantization.eval()  # Set the model to evaluation mode
model.to(device)
with torch.no_grad():  # No need to track gradients during evaluation
    for batch in test_dataloader:
                data, labels = batch
                data, labels = audio_data.to(device), labels.to(device)
                outputs = model(data)
</code></pre>
<h3><a name="versions-1" class="anchor" href="#versions-1"></a>Versions</h3>
<p>Collecting environment information…<br>
PyTorch version: 2.1.2+cu121<br>
Is debug build: False<br>
CUDA used to build PyTorch: 12.1<br>
ROCM used to build PyTorch: N/A</p>
<p>OS: Microsoft Windows 11 Pro<br>
GCC version: Could not collect<br>
Clang version: Could not collect<br>
CMake version: Could not collect<br>
Libc version: N/A</p>
<p>Python version: 3.9.18 (main, Sep 11 2023, 14:09:26) [MSC v.1916 64 bit (AMD64)] (64-bit runtime)<br>
Python platform: Windows-10-10.0.22621-SP0<br>
Is CUDA available: True<br>
CUDA runtime version: Could not collect<br>
CUDA_MODULE_LOADING set to: LAZY<br>
GPU models and configuration: GPU 0: NVIDIA RTX 2000 Ada Generation Laptop GPU<br>
Nvidia driver version: 538.27<br>
cuDNN version: Could not collect<br>
HIP runtime version: N/A<br>
MIOpen runtime version: N/A<br>
Is XNNPACK available: True</p>
<p>CPU:<br>
Architecture=9<br>
CurrentClockSpeed=2500<br>
DeviceID=CPU0<br>
Family=198<br>
L2CacheSize=11776<br>
L2CacheSpeed=<br>
Manufacturer=GenuineIntel<br>
MaxClockSpeed=2500<br>
Name=13th Gen Intel(R) Core™ i7-13800H<br>
ProcessorType=3<br>
Revision=</p>
<p>Versions of relevant libraries:<br>
[pip3] numpy==1.26.3<br>
[pip3] torch==2.1.2+cu121<br>
[pip3] torchaudio==2.1.2+cu121<br>
[pip3] torchprofile==0.0.4<br>
[pip3] torchvision==0.16.2+cu121<br>
[conda] numpy 1.26.3 pypi_0 pypi<br>
[conda] torch 2.1.2+cu121 pypi_0 pypi<br>
[conda] torchaudio 2.1.2+cu121 pypi_0 pypi<br>
[conda] torchprofile 0.0.4 pypi_0 pypi<br>
[conda] torchvision 0.16.2+cu121 pypi_0 pypi</p>

username: supriyar
cooked: <p>can you try moving the model to CPU before doing quantization to run all steps on the same device and see if that solves the error? It looks like the model is trying to run the cudnn apply_dynamic op.</p>

username: Rayne91
cooked: <p>Thank you for your suggestion! Sadly, the model is already on the ‘cpu’, the same applies also to the data and labels before running the model(data) command</p>
<pre><code class="lang-auto">next(model.parameters()).device
device(type='cpu')
</code></pre>

username: HDCharles
cooked: <p>can you try the following</p>
<ol>
<li>
<p>try just doing torch.quantization.quantize_dynamic(model) it looks like the qconfig spec might not be specified correctly in the above code.</p>
</li>
<li>
<p>replace the model with torch.nn.Sequential(torch.nn.Linear(32,32)) and see if that works, test by passing in random data like model(torch.randn(32,32)). The error might be specific to some aspect of the model.</p>
</li>
<li>
<p>can you take a look at the following test:</p>
</li>
</ol>
<aside class="onebox githubblob" data-onebox-src="https://github.com/pytorch/pytorch/blob/91d565da0c5c95a19ab0abbd47645507d7124340/test/quantization/eager/test_quantize_eager_ptq.py#L1088-L1125">
  <header class="source">

      <a href="https://github.com/pytorch/pytorch/blob/91d565da0c5c95a19ab0abbd47645507d7124340/test/quantization/eager/test_quantize_eager_ptq.py#L1088-L1125" target="_blank" rel="noopener nofollow ugc">github.com</a>
  </header>

  <article class="onebox-body">
    <h4><a href="https://github.com/pytorch/pytorch/blob/91d565da0c5c95a19ab0abbd47645507d7124340/test/quantization/eager/test_quantize_eager_ptq.py#L1088-L1125" target="_blank" rel="noopener nofollow ugc">pytorch/pytorch/blob/91d565da0c5c95a19ab0abbd47645507d7124340/test/quantization/eager/test_quantize_eager_ptq.py#L1088-L1125</a></h4>



    <pre class="onebox"><code class="lang-py">
      <ol class="start lines" start="1088" style="counter-reset: li-counter 1087 ;">
          <li>def test_single_layer(self):</li>
          <li>    r"""Dynamic Quantize SingleLayerLinearDynamicModel which has one Linear module,</li>
          <li>    make sure it is swapped to nnqd.Linear which is the quantized version of</li>
          <li>    the module</li>
          <li>    """</li>
          <li>    for dtype in [torch.qint8, torch.float16]:</li>
          <li>        model = SingleLayerLinearDynamicModel().eval()</li>
          <li>        qconfig = float16_dynamic_qconfig if dtype == torch.float16 else default_dynamic_qconfig</li>
          <li>        qconfig_dict = {</li>
          <li>            'fc1': qconfig</li>
          <li>        }</li>
          <li>        prepare_dynamic(model, qconfig_dict)</li>
          <li>        convert_dynamic(model)</li>
          <li></li>
          <li>        def checkQuantized(model):</li>
          <li>            self.checkDynamicQuantizedLinear(model.fc1, dtype)</li>
          <li>            self.checkScriptable(model, self.calib_data, check_save_load=True)</li>
          <li>            self.checkNoQconfig(model)</li>
          <li></li>
          <li>        checkQuantized(model)</li>
      </ol>
    </code></pre>


  This file has been truncated. <a href="https://github.com/pytorch/pytorch/blob/91d565da0c5c95a19ab0abbd47645507d7124340/test/quantization/eager/test_quantize_eager_ptq.py#L1088-L1125" target="_blank" rel="noopener nofollow ugc">show original</a>

  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>

<p>and see if that test runs successfully on your machine?</p>

username: Rayne91
cooked: <p>I have debugged the code and found the solution. The issue was that I had to move the dense model to CPU before applying quantization! Now everything works, thank you!</p>