Error quantizing LSTM autoencoder

Nick95: <p>Hi, I’m trying to quantize a trained LSTM autoencoder.<br>
Once reloaded the best model weights I used this call here to quantize the model:</p>
<pre><code class="lang-auto">torch.quantization.quantize_dynamic(model, {nn.LSTM}, dtype=torch.qint8)
</code></pre>
<p>However, I get this error:</p>
<blockquote>
<p>RuntimeError: bias should be a vector (1D Tensor)</p>
</blockquote>
<p>The model definition is quite simple:</p>
<pre><code class="lang-auto">class AE(nn.Module):

    def __init__(self, lstm1_h: int, batch_size: int):
        super().__init__()

        self.init_hidden(batch_size)

        self.encoder = nn.LSTM(input_size=1, hidden_size=lstm1_h, batch_first=True)
        self.decoder = nn.LSTM(input_size=1, hidden_size=lstm1_h, proj_size=1, batch_first=True)

        # initialize weights
        self.init_weights()

    def init_hidden(self, batch_size: int):
        self.h0 = torch.zeros(1, batch_size, 1)

    def init_weights(self):
        # set input weight and bias to force 1 as input to decoder
        self.decoder.bias_ih_l0.data.fill_(1).requires_grad_(False)
        self.decoder.weight_ih_l0.data.zero_().requires_grad_(False)
</code></pre>
<p>How can I solve this error?</p>

jerryzh168: <aside class="quote no-group" data-username="Nick95" data-post="1" data-topic="171128">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/n/8e7dd6/48.png" class="avatar"> Nick95:</div>
<blockquote>
<pre><code class="lang-auto">torch.quantization.quantize_dynamic(model, {nn.LSTM}, dtype=torch.qint8)
</code></pre>
</blockquote>
</aside>
<p>can you narrow done the problem by commenting out the modifications? e.g. what would happen if you don’t do <code>init_weights</code>?</p>

Nick95: <p>Thanks, I tried a couple of stuff:</p>
<ol>
<li>I tried to load that model from scratch (without using the saved version) and it works both with and without calling init_weigths</li>
<li>I tried to load the pre-trained model directly from the pth file and both with and without calling init_weigths it gives back again the old error.<br>
It looks like there is something in the trained version that does not works good. Any idea? I must use the trained weights</li>
</ol>

Nick95: <p>I managed to solve the issue by manually changing the size of the bias vector to make it a 1D Tensor.<br>
I have then exported the quantized model as pt file and I have imported it into libtorch, however I have not noticed any difference in the inference time. Is there something I should care of?</p>

Nick95: <p>Looking at the quantized model I have looked at something strange. I quantized the model with:</p>
<pre><code class="lang-auto">model_q = torch.quantization.quantize_dynamic(model, {torch.nn.LSTM, torch.nn.Linear}, dtype=torch.qint8)
</code></pre>
<p>But:</p>
<ol>
<li>if I print the quantized model I get:</li>
</ol>
<pre><code class="lang-auto">print("Post quantization: ", model_q)

// Output
Post quantization:  LSTMPacketAE(
  (encoder): DynamicQuantizedLSTM(1, 512, batch_first=True)
  (decoder): DynamicQuantizedLSTM(1, 512, batch_first=True)
)
</code></pre>
<p>Which seems to be quite different from the output shown in the <a href="https://pytorch.org/tutorials/advanced/dynamic_quantization_tutorial.html" rel="noopener nofollow ugc">tutorial</a>.</p>
<ol start="2">
<li>If then I print the named parameters I get an empty vector, while before quantization there were lots of parameters:</li>
</ol>
<pre><code class="lang-auto">print(list(model_q.named_parameters()))

// Output
[]
</code></pre>
<ol start="3">
<li>the quantize model expects now a 3D input, while the original one expected 2D inputs</li>
</ol>
<p>Anyone has some idea?</p>