LSTM (hidden_size), (num_layers) setting question

username: DonghunP
cooked: <p>Hi I’m a  newbie in LSTM and I want to ask basic question.</p>
<p>I’m training below network and <strong>xi</strong> is 64 bit data.</p>
<p><img src="https://discuss.pytorch.org/uploads/default/original/3X/2/e/2e930a7e26fcaa359d915366520af71695d53ac5.png" alt="image" data-base62-sha1="6E0ZQoY1NKc63HrVtsTLi8rfHwx" width="266" height="326"></p>
<p>This paper said<br>
“These bits(64bit data) are transformed by two non-recurrent hidden layers, each with 128 units and tanh activation functions.<br>
The output of the linear layers is fed into two recurrent LSTM layers, each with 512 units and tanh activation functions.”</p>
<p>and this is snippet code of me</p>
<pre><code class="lang-auto">        self.lstm = nn.LSTM(input_size=input_size, hidden_size=hidden_size,
                            num_layers=num_layers, batch_first=True)

        self.fc = nn.Linear(hidden_size, num_classes)

    def forward(self, x):
        h_0 = Variable(torch.zeros(
            self.num_layers, x.size(0), self.hidden_size)) #([1, 50656, 64])

        c_0 = Variable(torch.zeros(
            self.num_layers, x.size(0), self.hidden_size)) #([1, 50656, 64])

        # Propagate input through LSTM
        ula, (h_out, _) = self.lstm(x, (h_0, c_0))
</code></pre>
<p>My question is how should I have to set ‘<strong>hidden_size</strong>’ and ‘<strong>num_layers</strong>’ in this network??</p>
<p>How can I set fully connected layer in this case??</p>
<p>Actually, I can find <strong>Time-Series anomaly detection LSTM in single variable</strong> code in Google.</p>
<p>But, hard to find <strong>Time-Series anomaly detection LSTM in multivariate variable</strong> like my case.</p>
<p>This is not my homework and,</p>
<p>Any comment or existing tutorial will be appreciate.</p>

username: ariG23498
cooked: <p>Hey <a class="mention" href="/u/donghunp">@DonghunP</a><br>
With the screenshot shared I am thinking the model should look something like this</p>
<pre><code class="lang-python">class CustomModel(nn.Module):
    def __init__(self):
        super().__init__()
        self.projection = nn.Sequential(
            nn.Linear(64, 128),
            nn.Tanh(),
            nn.Linear(128, 128),
            nn.Tanh(),
        )
        self.LSTM = nn.LSTM(input_size=128,
                            hidden_size=512,
                            num_layers=2)
    def forward(self, x):
        pro = self.projection(x)
        output = self.LSTM(pro)
        return output
</code></pre>
<p>With an input of shape <code>(seq_leng, batch_size, 64)</code> the model would first transform the input vectors with the help of the projection layer, and then send that to the <code>LSTM</code> layer. Here the <code>hidden_size</code> of the LSTM layer would be 512 as there are 512 units in each LSTM cell and the <code>num_layers</code> would be 2. The num_layers is the number of layers stacked on top of each other.</p>
<p>Hope this makes sense to you! <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>

username: DonghunP
cooked: <p><a class="mention" href="/u/arig23498">@ariG23498</a> Very Thanks kind reply <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=12" title=":slight_smile:" class="emoji" alt=":slight_smile:" loading="lazy" width="20" height="20"></p>
<p>Anyway,</p>
<p><strong>Do you know why they expand input 64 dimension to 128 ( Why expand it?)</strong></p>
<ul>
<li>In my case, 64 dimension Input (actually 64bit size data) is partially correlated.</li>
<li><img src="https://discuss.pytorch.org/uploads/default/original/3X/b/7/b726f4d85f79e412712e012f38b25ac7f9c8746c.png" alt="image" data-base62-sha1="q8eQ8eo3saujeUD1Xv1D7d6AklC" width="199" height="155"></li>
<li>Is it okay to expand this data?</li>
</ul>
<p><strong>Do you know why they set num_layer as ‘2’? Any criterion?</strong></p>
<p>Any material that I can refer?</p>
<p>Thanks bro.</p>

username: ariG23498
cooked: <ol>
<li>
<p>Expansion of dimensions is mostly done to enhance the parameters (read weights and Biases) of a model. This might lead to better capture of the data distribution, but might also lead to over-fitting.</p>
</li>
<li>
<p>They set num_layers=2 to use two LSTM layer stacked one on top of the other. This way, they use recurrence of two layers. This is indeed an expensive operation, but using stacked LSTM helps in better recurrence capture of the sequence.</p>
</li>
</ol>
<p>I would want you to take a look at an article that I had written on LSTM to have a good understanding about it.<br>
The Article can be found <a href="https://wandb.ai/authors/rnn-viz/reports/Under-the-Hood-of-LSTMs--VmlldzoyNjY0MTg" rel="noopener nofollow ugc">here</a></p>