Dimension error, GRU model

Parkz: <p>Hello,</p>
<p>I am getting an index error in the model I made, and I’m not sure how to troubleshoot.<br>
Here is the code:<br>
class NLP_model(nn.Module):<br>
def <strong>init</strong>(self, vocab_size, embedding_dim, hidden_dim, num_classes):<br>
super(NLP_model, self).<strong>init</strong>()<br>
self.char_embedding = nn.Embedding(vocab_size, embedding_dim)<br>
self.gru = nn.GRU(embedding_dim, hidden_dim, num_layers = 5, bidirectional= True)<br>
self.fc = nn.Linear(hidden_dim*2,num_classes)<br>
<span class="hashtag">#self</span>.att = nn.MultiheadAttention(embed_dim, num_heads, …)</p>
<pre><code>def forward(self, x):
    x = self.char_embedding(x)
    output, hidden = self.gru(x)
    hidden = torch.cat((hidden[0][-2,:,:], hidden[0][-1,:,:]), dim=1)
    x = self.fc(hidden[0])

    return x
</code></pre>
<p>model = NLP_model(len(alphabet), 12, 24, assigned[‘fismaid’].nunique()) <span class="hashtag">#Find</span> the number of fismaids for last variable<br>
criterion = nn.BCEWithLogitsLoss()<br>
optimizer = optim.Adam(model.parameters(), lr = 0.001)</p>
<p>for epoch in range(100): <span class="hashtag">#Look</span> into DataLoader for batch processing<br>
y = list()<br>
z = list()<br>
for sentence, label in zip(ls_X_train, ls_y_train): <span class="hashtag">#training_data</span> should be an array of hostnames and labels<br>
model.zero_grad()<br>
output = model(sentence_to_id(sentence)) <span class="hashtag">#sentence</span> is the hostname, label is the fismaid<br>
<span class="hashtag">#print</span>(label)<br>
<span class="hashtag">#print</span>(output.shape)<br>
temp_label = label<br>
label = torch.zeros(assigned[‘fismaid’].nunique())<br>
label[temp_label] = 1.0<br>
<span class="hashtag">#label</span> = torch.tensor(label).unsqueeze(0)<br>
<span class="hashtag">#label</span> = torch.tensor([label]).unsqueeze(1)<br>
<span class="hashtag">#label</span> = torch.tensor(label).unsqueeze(1)<br>
<span class="hashtag">#print</span>(label.shape)<br>
loss = criterion(output, label)<br>
loss.backward()<br>
optimizer.step()<br>
y.append(loss.item())</p>
<p>The error I’m seeing is the following:</p>
<hr>
<p>IndexError                                Traceback (most recent call last)<br>
 in <br>
8     for sentence, label in zip(ls_X_train, ls_y_train): <span class="hashtag">#training_data</span> should be an array of hostnames and labels<br>
9         model.zero_grad()<br>
—&gt; 10         output = model(sentence_to_id(sentence)) <span class="hashtag">#sentence</span> is the hostname, label is the fismaid<br>
11         <span class="hashtag">#print</span>(label)<br>
12         <span class="hashtag">#print</span>(output.shape)</p>
<p>~\Anaconda3\lib\site-packages\torch\nn\modules\module.py in _call_impl(self, *input, **kwargs)<br>
887             result = self._slow_forward(*input, **kwargs)<br>
888         else:<br>
 → 889             result = self.forward(*input, **kwargs)<br>
890         for hook in itertools.chain(<br>
891                 _global_forward_hooks.values(),</p>
<p> in forward(self, x)<br>
10         x = self.char_embedding(x)<br>
11         output, hidden = self.gru(x)<br>
—&gt; 12         hidden = torch.cat((hidden[0][-2,:,:], hidden[0][-1,:,:]), dim=2)<br>
13         x = self.fc(hidden[0])<br>
14</p>
<p>IndexError: too many indices for tensor of dimension 2</p>

ptrblck: <p>Based on the error message it seems you are trying to index too many dimensions.<br>
Assuming <code>hidden</code> has 3 dimensions, then <code>hidden[0]</code> would return a tensor with 2 dimensions only.<br>
In that case, <code>[-2, :, :]</code> won’t work and you would have to remove one indexing dimension.</p>

vdw: <p>The hidden state of <code>nn.GRU</code> is not a tuple like for <code>nn.LSTM</code>, so <code>hidden[0]</code> is probably not what you want but simply <code>hidden</code>.</p>
<p>Also the shape of <code>hidden</code> is <code>(num_layers * num_directions, batch, hidden_size)</code>. I assume your <code>cat()</code> line is there to concatenate the forward in backward direction of the hidden state. If so, you probably want to do the following</p>
<pre><code class="lang-auto"># Split num_layers and num_directions (see official docs)
hidden = hidden.view(num_layers, num_directions, batch, hidden_size).
# Consider only the last hidden state (last w.r.t. to num_layer)
hidden = hidden[-1]
# Concatenate forward and backward direction
hidden = torch.cat((hidden[-2,:], hidden[-1,:]), dim=1)
</code></pre>
<p>The shape of hidden should now be <code>(batch, 2*hidden_size)</code>; otherwise try <code>dim=0</code>, I always get confused how to set this value correctly.</p>