LSTM Binary Classifier Backward() Error During Training

msabrii: <p>Hi, I am getting this error and I have looked at other forums but nothing has worked, not sure why this is happening in the first place.</p>
<p>Trying to backward through the graph a second time (or directly access saved tensors after they have already been freed). Saved intermediate values of the graph are freed when you call .backward() or autograd.grad(). Specify retain_graph=True if you need to backward through the graph a second time or if you need to access saved tensors after calling backward.</p>
<p>My code is as follows :</p>
<p>class LSTM(nn.Module):<br>
def <strong>init</strong>(self, embedding_dim, hidden_dim):</p>
<pre><code>    super().__init__()
    self.hidden_dim = hidden_dim
    self.lstm = nn.LSTM(embedding_dim, hidden_dim, batch_first=True)
    self.dropout = nn.Dropout(0.2)
    self.fc = nn.Linear(hidden_dim, 1)
    
def forward(self, src):
    lstm_out, (ht, ct) = self.lstm(src)
    return self.fc(output).squeeze()
</code></pre>
<p>model = LSTM(1462, 20)</p>
<p>optimizer = torch.optim.Adam(model.parameters(), lr=0.002)<br>
criterion = F.cross_entropy</p>
<p>for epoch in range(2):  # loop over the dataset multiple times</p>
<pre><code>running_loss = 0.0
for i, data in enumerate(train_loader, 0):

    # get the inputs; data is a list of [inputs, labels]
    inputs, labels = data

    # zero the parameter gradients
    optimizer.zero_grad()

    # forward + backward + optimize
    outputs = model(inputs)
    loss = criterion(outputs, labels)
    loss.backward()
    optimizer.step()

    # print statistics
    running_loss += loss.item()
    if i % 2000 == 1999:    # print every 2000 mini-batches
        print(f'[{epoch + 1}, {i + 1:5d}] loss: {running_loss / 2000:.3f}')
        running_loss = 0.0
</code></pre>
<p>print(‘Finished Training’)</p>

ptrblck: <p>Your code uses an undefined <code>output</code> tensor in <code>self.fc(output)</code>, so check where this tensor is coming from and if it’s causing the error.</p>

dreidizzle: <p>The above answer is right, I think you want to use the last hidden layer to do this, see the code below. Also, you might want to do gradient clipping before you call optimizer.step().</p>
<pre><code class="lang-auto">self.fc(ht).squeeze()
</code></pre>

mygithubid1: <p>Why use <code>ht</code> instead of <code>lstm_out</code> ?</p>

dreidizzle: <p>Actually maybe not but my interpretation was that you want to get a representation for the text (so use the last hidden state to represent the entire text you want to predict the sentiment of or something else). The output is for each state (time step). I’m guessing you don’t want to predict for each state unless you have a language model, just per sentence or text, so use the last hidden state as the representation. No? He could also combine the hidden states in some way and then also feed to the forward layer but unsure.</p>

mygithubid1: <p>In tensorflow, output of the last timestep is passed to a dense layer if required for prediction. Here’s the text to look for in this <a href="https://www.tensorflow.org/api_docs/python/tf/keras/layers/LSTM#args" rel="noopener nofollow ugc">link</a>.</p>
<blockquote>
<p>return_sequences	Boolean. Whether to return the last output in the output sequence, or the full sequence. Default: False.</p>
</blockquote>
<p>Here’s an <a href="https://www.tensorflow.org/text/tutorials/text_classification_rnn#create_the_model" rel="noopener nofollow ugc">example</a>.</p>

dreidizzle: <aside class="quote no-group" data-username="msabrii" data-post="1" data-topic="167413">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/msabrii/48/55360_2.png" class="avatar"> msabrii:</div>
<blockquote>
<p><code>lstm_out, (ht, ct) = self.lstm(src)</code></p>
</blockquote>
</aside>
<p>Aha right. So this line: lstm_out, (ht, ct) = self.lstm(src) returns the full sequence AND the last hidden and context states, per batch. If it’s batch first you have lstm_out[:, -1, :] == ht. So, I’m thinking he wants to use the last step’s hidden. PyTorch always returns (all sequence of hidden, (last hidden, last context)) … batch_first controls if you get N X L X D or L X N X D where L is the length in time and N is the batch size.</p>

mygithubid1: <p>Agreed (based on <code>batch_first=True</code>):  <code>(outputs[:, -1] == hidden_states[-1]).all()</code></p>
<p>Since this topic is on a binary classification task, using the output of the last time step makes sense to me as well.</p>

vdw: <aside class="quote no-group" data-username="mygithubid1" data-post="8" data-topic="167413">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/mygithubid1/48/55250_2.png" class="avatar"> mygithubid1:</div>
<blockquote>
<p>Agreed (based on <code>batch_first=True</code>): <code>(outputs[:, -1] == hidden_states[-1]).all()</code></p>
</blockquote>
</aside>
<p>Note that this only holds for <code>bidirectional=False</code>.</p>

mygithubid1: <p>You are right about that.</p>

msabrii: <p>Hi, could you please explain this? You said I should use the output of the last time step, right now I am just using the entire lstm_out as the input to my linear layer. Should I be using something different? How would this change if bidirectional = True? Thank you all for you’re helpful replies!</p>

vdw: <p>You can have a look at the <a href="https://discuss.pytorch.org/t/rnn-output-vs-hidden-state-dont-match-up-my-misunderstanding/43280">older post</a>.</p>
<p>I way to handle it can be found <a href="https://github.com/chrisvdweth/ml-toolkit/blob/master/pytorch/models/text/classifier/rnn.py" rel="noopener nofollow ugc">here</a>; search for <code># Handle directions</code> in the <code>forward()</code> method.</p>

dreidizzle: <p>This is tricky and you are using bidirectional=False (the default) but here is my info on this. This is the resource: <a href="https://pytorch.org/docs/stable/generated/torch.nn.LSTM.html" class="inline-onebox" rel="noopener nofollow ugc">LSTM — PyTorch 1.13 documentation</a> and you can look here: “For bidirectional LSTMs, h_n is not equivalent to the last element of output; the former contains the final forward and reverse hidden states, while the latter contains the final forward hidden state and the initial reverse hidden state.”</p>
<p>Here is an example.</p>
<p>Imagine no batches, so dimensions are L X D for everything.</p>
<p>Basically, imagine you do sentiment analysis and you have a sentence that tokenizes to [1,2, 3, 4] and you embed this into a 4 X 128 vector. If you want to do a forward RNN, you should use the last hidden layer. This is hidden or output[-1, :].</p>
<p>If you use a bidirectional RNN, you’d probably want to feed hidden and NOT output to your softmax. output in this case has [(h1_forward, h1_backward), (h2_forward, h2_backward), (h3_forward, h3_backward), (h4_forward, h4_backward)]. But, the backward RNN starts at step 4 so (h4_forward, h4_backward) contain’s the forward RNN’s encoding but the second element has almost no information (yet) as far as the backward RNN is concerned.</p>
<p>An “encoding” of the sentence is thus (h4_forward, h1_backward), the result of the forward RNN’s pass and the backward RNN’s pass. This is probably what you’d like to feed to the classifier head.</p>
<p>I.e., When you have bidirectional RNN, the last output IS NOT the hidden state returned.</p>