Calculate test loss on seq2seq GRU

username: Angelos_Papangelis
cooked: <p>Hello,</p>
<p>I am a beginner in ML. How can I calculate the <strong>Test</strong> loss on this tutorial?<br>
<a href="https://pytorch.org/tutorials/intermediate/seq2seq_translation_tutorial.html" class="onebox" target="_blank" rel="nofollow noopener">https://pytorch.org/tutorials/intermediate/seq2seq_translation_tutorial.html</a></p>
<p>Thank you very much in advance!</p>
<p>/Angelos</p>

username: ptrblck
cooked: <p>You could use the <a href="https://pytorch.org/tutorials/intermediate/seq2seq_translation_tutorial.html#evaluation">code snippet for evaluation</a> and provide your test sentences instead of randomly sampling some sentences.</p>

username: Angelos_Papangelis
cooked: <p>Basically I would like to illustrate that in the same graph having train loss and test loss.<br>
Right now in my implementation I have only Train loss.</p>
<p>/Angelos</p>

username: ptrblck
cooked: <p>To visualize the training and test loss inside the same figure, you could add the loss calculation to the mentioned code snippet and store the training and test losses in separate lists.<br>
Note that you should detach the (training) loss, if you are storing its value in a list to avoid an increasing memory usage (which might create an OOM) via:</p>
<pre><code class="lang-python">train_losses.append(loss.detach().cpu().item())
</code></pre>
<p>Once you have both lists, you can convert them to numpy arrays and plot via <code>matplotlib</code>.</p>

username: Angelos_Papangelis
cooked: <p>What should I put as target then?<br>
Will it be something liked this:</p>
<pre><code class="lang-auto">loss += criterion(decoder_output, decoded_words)</code></pre>

username: ptrblck
cooked: <p>I assume you have a test set with input data and the corresponding targets.<br>
If that’s the case, you should pass the test targets to the <code>criterion</code>.</p>
<p>If you don’t have test targets, could you explain the use case a bit?<br>
E.g. how did you collect the test target and how would you like to calculate the loss?</p>

username: Angelos_Papangelis
cooked: <p>There are no targets, that’s why I am a little bit confused. I feed the decoder’s predictions back to itself as the tutorial does.</p>

username: ptrblck
cooked: <p>The tutorial uses sentences in one language as the input and targets of the corresponding sentence in another language, no?<br>
If your test set doesn’t have the translations, you won’t be able to calculate the loss.</p>

username: Angelos_Papangelis
cooked: <p>Yes for the training.<br>
But for the testing it just feeds the decoder’s predictions back to itself in the evaluation function and it doesn’t calculate the test loss.<br>
So I am assuming that the target for the test set is the “decoded_words” which is the translation of the sentence. Am I right?</p>

username: ptrblck
cooked: <p>No, the <code>evaluate</code> method just prints the predictions and the target so that you could manually compare these.<br>
To calculate the loss, you would have to compute it with the <code>criterion</code> (as done during training):</p>
<pre><code class="lang-python">loss += criterion(decoder_output, target_tensor[di])
</code></pre>
<p>However, if you don’t have the targets, you won’t be able to calculate the loss and would have to output the predictions only.</p>

username: Angelos_Papangelis
cooked: <p>Yes that’s why asked you in a previous comments if this is right:</p>
<pre><code class="lang-auto">loss += criterion(decoder_output, decoded_words)
</code></pre>
<p>I understood that I should have the targets, my question is how I can have them.</p>

username: ptrblck
cooked: <p>Yeah, sorry for the confusion. I tried to explain that you would have to have the target values to calculate the loss.</p>
<aside class="quote no-group" data-username="Angelos_Papangelis" data-post="11" data-topic="87221">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/angelos_papangelis/48/24310_2.png" class="avatar"> Angelos_Papangelis:</div>
<blockquote>
<p>my question is how I can have them.</p>
</blockquote>
</aside>
<p>You would need to create them manually or download the targets, if you are working on a public dataset (they usually provide inputs and the corresponding targets).</p>