Something wrong with this official Pytorch tutorial?

username: Roman_Koshkin
cooked: <p>I followed the <a href="https://pytorch.org/tutorials/beginner/transformer_tutorial.html" rel="noopener nofollow ugc">Language Modeling with nn.Transformer and TorchText tutorial</a> exactly, but it doesn’t seem to work at all. After three epochs of training (as is in the tutorial code), the model’s loss is at about 4.0 and the model’s prediction (even on the training set) are gibberish:</p>
<pre><code class="lang-auto"># predict ONE batch

j = 0

src_mask = generate_square_subsequent_mask(bptt).to(device)
data, targets = get_batch(train_data, j) # i кратно bptt
output = model(data, src_mask)


i = 6
out = torch.argmax(output, axis=2)[:, i].tolist()
string = ' '.join(vocab.lookup_tokens(out))
print(i)
print(string)

out = targets.reshape(35, 20)[:, i].tolist()
string = ' '.join(vocab.lookup_tokens(out))
print(i)
print(string)

out = data[:, i].tolist()
string = ' '.join(vocab.lookup_tokens(out))
print(i)
print(string)

6
&lt;unk&gt; &lt;unk&gt; &lt;unk&gt; &lt;unk&gt; the &lt;unk&gt; &lt;unk&gt; of &lt;unk&gt; to of the &lt;unk&gt; &lt;unk&gt; the was a a the &lt;unk&gt; &lt;unk&gt; &lt;unk&gt; &lt;unk&gt; &lt;unk&gt; of &lt;unk&gt; to &lt;unk&gt; up to &lt;unk&gt; &lt;unk&gt; the &lt;unk&gt; to
6
&lt;unk&gt; was &lt;unk&gt; for a big loss &lt;unk&gt; losing some of the &lt;unk&gt; that he had gained with the previous play . &lt;unk&gt; state was unable to pick up another first down and sent in
6
&lt;unk&gt; &lt;unk&gt; was &lt;unk&gt; for a big loss &lt;unk&gt; losing some of the &lt;unk&gt; that he had gained with the previous play . &lt;unk&gt; state was unable to pick up another first down and sent
</code></pre>
<p>Even if I train the model for 300 epochs, the loss doesn’t go below about 3.6, and the model’s predictions even on the training set are gibberish. So the question is:</p>
<ul>
<li>Is there a bug in the tutorial code?</li>
<li>Is is true that this model just can’t be expected to produce any sensible predictions, and is only intended to just illustrate how to build a minimalistic encoder-only transformer?<br>
It seems to me that the second answer is more likely, as no example prediction (after training) is provided in that tutorial (as, for example is the case for this other tutorial (<a href="https://pytorch.org/tutorials/beginner/transformer_tutorial.html" rel="noopener nofollow ugc">LANGUAGE TRANSLATION WITH NN.TRANSFORMER AND TORCHTEXT</a>, in which a transformer (although a different architecture) trained for a few epochs does translate a phrase from German into English. Any help is much appreciated!</li>
</ul>

username: tom
cooked: <p>I would suspect that something is up. While the general methodology seems more or less sane (though word splitting is usually replaced by subword splitting these days), it would seem that one would need quite a bit more training to get anywhere. A notebook that does a very similar thing on a char-level basis (much “easier” to train) is in A. Karpathy’s minGPT: <a href="https://github.com/karpathy/minGPT/blob/master/play_char.ipynb" class="inline-onebox" rel="noopener nofollow ugc">minGPT/play_char.ipynb at master · karpathy/minGPT · GitHub</a> That trains a while (40 minutes for me) and then gives output that vaguely looks like Shakespeare…</p>
<p>Best regards</p>
<p>Thomas</p>

username: Roman_Koshkin
cooked: <p><strong>UPDATE</strong></p>
<p>I just trained it on my mid-level GPU for about 3 more hours</p>
<blockquote>
<p>| end of epoch 186 | time: 71.50s | valid loss  5.71 | valid ppl   302.70</p>
</blockquote>
<hr>
<p>| epoch 187 |   200/ 2928 batches | lr 0.02 | ms/batch 23.74 | loss  3.86 | ppl    47.48<br>
| epoch 187 |   400/ 2928 batches | lr 0.02 | ms/batch 23.63 | loss  3.87 | ppl    47.92<br>
| epoch 187 |   600/ 2928 batches | lr 0.02 | ms/batch 23.60 | loss  3.74 | ppl    41.98<br>
| epoch 187 |   800/ 2928 batches | lr 0.02 | ms/batch 23.67 | loss  3.82 | ppl    45.44<br>
| epoch 187 |  1000/ 2928 batches | lr 0.02 | ms/batch 23.65 | loss  3.84 | ppl    46.41<br>
| epoch 187 |  1200/ 2928 batches | lr 0.02 | ms/batch 23.65 | loss  3.84 | ppl    46.75<br>
| epoch 187 |  1400/ 2928 batches | lr 0.02 | ms/batch 23.63 | loss  3.82 | ppl    45.52<br>
| epoch 187 |  1600/ 2928 batches | lr 0.02 | ms/batch 23.57 | loss  3.87 | ppl    47.81<br>
| epoch 187 |  1800/ 2928 batches | lr 0.02 | ms/batch 23.61 | loss  3.87 | ppl    47.74<br>
| epoch 187 |  2000/ 2928 batches | lr 0.02 | ms/batch 23.65 | loss  3.86 | ppl    47.52<br>
| epoch 187 |  2200/ 2928 batches | lr 0.02 | ms/batch 23.64 | loss  3.72 | ppl    41.37<br>
| epoch 187 |  2400/ 2928 batches | lr 0.02 | ms/batch 23.63 | loss  3.80 | ppl    44.64<br>
| epoch 187 |  2600/ 2928 batches | lr 0.02 | ms/batch 23.62 | loss  3.82 | ppl    45.52</p>
<p>and it seems to improve:</p>
<pre><code class="lang-auto"># predict ONE batch

j = 0

src_mask = generate_square_subsequent_mask(bptt).to(device)
data, targets = get_batch(train_data, j) # i кратно bptt
output = model(data, src_mask)
</code></pre>
<blockquote>
<p>11<br>
, , the ’ the by the ’ s brother , and his doyle that was no evidence to the that be s him the jokes for the brother words jokes — s . with<br>
11<br>
authorities . hornung was angered by doyle ’ s action , and told him there was no need for him to ’ butt in ’ except for his own ’ satisfaction ’ . relations between<br>
11<br>
military authorities . hornung was angered by doyle ’ s action , and told him there was no need for him to ’ butt in ’ except for his own ’ satisfaction ’ . relations</p>
</blockquote>
<p>The top paragraph is the prediction on a sample from the training set (targets and inputs are the 2nd and third paragraphs, respectively). Maybe on the Wikitext2 this model must be trained for days? But then the learning rate should decay much slower. Please help.</p>

username: tom
cooked: <aside class="quote no-group" data-username="Roman_Koshkin" data-post="3" data-topic="135919">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/roman_koshkin/48/9630_2.png" class="avatar"> Roman_Koshkin:</div>
<blockquote>
<p>Maybe on the Wikitext2 this model must be trained for days? But then the learning rate should decay much slower. Please help.</p>
</blockquote>
</aside>
<p>It is depending on your hardware, but I would expect training to take a few days, yes.</p>
<p>The perhaps most classic reference with WikiText2 language modelling is <a href="https://openreview.net/forum?id=SyyGPP0TZ" class="inline-onebox" rel="noopener nofollow ugc">Regularizing and Optimizing LSTM Language Models | OpenReview</a> , so I would compare validation perplexities to that to see how you are doing.</p>
<p>Best regards</p>
<p>Thomas</p>