How can i check if my transformer blocks are working in my classifier? (with Colab Link)

Michael_Ringer: <p>Hey,</p>
<p>im currently working on a classification transformer and using the imdb sentiment dataset.<br>
And I am pretty unsure that my model is working because when i remove the transformer blocks the loss  is only 8% higher as with them.</p>
<p>I am using the  pretrained 100 dim glove word embeeding vectors. Thats why my hidden dim is also only 100.</p>
<p><strong>model specs:</strong></p>
<p>INPUT_DIM = 20000<br>
HID_DIM = 100<br>
OUT_DIM = 1<br>
N_LAYERS = 2<br>
N_HEADS = 4<br>
FF_DIM = 32<br>
MAXLEN = 100</p>
<p>optimizer = torch.optim.Adam(model.parameters())<br>
criterion = nn.BCEWithLogitsLoss()</p>
<p><strong>Google Colab Link:</strong></p><aside class="onebox allowlistedgeneric">
  <header class="source">
      <img src="https://colab.research.google.com/img/favicon.ico?vrz=colab-20210304-060047-RC01_361041602" class="site-icon" width="128" height="128">
      <a href="https://colab.research.google.com/drive/1QLJwMGuevBrgROhlFnnCe61xSfrtVX_h?usp=sharing" target="_blank" rel="noopener nofollow ugc">colab.research.google.com</a>
  </header>
  <article class="onebox-body">
    <img src="https://colab.research.google.com/img/colab_favicon_256px.png" class="thumbnail onebox-avatar" width="260" height="260">

<h3><a href="https://colab.research.google.com/drive/1QLJwMGuevBrgROhlFnnCe61xSfrtVX_h?usp=sharing" target="_blank" rel="noopener nofollow ugc">Google Colaboratory</a></h3>



  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>

<p>Can someone please tell me how to check if these attention algorithms are working?</p>

Unity05: <p>Hi,</p>
<p>as what I’ve seen, your MultiHeadAttention and EncoderLayer implementation should work. Could you show a continuous representation of both losses, please?</p>
<p>Regards,<br>
Unity05</p>

Michael_Ringer: <p>For that i need to train the model again. I can send you the losses later.<br>
But i forgot to mention that the model with transformer blocks seems to overfitt cause the validation_acc was in every 5 epochs about 80%. I think thats because i used the pretty small dim of the pretrained, glove word vectors (100) also as hid_dim.</p>

Unity05: <p>I’ve just seen that you only stack two of the encoder layers what might be too few. Talking about your hidden dimension, shouldn’t it be <code>hid_dim = embed_dim</code> as using different dimensions would complicate it. You could also try 300d for instance.</p>

Michael_Ringer: <p><strong>I removed the emb_dim and changed to the 300d pretrained word embeddings.</strong></p>
<p>INPUT_DIM = 20002<br>
HID_DIM = 300<br>
OUT_DIM = 1<br>
N_LAYERS = 4<br>
N_HEADS = 6<br>
FF_DIM = HID_DIM * 2<br>
MAXLEN = 100</p>
<p>Epoch: 01 | Epoch Time: 1m 15s<br>
Train Loss: 0.554 | Train Acc: 71.35%<br>
Val. Loss: 0.467 |  Val. Acc: 77.56%<br>
Epoch: 02 | Epoch Time: 1m 15s<br>
Train Loss: 0.402 | Train Acc: 81.82%<br>
Val. Loss: 0.457 |  Val. Acc: 79.22%<br>
Epoch: 03 | Epoch Time: 1m 15s<br>
Train Loss: 0.359 | Train Acc: 84.33%<br>
Val. Loss: 0.526 |  Val. Acc: 74.28%</p>
<p><strong>training results from old config (with transformerblocks)</strong></p>
<p>INPUT_DIM = 20002<br>
HID_DIM = 300<br>
OUT_DIM = 1<br>
N_LAYERS = 2<br>
N_HEADS = 2<br>
FF_DIM = 128<br>
MAXLEN = 100</p>
<p>Epoch: 01 | Epoch Time: 0m 30s<br>
Train Loss: 0.464 | Train Acc: 77.80%<br>
Val. Loss: 0.421 |  Val. Acc: 81.38%<br>
Epoch: 02 | Epoch Time: 0m 30s<br>
Train Loss: 0.314 | Train Acc: 86.78%<br>
Val. Loss: 0.442 |  Val. Acc: 80.23%<br>
Epoch: 03 | Epoch Time: 0m 30s<br>
Train Loss: 0.227 | Train Acc: 91.24%<br>
Val. Loss: 0.524 |  Val. Acc: 79.14%<br>
Epoch: 04 | Epoch Time: 0m 30s<br>
Train Loss: 0.159 | Train Acc: 94.36%<br>
Val. Loss: 0.643 |  Val. Acc: 77.77%<br>
Epoch: 05 | Epoch Time: 0m 30s<br>
Train Loss: 0.118 | Train Acc: 95.82%<br>
Val. Loss: 0.622 |  Val. Acc: 78.18%</p>
<p><strong>training results from old config (without transformerblocks)</strong></p>
<p>Epoch: 01 | Epoch Time: 0m 4s<br>
Train Loss: 0.607 | Train Acc: 69.78%<br>
Val. Loss: 0.521 |  Val. Acc: 76.43%<br>
Epoch: 02 | Epoch Time: 0m 4s<br>
Train Loss: 0.428 | Train Acc: 81.96%<br>
Val. Loss: 0.432 |  Val. Acc: 80.48%<br>
Epoch: 03 | Epoch Time: 0m 4s<br>
Train Loss: 0.343 | Train Acc: 85.94%<br>
Val. Loss: 0.409 |  Val. Acc: 81.34%<br>
Epoch: 04 | Epoch Time: 0m 4s<br>
Train Loss: 0.292 | Train Acc: 88.55%<br>
Val. Loss: 0.417 |  Val. Acc: 80.56%<br>
Epoch: 05 | Epoch Time: 0m 4s<br>
Train Loss: 0.255 | Train Acc: 90.13%<br>
Val. Loss: 0.412 |  Val. Acc: 81.23%</p>

Hammes: <p>I am so happy to see that the issue have been solved. i am looking for the same info.</p>
<p><a href="https://www.mymilestonecard.run/" rel="noopener nofollow ugc">mymilestonecard</a></p>