How do I format my transformer loss computation?

username: maxrivera
cooked: <p>I built a transformer from scratch for machine translation, where the src and trg dimensions are both <code>batch_size x seq_len</code>, and the output prediction dimensions are <code>batch_size x seq_len x trg_vocab_size</code>. I am using <code>torch.nn.CrossEntropyLoss()</code>, but I’m not sure how to format the dimensions correctly before inputting  into the loss function. Does anyone know what the exact dimensions of ‘trg’ and ‘out’ should be and have tips/suggestions on how to make those changes?</p>
<pre><code class="lang-auto"># train loop 
optimizer=torch.optim.Adam(params=model.parameters(),lr=1e-3)
loss_fn = torch.nn.CrossEntropyLoss() 
for epoch in range(0,EPOCHS+1):

    for i, (src,trg) in enumerate(train_data):

        src = torch.Tensor(src).to(DEVICE).long()
        trg = torch.Tensor(src).to(DEVICE).long()

        # forward pass 
        out = model(src,trg)
        print('pred size: ',out.size()) # size = batch_size x seq_len x target_vocab_size
        print('trg size: ',trg.size()) # size = batc_size x seq_len

        # # compute loss 
        # loss = loss_fn(out, trg)
        # print(loss)
        # optimizer.zero_grad()
        # loss.backward()
</code></pre>

username: basingse
cooked: <p>If I understand correctly you want the loss between ground truth <code>(batch_size*seq, 1)</code> and predictions <code>(batch_size*seq, vocab_size)</code></p>
<p>You can do something like I guess</p>
<pre><code class="lang-python">loss = loss_fn(out.view(-1, trg_vocab_size), trg.view(-1))
</code></pre>

username: maxrivera
cooked: <p>I tried the code you suggested, the loss function is decreasing slightly but does not go below 7.2. I have a mini data set for debugging purposes so I wouldn’t expect it to learn completely. Why don’t we want <code>out</code> and <code>trg</code> to be the exact same dimensions? In the code you suggested, the last dimension is still different.</p>

username: basingse
cooked: <p>Ideally we should one-hot encode the ground-truth to get the shape of <code>(batch_size, seq_len, trg_vocab_size)</code> where the entry will <code>1</code> for the index and <code>0</code> otherwise. But this is not needed as <code>nn.CrossEntropyLoss</code> can compute the entropy loss even when ground-truth is not one-hot encoded.  You can read the <a href="https://pytorch.org/docs/stable/generated/torch.nn.CrossEntropyLoss.html" rel="noopener nofollow ugc">docs</a>.</p>
<p>Can I know in ouput of model <code>out = model(src, trg)</code> did you apply <code>softwax</code> or not after the last linear layer ?</p>
<p>The reason I ask this is because in <code>nn.CrossEntropyLoss</code> we should give logits (before softmax) and not probabilities as the <code>predictions</code> ?</p>

username: maxrivera
cooked: <p>I did a torch.nn.functional.softmax() inside the last layer of the decoder. Now I removed that, and I get a loss curve looking this:<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/f/5/f5ac6c76047833ce47bcbd3932b25243ce8ec132.png" data-download-href="https://discuss.pytorch.org/uploads/default/f5ac6c76047833ce47bcbd3932b25243ce8ec132" title="Screen Shot 2022-08-20 at 9.19.42 AM"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/f/5/f5ac6c76047833ce47bcbd3932b25243ce8ec132_2_681x500.png" alt="Screen Shot 2022-08-20 at 9.19.42 AM" data-base62-sha1="z3kozj1wMzXyWtLoAbn0yqCPF6i" width="681" height="500" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/f/5/f5ac6c76047833ce47bcbd3932b25243ce8ec132_2_681x500.png, https://discuss.pytorch.org/uploads/default/optimized/3X/f/5/f5ac6c76047833ce47bcbd3932b25243ce8ec132_2_1021x750.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/f/5/f5ac6c76047833ce47bcbd3932b25243ce8ec132.png 2x" data-dominant-color="FBFCFC"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Screen Shot 2022-08-20 at 9.19.42 AM</span><span class="informations">1170×858 36.5 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>So it seems to be working, but I will read the docs. Also, if I add each sentence with zeros to the MAX_LENGTH of my dataset, is there anything I should account for with the loss function with regards to this?</p>

username: basingse
cooked: <p>For loss calculation you will need predictions and ground-truth label (index of word in vocabulary)</p>
<p>Suppose you pad the sentence (both source and target language in your case), at the time of calculation of loss it is recommended (it is your decision if want to ignore or not) to ignore the positions of pad.</p>
<p>You can manually do the above calculation like first extracting the predictions and ground truth for non-pad positions and then caclulate the loss.</p>
<p>Pytorch provides easy way to do this by having the parameter <code>ignore_index</code>  (by default it is <code>-100</code>). If your ground-truth has <code>-100</code> then <code>nn.CrossEntropyLoss</code> will ignore these values. For example</p>
<pre><code class="lang-python"># here I am using different ignore_index
loss_fn = nn.CrossEntropyLoss(ignore_index=-999)
loss = loss_fn(out.view(-1, trg_vocab_size), trg.view(-1))
</code></pre>
<p>You can choose different <code>ignore_index</code> but you have to consitent with this in your preprocessing scripts as well.</p>
<p>Let me know if you want more details on this.</p>
<p>Note : This is <strong>off topic</strong> but I have some questions</p>
<ul>
<li>what tokenizer are you using ?</li>
<li>Are you also building the tokenizer from scratch ?</li>
<li>For both input language and target language, you have same tokenizer or different tokenizer ?</li>
</ul>
<p>Thanks.</p>

username: maxrivera
cooked: <p>How would I ignore the index if the padding value is ‘0’? Would I say ignore_index=0?</p>
<ul>
<li>I am using word_tokenizer from nltk library</li>
<li>I’m using the same tokenizer for both the src and tgt languages, with slightly different lines of code for adjustments.</li>
</ul>
<p>I sent you a DM with more details!</p>