Am i using nn.transformer correctly?

Rohan_Kumar: <p>So this is the model i wrote for language modelling, (basically dialogue A to dialogue B generation), but if i use  key_padding_mask(s) the test loss becomes nan, if i dont use them the test loss starts increasing just after 1st epoch.</p>
<pre><code class="lang-auto">class Transfomermodel(nn.Module):
    def __init__(self):
        super().__init__()
        self.trans = nn.Transformer(d_model=300, nhead=6, num_encoder_layers=6, num_decoder_layers=6, dim_feedforward=2048, dropout=0.1)
        self.embedding = nn.Embedding(len(vocab)+1, 300)
        self.fc = nn.Linear(300, len(vocab)+1)
        self.srce_mask = None
        self.trg_mask = None
        self.src_pad_mask = None
        self.trg_pad_mask = None

    def forward(self,src,trg):
        self.srce_mask = self._generate_square_subsequent_mask(len(src)).cuda()
        self.trg_mask = self._generate_square_subsequent_mask(len(trg)).cuda()
        self.src_pad_mask = self._generate_padding_mask(src).cuda()
        self.trg_pad_mask = self._generate_padding_mask(trg).cuda()
        src_inp = self.embedding(src)
        trg_inp = self.embedding(trg)
        out = self.trans(src_inp,trg_inp, src_mask= self.srce_mask, tgt_mask= self.trg_mask, src_key_padding_mask= self.src_pad_mask, tgt_key_padding_mask= self.trg_pad_mask)
        return self.fc(out)
    
    def _generate_square_subsequent_mask(self, sz):
        mask = (torch.triu(torch.ones(sz, sz)) == 1).transpose(0, 1)
        mask = mask.float().masked_fill(mask == 0, float('-inf')).masked_fill(mask == 1, float(0.0))
        return mask

    def _generate_padding_mask(self, tsr):
        msk = torch.tensor((tsr.data.cpu().numpy() == 0).astype(int), dtype = torch.bool)
        msk = msk.permute(1,0)
        return msk
    
    def init_weights(self):
        initrange = 0.1
        self.encoder.weight.data.uniform_(-initrange, initrange)
        self.decoder.bias.data.zero_()
        self.decoder.weight.data.uniform_(-initrange, initrange)

</code></pre>
<p>I am using CrossEntropyLoss, and this is my training code</p>
<pre><code class="lang-auto">for epoch in tqdm(range(epochs)):
        model.train()
        epoch_loss = 0
        for batch in tqdm(train_iterator):
            src_inp = batch['src_sentence'].permute(1,0).cuda()
            trg_inp = batch['trg_sentence'].permute(1,0).cuda()
            in_tgt = trg_inp[:-1, :]
            exp_tgt = trg_inp[1:, :]
            optimizer.zero_grad()
            output= model(src_inp, in_tgt)

            output = output.view(-1, output.shape[-1])

            trg = exp_tgt.reshape(-1)
            
            loss = criterion1(output, trg)
            loss.backward()

            torch.nn.utils.clip_grad_norm_(model.parameters(), 1)

            optimizer.step()
            epoch_loss += loss.item()
        y.append(epoch_loss/len(train_iterator))
        print(y)
        epoch_loss = 0 
        model.eval()
        for batch in tqdm(val_iterator):
            src_inp = batch['src_sentence'].permute(1,0).cuda()
            trg_inp = batch['trg_sentence'].permute(1,0).cuda()
            in_tgt = trg_inp[:-1, :]
            exp_tgt = trg_inp[1:, :]
            optimizer.zero_grad()
            output= model(src_inp, in_tgt)

            output = output.view(-1, output.shape[-1])

            trg = exp_tgt.reshape(-1)

            loss = criterion1(output, trg)
            torch.nn.utils.clip_grad_norm_(model.parameters(), 1)
            epoch_loss += loss.item()
        y_val.append(epoch_loss/len(val_iterator))
        print(y_val)
        curr_loss = epoch_loss/len(val_iterator)
        if(curr_loss &lt; best_loss):
            best_loss = curr_loss
            torch.save(model.state_dict(), MODEL_DIR) 
</code></pre>
<p>i am using Adam optimizer, and I dont know where i am going wrong. the output predicted is gibberish or repetition of a single word. I dont know whats wrong, can anyone help me find out where i went wrong?</p>
<p>batch["xxx_sentence] is a long tensor of size [batch_size, sequence length]</p>

zhangguanheng66: <p>Why <code>exp_tgt</code> get <code>trg_inp[1:, :]</code>, rather than <code>trg_inp[:-1, :]</code>? The model will see the value <code>trg_inp[1:-1, :]</code>. If you see the WLM task, it predicts the next word and compares against <code>trg_inp[:-1, :]</code>.</p>

Rohan_Kumar: <p>lets say inp is [a , b, c ,d] then i guess i have to shift it it to the right, since its a transformer. then i will do this<br>
[‘sos’,a , b, c ,d , ‘eos’] this is trg inp, since it (seq len, bz ) dim, i do trg_inp[:1, :] which is [a,b,c,d, ‘eos’], as at the first position the transformer should see only ‘sos’ (from trg) and first word of source to output ‘a’.</p>
<p>ps - a,b,c,d are words of the sentence</p>

zhangguanheng66: <aside class="quote no-group" data-username="Rohan_Kumar" data-post="3" data-topic="82102">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/rohan_kumar/48/10228_2.png" class="avatar"> Rohan_Kumar:</div>
<blockquote>
<p>,d , ‘eos’] this is trg inp, since it (seq len, bz ) dim, i do t</p>
</blockquote>
</aside>
<p>Sorry. I’m confused. In a WLM problem, the target input is [a , b, c] and the output is [d-ish], which is compared against [d]. [d-ish] is something from the model during the training.</p>

Rohan_Kumar: <p>No i want to generate a sequence of text from (reply to a dialogue) from a give sequence of text (dialogue). and hence trg is (seq, bsz) .</p>

Rohan_Kumar: <aside class="onebox githubissue">
  <header class="source">
      <a href="https://github.com/pytorch/pytorch/issues/24816" target="_blank" rel="nofollow noopener">github.com/pytorch/pytorch</a>
  </header>
  <article class="onebox-body">
    <div class="github-row">
  <div class="github-icon-container" title="Issue">
	  <svg width="60" height="60" class="github-icon" viewBox="0 0 14 16" aria-hidden="true"><path d="M7 2.3c3.14 0 5.7 2.56 5.7 5.7s-2.56 5.7-5.7 5.7A5.71 5.71 0 0 1 1.3 8c0-3.14 2.56-5.7 5.7-5.7zM7 1C3.14 1 0 4.14 0 8s3.14 7 7 7 7-3.14 7-7-3.14-7-7-7zm1 3H6v5h2V4zm0 6H6v2h2v-2z"></path></svg>
  </div>

  <div class="github-info-container">
    <h4>
      <a href="https://github.com/pytorch/pytorch/issues/24816" target="_blank" rel="nofollow noopener">Transformer Encoder Layer with src_key_padding makes NaN</a>
    </h4>

    <div class="github-info">
      <div class="date">
        opened <span class="discourse-local-date" data-format="ll" data-date="2019-08-18" data-time="10:12:48" data-timezone="UTC">10:12AM - 18 Aug 19 UTC</span>
      </div>

        <div class="date">
          closed <span class="discourse-local-date" data-format="ll" data-date="2019-08-26" data-time="13:36:30" data-timezone="UTC">01:36PM - 26 Aug 19 UTC</span>
        </div>

      <div class="user">
        <a href="https://github.com/JunhyunB" target="_blank" rel="nofollow noopener">
          <img alt="JunhyunB" src="https://avatars3.githubusercontent.com/u/34757808?v=4" class="onebox-avatar-inline" width="20" height="20">
          JunhyunB
        </a>
      </div>
    </div>
  </div>
</div>

<div class="github-row">
  <p class="github-content">I'm using PyTorch 1.3 and this is sample code for my example.
import torch
import torch.nn as nn
enc = nn.TransformerEncoderLayer(3,1)
model = nn.TransformerEncoder(enc, 6)
x...</p>
</div>

<div class="labels">
    <span style="display:inline-block;margin-top:2px;background-color: #B8B8B8;padding: 2px;border-radius: 4px;color: #fff;margin-left: 3px;">module: nn</span>
    <span style="display:inline-block;margin-top:2px;background-color: #B8B8B8;padding: 2px;border-radius: 4px;color: #fff;margin-left: 3px;">triaged</span>
</div>

  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>

<p>This solved the issue</p>

SenSai_Onegai: <p>So, the problem about repetition of a single word is caused by the wrong way of using src_mask? I have been stuck on this problem for so long…</p>