How to use/train Transformer in Pytorch

username: paganpasta
cooked: <p>I followed the tutorial given <a href="https://pytorch.org/tutorials/beginner/transformer_tutorial.html" rel="nofollow noopener">here</a>. However, the <a href="https://pytorch.org/docs/master/_modules/torch/nn/modules/transformer.html#Transformer" rel="nofollow noopener">implementation</a> for Transformer is significantly different in the pytorch codebase. The latter being closer to the the proposed approach by the authors.</p>
<p>Can someone guide me how to use the pytorch transformer to do a sequence to sequence translation task. I have described below the problem in some detail.</p>
<p><code>Transformer(src, tgt)</code> parameters: <code>src: the sequence to the encoder (required), tgt: the sequence to the decoder (required).</code> <strong>EDIT: For example, English Language dataset</strong></p>
<p><code>src</code>: The dataset is <code>[32, 5, 256]</code> where <code>32</code> represents the total sentences in the database, <code>5</code> are the words in every sentence and <code>256</code> are the embeddings for each of the <code>5</code> words.</p>
<p><code>tgt</code>: <strong>I don’t know what to provide for this argument to the Transformer</strong>.</p>
<p><strong>EDIT: I have a similar dataset for French say the shape is [32, 7, 256]</strong></p>
<p>Assume that the positional encodings have been added to the above src</p>

username: zhangguanheng66
cooked: <p>If you use <code>TransformerEncoder</code>, you don’t need to provide <code>tgt</code>. For translation task, <code>tgt</code> is target language output, as the typical translation task.</p>

username: paganpasta
cooked: <p>Thanks <a class="mention" href="/u/zhangguanheng66">@zhangguanheng66</a>. I am looking for translation therefore, <code>tgt</code> is required. I was confused as to how to format the <code>tgt</code> tensor. However, trying with things I figured out the difference between <code>inference</code> and <code>test</code> time settings for translation. Thanks nonetheless.</p>

username: zhangguanheng66
cooked: <p>yes, the setup for inference and train are different</p>

username: dav-ell
cooked: <p><a class="mention" href="/u/paganpasta">@paganpasta</a> <a class="mention" href="/u/zhangguanheng66">@zhangguanheng66</a> Can you share how you handled the difference between training and test? I’m having a similar issue and haven’t figured it out yet.</p>
<p>Specifically, given that I’m training this way:</p>
<pre><code class="lang-python">opt.zero_grad()
# Model requires both "inputs" and "targets"
out = model(inp_emb, tgt_emb, src_mask=src_mask, tgt_mask=tgt_mask, src_key_padding_mask=inp_padding_mask, tgt_key_padding_mask=tgt_padding_mask)
loss = # ...
loss.backward()
opt.step()
sch.step()
</code></pre>
<p>How do I call the model for inference? I haven’t seen any code online that uses nn.Transformer with a decoder at inference time.</p>
<pre><code class="lang-python">model.eval()
with torch.no_grad():
  # What goes in "tgt_emb" and masks??
  out = model(inp_emb, tgt_emb, src_mask=src_mask, tgt_mask=tgt_mask, src_key_padding_mask=inp_padding_mask, tgt_key_padding_mask=tgt_padding_mask)
  # ...
</code></pre>

username: dav-ell
cooked: <p>Answered my own question on <a href="https://discuss.pytorch.org/t/nn-transformer-explaination/53175/19">this thread</a>.</p>
<p>The code I replace it with is like this:</p>
<pre><code class="lang-auto"># Model requires both "inputs" and "targets"
for i in range(2, targets.size(1)):
  opt.zero_grad()
  trimmed_tgt = targets[:, :i].contiguous()
  in_tgt = trimmed_tgt[:, :-1]
  exp_tgt = trimmed_tgt[:, 1:]
  # Some code missing here, assume in_tgt gets converted to in_tgt_emb
  out = model(inp_emb, in_tgt_emb, tgt_mask=tgt_mask, 
src_key_padding_mask=inp_padding_mask, tgt_key_padding_mask=tgt_padding_mask)
  loss = criterion(out, exp_tgt)
  loss.backward()
  opt.step()
  sch.step()
</code></pre>
<p>Not sure if I’m supposed to accumulate losses in the loop there or not, but this seems to be getting more realistic results than I was getting before.</p>

username: paganpasta
cooked: <p>Hi <a class="mention" href="/u/dav-ell">@dav-ell</a>.</p>
<p>For the training, you pass the complete <code>inp_emb</code> and <code>tgt_emb</code> updated with positional embeddings. <code>tgt_emb</code> is your ground-truth embeddings. The mask is computed by the Transformer module to have access to only the previous outputs in time.<br>
However, during inference you don’t have access to the ground truth <code>tgt_emb</code> and have to input <code>output[i-1]</code> as the <code>tgt_emb</code> at time <code>i</code>. How I did was something like this:</p>
<pre><code class="lang-auto">#let's assume batch_size = 1
initial_dec_input = zeros(1, 1, emb_dim) #All 0s
tgt_emb = zeros(1, tgt_size, emb_dim)
 tgt_emb[0,0, :] = initial_dec_input
for i in range(tgt_size):
  out = model(inp_emb, tgt_emb)
  tgt_emb[0, i+1, :] = out[0,i,:]
</code></pre>
<p>Cheers.</p>

username: dav-ell
cooked: <p>Ah, perfect! Yep, this makes sense. Thanks very much for explaining it.</p>
<p>I also want to offer a revision to my previous post. Turns out you get very bad model performance if you do that training sub-loop like I was. The correct way to do teacher forcing is just to pass the targets shifted left one. That way, since the attention mask already restricts the model from cheating by looking ahead, each output token will have to be correctly predicted based on all the previous tokens in the input. Effectively, the same thing is accomplished as my training sub-loop, but without the massive inefficiency and bad performance.</p>
<pre><code class="lang-auto"># Model requires both "inputs" and "targets"
opt.zero_grad()
in_tgt = targets[:, :-1]
exp_tgt = targets[:, 1:]
tgt_padding_mask = tgt_padding_mask[:, :-1]
# Some code missing here, assume in_tgt gets converted to in_tgt_emb
out = model(inp_emb, in_tgt_emb, tgt_mask=tgt_mask, src_key_padding_mask=inp_padding_mask, tgt_key_padding_mask=tgt_padding_mask)
loss = criterion(out, exp_tgt)
loss.backward()
opt.step()
sch.step()
</code></pre>

username: shamoons
cooked: <p>So then can the Transformer network only predict ONE tilmestep as an output at a time?</p>

username: shamoons
cooked: <p>What if the input and target have the same length?</p>

username: paganpasta
cooked: <p>During inference, for most of the usecases, <code>YES</code>.  For training, predictions happen in parallel.</p>

username: shamoons
cooked: <p>During training, I’m doing:</p>
<pre><code class="lang-auto">pred = model(x, y)
</code></pre>
<p>Where both have the same shape: <code>(batch, seq, features)</code>. <code>pred</code> has the same shape. But if it’s one at a time, then how do I get one time-step at a time?</p>

username: paganpasta
cooked: <p>This <a href="https://discuss.pytorch.org/t/nn-transformer-explaination/53175/20">answer</a> explains the working process. I am unaware of the problem you are facing, hopefully the responses on the other thread can help you out.</p>

username: shamoons
cooked: <p>Thanks so much! I think I got something working now. Only problem is that inference is super slow because it’s autoregressive.</p>
<p>I’m doing:</p>
<pre><code class="lang-auto">        tgt = torch.ones(x.size(0), 1, 128) * -1

        for i in range(x.size(1)):
            tgt = torch.randn(tgt.size())
            pred = reconstruct_spect_model(x, tgt)

            tgt = torch.cat((tgt, pred[:, -1, :].unsqueeze(1)), 1)
</code></pre>
<p>I assume I don’t need any <code>tgt_mask</code> during inference?</p>

username: muzikmonk
cooked: <p>Thanks all for the discussion! Has anyone had problems in inference despite training using the teacher-forcing with decoder input shifted left and loss computed on the target shifted right? I am beginning to suspect that too much teacher forcing during training is disrupting the ability of the network to independently estimate the sequence without using ground-truth priors. The inference on training data itself fails badly in my case despite a successful training. I am training the transformer model to generate spectrogram like features.</p>

username: paganpasta
cooked: <p>You might have to elaborate on what you mean from a <code>successful</code> training.<br>
In a nut shell, during training, the model is predicting the GT at <code>t+1</code> using GT from <code>0...t</code><br>
For inference, since you dont have GT have to iteratively feed in the <code>model output</code> to arrive at 1…t.</p>