Train transformer without teacher forcing

username: Himanshu_Chaudhary
cooked: <p>Has anyone tried training transformer without teacher forcing? I was thinking of doing it but could not understand how to implement it in batch.<br>
Here is my code, It works with batch size=1.</p>
<pre><code class="lang-auto">with torch.no_grad():
        for batch in test_loader:
            src, trg = batch
            imgs.append(src.flatten(0,1))
            src, trg = src.cuda(), trg.cuda()     
            memory = get_memory(model,src.float())                
            out_indexes = [tokenizer.chars.index('SOS'), ]            
            for i in range(max_text_length):
                mask = model.generate_square_subsequent_mask(i+1).to(device)
                trg_tensor = torch.LongTensor(out_indexes).unsqueeze(1).to(device)
                output = model.vocab(model.transformer_decoder(model.query_pos(model.decoder(trg_tensor)), memory,tgt_mask=mask))
                out_token = output.argmax(2)[-1].item()
                out_indexes.append(out_token)
                if out_token == tokenizer.chars.index('EOS'):
                    break
</code></pre>

username: dcaffo98
cooked: <p>I’m trying  transformer without teacher forcing for time series prediction and I came out with this snippet. Theoretically it works fine, but it’s tremendously slow and it keep crashing due to cuda running out of memory as soon as I use a batch size greater than 3,4 samples.</p>
<pre><code class="lang-auto"># src shape is [n_batch, src_length, d_model] --&gt; [N, S, D]
# tgt and gt (groundtruth) shape is [n_batch, tgt_length, d_model] --&gt; [N, T, D]

l = 0.
for src, tgt, gt in loader:
    src, tgt, gt = src.to(device), tgt.to(device), gt.to(device)
    optimizer.zero_grad()

    # first tgt element is the last src one, followed by a zero-filled tensor
    tgt = torch.cat((src[:, -1:, :], torch.zeros(tgt.shape[0], tgt.shape[1] - 1, tgt.shape[2], device=device)), dim=1)

    out = torch.zeros_like(tgt, device=device)
    memory = model.encode(src)
    for j in range(gt.shape[1]):
        out = model.decode(tgt, memory)
        l += loss(out[:, j, :], gt[:, j, :])
        tgt = tgt.detach()
        tgt = torch.cat((tgt[:, :j + 1, :], out[:, j:j+1, :], tgt[:, j + 1:gt.shape[1] - 1, :]), dim=1) 
    l.backward()
</code></pre>

username: dcaffo98
cooked: <p><em>cuda</em> issues was probably caused by <code>torch.autograd.set_detect_anomaly(True)</code>, which I was set for debugging purpose. The code should run fine.</p>

username: 774843769
cooked: <p>Thank you for your share. I want to know the meaning of the d_model. I adopt BART to try your code, but the size of input_ids is [n_batch, seq_length] but there is no d_model. Can you help me?</p>

username: dcaffo98
cooked: <p>I’m not an expert with BART, but I suppose what you’re referring to are the indexes of the embedding employed by the model. Each of those <em>seq_length</em> item is an index for a vocabulary of N embeddings, each of which being a tensor of size <em>d_model</em>. Those will make the input to your model. <a href="https://pytorch.org/tutorials/beginner/nlp/word_embeddings_tutorial.html" rel="noopener nofollow ugc">This</a> may be an helpful read.</p>