akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
With a Transformer, can teacher forcing be done in Parallel?

shamoons: <p>I have a sequence to sequence model with <code>nn.Transformer</code> and in my training loop, I have:</p>
<pre><code class="lang-auto">        for i in range(x.size(1)):
            pred = model(x, tgt[:, 0:i + 1, :])
            loss = loss_fn(pred, ground_truth[:, 0:i + 1, :])
            loss.backward()
            optimizer.step()
            total_loss += loss.item()
            cnt += 1
        return total_loss / cnt
</code></pre>
<p>Where <code>x.size()</code> is <code>[batch, seq, features]</code>. So my sequence increases in each iteration, but of course, this takes a LONG time Can this be done in parallel somehow?</p>

muzikmonk: <p>Hi, did you manage to find a way to parallelize the loop? I am also looking for a way to do that. Thanks!</p>
Pre processing the NSL-KDD dataset for RNNs

rcon313: <p>Hey guys is anyone able to help me out ? I am currently in my final year of university and for my final year project I am using recurrent neural networks within an IDS using the NSL-KDD dataset. If anyone has code on how to pre process the dataset and how you would implement the RNN into it as well, I will also be comparing this model against support vector machine for accuracy and false positive rate. I am a complete newbie to python and machine learning so any advice would be a huge help</p>
<p>Thank you,<br>
Ross</p>
Efficient gradient computation on sparsified/pruned network

aberk: <p>I am trying to train a sparsified network. I can construct that network using something like:</p>
<pre><code class="lang-python">masks = get_masks(my_network, quantile)
def prune_hook(module, input):
    for param, mask in zip(module.parameters(), masks):
        param.data = param.data * mask
my_network.register_pre_forward_hook(prune_hook)
# ~10% of weights will be nonzero during forward pass
</code></pre>
<p>This code snippet will guarantee that anytime I make a forward pass, the appropriate weights are “zeroed-out”. I’m wondering if PyTorch still performs the full gradient computation during each backward pass, or if there’s any way to inform PyTorch that it need only compute gradients for a subset of the weights. Part of the rationale of using a sparsified network is increased computational efficiency during training. I suppose I am wondering if that computational efficiency must be hand-rolled anew, or if it can be squeezed out of PyTorch.</p>