akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Why `criterion.cuda()` is not needed but `model.cuda()` is?

nivter: <p>When working on GPU, we need to do something similar to:</p>
<p><code>x.cuda()</code></p>
<p>where x can be a model or input variables.</p>
<p>I was wondering why is it not done for loss criteria?</p>
<pre><code class="lang-auto">criterion = nn.MSELoss()

# why is the below line not implemented?
if torch.cuda.is_available(): criterion.cuda()
</code></pre>
<p>Does the criterion somehow infer whether or not to use cuda from the model?</p>

albanD: <p>The impact of moving a module to cuda is actually to move all it’s parameters to cuda.<br>
Criterion don’t have parameters in general, so it is not necessary to do it.</p>

nivter: <p>Thank you.</p>
<p>So does that mean if I implement my own loss function which involves parameters, I should do <code>criterion.cuda()</code> to improve speed?</p>

albanD: <p>Yes if it has parameters, you will need to call <code>.cuda()</code> on it if the input you give to it is on the gpu.</p>

yashsharma0906: <p>Hey <a class="mention" href="/u/alband">@albanD</a>,</p>
<p>Apologies for the direct message.<br>
If I am using a dynamically created tensor for loss calculation, what would be the recommended approach for writing the loss function for optimizing Cuda command use? Ex. -</p>
<pre><code class="lang-auto">class KLDLoss(nn.Module):
    """KL-divergence loss between attention weight and uniform distribution"""
    def __init__(self):
        super(KLDLoss, self).__init__()
        
    def forward(self, attn_val, cluster):
        """
        Example:
          Input - attention value = torch.tensor([0.05, 0.1, 0.05, 0.1, 0.05, 0.1, 0.05, 0.05, 
                                0.1, 0.05, 0.1, 0.05, 0.1, 0.05])
                  cluster = [1, 1, 1, 1, 1, 1, 1, 2, 2, 2, 2, 2, 2, 2]
          Output - 0.0043
        """
        
        kld_loss = 0
        cluster = np.array(cluster)
        for cls in np.unique(cluster):
            index = np.where(cluster==cls)[0]
            if torch.cuda.is_available():
                kld_loss += F.kl_div(F.log_softmax(attn_val[index], dim=0)[None],\
                                torch.ones(len(index), 1)[None].cuda()/len(index),\
                                reduction = 'batchmean')
            else:
                kld_loss += F.kl_div(F.log_softmax(attn_val[index], dim=0)[None],\
                                torch.ones(len(index), 1)[None]/len(index),\
                                reduction = 'batchmean')                
        return kld_loss
</code></pre>
<p>Thanks!</p>

albanD: <p>Hey,</p>
<p>Side note, you might want to use the builtin KL: <a href="https://pytorch.org/docs/stable/nn.functional.html?highlight=kl#torch.nn.functional.kl_div" class="inline-onebox">torch.nn.functional — PyTorch 1.8.0 documentation</a></p>
<p>From your code, I think the main issue is that you’re using numpy arrays (which cannot be on GPU).<br>
You might want to keep everything as Tensors and when you create new Tensors, you can pass the <code>device=</code> kwarg. In this case, you want to match the device of the inputs I guess so <code>attn_val.device</code>.</p>
Does nn.Embedding detach tensor from computation graph?

rschaefer: <p>Hi,</p>
<p>I am currently trying to train a GAN model in the context of machine translation.</p>
<p>My problem is similar to: <a href="https://github.com/ngohoanhkhoa/GAN-NMT" class="inline-onebox" rel="noopener nofollow ugc">GitHub - ngohoanhkhoa/GAN-NMT: Generative Adversarial Networks in Neural Machine Translation</a> ; although they use TensorFlow and the data is different.</p>
<p>My problem is that during training the gradients of my generator are none, which basically means that the models do not train at all. The generator is based on a seq2seq (LSTM) model. The discriminator is a CNN. Both models rely on nn.embedding as a first step to encode the sequences I input into the models.</p>
<p>I tried to fix the problem for some time, without success. My assumption is that when I forward the generator output to the discriminator, which derives embeddings using nn.Embedding, some detachment takes place. Can this be the case?</p>
<p>Your help is highly appreciated. Thanks.</p>
<p>Best,<br>
Robin</p>

albanD: <p>Hi,</p>
<p>No detach takes place in that layer no.<br>
But if you most likely give it Tensors that don’t require gradients as input no?</p>

rschaefer: <p>Thanks for your reply. The input to the Generator has .requires_grad set to false.</p>
<p>I calculate the loss for the Discriminator output. .backward() creates the gradients for this model, but not for the generator which outputs the input for the Discriminator.</p>
<p>Let me give you some more details about the data processing:</p>
<ol>
<li>The sequences I input into the model are encoded.</li>
<li>Based on these encodings the embeddings are created inside the generator</li>
<li>The outputted data is ‘decoded’ again using the former character-encoding mapping; this is done as I give both the output of the generator and some encoded target sequences into the Discriminator; these target sequences do not pass through the Generator which means they are not embedded.</li>
<li>In the Discriminator sequences are again embedded</li>
</ol>
<p>I noticed that the output of the Generator has requires_grad set to true. When I transfer the data to the original encoding requires_grad is set to false. But I am not sure if this is the problem.</p>

albanD: <p>Well, the thing is that Embedding layers take indices as input (of integer types). And such Tensors cannot track gradients (because they are not contiguous). So I still think that you convert to an integer type between the two layers and thus don’t get gradients for your generator.</p>

rschaefer: <p>Ah, I get it now. I’ll check that. Thanks.</p>