akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
How to get a 2d tensor from a 3d tensor simply?

username: Raibows
cooked: <p>I have a 3d tensor outs whose shape is like [max_seq_len, batch_size, hidden_dim].<br>
For a simple example, the max_seq_len, batch_size, hidden_dim is set by 2, 3, 4. And i have a another 1d tensor sentence_len recording the real length of each sample in the batch that is [1, 1, 2].<br>
Now I need to choose the last state for each sample,<br>
1st sample, outs[1-1, 0, :]<br>
2nd sample, outs[1-1, 1, :]<br>
3rd sample, outs[2-1, 2, :]<br>
Then form these selected vectors to a 2d tensor that I want.<br>
Note that the sentence_len needs to minus 1 for the indexing.<br>
I have tried gather and index_select but failed and stuck with it for few days. Really hope you could help me! Thanks.</p>

username: Raibows
cooked: <p><img src="https://discuss.pytorch.org/images/emoji/apple/joy.png?v=9" title=":joy:" class="emoji" alt=":joy:"> I have worked it out by using slicing.<br>
<code>outs[sentence_len-1, torch.arange(outs.size(1)), :]</code></p>
How to use zero_grad with GANs

username: Rookie
cooked: <p>In the pix2pix code that can be found <a href="https://github.com/eriklindernoren/PyTorch-GAN/blob/master/implementations/pix2pix/pix2pix.py" rel="noopener nofollow ugc">here</a> first the generator is trained and then the discriminator. My question is, aren’t the gradients calculated loss_D.backward() accumulated in the training of the generator in the next loop? Shouldn’t we use zero_grad on the discriminator again before training the generator?</p>

username: matthewleigh
cooked: <p>loss_D.backward() wont accumulate any gradients for the generator parameters.</p>
<p>This is because of this line:</p>
<pre><code class="lang-auto">pred_fake = discriminator(fake_B.detach(), real_A)
</code></pre>
<p>The detach method prevents any gradient accumulation as it simply returns values.</p>

username: Rookie
cooked: <p>Thanks for the reply. What I was worried about was actually the adversarial part.  In the following part, the pixelwise loss and the loss from the discriminator is combined and backpropagation is called. Isn’t the generator also updated with the backpropagation path from the discriminator? But the gradients from the previous loop are not zeroed. I couldn’t understand thoroughly how the gradient accumulation works in this case.</p>
<pre><code class="lang-auto">        # GAN loss
        fake_B = generator(real_A)
        pred_fake = discriminator(fake_B, real_A)
        loss_GAN = criterion_GAN(pred_fake, valid)
        # Pixel-wise loss
        loss_pixel = criterion_pixelwise(fake_B, real_B)

        # Total loss
        loss_G = loss_GAN + lambda_pixel * loss_pixel

        loss_G.backward()

        optimizer_G.step()
</code></pre>

username: matthewleigh
cooked: <p>Remember that the parameters are updated through the optimizer’s step method.<br>
When the optimiser is initialised you pass it all the parameters it will modify.</p>
<p>So yes, there are still some uncleared gradients in the discriminator parameters, however:</p>
<pre><code class="lang-auto">loss_G.backward()
optimizer_G.step()
</code></pre>
<p>Wont be affected by them, as this will only modify the parameters of the generator based on its gradients wrt loss_G.</p>