akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
How to get gradients of output of activation function with respect to its input?

username: gebrahimi
cooked: <p>Hi,</p>
<p>I am wondering if there is a way to get gradients of output of activation function with respect to input to the activation. It is needed in backpropagation, so I am sure pytorch does that. I can probably use input values and take derivative as derivative of activation functions are easy, but is there a way to get that from pytorch? Also, I wonder what happens if the input is exactly 0 and our activation function is Relu?</p>
<p>Thanks.</p>

username: francois_rousseau
cooked: <p>Hi,</p>
<p>You can use autograd for this.</p>
<pre><code class="lang-auto">gradients = torch.autograd.grad(outputs=activations, inputs=input,
                          grad_outputs=torch.ones(activations.size()).to(device),
                          create_graph=True, retain_graph=True, only_inputs=True)[0]
</code></pre>
<p>Related question here. I’d like to compute the gradient of activations wrt inputs for several layers inside the network. So far, I’ve built several intermediate models to compute the gradients of the network output wrt input. I’m sure it can done more efficiently using hooks, but I don’t know how.</p>
<p>Any help ?<br>
Thanks.</p>
Display image in PIL format from torch.Tensor

username: rakshit_naidu
cooked: <p>Hello, I’m quite new to Pytorch. I was wondering how I could convert my tensor of size <code>torch.Size([1, 3, 224, 224])</code> to display in an image format on a Jupyter notebook. A PIL format or a CV2 format should be fine.</p>
<p>I tried using <code>transforms.ToPILImage(x)</code> but it resulted in a different format like this: <code>ToPILImage(mode=ToPILImage(mode=tensor([[[[1.3034e-16, 1.3034e-16, 1.3034e-16,  ..., 1.4475e-16,</code>. Maybe I’m doing something wrong <img src="https://discuss.pytorch.org/images/emoji/apple/no_mouth.png?v=9" title=":no_mouth:" class="emoji" alt=":no_mouth:"></p>

username: ptrblck
cooked: <p><code>ToPILImage()</code> should work.<br>
I’m not sure what you are passing as the <code>mode</code> argument, but this small code snippet works fine:</p>
<pre><code class="lang-python">x = torch.randn(1, 3, 224, 224)
trans = torchvision.transforms.ToPILImage()
out = trans(x[0])
out.show()
</code></pre>

username: rakshit_naidu
cooked: <p>Thanks a lot <a class="mention" href="/u/ptrblck">@ptrblck</a></p>
Accumulated gradients while hyper-parameter tuning

username: Shashank_Shekhar
cooked: <p>I am working on basic hyper-parameter tuning where my <strong>parameters are Learning rate [0.03], Batch [64, 88].</strong><br>
Individual training with [0.03, 64] parameter gives green loss curve, on the other hand for [0.03, 88] parameter gives pink loss curve. This seems correct as the pattern of loss curve remain same just their is shift because of the batch size.</p>
<p>While doing combined training one after the other, for [0.03, 64] parameter i still get green loss curve but for [0.03,88] parameter i am getting the Gray curve (which is at the bottom of image shown). I think this is because gradients learned form Run1 are not being deleted as a result causing Run2 to start with very low loss value.</p>
<p>But before starting Run2 i have reinitialised the instance of Optimiser, Model, Scheduler and after completion of Run 1 i have even deleted all the previous gradients using model.zero_grad(), optimizer.zero_grad(). Can somebody please help!!</p>
<p><img src="https://discuss.pytorch.org/uploads/default/original/3X/3/0/30df4b4f6b356a5abbbe7cba95e20af30e6149ee.png" alt="Screenshot from 2020-04-23 12-25-19" data-base62-sha1="6Ylk63HQ9jnt5wRjaqpuKlaI7Ia" width="405" height="274"></p>
PCA on GAN for training optimization

username: tomt
cooked: <p>Hello all, I want to perform PCA on the feature maps / weights in the hidden layers of GANs with for instance a single component, in order to see if there is possible to optimize the GANs architecture with PCA. Currently working with the ESRGAN on images, and training takes a long time.<br>
I was wondering how to perform PCA on convolution layers in the network.<br>
For instance if this is my model, where can I perform the PCA? Or should I do it in the training for loop?</p>
<pre><code class="lang-auto">class MyModel(nn.Module):
    def __init__(self):
        super(MyModel, self).__init__()
        self.conv1 = nn.Conv2d(1, 3, 3, 1, 1)
        self.pool1 = nn.MaxPool2d(2)
        self.conv2 = nn.Conv2d(3, 6, 3, 1, 1)
        self.pool2 = nn.MaxPool2d(2)
        
        self.conv_trans1 = nn.ConvTranspose2d(6, 3, 4, 2, 1)
        self.conv_trans2 = nn.ConvTranspose2d(3, 1, 4, 2, 1)
        
    def forward(self, x):
        x = F.relu(self.pool1(self.conv1(x)))
        x = F.relu(self.pool2(self.conv2(x)))        
        x = F.relu(self.conv_trans1(x))
        x = self.conv_trans2(x)
        return x
</code></pre>