RuntimeError: Function CudnnConvolutionBackward returned an invalid gradient at index 0 - got [1, 3, 134, 134] but expected shape compatible with [1, 3, 112, 112]

username: r_khanna
cooked: <p>I am trying to visualize CNN(Vgg-16) by optimizing a random input image so as to maximize the activation of a given channel in a given layer. But after every certain number of iterations, i want to upsample the image as well. After the first time upsampling happens, this error is presented -</p>
<p><strong>RuntimeError: Function CudnnConvolutionBackward returned an invalid gradient at index 0 - got [1, 3, 67, 67] but expected shape compatible with [1, 3, 56, 56]</strong></p>
<p>Here 56x56 is the original image size and 67x67 is size after upscaling.</p>
<p>Here is my code :<br>
<strong>Note</strong> :</p>
<ol>
<li>the forward method of model returns a dictionary with a layer’s index as key and layer’s output as the value. The layers in output are the relu layers.</li>
<li>n_channels is the number of channels from each layer we wish to vizualize</li>
</ol>
<pre><code class="lang-auto">layers = [29]
learning_rate = [.01]
n_channels = 1
iterations = 300
afterEvery = 50
start = afterEvery
upto = iterations
img_size = 56
random_low = 150
random_high = 180
upscaleFactor = 1.2
upscale = nn.Upsample(scale_factor=upscaleFactor,mode='bicubic')

for rate in learning_rate:
    for layer in layers:
        for ch_i in range(n_channels):
            image = (((random_high - random_low)*torch.rand(1,3,img_size,img_size) + random_low)/255).cuda()
            img = (image - mean)/std
            img.requires_grad = True
            optimizer = optim.Adam([img],lr=rate)
            out = model(img)
            channel = torch.randint(out[layer].size()[1],(1,))
            plt.figure(figsize=(30,70))

            for i in range(iterations):
                if(i!=0) :
                    out = model(img)
                kernel = out[layer][0][channel]

                loss = -kernel.mean()
                
                if loss == 0 and i==0:
                    print('Finding non-zero loss kernel')
                    while loss == 0:
                        channel = torch.randint(out[layer].size()[1],(1,))
                        kernel = out[layer][0][channel]
                        loss = -kernel.mean()
                        print('Tried channel index - ',channel.item(),' Loss - ',loss.item())

                optimizer.zero_grad()
                loss.backward()
                optimizer.step()

                if i%afterEvery == 0 or i==iterations-1:
                    print('Learning rate : ',rate,'\tLayer : ',layer,'\tChannel : ',channel.item(),'\titeration : ',i,' \tLoss : ',-loss.item())
                    print(img.size()[2])
                    plt.subplot(7,3,math.ceil(i/afterEvery)+1)
                    plt.imshow(transforms.ToPILImage()((img*std + mean).cpu()[0]))
                    if i &gt;= start and loss != 0 and i &lt;= upto:
                        img.data=upscale(img).data
                        print(img.size())
                        
                kernel=kernel.detach()
                for j in out:
                    out[j]=out[j].detach()
                    
            plt.show()  
</code></pre>
<p>I don’t know why is it still expecting gradient to be of shape [1, 3, 56, 56]. Any help would be appreciated.<br>
Thanks in advance <img src="https://discuss.pytorch.org/images/emoji/apple/wink.png?v=9" title=":wink:" class="emoji" alt=":wink:"></p>

username: ptrblck
cooked: <p>You shouldn’t use the <code>.data</code> attribute as it might yield these kind of unwanted side effects.<br>
Try to remove the <code>.data</code> usage and check, if Autograd raises any errors.</p>

username: r_khanna
cooked: <p>In that case, if I do <code>img = upscale(img)</code> , then the loss doesn’t change after the image is upscaled. <code>img</code> doesn’t remain a leaf node anymore, I check this with <code>img.is_leaf</code> and it returned False. I also checked  <code>img.grad</code> and it returned None. I used <code>img.data = upscale(img).data</code> to overcome this hurdle.</p>
<p>Can you please suggest how I may implement it?</p>

username: albanD
cooked: <p>Hi,</p>
<p>As a general rule, <code>.data</code> is never the solution you’re looking for <img src="https://discuss.pytorch.org/images/emoji/apple/smiley.png?v=9" title=":smiley:" class="emoji" alt=":smiley:"> Even though it can fix your issue, it is very likely to break many other things <img src="https://discuss.pytorch.org/images/emoji/apple/wink.png?v=9" title=":wink:" class="emoji" alt=":wink:"></p>
<p>Here you just want the python variable “img” to point to a new leaf that is built based on the upscaled image:</p>
<pre><code class="lang-auto">img = upscale(img).detach() # Detach from history to make it a leaf
img.requires_grad_() # Make it require gradients again
</code></pre>

username: r_khanna
cooked: <p>Thank you <a class="mention" href="/u/alband">@albanD</a>. That worked <img src="https://discuss.pytorch.org/images/emoji/apple/smile.png?v=9" title=":smile:" class="emoji" alt=":smile:"></p>