Backprop over kernels in a CNN

username: Srinjay_Sarkar
cooked: <p>Let’s say a convolutional layer takes an input 𝑋 with dimensions of 5x100x100 and applies 10 filters 𝐹 5x5x5, thus produces an output 𝑂 10 feature maps 96x96.<br>
During the backpropagation the layer receives 𝑑𝐸/𝑑𝑂 of shape 10x96x96.<br>
My question is how to compute 𝑑𝐸/𝑑𝐹 ?</p>
<p>According to [that article]<br>
(<a href="https://medium.com/@20!7csm1006/forward-and-backpropagation-in-convolutional-neural-network-4dfa96d7b37e" class="inline-onebox" rel="noopener nofollow ugc">Forward And Backpropagation in Convolutional Neural Network. | by Sujit Rai | Medium</a>)<br>
𝑑𝐸/𝑑𝐹 can be calculated as convolution between 𝑋 and 𝑑𝐸/𝑑𝑂<br>
Unfortunately, the article does not cover a case with multiple filters and multiple input channels.<br>
Since 𝑋 has shape 5x100x100 and 𝑑𝐸/𝑑𝑂 has shape 10X96x96 the depth of 𝑋 equals to 5 and the depth of 𝑑𝐸/𝑑𝑂 equals to 10. So the depth dimension does not match. How to compute convolution in that case ?</p>
<p>Link for question on stackoverflow:<a href="https://datascience.stackexchange.com/questions/38896/an-error-with-respect-to-filter-weights-in-cnn-during-the-backpropagation?answertab=votes#tab-top" class="inline-onebox" rel="noopener nofollow ugc">neural network - An error with respect to filter weights in CNN during the backpropagation - Data Science Stack Exchange</a></p>
<p>The author posted a solution to this problem as shown in the image.But this shows that the gradient of all filters will be the same across their channels which I could not reproduce with my code?</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/2X/d/dcfbb86fcb08b70ed104b9f12491023b399e3bfc.png" data-download-href="https://discuss.pytorch.org/uploads/default/dcfbb86fcb08b70ed104b9f12491023b399e3bfc" title="125"><img src="https://discuss.pytorch.org/uploads/default/optimized/2X/d/dcfbb86fcb08b70ed104b9f12491023b399e3bfc_2_541x500.png" alt="125" data-base62-sha1="vwUlPqsCgyNmNVeCSTxPnk3j2zy" width="541" height="500" srcset="https://discuss.pytorch.org/uploads/default/optimized/2X/d/dcfbb86fcb08b70ed104b9f12491023b399e3bfc_2_541x500.png, https://discuss.pytorch.org/uploads/default/optimized/2X/d/dcfbb86fcb08b70ed104b9f12491023b399e3bfc_2_811x750.png 1.5x, https://discuss.pytorch.org/uploads/default/original/2X/d/dcfbb86fcb08b70ed104b9f12491023b399e3bfc.png 2x" data-dominant-color="736742"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">125</span><span class="informations">1000×923 156 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>Is the method wrong or is something wrong with my code?</p>
<pre data-code-wrap="python"><code class="lang-python">import torch 
import numpy as np
import matplotlib.pyplot as plt
import torch.nn as nn 
import cv2
import matplotlib.pyplot as plt
ref_tensor1=torch.from_numpy(cv2.resize(cv2.imread("./trial_5.jpg",0).astype(np.float32),dsize=(225,225)))
ref_tensor1=ref_tensor1.unsqueeze(0).unsqueeze(0)
print(ref_tensor1.shape)
image1=cv2.imread("./trial_5.jpg").astype(np.float32)
image1=cv2.resize(image1,dsize=(256,256))
image1=np.rollaxis(image1,2)
image2=cv2.imread("./trial_8.jpg").astype(np.float32)
image2=cv2.resize(image2,dsize=(256,256))
image2=np.rollaxis(image2,2)
img_tensor2=torch.from_numpy(image2).unsqueeze(0)
img_tensor1=torch.from_numpy(image1).unsqueeze(0)
img_tensors=torch.cat((img_tensor1,img_tensor2),0)
print(img_tensors.shape)
print("Input_image_shape:",img_tensors.shape)
#print(img_tensors)


class torch_model1(torch.nn.Module):          
    def __init__(self,ic,oc,ks):     
        super(torch_model1,self).__init__()
        self.conv1 = torch.nn.Conv2d(in_channels=ic,out_channels=oc,kernel_size=ks,stride=1)
    def forward(self,x):
        x = self.conv1(x)
        return (x)

###1,3###
model1=torch_model1(3,3,32)
temp=torch.randn(img_tensors.shape)
op1=model1(img_tensor1)
print(op1.shape)
#assert(op1.shape==ref_tensor1.shape)
loss=torch.abs(op1-ref_tensor1).mean()
print(loss)
print("gradient_shape:",model1.conv1.weight.shape)
print("Before backprop:",model1.conv1.weight.grad)
loss.backward()
print("after backprop:",model1.conv1.weight.grad.shape)
#print("gradients:",model1.conv1.weight.grad)
print(model1.conv1.weight.grad)
##########RESULT(1,3,x,y):changes if seed is not set and the gradient is same for all channels#############
plt.subplot(131)
plt.imshow(model1.conv1.weight.grad[0,0,:,:])
plt.subplot(132)
plt.imshow(model1.conv1.weight.grad[0,1,:,:])
plt.subplot(133)
plt.imshow(model1.conv1.weight.grad[0,2,:,:])
plt.show()
</code></pre>

username: smth
cooked: <p>I dont know if it helps, but I wrote a post on it here: <a href="http://soumith.ch/ex/pages/2014/08/07/why-rotate-weights-convolution-gradient/" rel="nofollow noopener">http://soumith.ch/ex/pages/2014/08/07/why-rotate-weights-convolution-gradient/</a></p>
<p>It goes into the calculations in a bit more detail.</p>

username: Srinjay_Sarkar
cooked: <p>This post shows the calculation for gradient of the input.I need the  solution for calculation of the? gradient of the filter weights?</p>

username: Sia_Rezaei
cooked: <p>Hi <a class="mention" href="/u/srinjay_sarkar">@Srinjay_Sarkar</a>, did you figure this out?<br>
Adding padding, dilation, stride, and channels makes this a very mind twisting exercise!</p>

username: Srinjay_Sarkar
cooked: <p><a class="mention" href="/u/sia_rezaei">@Sia_Rezaei</a>, you might want to look at <a href="https://github.com/pytorch/pytorch/blob/master/torch/nn/grad.py" rel="nofollow noopener">this</a> .It has all the python implementations for calculating the gradients of weights,biases and inputs.But using this for a network makes it extremely slow.I used a C++ extension to call the cudnn backprop which is much faster.</p>

username: Sia_Rezaei
cooked: <p><a class="mention" href="/u/srinjay_sarkar">@Srinjay_Sarkar</a> thanks! Yes, I just found out about <code>grad.py</code> and yea, it is slow!<br>
Can you share how you call cudnn backprop? Or point me to the right direction? Thanks!</p>

username: Srinjay_Sarkar
cooked: <p><a href="https://discuss.pytorch.org/t/cuda-error-with-cudnn-convolution-backward-weight-function/41214">This</a> is a good starting point.</p>