Gradient is always none

hyunwookim: <p>Hi, I need some help trying to make my model pass through gradients properly.<br>
In my model, I have a series of conv layers, then linear layers. After the linear layers spit out an 2x8x8 grid, I apply  torch.abs, perform a cumulative sum operation on the grid, and upsample the grid to the size of 2x128x128. Then, I perform a grid_sample() operation to the source image using the upsampled grid, and finally the loss is computed by comparing the original image to the deformed image. In addition I have a l1 regularization term in the loss function, applied to the initial 2x8x8 grid as originally outputted by linear layers.<br>
But when I print the gradients through each of these consecutive operations, none of them has a gradient (although I set the retain_grad to true). Does anyone have suggestions as to what might be the problem here? Thanks!</p>
<p>aug_batch gradient: requires_grad(False) is_leaf(True) retains_grad(None) grad_fn(None) grad(None)<br>
source_batch gradient: requires_grad(False) is_leaf(True) retains_grad(None) grad_fn(None) grad(None)<br>
input_image gradient: requires_grad(False) is_leaf(True) retains_grad(None) grad_fn(None) grad(None)<br>
X gradient1: requires_grad(False) is_leaf(True) retains_grad(None) grad_fn(None) grad(None)<br>
X gradient2: requires_grad(True) is_leaf(False) retains_grad(True) grad_fn(&lt;ReluBackward0 object at 0x7f612bf44a90&gt;) grad(None)<br>
X gradient3: requires_grad(True) is_leaf(False) retains_grad(True) grad_fn(&lt;ViewBackward object at 0x7f612bf44a90&gt;) grad(None)<br>
X gradient4: requires_grad(True) is_leaf(False) retains_grad(True) grad_fn(&lt;ReluBackward0 object at 0x7f612bf44390&gt;) grad(None)<br>
X gradient5: requires_grad(True) is_leaf(False) retains_grad(True) grad_fn(&lt;AddmmBackward object at 0x7f612bf44fd0&gt;) grad(None)<br>
output_grid gradient: requires_grad(True) is_leaf(False) retains_grad(True) grad_fn(&lt;AddmmBackward object at 0x7f612bf44790&gt;) grad(None)<br>
output_grid gradient after view(): requires_grad(True) is_leaf(False) retains_grad(None) grad_fn(&lt;ViewBackward object at 0x7f612bf44450&gt;) grad(None)<br>
output_grid gradient after abs(): requires_grad(True) is_leaf(False) retains_grad(None) grad_fn(&lt;AbsBackward object at 0x7f612bf44390&gt;) grad(None)<br>
integrated_grid after cumsum(): requires_grad(True) is_leaf(False) retains_grad(None) grad_fn(&lt;PermuteBackward object at 0x7f612bf44790&gt;) grad(None)<br>
Upsampled_grid gradient: requires_grad(True) is_leaf(False) retains_grad(None) grad_fn(&lt;PermuteBackward object at 0x7f612bf44b90&gt;) grad(None)<br>
Target_est gradient: requires_grad(True) is_leaf(False) retains_grad(None) grad_fn(&lt;GridSampler2DBackward object at 0x7f612bf44fd0&gt;) grad(None)<br>
Warped_image gradient: requires_grad(True) is_leaf(False) retains_grad(None) grad_fn(&lt;SqueezeBackward1 object at 0x7f612bf44a90&gt;) grad(None)<br>
Warp_field gradient: requires_grad(True) is_leaf(False) retains_grad(None) grad_fn(&lt;CopySlices object at 0x7f612bf44390&gt;) grad(None)<br>
L_TV_Loss gradient: requires_grad(True) is_leaf(False) retains_grad(True) grad_fn(&lt;AddBackward0 object at 0x7f612bf44690&gt;) grad(None)<br>
Total_loss gradient: requires_grad(True) is_leaf(False) retains_grad(None) grad_fn(&lt;AddBackward0 object at 0x7f612bf44a90&gt;) grad(None)</p>

gphilip: <p>If you put your code between two lines with just three backticks, like this:</p>
<pre><code>```
your python code goes here
```
</code></pre>
<p>then it will get formatted nicely.</p>
<p>Edited to add: I <em>thought</em> what you quoted was code, but it looks like it is not. I am not sure how to interpret all the non-prose text in your post. Perhaps you could explain what those lines mean?</p>
<p>Also, I think gradients cannot pass across <code>torch.abs()</code>, but I may be remembering this incorrectly.</p>

hyunwookim: <p>Thanks. Since my model passed through a number of functions after the last linear layer, I rearranged the codes so you can see what is going on.</p>
<pre><code class="lang-auto">class Net(nn.Module):
  def __init__(self, grid_size):
    super().__init__()
    self.conv = get_conv(grid_size)
    self.flatten = nn.Flatten()
    self.linear1 = nn.Sequential(nn.Linear(80,20),nn.ReLU(),)
    self.linear2 = nn.Linear(20, 2*grid_size*grid_size)
    self.upsampler = nn.Upsample(size = [IMAGE_SIZE, IMAGE_SIZE], mode = 'bilinear')
    #self.linear2.bias = nn.Parameter(init_grid(grid_size).view(-1))
    #self.linear2.weight.data.fill_(float(0))
    self.grid_offset_x = torch.tensor(float(0), requires_grad=True)
    self.grid_offset_y = torch.tensor(float(0), requires_grad=True)
    self.grid_offset_x = nn.Parameter(self.grid_offset_x)
    self.grid_offset_y = nn.Parameter(self.grid_offset_y)
    self.grid_size = grid_size
  def forward(self, x, src_batch, checker_board=False):
    print(f'X gradient1: {get_tensor_info(x)}')
    x = self.conv(x)
    print(f'X gradient2: {get_tensor_info(x)}')
    x = self.flatten(x)
    print(f'X gradient3: {get_tensor_info(x)}')
    x = self.linear1(x)
    print(f'X gradient4: {get_tensor_info(x)}')
    x = self.linear2(x)
    print(f'X gradient5: {get_tensor_info(x)}')
    #enforce axial monotinicity using the abs operation
    x = torch.abs(x)
    print(f'X gradient after abs(): {get_tensor_info(x)}')
    batch, grid = x.shape
    x = x.view(batch, 2,self.grid_size,self.grid_size)
    #perform the cumsum operation to restore the original grid from the differential grid
    x = cumsum_2d(x, self.grid_offset_x, self.grid_offset_y)
    print(f'X gradient after cumsum(): {get_tensor_info(x)}')
    #Upsample the grid_size x grid_size warp field to image_size x image_size warp field
    x = self.upsampler(x)
    print(f'X gradient after upsampling: {get_tensor_info(x)}')
    x = x.permute(0,2,3,1)
    if checker_board:
      source_image = apply_checkerboard(src_batch, IMAGE_SIZE)
    #calculate target estimation
    x = nn.functional.grid_sample(src_batch.unsqueeze(0).permute([1,0,2,3]), x)

    return x```

The result of this gradient check is 
X gradient1: requires_grad(False) is_leaf(True) retains_grad(None) grad_fn(None) grad(None)
X gradient2: requires_grad(True) is_leaf(False) retains_grad(None) grad_fn(&lt;ReluBackward0 object at 0x7f5f1003d3d0&gt;) grad(None)
X gradient3: requires_grad(True) is_leaf(False) retains_grad(None) grad_fn(&lt;ViewBackward object at 0x7f5f123eed50&gt;) grad(None)
X gradient4: requires_grad(True) is_leaf(False) retains_grad(None) grad_fn(&lt;ReluBackward0 object at 0x7f5f123eed10&gt;) grad(None)
X gradient5: requires_grad(True) is_leaf(False) retains_grad(None) grad_fn(&lt;AddmmBackward object at 0x7f5f1004fe10&gt;) grad(None)
X gradient after abs(): requires_grad(True) is_leaf(False) retains_grad(None) grad_fn(&lt;AbsBackward object at 0x7f5f123eed10&gt;) grad(None)
X gradient after cumsum(): requires_grad(True) is_leaf(False) retains_grad(None) grad_fn(&lt;PermuteBackward object at 0x7f5f12350450&gt;) grad(None)
X gradient after upsampling: requires_grad(True) is_leaf(False) retains_grad(None) grad_fn(&lt;UpsampleBilinear2DBackward1 object at 0x7f5f1003d3d0&gt;) grad(None)
Total_loss gradient: requires_grad(True) is_leaf(False) retains_grad(True) grad_fn(&lt;MulBackward0 object at 0x7f5f10031b50&gt;) grad(None)
total_loss gradient: requires_grad(True) is_leaf(False) retains_grad(True) grad_fn(&lt;MulBackward0 object at 0x7f5f10031750&gt;) grad(None)

And this is my loss function
```def Total_Loss (target_image, warped_image, grid_size, Lambda):
  batch,W,H = warped_image.shape
  #print(f'Warp_field gradient: {get_tensor_info(warp_field)}')

  L2_Loss_f = nn.MSELoss()
  L2_Loss = 1/2 * L2_Loss_f(target_image, warped_image)
  
  Total_loss = L2_Loss
  Total_loss.retain_grad()
  print(f'Total_loss gradient: {get_tensor_info(Total_loss)}')

  return Total_loss```

Appreciate your help</code></pre>