Is torch.inverse differentiable?

username: Alessio_Bonacchi
cooked: <p>Hello everybody, i am still trying to solve the issue i have explained here, i will leave it here for someone brave enough <img src="https://discuss.pytorch.org/images/emoji/apple/smile.png?v=12" title=":smile:" class="emoji" alt=":smile:" loading="lazy" width="20" height="20"></p><aside class="quote quote-modified" data-post="1" data-topic="192284">
  <div class="title">
    <div class="quote-controls"></div>
    <img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/alessio_bonacchi/48/62645_2.png" class="avatar">
    <a href="https://discuss.pytorch.org/t/torch-nn-parameters-not-updating-gradient-remains-none/192284">Torch.nn.parameters not updating, gradient remains None</a> <a class="badge-category__wrapper " href="/c/autograd/7"><span data-category-id="7" data-drop-close="true" class="badge-category " title="A category of posts relating to the autograd engine itself."><span class="badge-category__name">autograd</span></span></a>
  </div>
  <blockquote>
    Hi everyone! 
I have a problem with my code, basically i am trying to do a camera optimization neural network which have basically 2 parameters rotation and traslation, that needs to update in order to minimize the loss between the rendered image and the target image. I will post my code here: 


Definition of the model: the parameter for rotation matrix is stored as 3 different arrays each representing a row of the matrix, and then in the forward i stack them. This because i saw that declaring …
  </blockquote>
</aside>
<p>
Btw, I am trying to decompose my problem in order to understand why my parameters do not update and the gradient mantains as None.</p>
<p>Just to give you a little bit of context my model is (partially) defined as:</p>
<pre><code>class Model(nn.Module):
    def __init__(self, device, initial_rotation, initial_traslation, gs, gt, fov, pipe, bg):
        super().__init__()
        
        # device and gaussian splat
        self.device = device
        self.gaussian_splat = gs
        
        self.criterion = nn.MSELoss()
        
        # Target image
        # self.image_ref = torch.from_numpy(gt).permute(2,0,1)
        # self.image_ref = self.image_ref.to(self.device)
        self.image_ref = PILtoTorch(gt, (472,837)).to(self.device)
        
        # Parameters
        self.r1 = nn.Parameter(
            torch.from_numpy(initial_rotation[0]).to(self.device))
        self.r2 = nn.Parameter(
            torch.from_numpy(initial_rotation[1]).to(self.device))
        self.r3 = nn.Parameter(
            torch.from_numpy(initial_rotation[2]).to(self.device))
        self.traslation = nn.Parameter(
            torch.from_numpy(initial_traslation).to(self.device))
</code></pre>
<p>and this is a part of the forward method:</p>
<pre><code>  def forward(self):
  
      R = torch.stack((self.r1, self.r2, self.r3))

      world_view_transform = getWorld2View_custom(self.device, R, self.traslation).transpose(0, 1).to(self.device)
</code></pre>
<p>where getWorld2View_custom is defined as:</p>
<pre><code>  def getWorld2View_custom(device, R, t, translate=torch.tensor([.0, .0, .0]), scale=1.0):
      Rt = torch.cat((torch.transpose(R, 0, 1), t.unsqueeze(1)), dim=1)
      Rt = torch.cat((Rt, torch.tensor([[0, 0, 0, 1]], dtype=torch.float32, device=device).expand(1, 4)), dim=0)
      
      C2W = torch.inverse(Rt)
      cam_center = C2W[:3, 3] * scale
      C2W[:3, 3] = cam_center
      Rt = torch.inverse(C2W)
      return Rt
</code></pre>
<p>R is given in the initialization of the model as a np.array with shape (3,3) and t as np.array with shape(3)<br>
Do you think that torch.inverse operation lead to an interruption in the backpropagation?<br>
I also printed the grad_fn of the result of this function world_view_trasform and i got &lt;TransposeBackward0 object at 0x7fb0884e3070&gt; which makes me think that is differentiable<br>
Another question, is it right to suppose that if a tensor have a non None grad_fn then it is the product of differentiable operations in any case?</p>

username: tom
cooked: <p>torch.inverse is decidedly differentiable.<br>
I didn’t immediately spot the problem in this or your last post, but the inplace operations look a bit tricky with the loop, I don’t know if that would work well (in the “really don’t know” not in the “that is not good but I’m polite” sense), maybe it would be worth while to try if creating a scaling matrix with all ones and scale in <code>[:3, 3]</code> and multiplying that to C2W works better. You could also see if you can use some autograd graph visualizer (my one is <a href="https://github.com/t-vi/pytorch-tvmisc/blob/master/visualize/bad_grad_viz.ipynb" rel="noopener nofollow ugc">very, very old</a> and I don’t know if I have published the more elaborate one from my course).</p>
<p>Best regards</p>
<p>Thomas</p>

username: Alessio_Bonacchi
cooked: <p>Where did you spot an inplace operation?<br>
Btw i simply removed the scale variable</p>
<pre><code>cam_center = C2W[:3, 3] 
</code></pre>
<p>but still do not work</p>

username: tom
cooked: <p><code>C2W[:3, 3] = ...</code> modifies C2W inplace. What happens if you remove that bit?</p>

username: Alessio_Bonacchi
cooked: <p>even if i remove that operation, the result is the same.<br>
Btw is there a method to do an operation like that in a safe way for backpropagation?<br>
EDIT: if i print C2W after computing that inplace operation i get:</p>
<pre><code>  tensor([[  ... ]], device='cuda:0', grad_fn=&lt;CopySlices&gt;)
</code></pre>

username: tom
cooked: <p>So the thing I’m a bit careful about is taking the values and then copying over them.<br>
In principle, modifying things in-place is not a problem unless the operation that calculated it (or something you calculated from the unmodified copy) wants to use it for the backward. In that case, using a clone and modifying that inplace instead works.</p>
<p>But are you sure the function is the problem? Can you provide a self-contained bit of code that shows the problem with dummy inputs?</p>
<p>Best regards</p>
<p>Thomas</p>

username: Alessio_Bonacchi
cooked: <p>Please check my topic here:</p><aside class="quote quote-modified" data-post="1" data-topic="192284">
  <div class="title">
    <div class="quote-controls"></div>
    <img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/alessio_bonacchi/48/62645_2.png" class="avatar">
    <a href="https://discuss.pytorch.org/t/torch-nn-parameters-not-updating-gradient-remains-none/192284">Torch.nn.parameters not updating, gradient remains None</a> <a class="badge-category__wrapper " href="/c/autograd/7"><span data-category-id="7" data-drop-close="true" class="badge-category " title="A category of posts relating to the autograd engine itself."><span class="badge-category__name">autograd</span></span></a>
  </div>
  <blockquote>
    Hi everyone! 
I have a problem with my code, basically i am trying to do a camera optimization neural network which have basically 2 parameters rotation and traslation, that needs to update in order to minimize the loss between the rendered image and the target image. I will post my code here: 


Definition of the model: the parameter for rotation matrix is stored as 3 different arrays each representing a row of the matrix, and then in the forward i stack them. This because i saw that declaring …
  </blockquote>
</aside>
<p>
here i have described pretty well my problem providing also the inputs that i send to the network.<br>
The main problem is that i am dealing with gaussian splatting repo:</p><aside class="onebox allowlistedgeneric" data-onebox-src="https://github.com/graphdeco-inria/gaussian-splatting">
  <header class="source">
      <img src="https://github.githubassets.com/favicons/favicon.svg" class="site-icon" width="32" height="32">

      <a href="https://github.com/graphdeco-inria/gaussian-splatting" target="_blank" rel="noopener nofollow ugc">GitHub</a>
  </header>

  <article class="onebox-body">
    <div class="aspect-image" style="--aspect-ratio:690/344;"><img src="https://opengraph.githubassets.com/ccc29f20871930994d1a5f3cfff3227079251285dcc2dfba968773be59a68d24/graphdeco-inria/gaussian-splatting" class="thumbnail" width="690" height="345"></div>

<h3><a href="https://github.com/graphdeco-inria/gaussian-splatting" target="_blank" rel="noopener nofollow ugc">GitHub - graphdeco-inria/gaussian-splatting: Original reference...</a></h3>

  <p>Original reference implementation of "3D Gaussian Splatting for Real-Time Radiance Field Rendering" - GitHub - graphdeco-inria/gaussian-splatting: Original reference implementation of &amp;qu...</p>


  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>
<p>
and the rendering functions are just taken from there, btw what i think is that the rendering part is identical to the one in the repository so i think that is not a problem.<br>
To be clear this is the complete forward method:</p>
<pre><code>def forward(self):

    R = torch.stack((self.r1, self.r2, self.r3))

    world_view_transform = getWorld2View_custom(self.device, R, self.traslation).transpose(0, 1).to(self.device)
    projection_matrix = getProjectionMatrix(znear=0.01, zfar=100.0, fovX=self.fovx, fovY=self.fovy).transpose(0,1).to(self.device)
    wvt = world_view_transform.unsqueeze(0)
    pm = projection_matrix.unsqueeze(0)
    full_proj_transform = torch.bmm(wvt, pm)
    fpt = full_proj_transform.squeeze(0)
    camera_center = torch.inverse(world_view_transform)
    cc = camera_center[3, :3]

    render_dict = render(self.fovx, self.fovy, 837, 472, wvt, fpt, cc, self.gaussian_splat, self.pipe, self.background)
    render_image = render_dict['render']

    
    lambda_dssim = 0.2
    Ll1 = l1_loss(render_image, self.image_ref)
    loss = (1.0 - lambda_dssim) * Ll1 + lambda_dssim * (1.0 - ssim(render_image, self.image_ref))
    
    return loss, render_image
</code></pre>
<p>What i mean is that from the render_dict creation onward the code is the same exposed in gaussian splatting repo , as you can see in the train.py file of such repository.<br>
Then there should be a problem with the part that ends with the creation of the camera center.<br>
I hope that i have been clear, thank you in advance</p>