Retaining gradients after comverting to numpy then Pytorch

Ali_Youssef: <p>Hello!</p>
<p>I had a question regarding retaining the gradients of tensors when converting to numpy arrays to do computations, for example in the code below, <code>kpts0</code> and<code> kpts1</code> are tensors which require gradients, unfortunately, the poselib library only accepts numpy arrays as inputs, so I detach the tenors, convert to numpy then convert the output to torch tensors with gradients set as true, but is it possible to convert to numpy arrays, pass to poselib and convert back to torch tensors while retaining the gradients?</p>
<p>Thank you!</p>
<pre><code class="lang-auto">
def estimate_pose(kpts0, kpts1, K0, K1, thresh, conf=0.99999):
    if len(kpts0) &lt; 5:
        return None
    
    kpts0 = kpts0.detach().cpu().numpy()
    kpts1 = kpts1.detach().cpu().numpy()
    K0 = K0.detach().cpu().numpy()
    K1 = K1.detach().cpu().numpy()

    fx_0, fy_0, cx_0, cy_0 = K0[0, 0], K0[1, 1], K0[0, 2], K0[1, 2]
    fx_1, fy_1, cx_1, cy_1 = K1[0, 0], K1[1, 1], K1[0, 2], K1[1, 2]
    
    camera_0 = {"model": "PINHOLE", "width": int(2 * cx_0), "height": int(2 * cy_0), "params": [fx_0, fy_0, cx_0, cy_0]}
    camera_1 = {"model": "PINHOLE", "width": int(2 * cx_1), "height": int(2 * cy_1), "params": [fx_1, fy_1, cx_1, cy_1]}

    M, info = poselib.estimate_relative_pose(kpts0, kpts1, camera_0, camera_1, {"max_epipolar_error": thresh, "success_prob": conf})
    R = M.R
    t = M.t

    R = torch.tensor(R, dtype=torch.float32, requires_grad=True).unsqueeze(0)
    t = torch.tensor(t, dtype=torch.float32, requires_grad=True).unsqueeze(0)

    P = torch.cat([R, t], dim=-1)
    return P
</code></pre>

ptrblck: <p>Yes, this is possible in custom <code>autograd.Function</code>s. You could use the same jumpy operation in its <code>forward</code>, but would also need to write the <code>backward</code> method for the numpy op.</p>

Ali_Youssef: <p>Thank you so much!</p>
<p>I tried following the pytorch documentation, and implemented the code below, I assume none of it is correct but I guess I am on the right track?</p>
<p>I had another question if you may, if I use the code in my initial post, then during backprogation, the gradients are computed until <code>R</code> and <code>t</code>, since the computational graph can not track any further than due to gradient detachement correct?</p>
<pre><code class="lang-auto">def estimate_pose(kpts0, kpts1, K0, K1, thresh, conf=0.99999):
    if len(kpts0) &lt; 5:
        return None

    fx_0, fy_0, cx_0, cy_0 = K0[0, 0], K0[1, 1], K0[0, 2], K0[1, 2]
    fx_1, fy_1, cx_1, cy_1 = K1[0, 0], K1[1, 1], K1[0, 2], K1[1, 2]
    
    camera_0 = {"model": "PINHOLE", "width": int(2 * cx_0), "height": int(2 * cy_0), "params": [fx_0, fy_0, cx_0, cy_0]}
    camera_1 = {"model": "PINHOLE", "width": int(2 * cx_1), "height": int(2 * cy_1), "params": [fx_1, fy_1, cx_1, cy_1]}

    M, info = poselib.estimate_relative_pose(kpts0, kpts1, camera_0, camera_1, {"max_epipolar_error": thresh, "success_prob": conf})
    R = torch.tensor(M.R, dtype=torch.float32).unsqueeze(0)
    t = torch.tensor(M.t, dtype=torch.float32).unsqueeze(0)
    P_est = torch.cat([R, t], dim=-1)
    return P_est


class PoseEstimation(torch.autograd.Function):
    @staticmethod
    def forward(ctx, mkpts_0, mkpts_1, K_0, K_1, P_GT=None, ordering="xy"):
        ctx.save_for_backward(mkpts_0, mkpts_1)
        mkpts_0 = mkpts_0.squeeze().detach().cpu().numpy()
        mkpts_1 = mkpts_1.squeeze().detach().cpu().numpy()
        K_0 = K_0.squeeze().cpu().numpy()
        K_1 = K_1.squeeze().cpu().numpy()
        P_est = estimate_pose(mkpts_0, mkpts_1, K_0, K_1, 0.5, 0.99999)
        return P_est
    
    @staticmethod
    def backward(ctx, grad_output):
        mkpts_0, mkpts_1 = ctx.saved_tensors
        return grad_output * mkpts_0, grad_output * mkpts_1, None, None, None, None

</code></pre>

ptrblck: <p>Yes, operations on <code>R</code> and <code>t</code> will be tracked again and gradients thus also computed up to their creation.</p>

Ali_Youssef: <p>Thank you very much!</p>

Ali_Youssef: <p>Hello! Sorry for the repetitive questions, but I managed to do the forward function correctly, however, during loss.backward(), everything freezes, for exmaple printing the gradients, everything keeps printing infitenly and freezes, I am not sure quite if it is due to the backward pass of the loss is incorrect?</p>
<pre><code class="lang-auto">self.pose_est = PoseEstimation.apply

P_est =  self.pose_est(mkpts_0,
                                        mkpts_1,
                                        batch["camera_intrinsic_matrix"],
                                        batch["camera_intrinsic_matrix"],
                                        0.5,
                                        0.99999)
                                        
                rot_loss, transl_loss = relative_pose_error(P_est,
                                                            batch["gt_relative_pose"],
                                                            train=True)
            loss += self.config["training"]["lambda_pose"]*(rot_loss + transl_loss)
            

        self.running_loss.append(loss.item())

        self.optimizer.zero_grad()

        loss.backward() &lt;------ everything freezes here

</code></pre>
<pre><code class="lang-auto">def estimate_pose(kpts0, kpts1, K0, K1, thresh, conf=0.99999):
    #if len(kpts0) &lt; 5:
    #    return None

    fx_0, fy_0, cx_0, cy_0 = K0[0, 0], K0[1, 1], K0[0, 2], K0[1, 2]
    fx_1, fy_1, cx_1, cy_1 = K1[0, 0], K1[1, 1], K1[0, 2], K1[1, 2]
    
    camera_0 = {"model": "PINHOLE", "width": int(2 * cx_0), "height": int(2 * cy_0), "params": [fx_0, fy_0, cx_0, cy_0]}
    camera_1 = {"model": "PINHOLE", "width": int(2 * cx_1), "height": int(2 * cy_1), "params": [fx_1, fy_1, cx_1, cy_1]}

    M, info = poselib.estimate_relative_pose(kpts0, kpts1, camera_0, camera_1, {"max_epipolar_error": thresh, "success_prob": conf})
    R = torch.tensor(M.R, dtype=torch.float32, requires_grad=True).reshape(3, 3).unsqueeze(0)
    t = torch.tensor(M.t, dtype=torch.float32, requires_grad=True).reshape(3, 1).unsqueeze(0)
    P_est = torch.cat([R, t], dim=-1)
    return P_est


class PoseEstimation(torch.autograd.Function):
    
    @staticmethod
    def forward(ctx, mkpts_0, mkpts_1, K_0, K_1, thresh, conf=0.99999):
        ctx.mkpts0, ctx.mkpts1 = mkpts_0, mkpts_1
        mkpts_0 = mkpts_0.squeeze().detach().cpu().numpy()
        mkpts_1 = mkpts_1.squeeze().detach().cpu().numpy()
        K_0 = K_0.squeeze().cpu().numpy()
        K_1 = K_1.squeeze().cpu().numpy()
        P_est = estimate_pose(mkpts_0, mkpts_1, K_0, K_1, thresh, conf)
        ctx.P_est = P_est
        return P_est
    
    @staticmethod
    def backward(ctx, grad_output):
        mkpts_0, mkpts_1, P_est = ctx.mkpts0, ctx.mkpts1, ctx.P_est
        grad_mkpts_0  = torch.autograd.grad(P_est, mkpts_0, grad_output, retain_graph=False)
        grad_mkpts_1  = torch.autograd.grad(P_est, mkpts_1, grad_output, retain_graph=False)
        return grad_mkpts_0, grad_mkpts_1, None, None, None, None 
</code></pre>

ptrblck: <aside class="quote no-group" data-username="Ali_Youssef" data-post="6" data-topic="194702">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ali_youssef/48/58359_2.png" class="avatar"> Ali_Youssef:</div>
<blockquote>
<p>for exmaple printing the gradients, everything keeps printing infitenly and freezes,</p>
</blockquote>
</aside>
<p>Does this mean the gradients are printed repeatedly? If so, your script might not be frozen but maybe stuck in a loop. I also don’t quite understand how <code>autograd.grad</code> should work on non-differentiable operations as I would expect to see the actual backwards operations in the <code>backward</code> method.<br>
If you need to store intermediate tensors you might also want to use <code>ctx.save_for_backward</code>.</p>

Ali_Youssef: <p>Yeah if I print the gradients in the backward pass, it keeps printing the same gradient repeatidly and everything freezes, even CTRL+C does not work, I have no idea why this is happening though. Printing everything up to loss.backward() works fine, then crashes during loss.backward(). Pretty certain it is mistake from my end on how I implemented the PoseEstimation backward pass, as everythign works fine if I do not include it.</p>
<p>Thank you again for your help! <img src="https://discuss.pytorch.org/images/emoji/apple/smiley.png?v=12" title=":smiley:" class="emoji" alt=":smiley:" loading="lazy" width="20" height="20"></p>
<pre><code class="lang-auto">P_est tensor([[[ 1.0000e+00, -4.1633e-17, -2.7756e-17,  2.7756e-17],
         [ 4.1633e-17,  1.0000e+00, -8.3267e-17,  1.0000e+00],
         [ 2.7756e-17,  8.3267e-17,  1.0000e+00,  2.7756e-17]]],
       grad_fn=&lt;PoseEstimationBackward&gt;)
rot_loss tensor(0.0010, grad_fn=&lt;MeanBackward0&gt;)
transl_loss tensor(1.6500, grad_fn=&lt;MeanBackward0&gt;)
</code></pre>

ptrblck: <p>I haven’t executed your code, but I speculate you might have created a ref cycle and might need to use <code>save_for_backward</code> as mentioned before and as seen in <a href="https://pytorch.org/tutorials/beginner/examples_autograd/two_layer_net_custom_function.html">this example</a>. Also, I still claim you need to write the actual backward operations as I would expect Autograd to fail since you are using numpy ops.</p>

Ali_Youssef: <p>Oh I understand now!, thank you very much!</p>