Forward Kinematics Using Pytorch

JeremySMorgan: <p>Hi There!</p>
<p>For my loss function, I need to calculate the forward kinematics of a robot’s end effector (the tip or hand of a robot) given joint angles returned by my network.</p>
<p>Basically: the network returns a vector describing the joint angle of each motor of the robot arm. I need to take those joint angles and calculate where the tip (end effector) of the robot is. The pose (position + rotation) of the end effector is then compared to a target pose in order to calculate a loss term.</p>
<pre><code class="lang-auto">for _, target_poses in dataset:
    joint_solutions = model(target_poses)
    poses_from_generated_solutions = forward_kinematics(joint_solutions)  # non-pytorch solver
    loss = fit_loss(poses_from_generated_solutions, target_poses)
</code></pre>
<p>The process of calculating the robots’ tip’s pose is called forward kinematics. I’m currently using a third party calculator that doesn’t use pytorch (Klampt: <a href="https://github.com/krishauser/Klampt" rel="nofollow noopener">https://github.com/krishauser/Klampt</a>).</p>
<p>The issue that i’m facing is that pytorch can’t calculate the gradient of the loss term w.r.t. the model parameters, because the computation chain is broken when the poses are calculated with the forwardkinematics calculator.</p>
<p>Questions:</p>
<ol>
<li>Is there anyway to get around this problem and still use the same setup (external non-pytorch forward kinematics solver)?</li>
<li>If using a non-pytorch forward kinematics solver isn’t possible, are there any pytorch forward kinematics solvers that I can use? (I’ve looked and haven’t found any myself)</li>
<li>Any other feedback here?</li>
</ol>
<p>I’ll write my own forward kinematics calulcator with pytorch if all else fails <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>
<p>Thanks so much!</p>
<ul>
<li>Jeremy</li>
</ul>

shartzog: <p>Hi, Jeremy!<br>
Let me preface this by saying I’m practically a noob with regard to pytorch in general, and less than that for robotics so feel free to ignore my thoughts… <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>
<p>My first thought: why are ML techniques required to infer the position of any part of the robot?  Isn’t this mathematically deterministic?</p>
<p>My second thought: granted my pytorch experience is limited, but I’m not sure why a custom function as a network layer <em>wouldn’t</em> work.  With that said, I’d need to see more of your code to give you solid feedback.</p>
<p>Good Luck!<br>
SEH</p>
<p><strong>EDIT</strong>: ahh, I think I get it… You’re using the net to give you the positions of your servos that will be required to meet each pose??</p>

JeremySMorgan: <p>Hey Sam!</p>
<p>’ You’re using the net to give you the positions of your servos that will be required to meet each pose’  - Yes exactly.</p>
<p>The network returns a vector of joint angles/ I.e. ‘positions of [my] servos’. What I need to do is convert (using pytorch) that joint angle vector into the position of the tip of the robot. That conversion is deterministic, it’s mostly trig and linear algebra/transformations.</p>
<p>My current approach is failing because the forward kinematics calculator doesn’t use pytorch, so the computation graph is broken, and the gradient w.r.t the loss can’t be backpropogated</p>

shartzog: <p>You’ve probably come across it already, but if not, you might find some inspiration from this post:</p><aside class="quote quote-modified" data-post="1" data-topic="29387">
  <div class="title">
    <div class="quote-controls"></div>
    <img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/jean_da_rolt/48/28290_2.png" class="avatar">
    <a href="https://discuss.pytorch.org/t/custom-loss-functions/29387">Custom loss functions</a> 
  </div>
  <blockquote>
    Hi, 
I’m implementing a custom loss function in Pytorch 0.4. Reading the docs and the forums, it seems that there are two ways to define a custom loss function: 

Extending Function and implementing forward and backward methods.
Extending Module and implementing only the forward method.

With that in mind, my questions are: 

Can I write a python function that takes my model outputs as inputs and use torch.* functions to compute my loss function (without extending Function or Module)? If not, wh…
  </blockquote>
</aside>


Cyril_Anderson: <p>So is your end goal here basically trying to train a neural network IK solver?</p>

joaoaccarvalho: <p>What about this library?<br>
<a href="https://github.com/UM-ARM-Lab/pytorch_kinematics" rel="noopener nofollow ugc">https://github.com/UM-ARM-Lab/pytorch_kinematics</a></p>

JeremySMorgan: <p>that’s pretty much what I was looking for! I ended up writing my own though: <a href="https://github.com/jstmn/jkinpylib" class="inline-onebox" rel="noopener nofollow ugc">GitHub - jstmn/jkinpylib: Jeremy's Kinematics (Python) Library</a></p>