Rodrigues Network for Learning Robot Actions

## Abstract

Abstract Understanding and predicting articulated actions is important in robot learning.
However, common architectures such as MLPs and Transformers lack inductive biases that reflect the underlying kinematic structure of articulated systems.
To this end, we propose the Neural Rodrigues Operator , a learnable generalization of the classical forward kinematics operation, designed to inject kinematics-aware inductive bias into neural computation.
Building on this operator, we design the Rodrigues Network (RodriNet) , a novel neural architecture specialized for processing actions.
We evaluate the expressivity of our network on two synthetic tasks on kinematic and motion prediction, showing significant improvements compared to standard backbones.
We further demonstrate its effectiveness in two realistic applications: (i) imitation learning on robotic benchmarks with the Diffusion Policy, and (ii) single-image 3D hand reconstruction.
Our results suggest that integrating structured kinematic priors into the network architecture improves action learning in various domains.

## 1 Introduction

We study the problem of understanding and predicting the actions of articulated actors.
Articulated actors (e.g. robots [1] and animated characters [40, 31]) are systems that use multiple rotational joints to connect moving links.
Their actions, including poses, motions, or control commands, are usually represented as values associated with all joints.
Learning with articulated actors usually involves predicting their actions while processing diverse sensory inputs.
Such a problem lies in a wide spectrum of intelligent systems, from whole-body controllers [34, 27], to dexterous grasp detectors [18, 53, 50, 56], to motion retargetting networks [2, 20].

To learn the actions of articulated actors, we seek a neural network architecture that can both understand robot structures and enforce structural constraints in its predictions. However, most existing network architectures, such as MLPs and Transformers [48], treat actions as unstructured tokens, ignoring the kinematic relationships that govern joint interactions.
A few prior works have incorporated structural priors like robot link connectivity, using graph convolutions [54, 16, 2] or masked attention [41].
Yet, the design of network architectures that fully capture and enforce the articulated kinematic structure of these actors remains underexplored.
This leads us to our central question: Can we design a neural network for action learning that naturally integrates articulated kinematics as an inductive bias in its architecture?

To answer this question, we reflect on how CNNs [26] led to a deep learning framework that achieved remarkable success in computer vision by generalizing classical convolution filters from image processing [6]. Convolution filters naturally respect image structure through translation equivariance and spatial locality, but when used with handcrafted features, they were limited to low-level image processing tasks. Building on this foundation, CNNs extend these filters into neural operators with optimizable weights, non-linearities, and high-dimensional feature channels. This results in a deep learning framework capable of learning high-level semantic features from images while preserving the key structural properties of classical image filters.

In a similar vein, we identify Rodrigues’ rotation formula as the basic operator in articulated forward kinematics and generalize it into a learnable form, which we call the Neural Rodrigues Operator.
Concretely, we separate the entries in the Rodrigues’ rotation formula into state-dependent parameters conditioned on the joint angles, and state-independent coefficients that only rely on the articulated structure. We then extend it into a neural operator by treating the state-dependent parameters as input features, and relaxing the state-independent coefficients into optimizable weights.
We further extend it into a multi-channel operator that applies to higher-dimensional features instead of 1D joint angles.

With our neural operator as the key component, we construct a complete network architecture, named the Rodrigues Network (RodriNet), for encoding, understanding, and synthesizing articulated actions.
It comprises three key components: a Rodrigues Layer for passing information from joints to links, built up our neural operator; a Joint Layer for passing information from links to joints; and finally, a Self-Attention Layer for global information exchange.
We also introduce a global token for processing other task variables, such as perception inputs.
Although starting with an operator in classical control theory, we end up with a deep neural network with modern designs while maintaining the structural bias in articulated kinematics, as illustrated in Figure [1](https://arxiv.org/html/2506.02618v1#S0.F1).

We evaluate our approach through three sets of experiments:
First, we demonstrate the strong expressivity of our network in representing forward kinematics and motions through toy experiments.
Second, we showcase our effectiveness in realistic robot-learning scenarios with imitation learning on 5 robot manipulation tasks, where our network achieves state-of-the-art results.
Finally, we also show our network achieving state-of-the-art results in human hand pose estimation from images, where the articulated actor is no longer a robot, highlighting its applications in computer vision and graphics.

## 2 Related work: articulation-aware robot learning

Articulated robots can be naturally modeled as graphs, making graph convolution [4, 36] a popular approach for processing articulated data.
This has been widely applied in character animation for tasks like skeleton-based action recognition [54, 13, 45, 46, 12, 5], pose estimation [16, 15, 55, 8], and motion retargeting [2].
Graph convolution effectively captures link connectivity and spatial locality, but it does not explicitly incorporate articulated kinematics, which is an essential aspect of articulated action understanding.
In contrast, our work introduces a novel operator derived from forward kinematics, providing networks with kinematics-informed inductive bias.

Transformer-based architectures [48] are widely used in policy learning [3, 57, 9, 39, 43], and recent work has explored incorporating structural bias via graph-aware positional embeddings [22] or masked attention [41].
However, these modifications do not fundamentally adapt the self-attention mechanism to suit kinematics.
We instead use standard self-attention layers for network capacity, but rely on our kinematics-inspired operator for inductive bias.

Forward kinematics provides a deterministic mapping from joint space to Cartesian space, but integrating it into neural networks remains non-trivial.
Prior methods have inserted analytical forward kinematics as a differentiable layer in neural networks [49] to help them reason about the Cartesian results of the robot action.
While this introduces kinematic awareness, it constrains the model’s flexibility.
Other methods apply Cartesian-space loss functions after computing forward kinematics on network outputs [38, 23, 30], but these approaches do not alter the network architecture and are thus orthogonal to our focus.
In contrast, our method derive a learnable operator from forward kinematics, thereby making the network kinematics-aware while maintaining the flexibility to learn high-level features.

## 3 Neural Rodrigues Operator

In this section, we derive the Neural Rodrigues Operator by making the Rodrigues’ rotation formula learnable and more generalized.

### 3.1 Background

#### Articulated robots

In this paper’s context, an articulated robot has a loop-free kinematic tree structure, with $D+1$ rigid links $L_{0},\cdots,L_{D}$ connected by $D$ one-DoF revolute joints $J_{1},\cdots,J_{D}$.
This definition encompasses a wide range of platforms such as robotic arms, dexterous hands, quadrupeds, and humanoids.
The kinematic tree has a root, denoted as the base link $L_{0}$, which can be either fixed or free-floating.
All links and joints have their local 3D frames.
Each joint $J_{j}$ connects a parent link $L_{\text{p}_{j}}$ with a child link $L_{\text{c}_{j}}$, where a rotation axis $\hat{\boldsymbol{\omega}}_{j}\in\mathbb{R}^{3}$ in the joint frame defines the rotational motion of the child link relative to the parent link, as well as a fixed transformation from the parent link’s frame to the joint’s frame $\mathbf{T}_{j}\in\mathrm{SE}(3)$.

Given the kinematic structure, the configuration of the robot is then determined by the joint angles $\boldsymbol{\theta}=[\theta_{1},\cdots,\theta_{D}]\in\mathbb{R}^{D}$, as well as the root pose $\mathbf{P}_{0}\in\mathrm{SE}(3)$ if its base link is free-floating (not required for fixed-base robots).
In most classical control systems, control commands are sent to the joints, specifying their joint angles, velocities, or torques, depending on the control modes.

#### Forward kinematics

To obtain the position and orientation of all links, including the end-effector, given a set of joint angles, we apply forward kinematics. Below, we briefly outline its computation using homogeneous coordinates.
We represent the pose of link $L_{i}$ as a homogeneous matrix $\mathbf{P}_{i}$ describing the transformation from the world frame to the link’s frame:

$$ $\mathbf{P}_{i}=\begin{bmatrix}\mathbf{R}_{i}&\mathbf{t}_{i}\\ \mathbf{0}_{1\times 3}&1\end{bmatrix}\in\mathbb{R}^{4\times 4}$ (1) $$

where $\mathbf{t}_{i}\in\mathbb{R}^{3\times 1}$ is the position and $\mathbf{R}_{i}\in\mathbb{R}^{3\times 3}$ is the orientation.
Given the pose of the base link $\mathbf{P}_{0}$, the poses of the descendant links can be computed recursively from parents to children.
More concretely, at joint $J_{j}$, using $\mathbf{T}_{j},\hat{\boldsymbol{\omega}}_{j},\theta_{j}$, we can compute the child-link pose $\mathbf{P}_{\text{c}_{j}}$ from the parent-link pose $\mathbf{P}_{\text{p}_{j}}$ through two transformations:
(i) a fixed coordinate change $\mathbf{T}_{j}$ from the parent frame to the joint frame;
(ii) a dynamic rotation $\mathbf{R}(\hat{\boldsymbol{\omega}}_{j},\theta_{j})\in\mathbb{R}^{3\times 3}$ around axis $\hat{\boldsymbol{\omega}}_{j}$ of angle $\theta_{j}$ in the joint frame.
Putting these together, the parent-to-children pose transformation is:

$$ $\begin{bmatrix}\mathbf{R}_{\text{c}_{j}}&\mathbf{t}_{\text{c}_{j}}\\ \mathbf{0}_{1\times 3}&1\end{bmatrix}=\begin{bmatrix}\mathbf{R}_{\text{p}_{j}} &\mathbf{t}_{\text{p}_{j}}\\ \mathbf{0}_{1\times 3}&1\end{bmatrix}\mathbf{T}_{j}\begin{bmatrix}\mathbf{R}( \hat{\boldsymbol{\omega}}_{j},\theta_{j})&\mathbf{0}_{3\times 1}\\ \mathbf{0}_{1\times 3}&1\end{bmatrix}.$ (2) $$

Here, transformation (i) only depends on the robot’s articulated structure and thus is fixed, and transformation (ii) is state-dependent with variable $\theta_{j}$. Therefore, we can abbreviate Equation [2](https://arxiv.org/html/2506.02618v1#S3.E2) as $\mathbf{P}_{\text{c}_{j}}=\mathbf{P}_{\text{p}_{j}}(\mathbf{T}_{j}\tilde{
\mathbf{R}}(\hat{\boldsymbol{\omega}}_{j},\theta_{j}))$, where $\tilde{\mathbf{R}}(\hat{\boldsymbol{\omega}}_{j},\theta_{j})\in\mathbb{R}^{4
\times 4}$ is the homogeneous matrix of the rotation. Essentially, forward kinematics is a hierarchical composition of fixed coordinate transformations and dynamic rotations in the axis-angle representation with variables $\theta_{1},\cdots,\theta_{D}$.

#### Rodrigues’ rotation formula

The Rodrigues’ rotation formula tells how to compute the rotation matrix $\mathbf{R}(\hat{\boldsymbol{\omega}},\theta)\in\mathbb{R}^{3\times 3}$ from the axis-angle representation:

$$ $\mathbf{R}(\hat{\boldsymbol{\omega}},\theta)=\mathbf{I}_{3}+\sin\theta[\hat{ \boldsymbol{\omega}}]+(1-\cos\theta)[\hat{\boldsymbol{\omega}}]$ (3) $$

where $[\hat{\boldsymbol{\omega}}]\in\mathbb{R}^{3\times 3}$ is the skew-symmetric cross-product matrix of the rotation axis $\hat{\boldsymbol{\omega}}=(\hat{\omega}_{x},\hat{\omega}_{y},\hat{\omega}_{z})$.

### 3.2 Neural Rodrigues Operator with learnable parameters

Observe from Equation [3](https://arxiv.org/html/2506.02618v1#S3.E3) that every entry in $\mathbf{R}(\hat{\boldsymbol{\omega}},\theta)$ is essentially a linear combination of $1$, $\cos\theta$, and $\sin\theta$, with fixed coefficients determined by the rotation axis $\hat{\boldsymbol{\omega}}$.
Therefore, every entry in $\mathbf{P}(\hat{\boldsymbol{\omega}}_{j},\theta_{j})$ and thereby $\mathbf{T}_{j}\mathbf{P}(\hat{\boldsymbol{\omega}}_{j},\theta_{j})$ are linear combinations of $1$, $\cos\theta_{j}$, and $\sin\theta_{j}$, with constant coefficients defined by the state-independent parameters $\mathbf{T}_{j},\hat{\boldsymbol{\omega}}_{j}$ of joint $J_{j}$.
Thus, we can re-parameterize Equation [2](https://arxiv.org/html/2506.02618v1#S3.E2) as:

$$ $\mathbf{P}_{\text{c}_{j}}=\mathbf{P}_{\text{p}_{j}}(\mathbf{A}_{j}+\mathbf{B}_ {j}\cos\theta_{j}+\mathbf{C}_{j}\sin\theta_{j})$ (4) $$

where $\mathbf{A}_{j},\mathbf{B}_{j},\mathbf{C}_{j}\in\mathbb{R}^{4\times 4}$ are coefficient matrices that only depend on the robot’s articulated structure.
Based on this, we construct our Neural Rodrigues Operator by replacing these fixed coefficients with learnable weights $W^{\text{bias}},W^{\text{cos}},W^{\text{sin}}\in\mathbb{R}^{4\times 4}$, resulting in:

$$ $F^{\text{out}}=F^{\text{in}}(W^{\text{bias}}+W^{\text{cos}}\cos\Theta+W^{\text {sin}}\sin\Theta)$ (5) $$

where $\Theta\in\mathbb{R}$ is a scaler feature of joint, $F^{\text{in}}\in\mathbb{R}^{4\times 4}$ and $F^{\text{out}}\in\mathbb{R}^{4\times 4}$ are the input and output link features, corresponding to the parent and child links.

This operator generalizes the classical transformation in Equation [2](https://arxiv.org/html/2506.02618v1#S3.E2). When applied recursively, it defines a hierarchical message passing along the robot’s kinematic tree.
On one hand, in the special case when $\Theta=\theta_{j}$ and all parameters in Equation [5](https://arxiv.org/html/2506.02618v1#S3.E5) are identical to those in Equation [2](https://arxiv.org/html/2506.02618v1#S3.E2), this learnable operator degenerate to the forward kinematics of the robot.
On the other hand, this generalization also provides a more expressive function space that can be potentially used to encode richer, high-level features beyond joint angles and link poses.
These properties enable the operator to combine the inductive bias of kinematic awareness with representational flexibility, making it well-suited for robot learning tasks involving complex actions and motions governed by articulated structures.

### 3.3 Multi-Channel Neural Rodrigues Operator

Directly derived from the Rodrigues’ rotation formula, the neural operator defined in Equation [5](https://arxiv.org/html/2506.02618v1#S3.E5) only applies to 1D joint features and $4\times 4$ link features.
We now extend it to a multi-channel operator to learn higher-dimensional features.
Specifically, we extend the link features from $4\times 4$ matrices to $F^{\text{in}}\in\mathbb{R}^{C_{L}\times 4\times 4},F^{\text{out}}\in\mathbb{R}
^{C_{L}^{\prime}\times 4\times 4}$, and similarily, the joint features $\Theta\in\mathbb{R}^{C_{J}}$.
Here $C_{L},C_{L}^{\prime},C_{J}$ are the channel numbers.
Accordingly, the learnable weights are extended to $W^{\text{bias}}\in\mathbb{R}^{C_{L}\times C_{L}^{\prime}\times 4\times 4}$ and $W^{\text{cos}},W^{\text{sin}}\in\mathbb{R}^{C_{L}\times C_{L}^{\prime}\times C
_{J}\times 4\times 4}$, giving a multi-channel extension of Equation [5](https://arxiv.org/html/2506.02618v1#S3.E5):

$$ $\displaystyle U[i,j]$ $\displaystyle=W^{\text{bias}}[i,j]+\sum_{c=1}^{C_{J}}\left(W^{\text{cos}}[i,j, c]\cos(\Theta[c])+W^{\text{sin}}[i,j,c]\sin(\Theta[c])\right)$ (6) $\displaystyle F^{\text{out}}[j]$ $\displaystyle=\sum_{i=1}^{C_{L}}F^{\text{in}}[i]U[i,j],\quad\text{where \ \ }U [i,j]\in\mathbb{R}^{4\times 4}$ (7) $$

For better expressivity of the network, we further learn another conjugate $\bar{U}[i,j]$ following Equation [6](https://arxiv.org/html/2506.02618v1#S3.E6) and extend Equation [7](https://arxiv.org/html/2506.02618v1#S3.E7) to include both left and right multiplications:

$$ $F^{\text{out}}[j]=\sum_{i=1}^{C_{L}}\left(F^{\text{in}}[i]U[i,j]+\bar{U}[i,j]F ^{\text{in}}[i]\right).$ (8) $$

Equation [6](https://arxiv.org/html/2506.02618v1#S3.E6), [8](https://arxiv.org/html/2506.02618v1#S3.E8) together define our Multi-Channel Neural Rodrigues Operator with learnable parameters $W^{*}=\{W^{\text{bias}},W^{\text{sin}},W^{\text{cos}},(\bar{W})^{\text{bias}},
(\bar{W})^{\text{sin}},(\bar{W})^{\text{cos}}\}$, where $W,\bar{W}$ are the parameters for learning $U,\bar{U}$ respectively. We abbreviate Equation [8](https://arxiv.org/html/2506.02618v1#S3.E8) as $F^{\text{out}}=\text{Rodrigues}~{}\left(F^{\text{in}},W^{*},\Theta\right)$ and will use it in the following sections.

## 4 Rodrigues Network

Figure: Figure 2: Rodrigues Block. It comprises three components: a Rodrigues Layer for passing information from joints to links, constructed with our Multi-Channel Neural Rodrigues Operator; a Joint Layer for passing information from links to joints; and a Self-Attention Layer for global information exchange with all the links and the global token.
Refer to caption: x2.png

Given the Rodrigues Operator, we are interested in building a complete neural network that leverages this operator while being versatile and expressive. To achieve that, we propose a basic building block called Rodrigues Block (Figure [2](https://arxiv.org/html/2506.02618v1#S4.F2)), which comprises the following three components:
(1) a Rodrigues Layer for passing information from joints to links, constructed with our Multi-Channel Neural Rodrigues Operator (Section [4.1](https://arxiv.org/html/2506.02618v1#S4.SS1)); (2) a Joint Layer for passing information from links to joints (Section [4.2](https://arxiv.org/html/2506.02618v1#S4.SS2)); (3) and a Self-Attention Layer (Section [4.3](https://arxiv.org/html/2506.02618v1#S4.SS3)) for global information exchange.

### 4.1 Rodrigues Layer

With the Multi-Channel Rodrigues Operator being the core component operating on each joint, we construct the Rodrigues Layer, which extends this operator to the full tree structure of an articulated robot.
We define the feature graph of an articulated robot as the collection of all link and joint features: $\{F_{l}\}_{j=0}^{D}$ for links and $\{\Theta_{j}\}_{j=1}^{D}$ for joints.
The Rodrigues Layer maintains a set of Rodrigues Kernels $\{W_{j}^{*}\}_{j=1}^{D}$, one for each joint, and computes the output link features $\{F_{l}^{\text{out}}\}$ from the input link features $\{F_{l}^{\text{in}}\}$ and joint features $\{\Theta_{j}^{\text{in}}\}$.
For each joint $J_{j}$, we retrieve its Rodrigues Kernels $W_{j}^{*}$, its joint feature $\Theta_{j}^{\text{in}}$, and the feature of its parent link $F^{\text{in}}_{\text{p}_{j}}$.
We then apply the Multi-Channel Neural Rodrigues Operator (Equation [8](https://arxiv.org/html/2506.02618v1#S3.E8)) to compute the transformed feature:

$$ $F^{\text{trans}}_{j}=\text{Rodrigues}~{}\left(F^{\text{in}}_{\text{p}_{j}},W_{ j}^{*},\Theta_{j}^{\text{in}}\right)$ (9) $$

This transformed feature is passed to the joint’s child link $L_{\text{c}_{j}}$, added to the child’s input feature $F^{\text{in}}_{\text{c}_{j}}$, and normalized:

$$ $F^{\text{out}}_{\text{c}_{j}}=\text{LayerNorm}~{}\left(F^{\text{in}}_{\text{c} _{j}}+F^{\text{trans}}_{j}\right)$ (10) $$

For the root link without a parent, its output feature is defined as the layer-normalized version of its input.
This formulation enables hierarchical, kinematics-aware feature propagation across the robot’s kinematic tree.

### 4.2 Joint Layer

While the Rodrigues Layer updates the link features, we still need a learnable mechanism to update the joint features.
The Joint Layer computes the output joint features $\{\Theta_{j}^{\text{out}}\}$ from the input joint features $\{\Theta_{j}^{\text{in}}\}$ and link features $\{F_{l}^{\text{in}}\}$.
For each joint $J_{j}$, we retrieve the feature of its child link $F^{\text{in}}_{\text{c}_{j}}$, apply a joint-specific linear transformation, and add it to the joint’s existing feature:

$$ $\Theta^{\text{out}}_{j}=\text{Linear}_{j}~{}\left(\text{Flatten}~{}\left(F^{ \text{in}}_{\text{c}_{j}}\right)\right)+\Theta^{\text{in}}_{j}$ (11) $$

The transformations $\text{Linear}_{j}:\mathbb{R}^{C_{L}\times 4\times 4}\to\mathbb{R}^{C_{J}}$ are independently learned for each joint, allowing the model to capture joint-specific information.
This operator enables the propagation of information from links to joints while preserving the locality of the robot’s articulation structure.

### 4.3 Other components and overall architecture

#### Self-attention layer

While the Rodrigues Layer and Joint Layer leverage the spatial locality inherent in articulated structures, they restrict information flow to consecutive links and joints. To enable direct communication across all links, we incorporate a Self-attention Layer. In this layer, each link feature is first projected into a token using a linear transformation. These tokens then interact through multi-head self-attention, allowing the model to aggregate information from all links regardless of their spatial distance. The attended tokens are subsequently projected back to the link feature space, followed by residual addition and layer normalization.

#### Global token

To capture and utilize global context, we optionally introduce a global token $G$. This token represents a learnable global feature and participates in self-attention alongside the link tokens. It is processed through its own projection, residual, and normalization steps, and joins the link tokens during multi-head self-attention. The global token enables the network to store and propagate task-wide information that is not tied to any specific joint or link. We optionally enable the global token in tasks that require predicting global outputs, such as base link pose estimation.

#### Overall architecture

We combine the above three components into a unified module called the Rodrigues Block. Each Rodrigues Block takes as input the link features, joint features, and an optional global token. It sequentially applies the Rodrigues Layer, Joint Layer, and Self-attention Layer to produce updated link features, joint features, and a global token. By stacking multiple such blocks, we construct the full Rodrigues Network (RodriNet), enabling deep, hierarchical reasoning over articulated structures. The overall architecture is drawn in Figure [2](https://arxiv.org/html/2506.02618v1#S4.F2).

## 5 Experiments

We evaluate our Rodrigues Network on a set of different tasks, ranging from forward kinematics and motion prediction (Section [5.1](https://arxiv.org/html/2506.02618v1#S5.SS1)), to imaging learning in robotics (Section [5.2](https://arxiv.org/html/2506.02618v1#S5.SS2)), to hand pose estimation for vision and graphics (Section [5.3](https://arxiv.org/html/2506.02618v1#S5.SS3)).
The experiments focus on two main questions on action learning:
(i) Does the structural prior of articulated kinematics make the Rodrigues Network better at representing robot actions?
(ii) Can this inductive bias improve the understanding and prediction of robot actions in realistic task scenarios?

### 5.1 Toy experiments on kinematics and motion

We begin by studying whether our neural operators help networks better represent the motions and actions of articulated actors with two synthetic tasks: forward kinematics fitting and (Cartesian-space) motion prediction.
These synthetic tasks provide a clean environment to directly evaluate network expressivity without other factors.

Figure: (a) MSE vs. backbones.
Refer to caption: x3.png

Figure: Figure 4: Visualization of forward kinematics prediction on an example configuration. Errors are plotted on each link with color scales, with darker colors indicating larger errors. MLP learns better positions but much worse orientations than GCN, leading to more aligned links but darker colors.
Refer to caption: x5.png

#### Forward kinematics fitting

As discussed in Section [3.1](https://arxiv.org/html/2506.02618v1#S3.SS1), forward kinematics maps the configuration of an articulated robot (including the root pose and joint angles) to the pose of each link.
To effectively control a robot, a neural network should, at a minimum, be capable of learning this mapping.
To evaluate whether the Neural Rodrigues Operator possesses strong representational capacity for kinematic modeling, we construct a specialized Rodrigues Network consisting solely of Rodrigues Layers and compare it to other neural backbones in their ability to fit the forward kinematics of a single robot.
We conduct this experiment on the LEAP Hand [42], a free-floating dexterous robotic hand with 16 joints and 17 links.
The network takes as input the configuration $(T,R,\theta)$ and predicts the pose matrices for all 17 links. For model hyperparameters and training settings, please refer to the supplementary material.

As shown in Figure [3(a)](https://arxiv.org/html/2506.02618v1#S5.F3.sf1), the Rodrigues Network achieves significantly lower prediction error than competing architectures, indicating superior precision in modeling forward kinematics.
Moreover, its training loss decreases much faster (Figure [3(b)](https://arxiv.org/html/2506.02618v1#S5.F3.sf2)), demonstrating better data efficiency.
We further visualize the networks’ predictions on a single robot configuration in Figure [4](https://arxiv.org/html/2506.02618v1#S5.F4).
Notably, MLP and GCN baselines produce visible artifacts, and all four baseline methods accumulate substantial error near the fingertips.
These findings suggest that fitting forward kinematics is not a trivial task; it requires modeling a structured function with spatial and hierarchical dependencies.
In contrast, our network closely matches the ground truth across all links, with minimal error, indicating that it effectively captures the underlying structure of the kinematic mapping.
We attribute this performance to the inductive bias introduced by the Neural Rodrigues Operator.
Its structural prior equips the network with an inherent advantage in learning kinematic functions.

#### Motion prediction in Cartesian space

**Table 1: Motion prediction in Cartesian space with trainset size $=10^{5}$.**
| Backbone | $\text{Error}_{T}$ (mm) | $\text{Error}_{R}$ (^∘) | $\text{Error}_{\theta}$ (^∘) | MSE ($1\mathrm{e}{-6}$) | Train MSE ($1\mathrm{e}{-6}$) |
| --- | --- | --- | --- | --- | --- |
| MLP | 3.49 $\pm$0.05 | 0.46 $\pm$0.05 | 0.17 $\pm$0.00 | 22.52 $\pm$0.95 | 12.47 $\pm$0.73 |
| GCN | 3.55 $\pm$0.05 | 0.48 $\pm$0.05 | 0.17 $\pm$0.01 | 18.52 $\pm$1.74 | 13.68 $\pm$1.87 |
| BoT | 2.92 $\pm$0.04 | 0.46 $\pm$0.04 | 0.15 $\pm$0.01 | 15.72 $\pm$1.21 | 13.04 $\pm$1.41 |
| Transformer | 2.89 $\pm$0.06 | 0.41 $\pm$0.06 | 0.14 $\pm$0.01 | 12.86 $\pm$1.25 | 10.50 $\pm$1.21 |
| Rodrigues | 1.21 $\pm$0.04 | 0.16 $\pm$0.04 | 0.06 $\pm$0.00 | 2.56 $\pm$0.39 | 1.93 $\pm$0.34 |

Figure: (a) Trajectory visualization. We visualize the trajectories of the end-point (marked in red) predicted by each model from the top-down view, interpolated with B-spline curves.
Refer to caption: x6.png

Although fitting forward kinematics highlights the benefits of incorporating kinematics-inspired inductive bias, the task itself has limited practical value.
In real-world robot learning scenarios, neural backbones typically process observations in 3D Cartesian space (e.g., point clouds) and output control commands as target joint angles.
To evaluate a network’s ability to bridge these modalities, we propose a Cartesian motion prediction task that challenges the model to reason about structured motion in Cartesian space while expressing its predictions in joint angle space.
This task uses a 6-DoF UR5 robotic arm.
We begin by randomly sampling two end-effector poses in 3D space and interpolating between them (linearly for translation and spherically for rotation) to generate a 16-frame Cartesian trajectory.
Using inverse kinematics, we convert this trajectory into corresponding joint configurations. The network is given the first 8 frames of joint angles and tasked with predicting the remaining 8, also in joint space.
For details on training and model parameter count control, please refer to the supplementary.

As shown in Table [1](https://arxiv.org/html/2506.02618v1#S5.T1), the Rodrigues Network achieves the lowest training loss and test errors when trained on a pre-collected dataset of $10^{5}$ trajectories.
Notably, its test MSE is lower than the train MSE of all baseline models, indicating that the Rodrigues Network not only fits the data more effectively but also generalizes better without overfitting.
Figure [5(b)](https://arxiv.org/html/2506.02618v1#S5.F5.sf2) further shows that our model consistently outperforms baselines across different training set sizes.
In Figure [5(a)](https://arxiv.org/html/2506.02618v1#S5.F5.sf1), we visualize the end-effector trajectories predicted by each network for a single example; our model’s trajectory aligns most closely with the ground truth.
These results confirm that the Rodrigues Network is more effective at bridging Cartesian space and joint angle space than conventional architectures.

### 5.2 Robotic manipulation with imitation learning

**Table 2: Baseline comparisons on the imitation learning benchmark.**
| Method | PushCube | PickCube | StackCube | PegInsertionSide | PlugCharger | Average |
| --- | --- | --- | --- | --- | --- | --- |
| Transformer-DP | 0.98 $\pm$0.02 | 0.63 $\pm$0.05 | 0.38 $\pm$0.02 | 0.18 $\pm$0.05 | 0.04 $\pm$0.02 | 0.44 |
| UNet-DP | 1.00 $\pm$0.00 | 0.85 $\pm$0.03 | 0.37 $\pm$0.04 | 0.56 $\pm$0.06 | 0.13 $\pm$0.06 | 0.58 |
| Rodrigues-DP | 1.00 $\pm$0.00 | 0.94 $\pm$0.02 | 0.44 $\pm$0.05 | 0.58 $\pm$0.04 | 0.10 $\pm$0.02 | 0.61 |

Next, we evaluate whether our method benefits realistic robotic applications. We integrate the Rodrigues Network as a backbone into the Diffusion Policy [14], one of the state-of-the-art imitation learning frameworks, and test on a manipulation benchmark in simulation.

Benchmark We construct a suite of five manipulation tasks from ManiSkill [35] using a 7-DoF Franka arm with a 1-DoF Panda gripper, simulated in SAPIEN [52]. For each task, we collect 100–500 demonstration trajectories using motion planning, with each trajectory spanning 200 steps. During training, the network receives proprioception and object state as input and outputs relative joint offsets used for PD control. Performance is measured by running 100 evaluation rollouts in simulation, and all models are trained with 5 random seeds to report the mean and standard deviation of success rates.

Methods We adopt Diffusion Policy [14] as our learning framework, using a 2-frame observation history and predicting 16 future steps, of which 8 are executed during deployment. To isolate the impact of the neural backbone, we keep the outer framework and all other components fixed, modifying only the denoising network. Baseline architectures include the U-Net and Transformer designs from the original Diffusion Policy paper. Our method replaces these with the Rodrigues Network, which takes the current observation, denoising timestep, and a noisy action as inputs and predicts the corresponding action noise. The gripper output is handled via the global token. All models are tuned to have approximately 17 million parameters for a fair comparison. Implementation details are provided in the supplementary material.

Results and analysis As shown in Table [2](https://arxiv.org/html/2506.02618v1#S5.T2), Diffusion Policy [14] with the Rodrigues Network backbone achieves overall state-of-the-art performance, demonstrating that our kinematics-inspired inductive bias improves imitation learning in realistic robotic tasks. In particular, we observe substantial gains in PickCube and StackCube, and comparable or slightly better performance in PushCube, PegInsertionSide, and PlugCharger. These results suggest that the benefits of the Rodrigues Decoder are task-dependent. PushCube is relatively simple, leading all backbones to achieve near-perfect success. In contrast, PegInsertionSide and PlugCharger involve complex contact dynamics, such as inserting a peg into a hole, that likely require tactile or force feedback, making the neural backbone less of a limiting factor. In summary, enhancing the backbone architecture with our kinematics-inspired inductive bias can yield significant gains when the network is the performance bottleneck.

### 5.3 3D hand reconstruction

**Table 3: Baseline comparisons on the FreiHAND dataset [59]. We use the standard protocol and report metrics on 3D joint and 3D mesh accuracy. PA-MPVPE and PA-MPJPE numbers are in mm.**
| Method | PA-MPJPE $\downarrow$ | PA-MPVPE $\downarrow$ | F@5 $\uparrow$ | F@15 $\uparrow$ |
| --- | --- | --- | --- | --- |
| I2L-MeshNet [32] | 7.4 | 7.6 | 0.681 | 0.973 |
| Pose2Mesh [15] | 7.7 | 7.8 | 0.674 | 0.969 |
| I2UV-HandNet [10] | 6.7 | 6.9 | 0.707 | 0.977 |
| METRO [28] | 6.5 | 6.3 | 0.731 | 0.984 |
| Tang et al. [47] | 6.7 | 6.7 | 0.724 | 0.981 |
| Mesh Graphormer [29] | 5.9 | 6.0 | 0.764 | 0.986 |
| MobRecon [11] | 5.7 | 5.8 | 0.784 | 0.986 |
| AMVUR [24] | 6.2 | 6.1 | 0.767 | 0.987 |
| HaMeR [37] | 6.0 | 5.7 | 0.785 | 0.990 |
| HaMeR (Reproduced) [37] | 6.2 | 5.9 | 0.774 | 0.989 |
| Ours | 5.9 | 5.6 | 0.793 | 0.991 |

Finally, we demonstrate that our Rodrigues Network is not limited to robot learning but also applies to the animatable models in computer vision and graphics, where articulated actors commonly exist, such as human-like characters and hands.
Specifically, we apply our method to 3D hand reconstruction from single-view RGB images, which involves predicting the rotations and positions of hand joints based on the kinematic structure defined by the MANO [40] hand kinematics.

Our network builds upon HaMeR [37] by replacing its vanilla transformer with the proposed Rodrigues Network (with modifications to suit MANO’s configuration representation). Additionally, we introduce a cross-attention layer following the self-attention layer to enable interactions between joint and link features and the input image tokens. Our network achieves a notable performance improvement while significantly reducing the number of trainable parameters (39.5M vs. ours: 10.7M). Further architectural details are provided in the supplementary material.

Table [3](https://arxiv.org/html/2506.02618v1#S5.T3) presents the quantitative results of our method. We follow the evaluation protocol and metrics established by HaMeR [37], and report performance on the FreiHand [59] dataset. For reference, we also include the results of our own reproduction of the HaMeR model.

As shown in Table [3](https://arxiv.org/html/2506.02618v1#S5.T3), our method achieves superior quantitative performance, surpassing the previous state-of-the-art. Compared to the strongest baseline, HaMeR, our approach outperforms both the results reported in the original paper and our reproduced implementation. This underscores the effectiveness of incorporating hand joint kinematics into the network, resulting in substantial improvements on kinematics-related tasks. Therefore, our approach is not limited to robotic applications, demonstrating its versatility and applicability to graphics-related tasks as well.

## 6 Conclusions and discussions

In this work, we design a neural network that addresses the kinematic structural priors in articulated robot action learning.
Core to it is the Neural Rodrigues Operator that extends the Rodrigues’ rotation formula into a learnable operator of more flexible forms, providing networks with an inductive bias that can better model kinematics-related features. With this neural operator as a key component, we build the Rodrigues Network with additional layer designs, resulting in a powerful and embodiment-aware network architecture applicable to diverse action-learning tasks.

Many state-of-the-art networks in robot learning build upon architectural designs originally developed for other domains, such as vision and language. With this work, we aim to encourage the exploration of action-centric neural network architectures that are tailored to the unique characteristics of robot learning, particularly those aspects that are underexplored in other fields.

#### Limitations and future work

First, the Rodrigues Network focuses on leveraging the kinematic structure at the level of joints, while overlooking the geometry of the links, such as their lengths and shapes. More effectively incorporating link information and geometry awareness could provide a more comprehensive representation of the kinematic structure, potentially improving performance on tasks that require precise contact reasoning.
Second, the current Neural Rodrigues Operator is designed for rotational joints; extending it to handle both rotational and translational joints could broaden the network’s applicability to a wider range of robotic systems.
Third, our robot learning experiments focused on imitation learning. An important direction for future work is to investigate how the network performs under reinforcement learning (RL), which could enhance its generality and effectiveness in closed-loop learning scenarios.

## 7 Acknowledgements

Leonidas Guibas, Yang You and Congyue Deng acknowledge support from the Toyota Research Institute University 2.0 Program, ARL grant W911NF-21-2-0104, a Vannevar Bush Faculty Fellowship, and a gift from the Flexiv corporation. Yang You is also supported in part by the Outstanding Doctoral Graduates Development Scholarship of Shanghai Jiao Tong University.
Pieter Abbeel holds concurrent appointments as a professor at UC Berkeley and as an Amazon Scholar, and this paper is not associated with Amazon.

## References

- sha [2005]
ShadowRobot.
URL [https://www.shadowrobot.com/dexterous-hand-series/](https://www.shadowrobot.com/dexterous-hand-series/), 2005.
- Aberman et al. [2020]
Kfir Aberman, Peizhuo Li, Dani Lischinski, Olga Sorkine-Hornung, Daniel Cohen-Or, and Baoquan Chen.
Skeleton-aware networks for deep motion retargeting.
*ACM Transactions on Graphics (TOG)*, 39(4):62–1, 2020.
- Brohan et al. [2022]
Anthony Brohan, Noah Brown, Justice Carbajal, Yevgen Chebotar, Joseph Dabis, Chelsea Finn, Keerthana Gopalakrishnan, Karol Hausman, Alex Herzog, Jasmine Hsu, et al.
Rt-1: Robotics transformer for real-world control at scale.
*arXiv preprint arXiv:2212.06817*, 2022.
- Bruna et al. [2013]
Joan Bruna, Wojciech Zaremba, Arthur Szlam, and Yann LeCun.
Spectral networks and locally connected networks on graphs.
*arXiv preprint arXiv:1312.6203*, 2013.
- Cai et al. [2021]
Jinmiao Cai, Nianjuan Jiang, Xiaoguang Han, Kui Jia, and Jiangbo Lu.
Jolo-gcn: mining joint-centered light-weight information for skeleton-based action recognition.
In *Proceedings of the IEEE/CVF winter conference on applications of computer vision*, pages 2735–2744, 2021.
- Canny [1986]
John Canny.
A computational approach to edge detection.
*IEEE Transactions on pattern analysis and machine intelligence*, (6):679–698, 1986.
- Chao et al. [2021]
Yu-Wei Chao, Wei Yang, Yu Xiang, Pavlo Molchanov, Ankur Handa, Jonathan Tremblay, Yashraj S Narang, Karl Van Wyk, Umar Iqbal, Stan Birchfield, et al.
Dexycb: A benchmark for capturing hand grasping of objects.
In *Proceedings of the IEEE/CVF conference on computer vision and pattern recognition*, pages 9044–9053, 2021.
- Chen et al. [2025]
Kaixin Chen, Lin Zhang, Zhong Wang, Shengjie Zhao, and Yicong Zhou.
Skeleton-aware graph-based adversarial networks for human pose estimation from sparse imus.
*ACM Transactions on Multimedia Computing, Communications and Applications*, 21(4):1–22, 2025.
- Chen et al. [2021a]
Lili Chen, Kevin Lu, Aravind Rajeswaran, Kimin Lee, Aditya Grover, Misha Laskin, Pieter Abbeel, Aravind Srinivas, and Igor Mordatch.
Decision transformer: Reinforcement learning via sequence modeling.
*Advances in neural information processing systems*, 34:15084–15097, 2021a.
- Chen et al. [2021b]
Ping Chen, Yujin Chen, Dong Yang, Fangyin Wu, Qin Li, Qingpei Xia, and Yong Tan.
I2uv-handnet: Image-to-uv prediction network for accurate and high-fidelity 3d hand mesh modeling.
In *Proceedings of the IEEE/CVF international conference on computer vision*, pages 12929–12938, 2021b.
- Chen et al. [2022]
Xingyu Chen, Yufeng Liu, Yajiao Dong, Xiong Zhang, Chongyang Ma, Yanmin Xiong, Yuan Zhang, and Xiaoyan Guo.
Mobrecon: Mobile-friendly hand mesh reconstruction from monocular image.
In *Proceedings of the IEEE/CVF conference on computer vision and pattern recognition*, pages 20544–20554, 2022.
- Chen et al. [2021c]
Yuxin Chen, Ziqi Zhang, Chunfeng Yuan, Bing Li, Ying Deng, and Weiming Hu.
Channel-wise topology refinement graph convolution for skeleton-based action recognition.
In *Proceedings of the IEEE/CVF international conference on computer vision*, pages 13359–13368, 2021c.
- Cheng et al. [2020]
Ke Cheng, Yifan Zhang, Xiangyu He, Weihan Chen, Jian Cheng, and Hanqing Lu.
Skeleton-based action recognition with shift graph convolutional network.
In *Proceedings of the IEEE/CVF conference on computer vision and pattern recognition*, pages 183–192, 2020.
- Chi et al. [2023]
Cheng Chi, Zhenjia Xu, Siyuan Feng, Eric Cousineau, Yilun Du, Benjamin Burchfiel, Russ Tedrake, and Shuran Song.
Diffusion policy: Visuomotor policy learning via action diffusion.
*The International Journal of Robotics Research*, page 02783649241273668, 2023.
- Choi et al. [2020]
Hongsuk Choi, Gyeongsik Moon, and Kyoung Mu Lee.
Pose2mesh: Graph convolutional network for 3d human pose and mesh recovery from a 2d human pose.
In *Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part VII 16*, pages 769–787. Springer, 2020.
- Ci et al. [2019]
Hai Ci, Chunyu Wang, Xiaoxuan Ma, and Yizhou Wang.
Optimizing network structure for 3d human pose estimation.
In *Proceedings of the IEEE/CVF international conference on computer vision*, pages 2262–2271, 2019.
- Dosovitskiy et al. [2020]
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al.
An image is worth 16x16 words: Transformers for image recognition at scale.
*arXiv preprint arXiv:2010.11929*, 2020.
- Duan et al. [2021]
Haonan Duan, Peng Wang, Yayu Huang, Guangyun Xu, Wei Wei, and Xiaofei Shen.
Robotics dexterous grasping: The methods based on point cloud and deep learning.
*Frontiers in Neurorobotics*, 15:658280, 2021.
- Fang et al. [2022]
Hao-Shu Fang, Jiefeng Li, Hongyang Tang, Chao Xu, Haoyi Zhu, Yuliang Xiu, Yong-Lu Li, and Cewu Lu.
Alphapose: Whole-body regional multi-person pose estimation and tracking in real-time.
*IEEE transactions on pattern analysis and machine intelligence*, 45(6):7157–7173, 2022.
- Geng et al. [2025]
Haoran Geng, Feishi Wang, Songlin Wei, Yuyang Li, Bangjun Wang, Boshi An, Charlie Tianyue Cheng, Haozhe Lou, Peihao Li, Yen-Jen Wang, Yutong Liang, Dylan Goetting, Chaoyi Xu, Haozhe Chen, Yuxi Qian, Yiran Geng, Jiageng Mao, Weikang Wan, Mingtong Zhang, Jiangran Lyu, Siheng Zhao, Jiazhao Zhang, Jialiang Zhang, Chengyang Zhao, Haoran Lu, Yufei Ding, Ran Gong, Yuran Wang, Yuxuan Kuang, Ruihai Wu, Baoxiong Jia, Carlo Sferrazza, Hao Dong, Siyuan Huang, Yue Wang, Jitendra Malik, and Pieter Abbeel.
Roboverse: Towards a unified platform, dataset and benchmark for scalable and generalizable robot learning, 2025.
URL [https://arxiv.org/abs/2504.18904](https://arxiv.org/abs/2504.18904).
- Hampali et al. [2020]
Shreyas Hampali, Mahdi Rad, Markus Oberweger, and Vincent Lepetit.
Honnotate: A method for 3d annotation of hand and object poses.
In *Proceedings of the IEEE/CVF conference on computer vision and pattern recognition*, pages 3196–3206, 2020.
- Hong et al. [2021]
Sunghoon Hong, Deunsol Yoon, and Kee-Eung Kim.
Structure-aware transformer policy for inhomogeneous multi-task reinforcement learning.
In *International Conference on Learning Representations*, 2021.
- Jiang et al. [2021]
Hanwen Jiang, Shaowei Liu, Jiashun Wang, and Xiaolong Wang.
Hand-object contact consistency reasoning for human grasps generation.
In *Proceedings of the IEEE/CVF international conference on computer vision*, pages 11107–11116, 2021.
- Jiang et al. [2023]
Zheheng Jiang, Hossein Rahmani, Sue Black, and Bryan M Williams.
A probabilistic attention model with occlusion-aware texture regression for 3d hand reconstruction from a single rgb image.
In *Proceedings of the IEEE/CVF conference on computer vision and pattern recognition*, pages 758–767, 2023.
- Jin et al. [2020]
Sheng Jin, Lumin Xu, Jin Xu, Can Wang, Wentao Liu, Chen Qian, Wanli Ouyang, and Ping Luo.
Whole-body human pose estimation in the wild.
In *European Conference on Computer Vision*, pages 196–214. Springer, 2020.
- Krizhevsky et al. [2012]
Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton.
Imagenet classification with deep convolutional neural networks.
*Advances in neural information processing systems*, 25, 2012.
- [27]
Yuxuan Kuang, Amine Elhafsi, Haoran Geng, Marco Pavone, and Yue Wang.
SkillBlender: Towards versatile humanoid whole-body control via skill blending.
In *CoRL 2024 Workshop on Whole-body Control and Bimanual Manipulation: Applications in Humanoids and Beyond*.
- Lin et al. [2021a]
Kevin Lin, Lijuan Wang, and Zicheng Liu.
End-to-end human pose and mesh reconstruction with transformers.
In *Proceedings of the IEEE/CVF conference on computer vision and pattern recognition*, pages 1954–1963, 2021a.
- Lin et al. [2021b]
Kevin Lin, Lijuan Wang, and Zicheng Liu.
Mesh graphormer.
In *Proceedings of the IEEE/CVF international conference on computer vision*, pages 12939–12948, 2021b.
- Liu et al. [2020]
Min Liu, Zherong Pan, Kai Xu, Kanishka Ganguly, and Dinesh Manocha.
Deep differentiable grasp planner for high-dof grippers.
*arXiv preprint arXiv:2002.01530*, 2020.
- Loper et al. [2023]
Matthew Loper, Naureen Mahmood, Javier Romero, Gerard Pons-Moll, and Michael J Black.
Smpl: A skinned multi-person linear model.
In *Seminal Graphics Papers: Pushing the Boundaries, Volume 2*, pages 851–866. 2023.
- Moon and Lee [2020]
Gyeongsik Moon and Kyoung Mu Lee.
I2l-meshnet: Image-to-lixel prediction network for accurate 3d human pose and mesh estimation from a single rgb image.
In *Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part VII 16*, pages 752–768. Springer, 2020.
- Moon et al. [2020]
Gyeongsik Moon, Shoou-I Yu, He Wen, Takaaki Shiratori, and Kyoung Mu Lee.
Interhand2. 6m: A dataset and baseline for 3d interacting hand pose estimation from a single rgb image.
In *Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part XX 16*, pages 548–564. Springer, 2020.
- Moro and Sentis [2019]
Federico L Moro and Luis Sentis.
Whole-body control of humanoid robots.
*Humanoid robotics: a reference*, pages 1161–1183, 2019.
- Mu et al. [2021]
Tongzhou Mu, Zhan Ling, Fanbo Xiang, Derek Yang, Xuanlin Li, Stone Tao, Zhiao Huang, Zhiwei Jia, and Hao Su.
Maniskill: Generalizable manipulation skill benchmark with large-scale demonstrations.
*arXiv preprint arXiv:2107.14483*, 2021.
- Niepert et al. [2016]
Mathias Niepert, Mohamed Ahmed, and Konstantin Kutzkov.
Learning convolutional neural networks for graphs.
In *International conference on machine learning*, pages 2014–2023. PMLR, 2016.
- Pavlakos et al. [2024]
Georgios Pavlakos, Dandan Shan, Ilija Radosavovic, Angjoo Kanazawa, David Fouhey, and Jitendra Malik.
Reconstructing hands in 3d with transformers.
In *Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition*, pages 9826–9836, 2024.
- Pavllo et al. [2020]
Dario Pavllo, Christoph Feichtenhofer, Michael Auli, and David Grangier.
Modeling human motion with quaternion-based neural networks.
*International Journal of Computer Vision*, 128:855–872, 2020.
- Radosavovic et al. [2024]
Ilija Radosavovic, Bike Zhang, Baifeng Shi, Jathushan Rajasegaran, Sarthak Kamat, Trevor Darrell, Koushil Sreenath, and Jitendra Malik.
Humanoid locomotion as next token prediction.
In *The Thirty-eighth Annual Conference on Neural Information Processing Systems*, 2024.
- Romero et al. [2022]
Javier Romero, Dimitrios Tzionas, and Michael J Black.
Embodied hands: Modeling and capturing hands and bodies together.
*arXiv preprint arXiv:2201.02610*, 2022.
- Sferrazza et al. [2024]
Carmelo Sferrazza, Dun-Ming Huang, Fangchen Liu, Jongmin Lee, and Pieter Abbeel.
Body transformer: Leveraging robot embodiment for policy learning.
*arXiv preprint arXiv:2408.06316*, 2024.
- Shaw et al. [2023]
Kenneth Shaw, Ananye Agarwal, and Deepak Pathak.
Leap hand: Low-cost, efficient, and anthropomorphic hand for robot learning.
*arXiv preprint arXiv:2309.06440*, 2023.
- Shridhar et al. [2023]
Mohit Shridhar, Lucas Manuelli, and Dieter Fox.
Perceiver-actor: A multi-task transformer for robotic manipulation.
In *Conference on Robot Learning*, pages 785–799. PMLR, 2023.
- Simon et al. [2017]
Tomas Simon, Hanbyul Joo, Iain Matthews, and Yaser Sheikh.
Hand keypoint detection in single images using multiview bootstrapping.
In *Proceedings of the IEEE conference on Computer Vision and Pattern Recognition*, pages 1145–1153, 2017.
- Song et al. [2020]
Yi-Fan Song, Zhang Zhang, Caifeng Shan, and Liang Wang.
Stronger, faster and more explainable: A graph convolutional baseline for skeleton-based action recognition.
In *proceedings of the 28th ACM international conference on multimedia*, pages 1625–1633, 2020.
- Song et al. [2022]
Yi-Fan Song, Zhang Zhang, Caifeng Shan, and Liang Wang.
Constructing stronger and faster baselines for skeleton-based action recognition.
*IEEE transactions on pattern analysis and machine intelligence*, 45(2):1474–1488, 2022.
- Tang et al. [2021]
Xiao Tang, Tianyu Wang, and Chi-Wing Fu.
Towards accurate alignment in real-time 3d hand-mesh reconstruction.
In *Proceedings of the IEEE/CVF international conference on computer vision*, pages 11698–11707, 2021.
- Vaswani et al. [2017]
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin.
Attention is all you need.
*Advances in neural information processing systems*, 30, 2017.
- Villegas et al. [2018]
Ruben Villegas, Jimei Yang, Duygu Ceylan, and Honglak Lee.
Neural kinematic networks for unsupervised motion retargetting.
In *Proceedings of the IEEE conference on computer vision and pattern recognition*, pages 8639–8648, 2018.
- Wan et al. [2023]
Weikang Wan, Haoran Geng, Yun Liu, Zikang Shan, Yaodong Yang, Li Yi, and He Wang.
Unidexgrasp++: Improving dexterous grasping policy learning via geometry-aware curriculum and iterative generalist-specialist learning.
*arXiv preprint arXiv:2304.00464*, 2023.
- Xiang et al. [2019]
Donglai Xiang, Hanbyul Joo, and Yaser Sheikh.
Monocular total capture: Posing face, body, and hands in the wild.
In *Proceedings of the IEEE/CVF conference on computer vision and pattern recognition*, pages 10965–10974, 2019.
- Xiang et al. [2020]
Fanbo Xiang, Yuzhe Qin, Kaichun Mo, Yikuan Xia, Hao Zhu, Fangchen Liu, Minghua Liu, Hanxiao Jiang, Yifu Yuan, He Wang, et al.
Sapien: A simulated part-based interactive environment.
In *Proceedings of the IEEE/CVF conference on computer vision and pattern recognition*, pages 11097–11107, 2020.
- Xu et al. [2023]
Yinzhen Xu, Weikang Wan, Jialiang Zhang, Haoran Liu, Zikang Shan, Hao Shen, Ruicheng Wang, Haoran Geng, Yijia Weng, Jiayi Chen, et al.
Unidexgrasp: Universal robotic dexterous grasping via learning diverse proposal generation and goal-conditioned policy.
*arXiv preprint arXiv:2303.00938*, 2023.
- Yan et al. [2018]
Sijie Yan, Yuanjun Xiong, and Dahua Lin.
Spatial temporal graph convolutional networks for skeleton-based action recognition.
In *Proceedings of the AAAI conference on artificial intelligence*, volume 32, 2018.
- Zeng et al. [2021]
Ailing Zeng, Xiao Sun, Lei Yang, Nanxuan Zhao, Minhao Liu, and Qiang Xu.
Learning skeletal graph neural networks for hard 3d pose estimation.
In *Proceedings of the IEEE/CVF international conference on computer vision*, pages 11436–11445, 2021.
- Zhang et al. [2024]
Jialiang Zhang, Haoran Liu, Danshi Li, Xinqiang Yu, Haoran Geng, Yufei Ding, Jiayi Chen, and He Wang.
Dexgraspnet 2.0: Learning generative dexterous grasping in large-scale synthetic cluttered scenes, 2024.
URL [https://arxiv.org/abs/2410.23004](https://arxiv.org/abs/2410.23004).
- Zhao et al. [2023]
Tony Z Zhao, Vikash Kumar, Sergey Levine, and Chelsea Finn.
Learning fine-grained bimanual manipulation with low-cost hardware.
*arXiv preprint arXiv:2304.13705*, 2023.
- Zimmermann and Brox [2017]
Christian Zimmermann and Thomas Brox.
Learning to estimate 3d hand pose from single rgb images.
In *Proceedings of the IEEE international conference on computer vision*, pages 4903–4911, 2017.
- Zimmermann et al. [2019]
Christian Zimmermann, Duygu Ceylan, Jimei Yang, Bryan Russell, Max Argus, and Thomas Brox.
Freihand: A dataset for markerless capture of hand pose and shape from single rgb images.
In *Proceedings of the IEEE/CVF international conference on computer vision*, pages 813–822, 2019.

## Appendix A Rodrigues Network for animated characters

Beyond articulated robots, animated characters such as MANO [40] and SMPL [31] also exhibit articulated structures and employ forward kinematics, which can similarly serve as inductive biases for neural networks.
However, a fundamental distinction exists between these models and articulated robots: each joint in MANO and SMPL is a free rotational joint with three degrees of freedom, as opposed to the single-axis, 1-DoF rotational joints typically found in robots.
As a result, adapting our approach to animated characters requires a fundamental modification of the Rodrigues Operator to accommodate unconstrained 3-DoF rotations.
We need to identify the basic operation in forward kinematics for animated characters, and develop a neural operator by turning that operation learnable.

#### Forward kinematics

At each 3-DoF joint, we can parameterize its configuration as a unit quaternion $\mathbf{q}=(q_{w},q_{i},q_{j},q_{k})$ where $\|\mathbf{q}\|=1$.
Following this, forward kinematics for animated characters is similar to articulated robots.
There are also two transformations at each joint $J_{j}$: (i) a fixed coordinate change $\mathbf{T}_{j}$ from the parent link frame to the joint frame; (ii) a dynamic rotation $\mathbf{R}_{\text{q2mat}}(\mathbf{q})\in\mathbb{R}^{3\times 3}$ in the joint frame.
The parent-to-children pose transformation can also be written as:

$$ $\begin{bmatrix}\mathbf{R}_{\text{c}_{j}}&\mathbf{t}_{\text{c}_{j}}\\ \mathbf{0}_{1\times 3}&1\end{bmatrix}=\begin{bmatrix}\mathbf{R}_{\text{p}_{j}} &\mathbf{t}_{\text{p}_{j}}\\ \mathbf{0}_{1\times 3}&1\end{bmatrix}\mathbf{T}_{j}\begin{bmatrix}\mathbf{R}_{ \text{q2mat}}(\mathbf{q})&\mathbf{0}_{3\times 1}\\ \mathbf{0}_{1\times 3}&1\end{bmatrix}.$ (12) $$

Forward kinematics for animated characters is essentially a hierarchical combination of Equation [12](https://arxiv.org/html/2506.02618v1#A1.E12).

#### Quaternions to matrix conversion

The key difference of computing forward kinematics for animated characters lies in converting a joint’s unit quaternions into a 3 by 3 rotation matrix.
This can be done using the following formula:

$$ $\mathbf{R}_{\text{q2mat}}(\mathbf{q})=\begin{bmatrix}1-2(q_{j}^{2}+q_{k}^{2})& 2(q_{i}q_{j}-q_{k}q_{w})&2(q_{i}q_{k}+q_{j}q_{w})\\ 2(q_{i}q_{j}+q_{k}q_{w})&1-2(q_{i}^{2}+q_{k}^{2})&2(q_{j}q_{k}-q_{i}q_{w})\\ 2(q_{i}q_{k}-q_{j}q_{w})&2(q_{j}q_{k}+q_{i}q_{w})&1-2(q_{i}^{2}+q_{j}^{2})\end {bmatrix}$ (13) $$

#### Neural Rodrigues Operator for quaternions

Observe from Equation [13](https://arxiv.org/html/2506.02618v1#A1.E13) every entry in $\mathbf{R}_{\text{q2mat}}(\mathbf{q})$ is a linear combination of ones and all 10 quadratic terms of $q_{w},q_{i},q_{j},q_{k}$, with constant coefficients.
Thus, we can mimic what we have done for the Rodrigues’ rotation formula and re-parameterize Equation [12](https://arxiv.org/html/2506.02618v1#A1.E12) as:

$$ $\mathbf{P}_{\mathrm{c}_{j}}=\mathbf{P}_{\mathrm{p}_{j}}\left(\mathbf{A}+\sum_{ x,y\in\{w,i,j,k\},x\neq y}\mathbf{A}_{xy}q_{x}q_{y}+\sum_{x\in\{w,i,j,k\}} \mathbf{A}_{xx}q_{x}^{2}\right)$ (14) $$

where $\mathbf{A},\mathbf{A}_{xy},\mathbf{A}_{xx}\in\mathbb{R}^{4\times 4}$ are coefficient matrices that only depend on the joint’s structure.
Similar to what we have done for axis angles, we construct our Neural Rodrigues Operator for quaternions by replacing these fixed coefficient with learnable weights $W^{\text{bias}},W^{xy},W^{xx}\in\mathbb{R}^{4\times 4}$, resulting in:

$$ $F^{\text{out}}=F^{\text{in}}\left(W^{\text{bias}}+\sum_{x,y\in\{w,i,j,k\},x \neq y}W^{xy}q_{x}q_{y}+\sum_{x\in\{w,i,j,k\}}W^{xx}q_{x}^{2}\right)$ (15) $$

where $F^{\text{in}}\in\mathbb{R}^{4\times 4}$ and $F^{\text{out}}\in\mathbb{R}^{4\times 4}$ are the input and output link features, corresponding to the parent and child links.
Similar to the original Neural Rodrigues Operator, we also extend this operator to support multi link feature channels and left multiplication.
But since there are already four feature channels for each joint, we do not extend it further.
We also add learnable weights $W^{x}\in\mathbb{R}^{4\times 4}$ to Equation [15](https://arxiv.org/html/2506.02618v1#A1.E15) to support representing linear terms of $q_{w},q_{i},q_{j},q_{k}$.

## Appendix B Experiment settings

### B.1 Forward kinematics fitting

#### Data preparation

In this experiment, the models learn to fit the forward kinematics mapping of a robot. Therefore, we do not need to generate a fixed-size training set. However, we do create a fixed-size validation set for selecting the best checkpoint, and a separate test set for evaluation.
We conduct this experiment on the LEAP hand, which is a fully-actuated robotic hand with four fingers, 16 rotational joints, and 17 links.
Each data point includes the input: the root link position $T\in\mathbb{R}^{3}$, the root link orientation matrix $R\in\mathbb{R}^{3\times 3}$, and joint angles $\theta\in\mathbb{R}^{16}$. The output is the position vectors and orientation matrices of all 17 links.
We sample $T$ uniformly in $[-0.05~{}\text{cm},0.05~{}\text{cm}]^{3}$, $R$ uniformly from $\mathrm{SO}(3)$, and $\theta$ uniformly within the joint limits. We then compute all link poses using forward kinematics.
The validation and test sets each contain 10,000 input-output pairs, generated using different random seeds.

#### Training details

We use the Adam optimizer with $\text{lr}=0.0003$ and without weight decay and train each model for 10,000 steps.
In each training step, we generate a new batch of 1024 input-output pairs using the same method as for the validation and test sets, instead of sampling from a fixed-size training set.
The model takes the input and predicts 17 position vectors and 17 orientation matrices, totaling 17 × (3 + 9) values.
We compute the mean squared error (MSE) loss between the predicted and ground truth outputs. The loss is averaged over the batch and all output values, and then used to update the model through gradient descent.
We evaluate the model on the validation set for every 500 steps, and pick the model with the lowest validation loss for final evaluation.
We also save a checkpoint every 1,000 iterations for plotting the training curve.
All hyperparameters are summarized in Table [4](https://arxiv.org/html/2506.02618v1#A2.T4).

**Table 4: Training hyperparameters for forward kinematics fitting experiment.**
| Parameter name | Value | Parameter name | Value | Parameter | Value |
| --- | --- | --- | --- | --- | --- |
| Training iterations | 100,000 | Optimizer | Adam | Learning rate | $0.0003$ |
| Batch size | 1024 | Validate every | 500 iterations | Weight decay | 0 |

### B.2 Motion prediction in Cartesian space

#### Data preparation

This experiment is conducted on a fixed-base 6-DoF UR5 robotic arm.
To make the inverse kinematics solution unique for all reachable end-effector poses, we limit the six joint ranges to $[0,\pi/2]$, $[-\pi/2,0]$, $[0,\pi/2]$, $[0,\pi/4]$, $[0,\pi/4]$, and $[0,\pi/4]$.
We sample two joint configurations $\boldsymbol{\theta}_{\text{start}},\boldsymbol{\theta}_{\text{end}}\in\mathbb{
R}^{6}$ within these limits, which serve as the start and end poses.
We then use forward kinematics to compute the corresponding 3D end-effector poses $\mathbf{P}_{\text{start}},\mathbf{P}_{\text{end}}\in\mathrm{SE}(3)$.
Next, we interpolate $16-2$ intermediate 3D poses between the start and end to form a trajectory of 16 poses: $\mathbf{P}_{1},\cdots,\mathbf{P}_{16}$.
Positions are interpolated linearly, and orientations are interpolated using spherical linear interpolation (slerp).
Finally, we apply inverse kinematics to get the corresponding joint configurations $\boldsymbol{\theta}_{1},\cdots,\boldsymbol{\theta}_{16}$ for the 3D pose trajectory.
The first 8 frames of joint configurations are used as input, and the models are asked to predict the last 8 frames.
In Cartesian space, the motion pattern is clear: positions follow a straight line, and orientations change smoothly with slerp.
However, this pattern is not obvious in joint space.
The model needs to learn to reason about this motion pattern from the joint configurations.
We generated four training sets with $10^{3}$, $10^{4}$, $10^{5}$, and $10^{6}$ input-output pairs. The validation and test sets each contain $10^{4}$ input-output pairs.

#### Training details

We use the Adam optimizer with a learning rate of 0.0001 and no weight decay, and train each model for 10,000 steps.
Each training session uses a fixed training set. In each step, we randomly sample a batch of 1,024 input-output pairs from this set.
The model takes the input and predicts $8\times 6$ values, which represent the joint configurations for the last 8 frames.
We compute the mean squared error (MSE) loss between the predicted and ground truth outputs, and use it to perform gradient descent.
We evaluate the model on the validation set every 500 steps, and select the checkpoint with the lowest validation loss for final testing.
Unlike the forward kinematics fitting experiment, this experiment trains models on a fixed-size training set and evaluates them on a separate test set drawn from the same distribution.
Therefore, it evaluates not only the model’s ability to fit the training data, but also its generalization performance.
All training hyperparameters are listed in Table [5](https://arxiv.org/html/2506.02618v1#A2.T5).

**Table 5: Training hyperparameters for motion prediction in Cartesian space experiment.**
| Parameter name | Value | Parameter name | Value | Parameter | Value |
| --- | --- | --- | --- | --- | --- |
| Training iterations | 100,000 | Optimizer | Adam | Learning rate | $0.0001$ |
| Batch size | 1024 | Validate every | 500 iterations | Weight decay | 0 |
| Input frames | 8 | Output frames | 8 | DoFs | 6 |

#### Evaluation

We apply forward kinematics to the predicted joint configurations to obtain the predicted end-effector 3D poses.
Then, we compare both the predicted joint configurations and the predicted end-effector poses with the ground truth. We report the following evaluation metrics:
1) $\text{Error}_{T}$ (mm): End-effector position error on the test set, averaged over the 8 predicted frames.
2) $\text{Error}_{R}$ (^∘): End-effector orientation error on the test set, averaged over the 8 predicted frames. We compute the relative rotation between the predicted and ground truth orientations, convert it to axis-angle representation, and take the angle.
3) $\text{Error}_{\theta}$ (^∘): Absolute joint configuration error on the test set, averaged over the 8 predicted frames and 6 joints.
4) MSE ($\times 10^{-6}$): Mean squared error on the test set, multiplied by $10^{-6}$ for better readability.
5) Train MSE ($\times 10^{-6}$): Mean squared error on the training set, also scaled by $10^{-6}$.
The first four metrics measure the model’s generalization ability, while the last one reflects its fitting ability.
For all metrics, lower values indicate better performance.

### B.3 Robotic manipulation with imitation learning

#### Data generation

We evaluate on five representative manipulation tasks from the ManiSkill Benchmark [35]: PushCube, PickCube, StackCube, PegInsertionSide, and PlugCharger.
For each task, we collect expert demonstration trajectories using the ManiSkill data collection API.
These trajectories are generated via motion planning algorithms and are limited to a maximum of 200 simulation steps per episode.
The number of demonstrations collected per task is listed in Table [7](https://arxiv.org/html/2506.02618v1#A2.T7).
During testing, the initial scene configurations are randomized using the same distribution employed during training data collection.
For additional details, please refer to the ManiSkill benchmark [35].

#### Training details

We follow the training setup in [14], including optimizer choice, learning rate scheduling, and exponential moving average (EMA) for network stabilization.
Specifically, we use the AdamW optimizer with a learning rate of 0.0001, $\beta=(0.95,0.999)$, and a weight decay of $1\times 10^{-6}$.
A cosine learning rate scheduler with 500 warm-up steps is applied.
To enhance training speed and stability, we maintain an EMA of the model weights with a decay factor of 0.75.
During training, we sample a batch of 1024 expert observation-action pairs from the demonstration dataset in each iteration and perform a gradient update using the diffusion loss.
All training hyperparameters are summarized in Table [6](https://arxiv.org/html/2506.02618v1#A2.T6).
The number of training iterations and demonstration trajectories varies across tasks, depending on task complexity; see Table [7](https://arxiv.org/html/2506.02618v1#A2.T7) for details.
For further implementation details, please refer to [14].

**Table 6: Training hyperparameters for imitation learning experiment (following [14]’s settings).**
| Parameter name | Value | Parameter name | Value | Parameter | Value |
| --- | --- | --- | --- | --- | --- |
| Optimizer | AdamW | Learning rate | $0.0001$ | Weight decay | 1e-6 |
| LR scheduler | Cosine scheduler | Batch size | 1024 | Episode steps | 200 |

**Table 7: Demo trajectories and training iterations for each task in imitation learning experiment**
| Task name | PushCube | PickCube | StackCube | PegInsertionSide | PlugCharger |
| --- | --- | --- | --- | --- | --- |
| Demo num | 100 | 100 | 100 | 500 | 500 |
| Training iterations | 30k | 30k | 60k | 100k | 300k |

### B.4 3D hand reconstruction

#### Dataset preparation

Following Hamer, we train our model on a large number of datasets that provide 2D or 3D hand annotations. Specifically, we use FreiHAND [59], HO3D [21], MTC [51], RHD [58], InterHand2.6M [33], H2O3D [21], DEX YCB [7], COCO WholeBody [25], Halpe [19] and MPII NZSL [44].

#### Training Details

We follow the training protocol described in Hamer [37]. We use the AdamW optimizer with learning rate 1e-5 and weight decay 1e-4, and train the model for 1,000,000 steps. The batch size is set to 64. Our model takes in a square image around the target hand, resized to 256 x 256, and output the 58 MANO [40] parameters of the human hand, specifically 48 pose parameters and the 10 shape parameters. We evaluate the model on the validation set for every 1,000 steps, and pick the model with the lowest validation loss for final evaluation. Similar to Hamer, we additionally estimate the camera parameters $\pi$. The camara $\pi$ corresponds to a translation $t\in\mathbb{R}^{3}$ that projects the 3D mesh and joints into the image. Hyperparameters are summarized in Table [8](https://arxiv.org/html/2506.02618v1#A2.T8).

**Table 8: Training hyperparameters for 3D hand reconstruction experiment.**
| Parameter | Value | Parameter | Value | Parameter | Value |
| --- | --- | --- | --- | --- | --- |
| Training iterations | 1,000,000 | Optimizer | AdamW | Learning rate | 1e-5 |
| Batch size | 64 | Validate every | 1,000 iterations | Weight decay | 1e-4 |

## Appendix C Method and implementation details

### C.1 Forward kinematics fitting

We construct four baseline methods using existing neural network backbones for comparison: MLP, Graph Convolution Network (GCN), Transformer, and Body Transformer (BoT) [41].

#### MLP

The MLP baseline concatenates the input root position, root orientation, and joint angles into a $(3+9+\text{DoF})$-dimensional vector ($\text{DoF}=16$ for the LEAP hand), and feeds it into a 7-layer MLP with the following shape:
$[28,768,768,768,768,768,768,204]$.
The output represents the positions and orientations of all $\text{DoF}+1=17$ links.
All hidden layers use ReLU activation, and no normalization layers are applied.

#### GCN

The GCN baseline uses $1+\text{DoF}$ separate linear transformations to encode the root pose and each joint angle into $1+\text{DoF}$ feature embeddings, each with 512 dimensions.
Each embedding corresponds to one robot link: the one derived from the root pose represents the root link, and each joint’s embedding corresponds to its child link.
We represent the robot as an undirected tree graph with $1+\text{DoF}$ nodes (links) and DoF edges (joints).
The GCN applies 11 layers of graph convolution to update the link features.
All hidden layers have 512 dimensions.
Finally, separate linear transformations are used to predict the pose (position and orientation) of each link.

#### Transformer

The Transformer baseline uses the same approach as the GCN baseline to encode the input root pose and joint angles into $1+\text{DoF}$ link feature embeddings, each with 256 dimensions.
After applying positional encoding, these embeddings are passed through a Transformer backbone consisting of 8 Transformer blocks.
Each block includes a feed-forward layer with a 256-dimensional hidden layer.
The Transformer outputs $1+\text{DoF}$ updated link features.
As in the GCN baseline, separate linear transformations are used to predict the pose of each link from these features.

#### Body Transformer (BoT) [ 41 ]

The BoT baseline shares a similar architecture with the Transformer baseline but replaces the Transformer backbone with a Body Transformer backbone.
It uses the same robot connectivity graph as in the GCN baseline to capture the structural relationships between links.

#### Rodrigues Network (ours)

Our method uses a specially customized Rodrigues Network as the neural backbone.
The network consists of 12 Rodrigues Blocks, and each block contains only a single Rodrigues Layer.
No Joint Layers, Self-Attention Layers, or Global Tokens are used.
Each Rodrigues Layer has 1 joint channel ($C_{J}=1$) and 3 link channels ($C_{L}=3$).
We first concatenate the input root position, root orientation, and joint angles into a $(3+9+\text{DoF})$-dimensional vector.
This vector is passed through DoF separate linear transformations to produce the input joint features, and through $1+\text{DoF}$ separate linear transformations to produce the input link features.
The Rodrigues Network updates the joint and link features through its 12 Rodrigues Blocks, resulting in the output joint and link features.
Finally, we concatenate all output link features and apply $1+\text{DoF}$ separate linear transformations to predict the pose of each link.

#### Parameter count comparison

All baseline networks have around 3 million parameters, while our network has only 0.2 million parameters.
Since this experiment focuses only on fitting ability and not generalization, this setup gives an advantage to the baseline methods with more parameters.
Even so, our network still performs better.

#### Compute resources

We train each model on a single Quadro RTX 6000 graphics card, and the approximate training time for 100,000 training iterations are listed in Table [9](https://arxiv.org/html/2506.02618v1#A3.T9).

**Table 9: Approximate training time of different methods for fitting forward kinematics**
| Method | MLP | GCN | Transformer | BoT | Rodrigues (ours) |
| --- | --- | --- | --- | --- | --- |
| Time | 17min | 1h 50min | 2h 20min | 2h 20min | 1h 18min |

### C.2 Motion prediction in Cartesian space

This experiment uses the same four neural backbones as in the forward kinematics fitting experiment: MLP, Graph Convolution Network (GCN), Transformer, and Body Transformer (BoT).

#### MLP

The MLP baseline takes all $8\times 6=48$ input values from the 8 input frames as input and feeds them into a 7-layer MLP with the following layer sizes:
$[48,768,768,768,768,768,768,48]$.
The output represents the joint configurations of the 8 predicted frames.
All hidden layers use ReLU activation, and no normalization layers are applied.

#### GCN

The GCN baseline uses $1+\text{DoF}$ separate linear transformations to encode the input into $1+\text{DoF}$ feature embeddings, each with 512 dimensions.
As in the forward kinematics fitting experiment, each embedding corresponds to one robot link, and the robot is modeled as an undirected tree graph with $1+\text{DoF}$ nodes (links) and DoF edges (joints).
The GCN applies 11 layers of graph convolution to update the link features.
Finally, for each joint, we extract the output feature of its corresponding child link and apply a separate linear transformation to predict the joint’s angles for all 8 output frames.

#### Transformer

The Transformer baseline follows the same procedure as the GCN baseline to encode the input into $1+\text{DoF}$ link feature embeddings, each with 250 dimensions.
Positional encoding is then applied to these embeddings, which are processed by a Transformer backbone consisting of 8 Transformer blocks.
Each block contains a feed-forward layer with a 250-dimensional hidden layer.
The Transformer outputs $1+\text{DoF}$ updated link features.
As in the GCN baseline, we extract the output feature corresponding to each joint’s child link and apply a separate linear transformation to predict the joint’s 8 output angles.

#### Body Transformer (BoT) [ 41 ]

The BoT baseline replaces the Transformer backbone of the Transformer baseline with a Body Transformer backbone, using the same number of blocks and hidden dimensions.

#### Rodrigues Network (ours)

Our method uses a full Rodrigues Network as the neural backbone, consisting of 12 Rodrigues Blocks.
Each block contains a Rodrigues Layer, a Joint Layer, and a Self-attention Layer.
In this experiment, Global Tokens are also not used.
Each Rodrigues Layer is configured with 4 joint channels ($C_{J}=4$) and 8 link channels ($C_{L}=8$).
Each Self-attention Layer operates on 256-dimensional embeddings with 8 attention heads.
To prepare the input, we apply DoF separate linear transformations to produce the input joint features, and $1+\text{DoF}$ separate linear transformations to generate the input link features.
These features are then processed by the Rodrigues Network, which updates them through all 12 blocks to produce the final joint and link features.
For each joint, we concatenate its output joint feature with the output link feature of its child link, and apply a separate linear transformation to predict the joint’s 8-frame output angles.

#### Parameter count comparison

All baseline methods and our approach have approximately 3 million parameters.
This ensures a fair comparison of both fitting and generalization abilities.

#### Compute resources

We train each model on a single Quadro RTX 6000 graphics card, and the approximate training time for 100,000 training iterations are listed in Table [10](https://arxiv.org/html/2506.02618v1#A3.T10).

**Table 10: Approximate training time of different methods for motion prediction experiment**
| Method | MLP | GCN | Transformer | BoT | Rodrigues (ours) |
| --- | --- | --- | --- | --- | --- |
| Time | 27min | 1h 12min | 1h 20min | 1h 20min | 2h 22min |

### C.3 Robotic Manipulation with Imitation Learning

This experiment follows the Diffusion Policy (DP) framework proposed in [14], and evaluates different neural backbones for denoising action samples.
The original paper [14] provides two backbone options, convolutional UNet and Transformer, which we directly adopt as baselines.
In the DP framework, the policy observes a history of 2 prior time steps, predicts a sequence of 16 future actions, and executes the first 8 during deployment.
Each action vector consists of 8 values: 7 for joint position control of the 7-DoF Franka arm and 1 for the 1-DoF Panda gripper.

#### UNet-DP

The UNet-DP baseline uses a convolutional UNet architecture that performs temporal convolution over the 16-step action sequence.
We follow the default structure, with the hidden dimensions of the downsampling layers set to $[320,320,344]$.
For further implementation details, please refer to [14].

#### Transformer-DP

The Transformer-DP baseline applies causal self-attention to the 16 action-step tokens.
We use an embedding dimension of 320, feed-forward hidden dimension of 1280, and a total of 10 Transformer blocks.
Additional details are available in [14].

#### Rodrigues-DP (ours)

Our method adopts a Rodrigues Network as the denoising backbone, composed of 12 Rodrigues Blocks with 16 link feature channels ($C_{L}=16$), 8 joint feature channels ($C_{J}=8$), and a self-attention embedding dimension of 256.
We also introduce Global Tokens of dimension 128 to model gripper actions.
The denoising network takes three inputs: a time step, a noisy action, and an observation vector.
We first embed the time step and concatenate it with the observation and noisy action to form an input feature vector.
This vector is then passed through DoF separate linear transformations to generate the input joint features, $1+\text{DoF}$ separate linear transformations to generate the input link features, and one additional linear transformation to produce the input global token.
The Rodrigues Network updates all features through its 12 Rodrigues Blocks, resulting in output joint features, output link features, and output global token.
For each joint, we concatenate its output joint feature with the output link feature of its corresponding child link, and apply a separate linear transformation to predict the joint’s 16-frame denoised action trajectory.
For the gripper, we use a linear transformation on the output global token to predict the 16-frame denoised gripper actions.

#### Parameter count comparison

All baseline networks and our network have approximately 17 million parameters to make comparisons fair.

### C.4 3D hand reconstruction

Architecture-wise, we employ a Vision Transformer [17] as the image-processing backbone, followed by our Rodrigues Network (RodriNet) head to regress both hand and camera parameters. Since there are 17 links and 16 joints in the MANO parameter model, to make the information pass through the entire kinematic chain, our RodriNet head comprises 18 Rodrigues blocks, each of which sequentially applies a Rodrigues Layer, a Joint Layer, and a Self-attention Layer to update the link features, joint features.
The Rodrigues Layer uses 4 link channels, and the Self-attention Layer uses 64 embed dimensions.
We enable Global Tokens in this experiment to model the MANO’s shape parameters.

In addition to the standard components, we append a Cross-Attention Layer at the end of each block. This layer processes the link features and the input image tokens through a standard cross-attention transformer module, enriching the link features with visual information from the input image.

The training takes approximately 7 days on a single Quadro RTX 6000 graphics card.

## Appendix D Additional results

We conduct additional experiments to further support the claims made in the main paper.

### D.1 Ablation studies

In the main paper, we benchmark the Rodrigues Network against several baselines for motion prediction in Cartesian space using a fixed training set of $10^{5}$ trajectories. Our method demonstrates a significant performance advantage over all baselines.
To investigate the contributions of different components of our model, we perform an ablation study by individually removing the Rodrigues Layer, Joint Layer, or Self-attention Layer from the original architecture, and evaluate the resulting performance changes.

The results are presented in Table [11](https://arxiv.org/html/2506.02618v1#A4.T11).
First, we observe that the Self-attention Layers comprise more than half of the model parameters. However, removing them results in only a slight increase in training error and even a slight decrease in test error. This suggests that while the Self-attention Layer enhances the model’s ability to fit the training data, it may slightly hinder generalization. Nonetheless, we retain this component in our default architecture to ensure sufficient model capacity for more complex tasks.
Second, the Joint Layers contribute negligibly to the overall parameter count, yet their removal consistently degrades both training and test performance. This highlights their critical role in the effectiveness of the Rodrigues Network.
Third, the Rodrigues Layers constitute a substantial portion of the model’s parameters.
Removing them causes the greatest drop in both training and test performance among all ablation settings.
This indicates that the Rodrigues Layer is the most critical component for the model’s success.
Since the Rodrigues Layer encodes the core inductive bias of our architecture, these findings strongly support our central claim: embedding structural prior of articulated kinematics directly into the network architecture improves learning the actions and motions of articulated actors.

**Table 11: Ablation studies for motion prediction in Cartesian space with trainset size $=10^{5}$. We remove the Rodrigues Layer (R Layer), Joint Layer (J Layer), or Self-attention Layer (S Layer) respectively from the original Rodrigues Network, and evaluate the MSE on train/test sets.**
| R Layer | J Layer | S Layer | Params (M) | Train MSE ($1\mathrm{e}{-6}$) | Test MSE ($1\mathrm{e}{-6}$) |
| --- | --- | --- | --- | --- | --- |
| $\checkmark$ | $\checkmark$ | $\checkmark$ | 3.04 | 1.93 $\pm$0.34 | 2.56 $\pm$0.39 |
| $\checkmark$ | $\checkmark$ |  | 1.44 | 1.94 $\pm$0.26 | 2.33 $\pm$0.26 |
| $\checkmark$ |  | $\checkmark$ | 3.01 | 2.33 $\pm$0.56 | 2.80 $\pm$0.62 |
|  | $\checkmark$ | $\checkmark$ | 1.69 | 5.57 $\pm$0.55 | 6.19 $\pm$0.57 |

### D.2 Tuning the baselines for motion prediction

Figure: Figure 6: Comparing our method to different baseline configurations in motion prediction with trainset size $=10^{5}$.
Refer to caption: x8.png

While our method outperforms all baselines on the motion prediction task under the 3-million-parameter setting, we further investigate whether these baselines have been sufficiently tuned. Therefore, we construct three additional configurations for each of the four baselines (GCN, MLP, BoT, and Transformer) with approximately 1M, 10M, and 30M parameters.

The results are shown in Figure [6](https://arxiv.org/html/2506.02618v1#A4.F6). For all baselines, the training error consistently decreases as model size increases. However, both GCN and MLP begin to overfit with larger models, as indicated by rising test errors. In contrast, BoT and Transformer exhibit test error saturation, but not overfitting, as parameter count increases to 30M.
Importantly, across all parameter scales, our 3M-parameter model outperforms all baseline configurations on both training and test errors. This confirms that the benchmark results reported in the main paper are reliable and that the baselines have been appropriately tuned.
Furthermore, even the best-performing baseline (Transformer) with 30M parameters yields a training error that remains higher than the test error of our 3M model. This highlights the superior fitting capability, generalization performance, and parameter efficiency of our proposed approach.

### D.3 Hyperparameter sensitivity analysis for the Rodrigues Network

**Table 12: Hyperparameter sensitivity analysis for the Rodrigues Network on motion prediction in Cartesian space with trainset size $=10^{5}$.**
| Variation | $C_{J}$ | $C_{L}$ | $B$ | Params (M) | Train MSE (1e-6) | Test MSE (1e-6) |
| --- | --- | --- | --- | --- | --- | --- |
| Default | 4 | 8 | 12 | 3.04 | 1.93 | 2.56 |
| Joint channels | 2 |  |  | 2.43 | 2.18 | 3.00 |
| Joint channels | 8 |  |  | 4.26 | 1.32 | 1.96 |
| Link channels |  | 4 |  | 1.19 | 3.64 | 4.35 |
| Link channels |  | 16 |  | 8.73 | 1.51 | 2.18 |
| Num blocks |  |  | 6 | 1.55 | 2.24 | 2.76 |
| Num blocks |  |  | 24 | 6.03 | 3.47 | 4.15 |

We investigate the robustness of our network’s performance with respect to variations in key architectural hyperparameters. Specifically, we focus on three core components of the Rodrigues Network: the number of joint channels ($C_{J}$), link channels ($C_{L}$), and the number of Rodrigues Blocks ($B$). Starting from the default configuration used in the main paper, we create six variants by either halving or doubling the value of each hyperparameter. We then evaluate these configurations on the motion prediction task to determine whether the network maintains stable performance across these variations. This analysis helps assess the sensitivity of our model to design choices and the general robustness of its architecture.

The results, summarized in Table [12](https://arxiv.org/html/2506.02618v1#A4.T12), show that increasing the number of joint channels from 2 to 8 consistently improves both training and test errors. A similar trend is observed for link channels, where increasing from 4 to 16 leads to better performance. In contrast, increasing the number of Rodrigues Blocks from 6 to 24 initially improves train and test errors, but further depth degrades them both, indicating optimization challenges in deeper configurations.
Overall, across all hyperparameter variations (up to 4× changes), performance remains within a reasonable range, demonstrating the robustness of our architecture. Furthermore, all variants significantly outperform the strongest baseline from the main paper, suggesting that our method is not only robust but also easily tunable for high performance.

## Appendix E Accelerating the Multi-Channel Rodrigues Operator with CUDA

The Multi-Channel Rodrigues Operator is a novel component in our architecture, and we found that implementing it directly in PyTorch led to suboptimal speed and memory efficiency. To address this, we developed a custom CUDA kernel that significantly accelerates both the forward and backward passes while reducing memory overhead.

In the forward pass, we assign each CUDA thread block to compute a single output channel over a sub-batch of link features. Each thread accumulates contributions from all input joint and link channels to produce the output feature. This design eliminates the need to explicitly compute and store intermediate tensors such as $U$ and $\bar{U}$ (see Equation 6 in the main paper), thereby saving memory bandwidth and improving runtime performance.

The backward pass adopts a similar strategy of accumulation to avoid intermediate computations. However, it additionally requires explicit gradient computations for both the input joint features and the Rodrigues Kernels, which we handle within the same memory-efficient framework.

This CUDA implementation significantly improves the efficiency of our experiments.
For instance, training a 12-block, 16-DoF Rodrigues Network composed solely of Rodrigues Layers, with 16 joint channels and 16 link channels (approximately 52 million parameters), for 100,000 iterations at a batch size of 1024, would take over 100 hours using our PyTorch implementation on a Quadro RTX 6000 GPU.
In contrast, using our CUDA kernel to compute the Multi-Channel Rodrigues Operator reduces the training time to approximately 15 hours (over 6x speed-up).

Nonetheless, we acknowledge that existing operators (such as graph convolutions, temporal convolutions, and multi-head attention) have benefited from extensive optimization over the years. Our current implementation does not yet reach that level of maturity. We will release the CUDA source code upon acceptance and hope it serves as a foundation for further research into optimized implementations of the Multi-Channel Rodrigues Operator.

## Peer Reviews

confidence: 3
contribution: 3
initial_rating: 8
presentation: 3
questions: -
soundness: 3
strengths: - Introduces a learnable generalization of Rodrigues rotation to embed articulated-kinematics structure into neural networks, giving a principled inductive bias for articulated action learning

- Presents a clear architecture (Rodrigues Layer, Joint Layer, self-attention) that mixes local kinematic structure with global information exchange

- Demonstrates strong improvements across domains: forward kinematics, Cartesian-to-joint motion prediction, robotic imitation learning, and 3D hand reconstruction

- Shows faster convergence and better data efficiency than standard architectures like MLPs, GCNs, and Transformers in modeling articulated motion
summary: This paper introduces RodriNet, a neural architecture that embeds articulated-kinematics structure directly into its computation by learning a generalized version of Rodrigues’ rotation formula. Instead of treating joint values as flat vectors like MLPs or Transformers, the proposed Neural Rodrigues Operator replaces fixed trigonometric coefficients in classical forward-kinematics equations with learnable weights, enabling structured message passing along a robot's kinematic tree. RodriNet stacks these operators with joint-update and attention layers, yielding a model that naturally learns hierarchical motion patterns. Experiments show large gains in fitting forward kinematics, predicting 3D joint trajectories from Cartesian motions, improving imitation-learning performance when used as the backbone in Diffusion Policy, and achieving state-of-the-art results on 3D hand pose estimation, all with fewer parameters. Overall, the work argues for architectural priors tailored to robot embodiment as a path to more efficient and generalizable action learning.
weaknesses: - Requires per-joint learnable parameters and tree-structured computation, which may introduce higher architectural complexity compared to simpler universal backbones

- No ablation studying how much each component (Rodrigues layer vs joint layer vs attention) contributes, leaving uncertainty about which parts drive performance

---

confidence: 4
contribution: 3
initial_rating: 8
presentation: 3
questions: Q1: What's the reason for not comparing it to the Denavit–Hartenberg formulation of kinematic structure? A quick search reveals that the Rodrigues formulation does have some technical advantages, but this seems the more appropriate comparison. 

Q2: What kind of features are provided to the Rodriguez layers? Do the transformations process non-kinematic related features in some way? How would these layers work with non-kinematic associated features, for instance? 

Q3: What about the author's current work that limits their representation to only spherical joints as opposed to translational joints? 

Q4: What distinguishes the Rodriguez layer from running system identification to find the model's kinematics?
soundness: 3
strengths: > Incorporating learnable kinematic models in robotics makes intuitive sense and seems to provide measurable improvements for applying machine learning on rigid body structures

> The paper is well written and presents the proposed model clearly. The diagrams help summarize the approach. 

> The experiments show compelling results to validate the efficacy of the Rodriguez layer for rigid body agents. The inclusion of both robots and animated characters helps demonstrate the model's potential across several settings of embodied agents.
summary: The authors investigate incorporating structural inductive biases from kinematic models in neural network design. They propose the Rodriguez layer, which combines projecting joint features through the learned Rodriguez operator and aggregating features globally with self-attention. Experimentally, the authors show that for problems where agents have a rigid body structure (e.g., forward kinematics or solving manipulation tasks with imitation learning), their model design performs substantially better than neural architectures that lack these inductive biases.
weaknesses: > Some aspects of the work are a bit obvious. The results for forward kinematics are not surprising, particularly as the Rodriguez matrix is one of two major approaches to modelling rigid-body kinematics—the other being Denavit-Hartenberg parameters. It could strengthen the paper to show how directly predicting the kinematic model's parameters compares in this problem. 

> Some results could benefit from statistical hypothesis testing to confirm whether the observed benefits are statistically significant (e.g. the success rates in Table 2). 

> Something that might be helpful would be to include results for failure cases. Presumably, there are data modalities or scenarios in which assuming a kinematic relation negatively impacts performance. 

Writing Opinions:

Line 063 - 068:  This paragraph can be cut with almost no loss of meaningful content to the paper. 

Related Work: Consider looking at cross-embodiment works as potential related work. These models also consider agent structure as necessary to model explicitly. 

[1] Gupta, Agrim, et al. "Metamorph: Learning universal controllers with transformers." arXiv preprint arXiv:2203.11931 (2022).

[2] Xiong, Zheng, Jacob Beck, and Shimon Whiteson. "Universal morphology control via contextual modulation." International Conference on Machine Learning. PMLR, 2023.

> Figure 3: We suggest replacing "backbone" with "architecture" unless these models are pretrained, which, in our experience, is where this term is used more frequently. 
 
>line 353: replace backbone with "architecture" 

> Figure 5: Provide more information in the caption on what is shown in the diagram.

---

confidence: 3
contribution: 3
initial_rating: 6
presentation: 3
questions: - Equivariance and structure validity: Does RodriNet satisfy any formal invariance or equivariance guarantees under changes of the base frame or re-rooting of the kinematic tree? Could initializing weights from analytical FK coefficients and applying soft orthogonality constraints improve SE(3) consistency while maintaining flexibility?

- Comparison to SE(3)-aware models: How does RodriNet perform against SE(3)-equivariant architectures (e.g., SE(3)-Transformer, E3NN) or MLP/Transformer baselines with Fourier angle encodings and identical kinematic-graph wiring? This would clarify whether the Neural Rodrigues Operator contributes beyond periodic encodings and structural locality.

- Cross-morphology generalization: Can the per-joint kernels be shared across joints of the same type or conditioned on metadata (axis, link transforms) to enable zero-shot transfer to unseen kinematic trees? A small multi-embodiment study would help test generality.

- Feature interpretation: Do the learned 4×4 “link features” converge toward SE(3)-like structures in practice? Visualizing the 3×3 rotation blocks (e.g., via polar decomposition) could reveal whether the network implicitly maintains orthogonality.

- Compute tradeoffs: RodriNet requires longer training time in the UR5 task. With the CUDA kernel acceleration, what is the effective wall-clock speedup compared to a Transformer at matched accuracy? Are there memory or scaling limits for large CL/CJ?

- Simulation-to-real transfer: In tasks where simulated gains are limited (e.g., PegInsertionSide, PlugCharger), how does RodriNet handle tactile or force-feedback inputs? Any early evidence of transfer performance on hardware would strengthen the empirical case.
soundness: 3
strengths: - Principled inductive bias: The formulation of a learnable operator directly from Rodrigues’ rotation rule is mathematically clean and physically grounded. By explicitly modeling the basis (1, sin θ, cos θ), RodriNet captures rotational structure while preserving the kinematic tree topology.

- Strong expressivity for kinematic mapping: On forward-kinematics fitting, RodriNet (Rodrigues Layers only) achieves orders-of-magnitude lower MSE and faster convergence than MLP, GCN, Transformer, or BoT baselines, demonstrating that the inductive bias aligns well with articulated geometry.

- Effective bridging of Cartesian and joint spaces: The UR5 trajectory prediction task is a well-controlled test of inverse kinematics. RodriNet achieves lower train/test MSE and maintains accuracy under reduced data regimes, showing improved data efficiency.

- Practical gains in imitation learning: Integrated into Diffusion Policy, RodriNet improves average success rate (0.61 vs. 0.58 for UNet-DP and 0.44 for Transformer-DP) across five ManiSkill tasks, with the largest gains on PickCube and StackCube. The global token cleanly handles gripper control.

- Cross-domain transfer: When replacing the transformer head in HaMeR, RodriNet attains consistent SOTA on FreiHAND with far fewer parameters (10.7 M vs. 39.5 M), indicating that the inductive bias generalizes beyond robotics.

- Thorough analysis and solid engineering: Ablations identify the Rodrigues Layer as the main contributor. Performance is stable under architectural scaling, and the custom CUDA implementation yields ~6× speedup in large configurations.
summary: This paper introduces Rodrigues Network (RodriNet), a neural architecture that explicitly embeds articulated kinematic structure into its design. The key component is the Neural Rodrigues Operator, derived from Rodrigues’ rotation formula by separating “state-dependent” (sin θ, cos θ) terms from “structure-dependent” coefficients and making the latter learnable. This transforms parent–child relations in a kinematic tree into a learnable message-passing rule. A multi-channel variant updates per-link 4×4 features from per-joint inputs using both left and right multiplications. Stacking these operators forms Rodrigues Layers, complemented by a Joint Layer and a Self-Attention Layer for global interaction, with an optional global token handling shared variables such as base pose or gripper state.
Empirically, RodriNet (1) achieves superior fitting accuracy on forward kinematics for the LEAP Hand and improved Cartesian-trajectory prediction on the UR5 arm, (2) boosts simulated success rates in Diffusion Policy across five ManiSkill tasks, and (3) improves single-image 3D hand reconstruction on FreiHAND by replacing the transformer head in HaMeR, yielding small but consistent SOTA gains with significantly fewer parameters.
weaknesses: - Lack of equivariance guarantees: The learnable operator applies unconstrained 4×4 left/right multiplications, offering no guarantee of SE(3) consistency or frame equivariance. Layer composition may not preserve physically valid transformations. The method serves as an inductive bias rather than a structured constraint; adding formal regularization (e.g., orthogonality or SE(3)-aware constraints) could strengthen the theoretical foundation.

- Limited novelty relative to existing basis encodings: The operator performs a learned linear combination over {1, sin θ, cos θ} (and quadratic quaternion terms for MANO), similar in spirit to Fourier or harmonic encodings. The absence of comparisons to SE(3)-equivariant architectures (e.g., SE(3)-Transformer, EGNN) or differentiable FK layers leaves it unclear whether gains arise from kinematic-tree structure or from providing the correct trigonometric basis.

- Restricted embodiment generalization: Kernels are defined per joint within a fixed kinematic tree, and experiments are conducted per robot (LEAP Hand, UR5, Franka). There is no evaluation of cross-morphology transfer, shared parameters across repeated structures, or conditioning on structural metadata for multi-embodiment generalization.

- No real-world validation: Results in imitation learning are limited to simulation. For contact-rich tasks such as PegInsertionSide or PlugCharger, performance gains are minimal, suggesting limited robustness to real sensor or actuation noise.

- Compute tradeoffs insufficiently analyzed: Training RodriNet for motion prediction requires longer runtime (2 h 22 m vs. ~1 h 20 m for Transformer/BoT), yet compute–accuracy tradeoffs are not fully discussed.

- Scope limitations: The model handles only rotational joints and omits link geometry or prismatic motion. Its focus on imitation learning rather than closed-loop control narrows applicability to contact-rich or mobile-manipulation domains.

---

confidence: 4
contribution: 2
initial_rating: 2
presentation: 3
questions: (I list all questions in the weakness section)
soundness: 2
strengths: 1. The overall presentation is very clear and straightforward, give enough background to understand the proposed method. 
2. The forward/inverse kinematic fitting experiments clearly demonstrate that the proposed architecture captures the inductive bias of articulated chains. Before that, I hesitated on this because GNN message passing is very different from sequential forward kinematic computation. 
3. The Rodriguez equation is an interesting perspective, and the authors conduct experiments over a diverse range of tasks. I very appreciate the authors' efforts.
summary: This paper proposes Rodrigues network (rodrinet), a novel graph neural network architecture that encodes the kinematics-aware inductive bias. The authors evaluate the proposed architecture on forward/inverse kinematic fitting, robot manipulation, and hand reconstruction tasks, which show that rodrinet outperforms or is on par with existing popular architectures.
weaknesses: My major concerns (questions) are two below, which also link to the core weaknesses of this paper in my opinion.

1. Why robot-kinematic-inductive bias matters? 
2. Does the performance gain from RodriNet justify the cost of an unorthodox design?

[Q1]: Most of the current robotic applications use task space control (i.e., predicting the end-effector pose), aka the eef pose, which can be easily translated to joint torques with an op-space controller. In this case, the model does not need to consider the robot's own kinematic structure and can leave that to controller. 

In this paper's experiment results, the most compelling results come from the forward/inverse kinematics fitting, which, however, has faster and flexible analytical solutions. On other tasks, the benefits are very marginal. Like in the robot manipulation experiments, even when Rodrinet outperforms the diffusion policy on PickCube and StackCube, I am not entirely convinced because these two tasks should not heavily depend on the robot kinematic chain. In the hand reconstruction task, the authors explicitly mention that modifications are made to RodriNet to suit MANO's configuration space (for example, fitting the hand shape beta parameters, which I assume are not tied to joint/link). This even make the results look less convincing. 


[Q2] Even if robot-kinematic-inductive bias may not matter in every task, having them for free does not hurt. However, this proposed architecture, which looks like another GNN variant, has limitations and cannot be used for free. 

For example, all robot manipulation experiments are conducted in state space, rather than using visual inputs. This is a critical constraint. Additionally, using a GNN instead of a general transformer could make it challenging to interpolate with other transformer-based backbones. Using joint space and not task space actions can also make the prediction less visually grounded. In sum, these constraints outweigh the marginal performance gain.

A side note: joint controllers are widely used in dexterous hands; maybe this technique can find its place in hand manipulation tasks, instead of parallel grippers.


Decision: Accept (Oral)