Scaling Manipulation Learning with Visual Kinematic Chain Prediction
Authors: Xinyu Zhang
Yuhan Liu
Haonan Chang
Abdeslam Boularias
Venue: CoRL 2024 (2024)
Keywords: Multi-Task Robot Learning
Manipulation
TL;DR: We propose visual kinematics transformer, a convolution-free architecture trained with forecasting kinematic structures through optimal point set matching, and demonstrate its superior performance as a general agent in diverse environments.

Abstract: Learning general-purpose models from diverse datasets has achieved great success in machine learning. In robotics, however, existing methods in multi-task learning are typically constrained to a single robot and workspace, while recent work such as RT-X requires a non-trivial action normalization procedure to manually bridge the gap between different action spaces in diverse environments. In this paper, we propose the visual kinematics chain as a precise and universal representation of quasi-static actions for robot learning over diverse environments, which requires no manual adjustment since the visual kinematic chains can be automatically obtained from the robot’s model and camera parameters. We propose the Visual Kinematics Transformer (VKT),  a convolution-free architecture that supports an arbitrary number of camera viewpoints, and that is trained with a single objective of forecasting kinematic structures through optimal point-set matching. We demonstrate the superior performance of VKT over BC transformers as a general agent on Calvin, RLBench, ALOHA, Open-X, and real robot manipulation tasks. Video demonstrations and source code can be found at https://mlzxy.github.io/visual-kinetic-chain.

Scaling Manipulation Learning with Visual Kinematic Chain Prediction
Xinyu Zhang
Yuhan Liu [xz653, y11834, hc856, ab1544) @rutgers.edu
Haonan Chang
Abdeslam Boularias
Rutgers University
Learning general-purpose models from diverse datasets has achieved great success in machine learning. In robotics, however, existing methods multi-task learning are typically constrained a single robot and workspace, while recent work such as RT-X requires non-trivial action normalization pro- cedure to manually bridge the gap between different action spaces in diverse en- vironments. In this paper, we propose the visual kinematics chain as a precise and universal representation of quasi-static actions for robot learning over diverse environments, which requires no manual adjustment since the visual kinematic chains can be automatically obtained from the robot's model and camera param- eters. We propose the Visual Kinematics Transformer (VKT), a convolution free architecture that supports arbitrary number of camera viewpoints, and that is trained with single objective of forecasting kinematic structures through optimal point-set matching. We demonstrate the superior performance of VKT over BC transformers as general agent on Calvin, RLBench, ALOHA, Open-X, and real robot manipulation tasks. Video demonstrations and source code can be found at https://mlzxx: .gi tatitb..ooviiiiii io/visuall http://isaal-iieiii--heeiie -inetic-chainn
Abstract:
Keywords: Multi-Task Robot Learning, Manipulation
Introduction
There are numerous techniques in machine learning and computer vision that can successfully learn a single general-purpose model from multiple diverse datasets [1]. In robotics, however, despite the recent advances in multi-task learning that enable single policy to perform various tasks through imitation learning with language instructions [2. 3], these methods are typically constrained single robot and workspace. Some recent work, such RT-X [4, 5], leverages large vision- language model (VLM) directly train policies with Behavioral Cloning (BC) on the Open-X Embodiment [5], a collection of datasets crowd-sourced from various environments and robots. However, these techniques require a non-trivial action normalization procedure to bridge the gap between the different action spaces in diverse sctups, such as end-effector poses, joint positions, and velocities in various world frames. This manual engineering procedure currently custom custom-designed designed for each training dataset, which affects the generalization and interpretability of these models.
Therefore, a key question is: can we find action representation that is precise and universal for various setups and robots? To this end, we propose the visual kinematic chain, which is the projection of the robot's high- high-dimensional kinematic structure into the image plane as pixels. Instead of predicting the low-level robot actions, we propose to learn and visually forecast the movement of robots' kinematic chains in the image plane. This visual approach provides unified action representation for different robots, and requires manual adjustment since the visual kinematic chains can be automatically obtained from the robot's model and camera parameters.
To forecast the kinematic structures of various robots, we render the kinematic chains into point sets by sampling points along links and performing optimal matching [6] to minimize the earth moving
8th Conference Robot Learning (CoRL. 2024), Munich, Germany.

---

Diverse Robots and Enviromments
Visual Kinematic Chain as Universal Action Representation
Righ
Different workspaces, world frames, camera viewpoints, robot configurations.
Deploying to a Specifie Environment
S Chain Forecasting with Visual Kinematies Transformer
RBG Images
Language Instruction
Kinematics Transtormer
Visual Kinematics Transformer
RBG Images
Tiny Head
Open top drower cups
Positions'
EEP Pese Velacity,
Language Instruction
Commands
Figure 1: Overview of the proposed framework. We use visual kinematic chain universal action representation across diverse robots and setups. We propose the Visual Kinematics Transformer (VKT), an architecture based solely on attention layers, which predicts the future movements of the visual kinematic chain in images from multiple viewpoints, Our VKT trained with the earth- moving distance as the single learning objective without knowing any low-level robot states actions. When deployed a specific environment, we freeze the VKT as a backbonc and attach a tiny head to project the VKT output to actual robot commands.
distance [7, 8] between the predicted point sets and ground-truth kinematic structures. Further, propose the Visual Kinematics Transformer (VKT), convolution-fi architecture that is solely based on the attention mechanism in the RGB space, which supports arbitrary number of camera viewpoints. Our VKT is trained with single objective of forecasting kinematic structures through optimal point set matching without seeing any low-level robot actions. VKT is deployed to specific environment by simply training tiny head while freezing the backbone. VKT demonstrates superior performance over BC transformers as general agent on Calvin (average length 1.46 vs 0.48), RLBench (success rate 61.7% vs 24.3%), ALOHA (success rate 55.3% 12%). Open-X, and real robot manipulation tasks.
To summarize, our contributions are threefold. (1) We propose the visual kinematic chain as precise and universal representation of quasi-static actions for learning from diverse robot configu- rations. (2) We propose VKT, a convolution-free architecture that supports an arbitrary number of camera viewpoints and is trained with single objective forecasting kinematic structures through optimal point matching. (3) We present thorough empirical study of the performance of VKT specialized and general agents over diverse set of -andiiee-ooditioneed tasks and environments.
2 Related Work
Robot Learning from Diverse Environments. Learning single universal multi-task policy on various robots, cameras, and task configurations is challenging problem that receiving increas- ing attention [9]. One recent strategy solving this problem consists in training network that takes input robot's ss kinematic structure [10, 11]. For instance, MetaMorph [12] tokenizes the

---

robot's kinematic tree, which is then used as transformer prompt to produce per-joint action. How- ever, these methods are only evaluated in simple walking environments of Mujoco [13] without any variations in the task, the camera viewpoint, and the workspace. Several recent works have also looked into leveraging large vision-language model (VLM) to directly train robots, through be- havior cloning, on collection of crowd-soureed datasets such as Open-X Embodiment [14, 15, 5]. Despite their impressive results, these methods require non-trivial action normalization procedure to reduce the gap between action spaces such as end-effector poses, joint positions, and velocities in various world frames. However, this manual engineering procedure is custom- designdd for each training dataset, which severely affects the generalization and interpretability of these models. In contrast, our method directly learns and visually predicts the movement of robots' kinematic chains.
Transformers for Manipulation Learning. Transformers are widely used in manipulation learn- ing [4, 16]. However, existing architectures require fixed number of camera viewpoints, which is not guaranteed when learning is performed across various environments. Both RT-1/2 and RT-X select one canonical view from each Open-X dataset [5, 4, 17J. RVT [18] supports multiple but pre- defined viewpoints. PerAct [3] trains transformers over point-clouds and is not limited by camera viewpoints. But methods based on point clouds still suffer from the lack of datasets with depth infor- mation, as well as scale discrepancies [19, 20). In contrast, our visual kinematics transformer (VKT) convolution-free architecture that solely based on attention mechanisms in the RGB space, without the need for depth data. VKT also supports an arbitrary number of camera viewpoints.
Intermediate Action Representations. Instead of only using low-level robot commands, incorpo- rating intermediate actions into control policies has been shown to increase data efficiency [21]. J. RT- H is a hierarchical policy that predicts language motions such as "move arm forward" [22]. SWIM uses affordance maps to unify grasping actions of humans and robots [21]. General Flow employs labeling process translate end-effector motions into point-cloud movements [23]. IMOP [24] uses
invariant regions to describe key manipulation poses. RT-Trajectory predicts the future 2D trajectory of the end-effector's ss center in given image [25]. In contrast, our method forecasts the movements of the entire kinematic chain in multiple camera viewpoints. This representation provides a visually grounded action abstraction, while capturing detailed and small motions at the same time.
Recently, designing universal action representation has received growing attention. ATM [26] de- signs track-guided policy that accepts set of point trajectories the image plane as guidance for predicting robot actions, which enables policy learning from both robot and human videos. How- ever, compared to our visual kinematic chain, point trajectory is ambiguous because trajectories are not guaranteed to correlate with the desired robot actions. UniPi [27] solves robot tasks by first generating videos and then deriving robot actions from the video generated. Yet, using videos as an action representation inefficient because video generators are orders larger than common policy networks [28. 29]. Moreover, UniPi [27] is evaluated on single planar environment [30] because deriving 3D actions or joint positions from videos is not trivial.
3 Visual Kinematic Chain Forecasting
We consider the problem of learning a single univer- sal multi-task policy from diverse robots and workspaces through a unified action space. Our key insight lies in the use of the visual kinematic chain, which is the projection of the robot's high-dimensional kinematic structure into set of pixels in the image 2D plane. The visual kinematic chain can be analytically generated with camera param- eters and robot models without any manual engineering. We show that this simple representation allows us to train a single visual language-conditiones pol- icy over multiple environments. The VLM policy is trained with a single objective of forecasting the future visual kinematic chain movements without knowing any low-level robot actions. Our proposed transformer architecture is built entirely with attention layers and can predict consistent
Sample
Points
Figure 2: Rendering kinematic chain as set of pixels in RGB image
3

---

T X'X
Multi-Viow Dual Attention Block
Query
Kinematics Takens
other cameras
Groundrrut
"Open the hop dhawer"
Text +
Query
Point Sets Matching
Self Altn
Giross
Cross
Tokknn
Sample Points
Encoder
MLP
Text
Query Tokens
Vision
Vison
Cross Altin
Croas Attin
Token
M
Encoder
Miniizze Earth Moving Distance
Camera
Vision Tokens
other cameras
Figure 3: Overview of our proposed visual kinematics transformer (VKT). For each camera input, we encode the language instruction and RGB image as text-and-vision tokens with CLIP [32]. Then, we concatenate the text tokens and the kinematics tokens as query tokens. The kinematics tokens are learned parameters. Next, the query and vision tokens are interweaved with a sequence of our proposed multi-viww dual attention block. For each block, the query tokens first updated with self-attention (orange). Then, cross attention is applied with the query tokens as queries and the vision tokens keys and values (green), A cross-attention layer updates queries with keys and values, we use - to denote queries and to denote keys and values. Then, both the query tokens and the vision tokens are updated through viewpoints (blue). Next, the vision tokens are updated by cross
attentins with tokens from other camera cross-attention attention with query tokens as keys and values (green). Finally, the T' kinematics tokens are projected into T point representing the visual kinematic chain in the current and the future 7 - 1 steps. The predicted point-sets are optimized through point-set matching with the ground-truth, shown in Equation 1.
through an MIP,
multi-view chains from an arbitrary number camera viewpoints. To deploy the multi-task policy in specific environment, one simply needs freeze the transformer and train a tiny head project the actions into actual robot commands. An overview of our framework illustrated in Figure 1.
Fitting any Structure with Point-Set Matching. The kinematic structures of various robots differ significantly. To forecast different structures, we propose to render them to point sets by uniformly sampling points along links, as illustrated in Figure 2. Let P [p denote the point-set rendered from the ground-truth configuration of kinematic chain, and Q = (q
1 denote the point-set predicted by VKT, where pi R². q, E R². We first compute the pairwise euclidean distance matrix CE RMN, where Cj =Il P; --- q, ll. We then minimize the earth-moving distance EMD(P, Q) Eij Cij. where y" is stochastic assignment matrix that matches points in P and Q. We find y" using the Kankhorn-Knop algorithm to solve the following optimization problem online for each mini-batch [31.8] 8).
= arg min Mat Enj YiJCi Cij. s.t. yl = 1;y 1 1:y 2 0.
(1)
Visual Kinematics Transformer (VKT). Our VKT network accepts language instruction and an RGB image input and predicts sequence of T point-sets. The dimensions of the kinematics tokens are 7' x D, where D) denotes the embedding size, and each token predicts one point-set. The first point-set in the predicted sequence describes the current state of the kinematic chain in the given RGB image. The remaining point-sets are predictions the future states of the kinematic chain in the next each point-set the ground-truth kinematic structure at the corresponding time-step by minimizing the earth-moving distance. The same process is applied multiple RGB images taken from differ- ent viewpoints. The network is also trained to make the future point-sets predicted from different viewpoints consistent with each other. The kinematics tokens are concatenated with the text tokens as query tokens. The concatenated query and vision tokens are forwarded to our proposed multi- view dual attention block, to interweave the information between query and vision tokens and across multiple camera viewpoints.
time-steps, i.e., future robot movements. The network is trained to match
The multi-vies view dual attention block performs dual attentions between query and vision tokens within each single view, indicated by the green blocks in Figure 3, and multi-view attentions for query

---

and vision tokens independently but across multiple camera viewpoints, shown as the blue blocks in Figure 3. This dual attention enables the query token to learn the visual kinematic chain movements. The multi-view attention enables a spatially-conse consistent prediction across multiple viewpoints. The output kinematics tokens are projected to T point-sets using a small MLP network. Each point-set contains N 2D points on the imput image. The VKT architecture is illustrated Figure 3.
Compared to existing robot learning transformers [18, 17]. our proposed VKT is convolution- free and built solely with attention layers. Therefore, VKT supports an arbitrary number of camera view- points, which an important advantage as camera setups are different among various environments.
Deploying to a Specific Environment. We freeze the trained VKT, drop the point-set prediction branch, and only use the kinematics tokens. We ap- ply a 1D convolution to project the 7' kinematics to- kens into low-level robot actions for the next 7 time- steps. Unlike the VKT backbone which is trained from multiple environments, the ID convolution head is trained separately for each environment. The corresponding proposed architecture is illustrated in Figure 4.
Visual Kinematics
Convid Mead
Low-level Roboot actions slaps.
Transformer
"Lift the block to Target"
Kinematics Tokens
Figure Projecting the kinematics tokens into robot actions with 1D convolution.
Table 1: Comparison of Behavioral Cloning Transformer (BCT) and our Visual Kinematics Transformer (VKT) on Calvin, RLBench, and ALOHA. BCT and VKT share the same archi- tecture except that BCT directly predicts the robot's s low-level actions. VKT outperforms BCT in isolated environments (specialized). When trained in all environments (general), VKT retains competitive performance but BCT suffers drastic performance drop.
Specialized Agent
General Agent
Environment
Metric
BCT
VKT (Ours)
BCT
VKT (Ours)
Calvin [33]
Avg. Length 1
1.36
1.58
0.48
1.46
RLBench [34]
36.4
55.5
24.3
61.7
Success Rate (%) f
ALOHA [35]
60.7
63.3
12
55.3
4 Experiments
We evaluate VKT on Calvin [33], RLBench [34], and ALOHA [35]. Calvin and RLBencc are stan- dard benchmarks in language-conditioned multi-task manipulation learning. ALOHA is a bimanual manipulation environment. Further, we evaluate VKT in a subset of Open-X Embodiment [4] and real robot experiments. We aim to answer the following questions: (1) Can visual kinematics fore- casting improve specialized agents in each environment?
(2) Can VKT serve as a strong general agent multiple environments? (3) Can VKT be efficiently trained with real-world demonstrations and solve manipulation tasks with real robots?
Setup. We adopt the setting of RT-X [5] for imitation learning from RGB images without depth. RL- Bench contains various task categories, such as pick-and-place tool use, high-precision operations, screwing, tasks with visual occlusion, and long-term manipulation. RLBench provides five RGBD cameras. We use the RGB signal from the front, left, and right cameras and exclude the cameras installed on the shoulder and gripper of the robot because they do not take images of the robot. We choose 17 tasks based camera visibility and the task categorization of Hiveformer [36]. We use 100 recorded trajectories per task for training, evaluate the trained agent on 25 independent trials, and report the average success rate. Calvin requires policies to accomplish multiple tasks sequen- tially in each episode. We use the training set and evaluation procedure of the Calvin environment and report the average number of successful tasks per episode (average length) [33, 37]. ALOHA requires high-frequency control of two robot arms through joint positions solve tasks collabo- the
ratively. We chose the cube transfer task that requires picking up the cube and transferring other anm. We follow the convention 38] to report the metrics Calvin and ALOHA with three independent runs of 1000 and 100 episodes, respectively. We report the success rate of RLBench
5

---

Prediction
Result
Prediction
Result
Prediction
Result
Rotate the pink black the right
Move the steak out of grill
Pick the cute with right arm and hand it to left arm
Slide the cabinet door to the left
Push the block with stick to target (different camera viewpoints)
Figure 5: Predicted visual kinematic chains returned by our VKT for different robots in Calyin (left), RLBench (right), and ALOHA (top right). The predicted kinematic chain of the current frame is colored in red and the forecast one for the next time-step is in blue.
with five independent runs of 25 episodes [3]. Examples of Calvin, RLBench, and ALOHA tasks shown in Figure 5. Additional visualizations provided in the supplementary material.
Implementation Details. We use CLIP ViT-B/16 [32] as text and vision encoders. We resize all RGB inputs to 224 x 224. We use 8 multi-view dual attention blocks (L = 8) and predict visual kinematic chains for a horizon of 20 time-steps (7 = 20) with 100 points (N 1000 at each step and set size to 512 in all experiments. RLBench has 3 camera viewpoints. ALOHA and Calvin have single viewpoint each. During training, each minibatch consists of samples from all these three environments. Moreover, the three viewpoints of RLBench are randomly shuffled for every sample. The number of parameters is 14.8M in total and 32.5M without the ViT-B vision encoder. We adopt LoRA finetuning [39, 40] on the vision encoder. The original weights of the ViT-B vision encoder are kept frozen. The convolution head has only 96K parameters, less than 0.1% the entire network. The structure of the convolution head illustrated in Figure A4. We train the VKT for 50000 steps with a batch size of 96, which takes roughly 22 hours on a dual NVIDIA A6000 GPU server. The inference throughput is 0.02 secs/img NVIDIA 3060 GPU.
4.1 Learning from Multiple Environments
4.1.1
Simulation
Baselines. We compare the performance of our visual kinematics transformer (VKT) with a behav- ioral cloning transformer (BCT). The BCT is directly trained to predict the robot actions such as 6-DoF end-effector poses and joint values. In contrast, VKT is trained to forecast visual kinematic chains first, then project the kinematics tokens robot actions using convolution head with the backbone frozen, as shown in Figure 4. Note that the BCT shares the same neural architecture with VKT, including the convolution head for each specific environment. The specialized agent is trained each environment separately. The general agent is trained using demonstrations from all environ- ments. We use the 6-DoF end-effector pose and binary gripper states as the robot action space for both Calvin and RLBench, and the radian values of the 14 robot joints for ALOHA. We also com- pare the performance of our VKT with existing environment-speene -specffe methods. Calvin, recent methods go beyond our imitation learning setup and adopt model learning for long-horizon latent planning. Therefore, we choose the goal-condition behavior cloning (GCBC) baseline from [41]. In RLBench, choose PerAct [3] and retrain the model with the same camera setup as VKT. In ALOHA, we choose the action chunking transformer (ACT) from the origin ALOHA paper[35].
Results. Table and A3 summarize the comparison between VKT, BCT, and environment-spe specific methods. By forecasting the visual kinematic chains, VKT outperforms BCT in both specialized and general agents. This indicates that the kinematics tokens capture sufficient information about the robot's action, despite not seeing any low-level robot actions. When trained in all environments, the general BCT agent suffers catastrophic performance drop (1.36 0.48, 36.4% - 24.3%, 55.3% + 12%). This indicates the challenge of policy learning from different action spaces. In

---

Prediction
Transformer (Ours)
0.25
Put blue the cup.
Pick glass
0:10
005
D00
Berkeley
Edan
Fractal
TOTO
Relative Precision
Close the drower
Pour the cup
Figure 6: Results on Open-X Embodiment. Visual kinematic chains of different robots predicted by our VKT in Open-X Embodiment (left). The right figure compares the relative precision of BCT and VKT the four selected datasets. BCT is randomly initialized and VKT is initialized from weights that are pretrained with visual kinematics forecasting. Relative precision is defined as an inverse of LI error robot actions, which is detailed in Section 4.1.2.
contrast, the general VKT agent has much smaller performance drop in Calvin (1.58 - 1.46) and ALOHA (63.3% + 55.3%) ) but its performance increases (55.5% - 61.7%) in RLBench. This indicates that by learning actions in the image planes, training becomes more robust to distribution shifts. This is further verified by our real-robot experiments in Section 4.2.
Failure Case Analysis. We list the per-task success rates of RLBench and Calvin in Table A4 and A5. Overall, VKT outperforms BCT. We empirically find that VKT is more robust to changes in object layout because of the visually grounded action space. However, VKT has a lower success rate in the "turn tap" task. This task requires some specific end-effector rotations which involve large variations in the Euler angles the end-effector but small changes in the visual kinematic chain from all camera viewpoints, as illustrated in the fourth row of Figure A5. Figure A5 summarizes the four categories of failure cases of VKT.
4.1.2 Open-X Embodiment
Setup. Open-X Embodiment (Open-X) [5] is collection of video demonstrations of real-world manipulation tasks crowd-sourced from diverse environments. Since Open-X does not provide robot models such as URDF, and camera parameters are not available in most datasets, we select four datasets from Open-X and manually annotate the robot joint positions from 80 demonstration videos with 1, 1359 frames. future work, it is possible to apply robot keypoint detectors [42, 43] to extract kinematic chain without URDFs. Using the annotated videos, we train single VKT for the four datasets. Then, we train the network predict low-level robot actions as in Figure 4. However, instead of freezing the VKT, continue finetuning the entire network with the trained VKT as initial weights. For the BCT, we use the same training setup but randomly initialize the network.
Figure shows qualitative examples of the predicted kinematic chains (left) and compares the relative precision BCT and VKT on the validation set (right). Let Lvkr and Laer denote the error of VKT and BCT, the relative precision of VKT is computed as [Likr Lxel/ Likr Figure shows that VKT learns to predict kinematic chains from diverse real-world data, and visual kinematics forecasting serve pre-training objective improve imitation learning.
Results.
4.2 Real Robot Experiments
Setup. We evaluate our VKT in real-world language-conditioned pick-and-place task. The task input includes an RGB image and text instructions such as "put the cup into the blue box". We adopt nine YCB objects such as balls, chips, and two boxes as place targets. Examples of this task and the forecasted kinematic chains are shown Figure/ We implement scripted policy using a DINOv2-based object detector [44] collect 90 demonstrations as training data, with only 5 demonstrations for each object-containet pair. We automatically compute the ground-truth visual kinematic chain with the robot's 's URDF and the calibrated camera parameters. We use Kuka LBR

---

Prediction
Result
Prediction
Result
Table 2: Success rates of VKT
for solving the real-world language- conditioned pick-and-place task.
Object
BCT
VKT (Ours)
Put orange red the brick blue box
Overall (%)
41.5
69.2
Crackers
3/7
4/7
Cup
3/7
4/7
Ball
3/8
7/8
Chips
1/7
3/7
Brick
Tomato Can
4/7
6/7
Put tomato can into red box
crackers blue box
Mustard
0/7
4/7
Sugar Box
4/7
6/7
Figure 7: Visual chains predicted by VKT in language- conditioned pick-and-place task with a real robot. The chains of pick and place actions are in red and blue, respectively.
Orange
4/7
4/7
iiwa robot. For the convolution head, we train the model to directly predict the pick and the place poses instead predicting the full trajectory for simplicity, and use Movelt [45] for path generation.
Results. Table 2 shows the success rates of VKT in the real-world task. The overall success rate is the average of 65 independent the same training and evaluation setup with VKT, but VKT outperforms BCT by significant margin. Further, we discover that visual kinematic forecasting enables the use of 2D image augmentation for manipulation learning because the action space also resides in image planes, which important to prevent overfitting in our experiment due to the small training set. We randomly augment the images through size scaling, translation, rotation, and perspective transform. In comparison, existing manipulation learning augmentations are applied in the 3D space [46], which requires depth that less available than RGB data, and has fewer categories than 2D image augmentations [47].
with different object layouts. The BC Transformer (BCT) shares
Details of analyses and ablation studies are included in Appendix A.1 due to space limit.
Discussion and Future Possibilities
We have shown that visual kinematics forecasting of quasi-static robot movements can be a building block technique for developing general agents across diverse robot learning environments. In the following, discuss the limitations and potential future directions.
3D Kinematic Chain. In Figure A5, VKT shows an inferior performance when the desired robot actions only produce small visual changes in the kinematic chain. This discrepancy is caused by the ambiguity the 3D to 2D projection and prohibits VKT from more dexterous or precise tasks. One possible solution is to project the 2D kinematic chain back normalized 3D space with virtual camera. This ensures the 3D space is consistent across different environments, provides the necessary precision in the 3DD space, and still be compatible with 2D kinematic chain method.
General Visual Encoding of Robot Actions. From an encoding perspective, the visual kinematic chain can be viewed as a visual encoding strategy for robot actions. The benefit visual encoding is that it transforms robot learning tasks into vision tasks, where more generalized and robust solutions are often discovered. One possible future direction explore more visual encoding strategies with better learnability, higher precision, and reliable even analytical translation into robot actions.
Beyond Universal Action Representation. Despite our advancements in unifying the action spaces across diverse environments, there still a considerable gap in performance between our general agent VKT and those environment-specifie state-of the-art solutions. Different environments not only differ greatly action spaces more so in types of tasks and the required roboties skills. These skills acquired separately by those environment-specific solutions. It potentially promising future direction not just focus universal action representation, but designing uni- versal learning architecture that can acquire various robot skills for diverse environments.

---

Acknowledgement
This work is partially supported by NSF awards 1846043 and 2132972.
References
[1] R. Bommasani, D. A. Hudson, E. Adeli, R. Altman, S. Arora, S. von Arx, M. S. Bemsstenn, Bohg, A. Bosselut, E. Brunskill, et al. On the opportunities and risks of foundation models. arXiv preprint arXiv:2108.0723 2021.
[2] M. Shridhar, L. Manuelli, and D. Fox. Cliport: What and where pathways for robotic manipu- lation. In Conference on robot learning, pages 894-906. PMLR, 2022.
[3] M. Shridhar, L. Manuelli, and D. Fox. Perceiver-actor: A multi-task transformer for robotic manipulation. In Conference on Robot Learning, pages 785-799, PMLR, 2023.
[4] A. Brohan, N. Brown, Carbajal, Y. Chebotar, X. Chen, Choromanski, T. Ding, D. Driess,
A. Dubey, Finn, al. Rt-2: Vision-language-actionn models transfer web knowledge to robotic control. arXiv preprint arXhv:2307.15816 2023.
[5] A. Padalkar, A. Pooley, A. Jain, A. Bewley, A. Herzog, A. Irpan, A. Khazatsky, A. Rai,
A. Singh, A. Brohan, al. Open x-embodiment: Robotic learning datasets and rt-x mod- els. 2023.
[6] R. Flamary, N. Courty, A. Gramfort, M. Z. Alaya, A. Boisbunon, Chambon, L. Chapel, A. Corenflos, Fatras, N. Fournier, L. Gautheron, N. T. Gayraud, H. Janati, Rakotoma- monjy, I. Redko, A. Rolet, A. Schutz, V. Seguy, D. J. Sutherland, R. Tavenard, A. Tong, and T. Vayer. Pot: Python optimal transport. Journal of Machine Learning Research, 22(78):1-8, 2021. URL http://jmlr arggppapers//22/222 20122000-5111 1hmml.
[7] Y. Rubner, C. Tomasi, and L. Guibas. The earth mover's distance as metric for image retrieval. International journal of computer vision, 40:99-121, 2000.
[8] J. Feydy. Geometric data analysis, beyond convolutions. Applied Mathematics, 2020.
[9] S. Dasari, F. Ebert, S. Tian, S. Nair, B. Bucher, K. Schmeckpeper, S. Singh, S. Levine, and
C. Finn. Robonet: Large-scale multi-robot learning. In L. P. Kaelbling, D. Kragic, and K. Sugiura, editors, Proceedings of the Conference on Robot Learning, volume 100 of Pro- ceedings of Machine Learning Research, pages 885-897. PMLR, 30 Oct-01 Nov 2020. URL https:/pproceedingg mlr, prress/1100ddasart// http:///........eees/ooo/////eee
html.
[10] W. Huang. I. Mordatch, and D. Pathak. One policy to control them all: shared modular policies for agent-agnostic control. In Proceedings of the 37th International Conference on Machine Learning, pages 4455-4464, 2020.
[11] V. Kurin, M. Igl, T. Rocktäschel, W. Boehmer, and S. Whiteson. My body is a cage: the role of morphology in graph-based incompatible control. In International Conference on Learning Representations, 2020.
[12] A. Gupta, L. Fan, S. Ganguli, and L.. Fei-Fei. Metamorph: Learning universal controllers with transformers, In International Conference on Learning Representations, 2021.
[13] E. Todorov, T. Erez, and Y. Tassa. Mujoco: A physics engine for model-based control. In 20/2 IEEE/RSJ international conference intelligent robots and systems, pages 5026-5033. IEEE, 2012.
[14] S. Reed, K. Zolna, E. Parisotto, S. G. Colmenarejo, A. Novikow, G. Barth-Maron, M. Gimenez, Y. Sulsky, J. Kay, J. T. Springenberg, et al. A generalist agent. arXiv preprint 0rXiv:2205.06175, 2022.

---

[15] O. M. Team, D. Ghosh, H. Walke, K. Pertsch, K. Black, O. Mees, S. Dasari, J. Hejna, C. Xu,
J. Luo, et al. Octo: An open-source generalist robot policy. In Proceedings of Robotics: Science and Systems, Delft, Netherlands, 2023.
[16] C. Chi, S. Feng, Y. Du, Z. Xu, E. Cousineau, B. Burchfiel, and S. Song. Diffusion policy: Visuomotor policy learning via action diffusion. arXiv preprint arXiv:2303.04133, 2023.
[17] A. Brohan, N. Brown, J. Carbajal, Y. Chebotar, J. Dabis, C. Finn, K. Gopalakrishnan, K. Haus- man, A. Herzog, J. Hsu, et al. Rt-1: Robotics transformer for real-world control at scale. arXiv 2022.
preprini wrXiv:22/2.068//
[18] A. Goyal, J. Xu, Y. Guo, V. Blukis, Y.-W. Chao, and D. Fox. Rvt: Robotic view transformer for 3d object manipulation. arXiv preprint arXiv:2306. /4896, 2023.
[19] C. Xu, S. Yang. Galanti, B. Wu, X. Yue, B. Zhai, W. Zhan, Vajda, K. Keutzer, and M. Tomizuka. Image2point: 3d point-cloud understanding with 2d image pretrained models. In European Conference on Computer Vision, pages 638 638-656. Springer, 2022.
[20] H. Zhu, H. Yang, X. Wu, D. Huang, S. Zhang, X. He, T. He, H. Zhao, C. Shen, Y. Qiao, et al. Ponderv2: Pave the way for 3d foundataion model with a universal pre-training paradigm. arXiv preprint arXiv:2310.085560 2023.
[21] R. Mendonca, S. Bahl, and D. Pathak. Structured world models from human videos. arXiv preprint ar3iv:2308. 10901, 2023.
[22] S. Belkhale, T. Ding, T. Xiao, P. Sermanet, Q. Vuong, J. Tompson, Y. Chebotar, D. Dwibedi, and D. Sadigh. Rt-h: Action hierarchies using language. arXiv preprint arXiv:2403.0182. 2024.
[23] C. Yuan, C. Wen, T. Zhang, and Y. Gao. General flow foundation affordance for scalable robot learning. arXiv preprint arXiv:2401. 2024.
[24] X. Zhang and Boularias. One-shot imitation learning with invariance matching for robotic manipulation. In Proceedings of Robotics: Science and Systems, Delft, Netherlands, 2024.
[25] Gu, S. Kirmani, P. Wohlhart, Y. Lu, M. G. Arenas, K. Rao, W. Yu, C. Fu, K. Gopalakrishnan, Z. Xu, al. Rt-trajectory: Robotic task generalization via hindsight trajectory sketches. arXiv preprint arXiv:2311.01977 2023.
[26] C. Wen, X. Lin, J. So, K. Chen, Q. Dou, Y. Gao, and P. Abbeel. Any-poim trajectory modeling for policy learning. arXiv preprint arXiv:2401 000000000055000 2023.
[27] Y. Du, S. Yang. B. Dai, H. Dai, O. Nachum, J. Tenenbaum, D. Schuurmans, and P. Abbeel. Learning universal policies via text-guided video generation. Advances in Neural Information Processing Systems, 36, 2024.
[28] Z. Zheng, X. Peng, T. Yang, C. Shen, S. Li, H. Liu, Y. Zhou, T. Li, and Y. You. Open-sora: Democratizing efficient video production for all, March 2024. URL Pecaatecch/pee.. Sora.
[29] A. Mandlekar, D. Xu, J. Wong, S. Nasiriany, C. Wang, R. Kulkarni, L. Fei-Fei, S. Savarese, Y. Zhu, and R. Martin-Martin. What matters in learning from offline human demonstrations for robot manipulation. In arXiv preprint arxiv:2 03.9998 2021.
[30] A. Zeng, P. Florence, J. Tompson, S. Welker, J. Chien, M. Attarian, T. Armstrong, I. Krasin, D. Duong, V. Sindhwani, et al. Transporter networks: Rearranging the visual world for robotic manipulation. In Conference on Robot Learning, pages 726-747, PMLR, 2021.
[31] M. Cuturi. Sinkhom distances: Lightspeed computation of optimal transport. Advances in neural information processing systems, 26, 2013.
10

---

[32] A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, Mishkin, J. Clark, et al. Learning transferable visual models from natural language supervi- sion. In International Conference on Machine Learning, pages 8748-8763. PMLR, 2021.
[33] O. Mees, L. Hermann, E. Rosete-Beas, and W. Burgard. Calvin: A benchmark for language- conditioned policy learning for long-horizon robot manipulation tasks. IEEE Robotics and Automation Letters (RA-L), 7(3):7327-7334, 2022.
[34] S. James, Z. Ma, D. R. Arrojo, and A. Davison. Rlbench: The robot learning benchmark & learning environment. IEEE Robotics and Automation Letters, 5(2):3019-30266 2020.
[35] Z. Zhao, Kumar, Levine, and Finn. Learning fine-grained bimanual manipulation with low-cost hardware. arXiv preprint arXiv:2304. 2023.
[36] P.-L. Guhur, S. Chen, R. G. Pinel, M. Tapaswi, I. Laptev, and C. Schmid. Instruction-driven histery-aware policies for robotic manipulations. Conference on Robot Learning, pages 175-187. PMLR, 2023.
[37] O. Mees, L. Hermann, and W. Burgard. What matters in language conditioned robotic imitation learning over unstructured data. IEEE Robotics and Automation Letters, 7(4): 1e1111100-11111 2022.
[38] R. Cadene, S. Alibert, A. Soare, Q. Gallouedec, A. Zouitine, and T. Wolf. Lerobot: State- of-the-art machine learning for real-world robotics in pytorch. https://github.ce com/ huggingface/lerobos 2024.
[39] S. Mangrulkar, Gugger, L. Debut, Y. Belkada, S. Paul, and B. Bossan. Peft: State-of-the-ar pframeter-efficient fine-tuning methods. https://gi thub. com/huggingfacc/ppeteeeeoo 2022.
[40] E. J. Hu, Y. Shen, P. Wallis, Z. Allen-Zhu, Y. Li, S. Wang, L. Wang, and W. Chen. Lora: Low-rank adaptation large language models. arXiv preprint arXiv: 106.09655, 2021.
[41] C. Lynch, M. Khansari, T. Xiao, V. Kumar, J. Tompson, S. Levine, and P. Sermanet. Learning latent plans from play. In Conference on robot learning, pages 1113-1132. PMIR, 2020.
[42] J. Lu, F. Richter, and M. C. Yip. Pose estimation for robot manipulators via keypoint opti- mization and sim-to-real transfer. IEEE Robotics and Automation Letters, 7(2):4622-4629, 2022.
[43] Y. Tian, Zhang, Yin, and Dong- Robot structure prior guided temporal attention for camera-to-robot pose estimation from image sequence. In Proceedings the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 8917-8926, 2023.
[44] X. Zhang, Y. Liu, Wang, URL https: //apxii////xvv..e////
Boularias. Detect everything with few examples, 2024.
orgg/bs/233999 http:////aaaa..oo////////////// 12969.
[45] D. Coleman, I. Sucan, S. Chitta, and N. Corell. Reducing the barrier to entry of complex robotic software: a moveit! case study. arXiv preprint 104.:75 2014.
[46] P. Mitrano and D. Berenson. Data augmentation for manipulation.
arXiv preprint
arXiv:2205.02886. 2022.
[47] M. Xu, S. Yoon, A. Fuentes, and D. S. Park. A comprehensive survey of image augmentation techniques for deep learning. Patlern Recognition, 137:109347, 2023.
[48] F. Williams. Point cloud utils, 2022. http://wwww.giillbooo///liliiiis
[49] C. Lynch and P. Sermanet. Language conditioned imitation learning unstructured data. arXiv preprint arXiv:2005.07648, 2020.

---

A Appendix
A.1
Ablation Studies
Table A1: General agent's performance on forecasting the full kinematic chain vs. only the end-effector
Table A2: Effect of multiple
Calvin
RLBench
ALOHA
viewpoints in RLBench
Configuration
Avg. Length
Success Rate (%6
Configuration
Success Rate
BCT
0.48
24
12
Single View 26.8%
Full Chain
1.460
55,5
55.3
Multi View 55.5%
VKT (Ours)
End-Effector
1.24
54.1
28
Predicting the Full Chain versus Predicting Only the End-Effector. We compare the effects of forecasting the full chain and only the end-effector in Table A1. Table A1 shows that forecasting the full chain improves the performance on all environments and especially the bimanual manipulation ALOHA benchmark. Moreover, despite the reduced capacity to represent rotations, end-effector VKT also delivers stronger performance as general agent than BCT. This indicates that encoding robot actions visually in the image plane is the key countering distribution shifts from different action spaces when building general agents across diverse environments. Furthermore, the end- effector predictions can be useful in special cases when robot models are unavailable. This suggests a broader applicability for visual kinematics forecasting.
Behavior Cloning Loss (L1)
VKT Pretrained Steps
0kc
0.05
10k
20k
0.04
30k
40k
0.03
50k
0.01
5000
10000
15000 20000
25000
30000
Training Steps of the Convolution Head
Figure A1: Behavior cloning Loss curves of the convolution head for Calvin. The VKT backbones trained with different steps. We find strong correlation between the VKT performance (repre- sented by the pretrained steps) and the imitation learning convergence.
VKT Performance and Imitation Learning Convergence. We study the impacts of VKT perfor- mance imitation learning convergence for the low-level robot actions in Figure A1. We train the VKT for different number of steps. Next, we use the pretrained VKTs backbones and train the convolution head for Calvin. Figure Al demonstrates a strong correlation between the VKT pretrained steps and the convergence quality of imitation learning on robot actions.
Earth Moving Distance versus Chamfer Distance. Figure A2 compares the predicted visual kine- matic chains using our proposed EMD and Chamfer Distance [48]. Unlike EMD, Chamfer Distance is not density-aware and only minimizes the distance between each point and its nearest neighbor. Figure A2 shows EMD produces fuzzier but more accurate kinematic chains, but Chamfer Distance produces sharper but sparser kinematic chains that lack structural integrity.
Percentages of Training Data for the Convolution Head. Figure A3 shows the average lengths on Calvin with the convolution heads trained different data percentages. Similar levels of perfor- mance can be obtained with 5% data, even 1% data at minor performance drop. Calvin includes 34 tasks and 5 5123 episodes as the training set for D D challenge. 1% data only leaves roughly
12

---

EMD
Chamfer
Groundtruth
EMD
Chamfer
Groundtruth
Swipe dirt to the large dustpan
Use the right arm to pick the red block
Put money the middle shelf
Close the box
Figure A2: Comparison of Predicted Visual Kinematic Chain using EMD and Chamfer Dis- tance [48]. The kinematic chains predicted by EMD are visually fuzzier but more accurately de- scribe the structure of the robot arm. The chains predicted by Chamfer Distance lack structural integrity despite being visually sharper. Unlike EMD, Chamfer Distance not density-aware and only minimizes the nearest neighbor distance for each point.
Avg. Length
Input tokens
1.5
5-10,
25.
50
100.
pointwise
bn + elu
1.0
depthwiss (K=5),
bn elu
0.5
20
40
60
80
100
pointwise 128, 128
Percentage of Training Data for the Conv head (%)
linear, 128, action_dim
Figure A3: Average lengths on Calvin by training the convolution head with different data percentages. Calvin includes 34 tasks and 5123 episodes as the training set for D - D challenge. Sim- ilar levels of performance can be obtained with 5% data, or even 1 1% data at minor performance drop.
output actions
Figure A4: Architecture of the convolution head.
episode per task. This shows that kinematics tokens already encode robot action information. Only a small amount of data needed to train a decoder to map kinematics tokens to actions.
Multi- View versus Single-View Predictions. We study the impacts of multi-view forecasting in Table A2. We remove the multi-view attention layers create the single-view VKT. We average the low-level robot actions predicted from each view as the final prediction. Table A2 shows that VKT improves its 3D understanding significantly with our multi-view dual attention block by predicting consistent kinematic chains in different camera viewpoints.
13

---

A.2 Additional Tables and Figures
Table A3: Comparison our VKT environment-speeifie methods. We show that our general agent VKT can outperform GCBC [41], and reach performance levels comparable to PerAct [3], two works published in 2020 and 2022, respectively. VKT also shows competitive performance against
ACT [35], the SOTA method on ALOHA.
Specialized Agent
VKT (Ours)
Environment
Metric
GCBC [41]
PerAct [3]
ACT [35]
General
Specialized
Calvin [33]
Avg. Length
1.11
1.46
1.58
RLBench [34]
60.4
61.7
55.5
Success Rate (%) 1
ALOHA [35]
76
55.3
63.3
Table A4:
Performance on RLBench. We report the success rates on each task and the average
overall success rate.
Agent
Method
Avg. Success
Put In Drawer
Reach Drag
Turn Tap
Slide Blocks
Open Drawer
Money In Safe
Place Wine
Sweep
To Pan
PerAct [3]
60.4
60.4
60.0
64.8
52.2
68.0
34,8
10.0
46.0
Specialized
BCT
36.4
1.6
77.6
63.2
12.0
20.8
44.8
44.0
5.6
VKT (Ours)
55.5
52.0
88.8
40.8
22.4
56.0
76.8
31.2
48.0
General
BCT
24.3
0.0
28.4
68.8
8.0
8.6
8.0
4,0
0.0
VKT (Ours)
61.7
48.8
77.6
45.6
12.8
73.6
81.6
89.6
50.4
Meet Off Grill
Phone Base
Lid Off
Close Microwave
Close Box
Ball In Hoop
Push
Lift Block
Close
Jar
PerAct [3]
40.4
54.8
87.2
84.8
71.6
78.8
80.8
62.0
24.4
Specialized
BCT
32.0
37.6
47.2
93.6
84.0
33.6
21.6
0.0
0.0
VKT (Ours)
41.6
68.0
92.0
64.0
93.6
85.6
64.0
10.4
General
BCT
21.6
0.0
36.4
0.0
VKT (Ours)
33.6
72,8
98.4
92.0
84.8
93.6
69.6
11.2
12.8
Table A5: Performance on Calvin. Calvin requires agents to accomplish 5 tasks sequentially at each episode. We report the step-level success rate, average length and standard deviation average length.
Success rates at each task (%)
Agent
Method
Avg. Length f
2
4
5
MCIL [49]
0.41
34.4
5.8
1.1
0.2
0
GCBC [41]
1.11 (0.3)
64.7
28.4
12.2
4.9
1.3
Specialized
BCT
1.36 (0.06)
63.2
39.6
18.6
9.7
4.9
VKT (Ours)
1.58 (0.06)
67.6
44.3
24.1
14.5
7.5
BCT
0.48 (0.1)
36.4
8.6
2.6
0.4
General
VKT (Ours)
1.46 (0.08)
64.9
41.4
22.3
12.8
6.4
14

---

Precision
Predicted visually the action sometimes fails to hold the stick.
Next action is gripper top of red cube.
Next action to grab the thin stick.
action sometimes pushes the red gube off the table.
visually correct,
Drag the cube with stick
Stack the blue cube on top of red cube
Incorrect Kinematics Chain Prediction
action move to the left side of the
But the predicted chain chooses to go backwards.
dear and bady, a case unseen during training.
Fall from failure.
Slide the block to blue target
Close the microwave
Ambiguity of Projection 3D to 2D
However, facing camera, opaniciose ambiguous from 2d chain.
Next action is transfer the cube
the groundtruth behavior
The kinematics chain
is predicted correctly
Pick up the red block with right arm, and transfer it to left arm
Rotations with Tiny Visual Changes
But predicted chain aligns with current rotation).
action rotate the tap
But predicted chain shows no rotations.
the lid through rotation
Close the red jar
Turn the left tap
Figure A5: Failure Case Analysis. The predicted kinematic chain of the next step is colored in red. The expected action shown in green arrow. The first row shows tasks: drag with stick and stack blocks. The kinematic chain is visually correct represent the grabbing and stacking action. However, the predicted action sometimes lacks the necessary precision. The second row shows tasks: push blocks and close microwave, which requires spatial reasoning. However, the predicted chain represents the wrong actions. The third row shows the cube transfer task. The next action to move two arms closer with the right reasonable. However, the gripper state is ambiguous when the right arm faces the camera from the side. When the right drops the cube, the task fails. The fourth shows the tasks: close jar and turn tap, which require rotation actions with only small changes the visual kinematic chain. The predicted chain does not show rotation actions.
grasping the cube. The predicted chain is visually
15


Reviews:
[{"review_id": "DXchQZRhCq", "reviewer": "Reviewer_Lf1f", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": 3, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "This paper proposes the visual kinematics chain as a novel action representation to enable manipulation policy training on diverse data with various embodiment and environments. The authors also introduce the Visual Kinematics Transformer (VKT) to predict kinematics structure and use it to generate actions for downstream tasks.", "summary_of_recommendation": "This paper is overall a good paper and novel. If authors could clarify their experiment setting and add more experiment analysis, I lean to accept it.", "technical_quality": 3, "clarity_of_presentation": 3, "potential_impact": 3, "robotics_focus": 4, "quality_of_limitations": 3, "questions_for_rebuttal": "- Has the authors tried different losses for VKT? Is earth-moving loss significantly better than other loss, like Chamfer distance?\n- Is N (the number of predicted points) a fixed number?\n- For specific environment deployment, how much data is needed to train the tiny head?\n- How scalable this method is with respect to the number of camera viewpoints? For example, why only three viewpoints out of five are chosen in RLBench?\n- Is it possible to raise visual kinematics to 3D to resolve the rotation motion problem?", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "hwbuE86Bel", "reviewer": "Reviewer_t5uW", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": 2, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "Action spaces are frequently not aligned between different robotics datasets, and this paper proposes a method to address this problem. Instead of training a model to predict the 6DOF pose of the end-effector, they train a model to predict, in pixel space, the motion of the kinematic chain of the robot. This kinematic chain can be obtained using the robot’s URDF and a calibrated camera. The performance of this method is evaluated both in simulation and on real hardware and is compared to a BC transformer baseline that predicts the robot’s end-effector pose directly.", "summary_of_recommendation": "The paper addresses an important problem and shows some positive results. The emphasis on predicting the entire visual kinematic chain despite the negative results of the ablation study is very confusing. The paper needs some clarification on some of the results as described in the main review.", "technical_quality": 2, "clarity_of_presentation": 4, "potential_impact": 2, "robotics_focus": 4, "quality_of_limitations": 3, "questions_for_rebuttal": "Lines 215 and 216 refer to 2D image augmentation for manipulation learning. What augmentations are used?", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "eV6vGM8kjI", "reviewer": "Reviewer_TuBj", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": 3, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "The paper introduces Visual Kinematic Chains as a universal action space for multi embodiment robot training. The main idea is to track the kinematic chain of the robot arm that is robot and end-effector agnostic and can be pre-trained on diverse robot datasets such as Embodiment. They use a transformer to track the kinematic chain and for domain adaption finetune a small action head. The proposed method is evaluated on several simulation and real robot environments.", "summary_of_recommendation": "Overall, I really like the creative and novel contributions of the paper and appreciate the evaluation across several benchmarks. However, missing relevant baselines and low performance on sim environments and the limitation section are open issues, that need to be addressed. Thus, I only recommend weak reject for now.", "technical_quality": 3, "clarity_of_presentation": 4, "potential_impact": 3, "robotics_focus": 4, "quality_of_limitations": 2, "questions_for_rebuttal": "- How would you try to solve the issue of missing URDF files to enable training VKT on OXE on more diverse data for future work? \n- Can you add some relevant baselines mentioned in the weakness section?\n- Why is the average performance of VKT on CALVIN so low compared to other methods? \n- Did you ever try to combine your kinematic chain prediction with a separate more powerful low-level transformer policy as used in [1,2] and only use the kinematic chain as a visual conditioning for the low level policy? Similar how [3] does it with their track conditioning? While I understand the idea to have a single model for everything, I believe that the performance of the proposed method can be enhanced a lot with a better low level policy. In my experience a small action head is not enough for challenging benchmarks as used in the paper.\n\n\n—-\n\nPost rebuttal \n\nThe added experiments and the updated limitations section further strengthen the paper. While the performance gap to specialist models remain on some benchmarks, I believe it’s mostly related to architecture design decisions and are not related to the proposed method itself. Overall, I believe the paper has timely and relevant contributions for generalizable action spaces that are interesting for the wider robot learning community. Thus I changed to weak accept. For a strong accept the authors would have required better performance across benchmarks. However, I believe the paper is still timely and relevant.", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper introduces the Visual Kinematic Chain as a more general action space to scale up learning in robotic manipulation. The reviewers all recognize the novelty and importance of this approach.

Before the rebuttal, the reviewers raised several concerns. The experimental analysis is criticized for unfair comparisons and lack of detailed analysis. The paper does not clearly show the quantitative benefits of predicting the entire kinematic chain versus just the end-effector. Additionally, the claim of bridging different action spaces is not well-supported, and the reason for VKT's improvement over BCT is unclear.

The evaluation methods and metrics also need clarification, especially the success length metric. Important details are missing, and the reliance on URDF files might limit generalizability. Comparisons with other action spaces like track points and video are insufficient.


During the rebuttal, the authors did an excellent job addressing the reviewers’ concerns. As a result, all reviewers now rate the paper as a Weak Accept. However, Reviewer t5uW still has concerns regarding the explanation of Figure 6 and the claims about performance improvements. Nonetheless, the reviewer acknowledges that the overall idea is interesting and valuable for the field. I strongly encourage the authors to consider the reviewers’ feedback both during and after the rebuttal to strengthen their paper.

I’m including Reviewer t5uW’s comment from the discussion below.

========

Thank you to the authors for their response.

One of my main concerns previously was the ablation study in Table A1 which previously showed no improvement for forecasting the full kinematic chain instead of only the end-effector. I stated that “the success rate for predicting the full chain is 55.5% and 54.1% for predicting the end-effector. It’s unclear for this table how many trials were used, but assuming 1000 trials were used, the p-value for this result is 0.5 which is not statistically significant.” I’m also not convinced that for the other environment a difference in average length of 1.24 vs 1.46 is a meaningful improvement. The authors need to perform a statistical comparison, or at the very least, provide a measure of variance of their reported performance. Despite this, the authors kept the claim in their revised draft that “forecasting the full chain improves performance on all environments” which I strongly disagree with given the p-value analysis I provided. The authors should remove the sentence in lines 411-413. I appreciate the new results on the ALOHA environment that show a meaningful improvement in performance for predicting the full kinematic chain. However, this means that predicting the full kinematic chain only leads to improvement in 1 out of the 3 tasks studied and no explanation was provided on why that is. The paper would be stronger if more analysis was provided to explain when and why predicting the full kinematic chain actually helps.

The authors’ response to my other concerns in W3 were helpful and I agree with their comments.

My other major concern are the results presented in Figure 6. I asked for more explanation from the authors for this figure, but the additional explanation provided merely repeated what was already said. In its current state, Figure 6 does not make sense. The text says that Figure 6 shows that VKT can predict kinematic chains from real world data. But all Figure 6 shows is a comparison to the BCT baseline.

The “relative precision” metric does not make sense — according to 4.1.2, it’s a relative comparison between two methods. By this definition, VKT and BCT could both be equally terrible. No data is provided to evaluate the absolute performance of either method, therefore a comparison between them is not helpful.

Also, if relative precision is indeed defined as in 4.1.2, it doesn’t make sense to show BCT and VKT as separate bars. Isn’t the whole point of the relative precision metric to produce a single scalar comparing the two methods?

The text in the authors’ rebuttal also does not make sense to me.

> “Figure 6 compares the relative precision (an inverse of L1 error on regression of robot actions, explained in lines 234-235) of BCT and VKT on validation splits of the selected datasets.”

- “relative precision” as defined in 4.1.2 is not the inverse of L1 error.

> “It demonstrates that visual kinematics forecasting, as a pre-training objective, contributes to the accuracy of action regression.”

- The authors do not show this. Are they trying to argue VKT is better than BCT? The actual paper text does not make this claim, and the figure does not support it.

I am clearly still missing what the authors are trying to show. The current figure and text in 4.1.2 do not make sense together even after the authors attempted to clarify, in either the rebuttal or the text. If the goal is for the authors to show that VKT can produce kinematic chains, why do they not simply report L_vkt?

This is unfortunate because Figure 6 is fairly incidental to the overall story the authors are telling. I am fine with accepting this paper if the authors can either explain Figure 6 in a coherent fashion or simply delete it from the paper.

I am changing my evaluation to a weak accept because I think the ideas proposed in the paper are interesting and valuable for the field and the ALOHA results have strengthened the paper, but given my remaining concerns on the quality of the execution of the paper, I would be very hesitant to publish this paper unless

(1) the authors can coherently explain Figure 6 (or simply delete it)

(2) the authors remove the claim that a success rate of 55.5 versus 54.1 is meaningful in Table A1 (lines 411-413). It is simply not.
Decision: Accept