What's the Move? Hybrid Imitation Learning via Salient Points
Authors: Priya Sundaresan
Hengyuan Hu
Quan Vuong
Jeannette Bohg
Dorsa Sadigh
Venue: ICLR 2025 Poster (2025)
Keywords: Imitation Learning
Robot Learning
Robot Manipulation
Robotics
Primary area: applications to robotics, autonomy, planning
TL;DR: We propose an imitation learning algorithm for complex robot manipulation with visuospatial generalization; it substantially outperforms SOTA existing methods across 4 real-world tasks and 2 simulated benchmarks.

Abstract: While imitation learning (IL) offers a promising framework for teaching robots various behaviors, learning complex tasks remains challenging. Existing IL policies struggle to generalize effectively across visual and spatial variations even for simple tasks. In this work, we introduce **SPHINX**: **S**alient **P**oint-based **H**ybrid **I**mitatio**N** and e**X**ecution, a flexible IL policy that leverages multimodal observations (point clouds and wrist images), along with a hybrid action space of low-frequency, sparse waypoints and high-frequency, dense end effector movements. Given 3D point cloud observations, SPHINX learns to infer task-relevant points within a point cloud, or *salient points*, which support spatial generalization by focusing on semantically meaningful features. These salient points serve as anchor points to predict waypoints for long-range movement, such as reaching target poses in free-space. Once near a salient point, SPHINX learns to switch to predicting dense end-effector movements given close-up wrist images for precise phases of a task. By exploiting the strengths of different input modalities and action representations for different manipulation phases, SPHINX tackles complex tasks in a sample-efficient, generalizable manner. Our method achieves **86.7%**  success across 4 real-world and 2 simulated tasks, outperforming the next best state-of-the-art IL baseline by **41.1%** on average across **440** real world trials. SPHINX additionally generalizes to novel viewpoints, visual distractors, spatial arrangements, and execution speeds with a **1.7x** speedup over the most competitive baseline. Our website (http://sphinx-manip.github.io) provides open-sourced code for data collection, training, and evaluation, along with supplementary videos.

Published as a conference paper at ICLR 2025
WHAT'S THE MOVE?
HYBRID IMITATION LEARNING VIA SALIENT POINTS
Priya Sundaresan" Hengyuan Hu"!, Quan Vuong", V Jeannette Bohg', Dorsa Sadigh'
"Equal contribution.
Stanford University, PPhysical Intelligence
ABSTRACT
While imitation learning (IL) offers a promising framework for teaching robots various behaviors, learning complex tasks remains challenging. Existing IL poli- cies struggle to generalize effectively across visual and spatial variations even for simple tasks. In this work, introduce SPHINX (Salient Point-Based Hy- brid ImitatioN and eXecution), flexible IL policy that leverages multimodal ob- servations (point clouds and wrist images), along with hybrid action space of low-frequency. sparse waypoints and high-frequency, dense end effector move- ments. Given 3D point cloud observations. SPHINX learns to infer task-relevant points within a point cloud, or salient points, which support spatial generaliza- tion by focusing on semantically meaningful features. These salient points serve as anchor points to predict waypoints for long-range movement, such as reaching target poses in free-space. Once near a salient point, SPHINX learns to switch to predicting dense end-effector movements given close-up wrist images for precise phases of a task. By exploiting the strengths of different input modalities and ac- tion representations for different manipulation phases, SPHINX tackles complex tasks in a sample-efficient, generalizable manner. Our method achieves 86.7% success across 4 real-world and 2 simulated tasks, outperforming the next best state-of-the-art II. baseline by 41.1% on average across 440 real world trials. SPHINX additionally generalizes novel viewpoints, visual distractors, spatial arrangements, and execution speeds with a 1.7x speedup over the most compet- itive baseline. Our website contains code for data collection and training code along with supplementary videos: http: http:://phhnxxmmnipp
manip. gi thub. 1o.
INTRODUCTION
Imitation learning (IL. of visuomotor policies is widely used framework for teaching robots ma- nipulation tasks given demonstrations collected by humans (Schaal, 1996). While prior works have shown that IL policies can learn range of behaviors with sufficient data, from simple object pick- and-place to more complex tasks, they typically succeed only in highly controlled settings with low variation. Generalizing to realistic visual and spatial variations remains significant challenge. Consider teaching robot to make a cup of coffee in the morning, which demands precision, long- horizon reasoning, and tolerance environment variations. The robot must first carefully grasp mug handle, position it under the machine, insert a pod into a narrow slot, close the lid, and press a button all with very little margin for error (Fig. 1). Even after mastering this sequence, the policy might struggle with spatial changes like moving the machine, or visual changes such as new coffee pods, spilled grounds, a different camera angle, or varying lighting conditions. This underscores the need for IL. policies that can learn complex tasks from a limited number of demonstrations while effectively generalizing to natural and expected variations in the real-workd.
Conventional IL policies often struggle with both performance and generalization, largely due to limitations in their input and output representations. First, they tend to rely heavily on visual inputs like RGB images, treating irrelevant details like the background, lighting, viewpoint the same as task-relevant information. This can cause a policy to memorize specific scenes, making brittle to visual variations (Zhao et al., 2023; Chi et al., 2023). Second, these policies usually predict actions
Correspondence prriyasun@stanfodd edu, hengyuan, hhu@gma com

---

Published as a conference paper at ICLR 2025
Observations
Actions
Point Cloud
SPHINX
Next Modle
A
271
Woypsiot
paypp
Offect
EU2
Waypooist
Mode
Salient Point
a
lense
Dense Action
π
DOO
171
Mode
Wrist Image
Figure 1: SPHINX is hybrid IL agent which learns to switch amongst different modes (m²) of execution to tackle complex tasks with visuospatial generalization. In waypoint mode, n waypt takes a point cloud as input, and predicts single waypoint Wy coffee pod, etc. denoted). After reaching a waypoint via controller, the policy uses learned switching a dense policy x= deess which takes wrist-camera images as input and outputs dense actions (ac) for precise manipulation around a salient point. On the right, the policy interleaves both modes of execution to complete a long-horizon coffee- coffee-making task guided by salient points and mode switches
offset (σx) to task-relevant salient point 2, (i.c. mug handle,
for the next immediate timestep, which hampers spatial reasoning. Simple spatial movements, like reaching, are predicted through hundreds of end-effector actions, increasing the risk of veering off course. To address these limitations, recent works explore 3D scene representations, such as point clouds and voxel grids, to offer better spatial awareness, and propose predicting actions as end- effector poses (waypoints) reachable through a controller or motion planner (Goyal et al., 2023; Sundaresan al., 2023; Yang et al., 2024a;b; Shridhar et al., 2023). This can drastically shorten the action prediction horizon and enable better spatial generalization. However, these methods often lack precision, point clouds typically lack the necessary resolution to capture small object details preserved in images. On the other hand, recent image-based II. policies attempt to remedy spatial generalization using hybrid action spaces (Belkhale et al., 2023; Shi et al., 2023). Here, the policy has two potential modes of execution: waypoints for long-range motions like reaching, or dense actions end-effector movements predicted per-timestep
only when precision or reactivity is required. These works ultimately consider training policies with a single input modality type, and optionally hybrid actions. In reality, different phases of task may lend themselves more favorably to different visual inputs or action modes. A policy which can effectively choose and interchange both the input modality and the underlying action representation during execution remains underexplored.
Our key insight is that by encouraging the policy to attend to salient taints-task-relvaan 3D points-wa we enable to choose between different input modalities as well as different action modes when appropriate, improving performance and generalization. Fig. 1 illustrates example salient points in the coffee-making task red (i.c. the mug handle, pod, etc.). We introduce SPHINX: Salient Point-based Hybrid ImitatioN and eXecution, hybrid IL agent which learns to switch amongst a waypoint policy which predicts waypoint actions given point clouds, and a dense pol- icy which predicts dense actions given close-up wrist-camera images. Specifically, the waypoint policy manages long-range movements by first predicting salient points that narrow the search space of actions around spatially relevant features, promoting spatial generalization. It then predicts way- point actions relative to these points. After reaching a waypoint, SPHINX switches to a dense policy which takes wrist-camera images as input. This policy captures close-up object details for precise manipulation and supports visual generalization by staying agnostic to broader scene changes. To support training SPHINX, we develop a flexible data collection interface that allows demonstrators to specify salient points and switch modes in real-time during teleoperation.
Empirically, we show that SPHINX can tackle a range of precise, morizooizon manipulation tasks, including four real-world scenarios (drawer-opening. cup-stacking, coffee-making, toy train assem- bly) and two simulated ones. SPHINX achieves 86.7% success and outperforms the next best II. baseline by 41.1% on average, while generalizing better to visual distractors, viewpoints, spatial rangements, and execution speeds. We open-source our web-based data collection interface for spec- ifying salient points and hybrid teleoperation, alongside code, supplementary material, and videos at: http:: : sphi i nnn-nnnnipppp.mmmmmm wwe.ni..giiiii github.
2

---

Published as a conference paper at ICLR 2025
2 RELATED WORK
Imitation learning has long been a foundational ap- proach in roboties for teaching robots to replicate human demonstrations (Schaal, 1996; Atkeson & Schaal, 1997; Pomerleau, 1988). Robotic imitation learning policies typically take images as input and output motor commands, such as joint positions, velocities, or Cartesian end-effector poses. Recent works of that type (Reuss et al., 2023; Chi et al., 2023; Zhao et al., 2023) have demonstrated strong performance on tasks in controlled settings with a limited initial state distribution. However, they struggle to generalize to unseen visual or spatial variations. To address visual generalization, some works augment vision-based policies with diffusion -ennerrtee image observations (Yu et al., 2023; Bharadhwaj et al., 2024). While useful and complementary to our approach, these augmen- tations do not directly enable spatial generalization. Other works propose replacing image inputs with 3D scene representations such point clouds and voxel grids, and outputting actions as way- points, 6-DoF poses reachable through motion planning (Sundaresan et al., 2023; Shridhar et al., 2023; Yang et al., 2024b). While this reduces the complexity of action prediction from hundreds of actions to a single pose, 3D representations such as point clouds often lack the resolution to enable precise manipulation of small objects. Other recent approaches like HYDRA (Belkhale et al., 2023) and AWE (Shi et al., 2023) take image inputs but propose a hybrid output action space of waypoints and dense actions. These distinct action modes are intended for long-range and precise movements, respectively, Our method builds on these approaches by leveraging salient points to bridge hybrid input space of point clouds and wrist-camera images, and hybrid output action space of waypoints and dense actions.
Imitation Learning for Robotics Control:
Action Representations: Most visual imitation learning works rely on standard 6-DoF action spaces, but recent efforts explore alternatives for better spatial generalization. One approach in- volves predicting actions as parameterized manipulation primitives instead of low-level end-effector movements. This reduces the dimensionality of the action space and improves sample efficiency, but often requires task-specific engincering (Dalal et al., 2021; Sundaresan et al., 2023; Nasiriany et al., 2022; Agia et al., 2022). Other methods exploit equivariance, ensuring that transformations of visual inputs (e.g., rotations or scaling) are reflected in output actions (Wang et al., 2024; Yang et al., 2024a;b). However, these works often rely on limiting assumptions like access object states via segmentation, or single-object tasks. In the grasping domaín, many policies consider point clouds as inputs and output action space defined as per-point predictions for the end-effector pose. This has proven effective for learning sample-efficient and generalizable grasping policies (Saxena et al., 2008: Sundermeyer et al., 2021). Inspired by this, our method also parameterizes waypoint actions as offsets to salient points in a point cloud, but we critically learn a hybrid policy which predicts both waypoint and dense actions to tackle longer-horizon and precise tasks beyond grasping.
Data Collection for Imitation Learning: Despite the advancements in action representations and spatial generalization, the success of visual imitation learning policies still hinges on the quality of teleoperated demonstrations. Human operators typically collect robot data using interfaces like virtual reality controllers (Jedrzej Orbik, 2021), ) handheld devices (Chi et al., 2024), puppeteering setups (Zhao et al., 2023), or 3D mice (e.g., Spacemouse). However, these interfaces map demon- strator controls directly to robot actions on a per-timestep basis, which presents two key limitations. First, the recorded data only captures (observation, dense action) pairs, lacking compatibility with waypoint actions or useful metadata such as salient points. Second, directly controlling long-range movements can be inefficient, noisy, and tiring for demonstrators. To address these issues, design an interface (Fig. 2) that seamlessly integrates both waypoint and dense action modes. A custom web-based GUT supports waypoint mode, allowing demonstrators to specify salient points and way- points with the ease of simple clicks and drags. A controller can then reach the specified waypoint automatically, removing the need for constant teleoperation from the demonstrator. Additionally, the interface is compatible with any external device for dense actions, allowing for easy switching between the computer mouse and the device, as long as it on hand. This provides flexible and efficient data collection process for high-quality hybrid datasets, with no post-hoc labeling required.
3 PROBLEM STATEMENT
In standard IL, we are given a dataset D of N trajectories of expert demonstrations [ }. Each trajectory is a sequence of observation action pairs (on, ao:-... or, ar). The goal is to learn or,,
3

---

Published as a conference paper at ICLR 2025
policy x(aloo (ot) that matches the expert distribution using the following loss -E,~D log IT (aylor). However, this formulation can easily lead to compounding errors for long-horizon tasks where episodes may span hundreds of steps. In this paper, instead consider hybrid imitation learning setting where the policy can either output dense action dę E R? short-range end-effector pose reachable by
waypoint w; RR, long-range end-effector pose reachable by series of interpolated movements. Both waypoint and dense actions capture the end-effector pose, but waypoint action wy specified timestep is translated fav,..... 4 by a controller a; = C(o we) based on the current pose of and the target waypoint Wy specified t'. In practice, we use a simple controller that linearly interpo- lates between current pose and target waypoint. Then, we can record timestep in the dataset as (ot, de; my, [wyr] where m, (waypt, dense, termi nate) } is the mode and wy is the optional target waypoint only when my = waypt. Each we spans the next ky steps decided by the con- troller. Our goal is thus to learn hybrid policy (o;)) that first predicts a mode p(m)leo (or) and then predicts either waypoint from *aypr (weloz) a dense action from π dense (ac (0t).
sequence of ke interpolated actions
SPHINX: SALIENT POINT-BASED HYBRID IMITATION AND EXECUTION
We introduce SPHINX: Salient Point-based Hybrid ImitatioN and eXecution, a framework for learn- ing sample-efficient, generalizable imitation policies capable of handling complex, long-horizon manipulation tasks across diverse initial conditions. SPHINX combines a high-level waypoint pol- icy **ypp= for long-range movements and dense policy dense for precise manipulation (Fig. 1). The waypoint policy takes point clouds as input, classifies semantically meaningful salient points, 2, R.. and regresses waypoint actions wy RR, Importantly, *aypt predicts the positional component of W, as an offset to the salient point 2t. This grounds the desired interaction around salient point, such as learning reach for a mug by first identifying the handle. The dense policy o dense takes over only for precise actions around salient point, like carefully inserting coffee pod into its slot (Fig. 1). Since the waypoint policy handles long-range movements, it uses point clouds to provide spatial context. The dense policy uses wrist camera images input, capturing detailed object features for precise manipulation and enabling visual generalization to variations in the surrounding scene. Both policies also predict the next mode m4+1 to decide which policy to after completing the current movement. Without loss of generality, we initialize my to waypoint mode.
To train SPHINX, we first need to collect demonstrations using the two modes and annotate salient point for each waypoint. In Section 4.1, introduce intuitive web GUI to easily collect such demonstrations in the hybrid format and record salient points with no additional overhead. Then, we discuss how to learn waypt (w.) and π denss (azlo:) ) in Section 4.2 and Section 4.3 respectively.
Waypoint Mode
Specify Salient Point
Specify Waypoint
Reach Waypoints via Controller (C)
ped
W
0g
Rotation
Dranslation
wrist
OF
Mode
place train on bridge
grasp and tilt bridge
The demonstrator visualizes point cloud obedd in web GUL, where they can click a salient point 2 and specify waypoint action w/ by clicking and dragging to rotate or translate digital twin of the gripper. After the controller C reaches the waypoint to grasp the train, the process repeats for waypoint above the bridge. The demonstrator then switches to providing dense actions a, with a 3D SpaceMouse to carefully place the train bridge and it, causing the train to roll.
Figure 2: Data Collection Interface:

---

Published as a conference paper at ICLR 2025
4.1 DATA COLLECTION INTERFACE FOR SPHINX
Without an existing interface that satisfies our need, we design a data collection system to support waypoint specification, salient point annotation, and mode switching seamlessly. Our hardware setup includes two third-person cameras to provide RGB-D observations to construct a colorized point cloud of and one wrist-mounted camera to provide RGB wrist images ofrist. We develop a custom web-based GUI for specifying waypoints and salient points in waypoint mode. To provide dense actions instead, a demonstrator can seamlessly switch from the computer mouse to any dense teleoperation device like a VR/game console controller or a 3D mouse (Spacemouse) as in this work.
The top row of Fig. 2 visualizes the web-based GUI and the process of recording a waypoint ac- tion. The GUI streams the point cloud of N points god = {c1, C CN} } to the browser in real for each waypoint by clicking within the point cloud, (e.g. the red dot on the toy car next to the mouse cursor.) After clicking on the salient point, a digital twin of the gripper appears near the salient point to facilitate waypoint specification. The demonstrator can use click and drag interactions on the virtual grip- per to set waypoints relative to these salient points. The salient point E R$ specifies the region of interest for interaction while the waypoint, 7 DoF target end-effector pose, captures how to in- teract with it. After specifying a waypoint, the linear controller defined above interpolates and exccutes actions to reach the waypoint. Critically, this removes the need for the demonstrator to manually teleoperate long-range movements. The entire waypoint motion is recorded as a sequence {(o,, dg, waypt, W 2e hi (*.....++++)) where f' is the timestep when the waypoint is specified and by is the number of steps that the controller takes to complete the waypoint Wyr
time and allows demonstrator to select salient point 2t E {I,.. [I,...,N) N
Once the controller reaches waypoint, the demonstrator may specify another waypoint switch to dense mode for precise manipulation. To take over with dense mode, the demonstrator simply operates the teleoperation device, such as 6DoF joystick, and its movements are automatically detected and mapped to delta end-effector movements. This is illustrated by the bottom row of Fig. 2, where the operator uses the telcoperation device (Spacemouse in this case) to precisely align the toy car above the narrow bridge. Each step in dense mode is recorded as (0t, at dense). Note that regardless of the mode, we record the full set of observations O which includes all camera views and proprioception to facilitate data augmentation and compatibility with any IL policy.
4.2 THE WAYPOINT POLICY OF SPHINX
The waypoint policy "aypp in SPHINX takes a point cloud of as input and outputs a 7-DoF end- effector pose Wy for the robot to reach via a controller. We utilize point clouds as input to cast part of the action prediction problem to learning a salient map over the points. This encourages the policy to attend to the important spatial features (i.e. the handle on a mug) rather than memorize actions.
The detailed design of the waypoint architecture illustrated in Fig. 3. At high level, we would like to have per-point predictions, such as the probability of point to be salient and the translational offset between the point and the target location the end-effector, as well as other predictions whose targets are not expressed relative to the points, such as rotation, gripper state, and mode. We use transformer to process the points and add additional tokens for point-agnostic predictions. We first use farthest-point ppoint-samplin (FPS) (Qi et al., 2017) to downsamples raw point cloud to D 1024 points, and then convert the points G E R66 to tokens e, E Rdd via a shared linear projection layer. Then we feed the entire set of tokens into transformer (Vaswani et al., 2017; Radford et al., 2019) to get output embeddings. Since the points in a point cloud are unordered, the transformer has no positional embedding and does not use causal mask. We pass each point embedding through shared linear layer to get two predictions per point: one for the probability of the point being a salient point P and the other for the offset Φ. (xx, Ui, =;) between the point position and target waypoint position, illustrated by the middle "Prediction" panel of Fig. 3.
Instead of using a hard one-hot target for salient point prediction, we construct a soft salient map over points where the probability of each point is given by:
P: x ||| - llll if ||q: - cxlle S r else 0
Here, ls is the index of the point selected by the user and is a hyperparameter defining a neighbor- hood of points that are salient. Within this radius, the probability of saliency decreases with distance
5

---

Published as a conference paper at ICLR 2025
Input
Prediction
Ground Truth
Desenstmpingg (PP5) & Linear Projection
Offset
Salient
Pred.
Pred.
Action, Salient Point
Pi
356
Waypoint
Transtormer n seappt
Offieet
Rut Tokem
Rot P.vd.
I. uuiinnt
Gripper loken
Gripper Pred.
ModePPodd
Architecture & Training Objectives: SPHINX takes downsampled point clouds as input, generating per-point tokens es, and uses a Transformer-style architecture to predict salient points and waypoint actions (position, orientation, gripper state). Specifically, SPHINX predicts the waypoint's positional component as offset from salient point. The model outputs per-point translational offset é;,, but only penalize the offset loss salient points (shaded) during training. Salient point prediction supervised using cross-entes entsopy loss (Laalien) between predicted P. and ground truth p. salient probabilities.
Figure 3:
to the ground-truth point. In practice, we set to r to 5em in all of our experiments. This target distribution salient points illustrated by the red points on the right-most panel in Fig. 3, where different shades of red represent the magnitudes of the probability pt. We train saliency prediction using cross entropy loss on the predicted salient probability p.:
Lsalient - Σ,
Ps log P.
(2)
The goal for the offset prediction is to recover the target end-effector location relative to a predicted salient point, which intuitively aims to ground the waypoint prediction in task-relevant features. This can potentially provide more spatial grounding and awareness that can help with the policy more effectively generalizing. Since we only care about the predicted offset from points with high saliency at inference time, we only penalize the offset loss on the points that matter, whose ground- truth salient probabilities p; are nonzero. Assuming & is the position the target end-effector location, ;P; is the location of the i-th point, the loss for offset prediction &; is:
Loffset
1p 0} E, fi > 0) ||E -;; - -611
(3)
The prediction targets for rotation, gripper state and next mode are the same regardless of which point is selected. Therefore, we predict them from the output embedding their respective tokens. We use the mean squared error (MSE) loss for rotation (BCE) loss for the binary gripper state Lgripper. and negative log-likelihood (NLL) loss for next mode classification Lmode- Since these losses are standard, we move them to Appendix A for conciseness. Finally, the full waypoint loss is the sum of all terms, and we find that simple unweighted sum works well in practice, eliminating the need of additional hyperparameters:
euler angles Love, binary cross entropy
Twayypoitt = L-galient + Toffsst + Tupo + Lgripper + I.mode
By default, the waypoint dataset only consists of {or, a,, waypt, We, 2)} for timesteps t where the demonstrator explicitly specified a waypoint. However, because we use a controller to move the end-effector between its current pose and the waypoint via h, interpolated actions, interpolated steps as additional data points to train on with the same target waypoint action label. We can expand each observation-action pair a sequence:
Wr,
treat
{(o,, dr. waypt, we, 2x) stst' akr)
(5)
where a is hyperparameter specifying how much of the interpolated data to train on.
4.3 THE DENSE POLICY OF SPHINX
The waypoint policy л "aypt (wylo:) guides the robot to reach objects in a desired pose. How- ever, long-horizon tasks often contain sub-tasks like insertion alignment that require finer- grained actions. These parts of the task would be easier to accomplish through direct, per-time- step teleoperation, rather than sequence of waypoints. To address this, we train dense policy,

---

Published as a conference paper at ICLR 2025
d (aclop which takes over near salient point to perform precise manipulation before handing control back to the waypoint policy. Note that this policy uses the wrist camera instead of the point cloud of for nigher-resolutin perception of close-up object details. Ignoring global observations also encourages generalization to scene variations since nT densee only operates locally,
We instantiate the dense policy of SPHINX with diffusion policy (Chi et al., 2023) which has been shown to work well in wide range of manipulation tasks. To allow the dense policy to switch back to waypoint mode, we augment its action with an additional mode prediction dimension that predicts the next mode m, {0, 1, 2) corresponding to {waypt, dense, terminate) modes. Fig. illustrates how the dense policy fits into the entire SPHINX framework. We train the dense policy using the entire dataset and use the interpolated steps to augment our data. This provides the policy with more data and encourages it to be robust to slightly early or late mode switches.
5 EXPERIMENTS
In this section, we evaluate how SPHINX'S attention to salient points and hybrid policy architec- ture impacts its performance and generalization on suite of four challenging real-world and two simulated tasks. In all experiments, we assume access to two external camera viewpoints and a wrist-mounted camera on a Franka Panda robot. See Appendix B for implementation details.
5.1 EVALUATION ON PRECISE AND LONG-HORIZON TASKS
We first evaluate SPHINX'S performance on complex, long-horizon tasks that demand precision. We hypothesize that by interleaving waypoint actions predicted from point clouds, and dense tions predicted from close-up wrist-camera images once near a salient point, SPHINX will more effectively be able to complete this class of tasks compared baselines which do not exploit salient points or a hybrid mode-switching policy. To assess this, we consider 3 challenging real-world tasks which we found to be infeasible teleoperate in waypoint-mode alone due to the required degree of precision and reactivity: Cup Stack (30 demonstrations), Train Track (30 demonstrations), and Coffee Making (60 demonstrations). See Fig. 5 for visualizations of each task and example rollouts. For each task, we consider a large and diverse space of initial configurations (Fig. 6), varying the relative locations of objects (cups, mugs, coffee pods, machine, train track).
Baselines: We compare against three baselines, the first being Diffusion Policy (DP) from Chi et al. (2023) with images from all three cameras as input. We intend for this baseline to demonstrate the benefit of waypoint modes (or lack thereof) for challenging manipulation tasks. The second baseline is HYDRA (Belkhale et al., 2023), hybrid IL. policy which takes images from all three views as input and outputs waypoint and dense actions. Given that it is mmage-based, HYDRA uses a multihhaded policy with a shared image encoder input to waypoint, dense, and mode prediction heads. Its waypoint prediction head outputs a waypoint without any intermediate salient representation. This baseline aims to explore the effects of separate input modalities (point cloud vs. wrist image) for waypoint and dense modes, as well as the benefit of using salient points for grounding. The original HYDRA implementation used simple MLP for the dense head, but update it to a diffusion policy for fairness. The last baseline is Fine-tuned OpenVLA where we fine-tune the recent OpenVLA model (Kim al., 2024) on our single-task datasets. We use it as an independent baseline given the recent trend of finetuning models trained on large prior datasets. Due to its design restrictions, this model can only take a single third-person image as input.
In Fig. 4 (left), SPHINX achieves the best performance across all three challenging tasks. Fine-tuned VLe struggles to achieve the required precision, lacking mode switches and close-up wrist images. HYDRA shows nonzero performance but suffers from inaccurate waypoint predictions without salient point attention and point clouds. Diffusion policy performs the best among the baselines but struggles to generalize across initial configurations without the waypoint mode. As shown in Fig. 6, SPHINX generalizes better across various object placements, while baselines tend to memorize few arrangements. Overall, these tasks are highly unforgiving of grasping failures and imprecision. Baseline methods particularly struggle to make progress in the train and coffee tasks where early mistakes (missed grasps placements with the mug, coffee pod, train) derail an entire rollout. SPHINX's 's use of salient point attention and close-up wrist images allows the policy to carefully proceed through difficult task phases, leading higher success across scenes.
7

---

Published as conference paper at ICLR 2025
Hybrid Tasks
Waypoint-Oni Tasks
93.5
$6.5
16/20
14/20
67.5
20.0
65.0
60
9/20
7/20
60
56.0
29.0
40.0
20
2/20
w20 1/20
0/20
3/20
30
13.0
Cup Stack (30 demos)
Train Track &
Coffee
Robomimic
Robomimic Square O (50 demos)
Drawer (20 demos)
(50 demos) ami
(60 demos)
(20 demos)
OLenVLA
HYDRA
DP
3D Vanilla Waypoint
Vanilla Waypoint Aux. Salient SPHINX
Figure 4: Success Rates Across Tasks: Left: SPHINX outperforms image-only dense baselines VLP, diffusion policy) well as hybrid baseline (HYDRA) across 3 challenging real-world tasks (Fig. 5) collected with hybrid mode teleoperation. Train Track requires degree of precision that baselines lack, while SPHINX's use salient points and hybrid actions enables precise, long-horizon manipulation. Right: SPHINX performs 1.6x better than the SoTA image or point-cloud based diffusion policies across tasks teleoperated in only waypoint mode. Comparisons with the two vamilla waypoint baselines also show that hoth saliency prediction and the relative waypoint action representation contribute to SPHINX's 's strong performance.
Waypoint Mode
Dense Mode
Drawer Open
plck
aliiga place cup
Cup Štack
conneul (reck to bridge
place train
lift bridge to push Iruin
Train Track
pick & place mug
Insert
Coffee
Figure 5: SPHINX Rollouts: We evaluate SPHINX across suite of challenging real-world tasks subject
wide initial state variations. SPHINX'S waypoint mode alone precise enough to handle tasks like drawer opening, while the full hybrid policy leverages different action modes to tackle complex tasks such as cup stacking, building and playing with toy train set, and making coffee.
5.2 WAYPOINT POLICY ABLATIONS
Although SPHINX'S performance in challenging real-world tasks relies on the dense policy, the waypoint policy is crucial for strong performance and generalization as it reliably guides the robot task-relevant locations. In this section, we validate the design choices behind SPHINX'S waypoint policy through ablations and comparisons against state-of-the-an (SoTA) policies. To isolate the impact of the waypoint policy from that of the dense policy, conduct these experiments exclusively on tasks that can be teleoperated solely waypoint mode, without the use dense mode.
We posit that SPHINX achieve higher task success rate across wide range of initial spatial configurations by effectively using waypoints (via salient points and offsets) to reduce the action prediction horizon and maintain precision and spatial awareness. We consider three tasks, real
8

---

Published as a conference paper at ICLR 2025
DP (Dense Onlv)
DP (Dense Only)
HYDRA
DP (Dense Only)
Best Baseline
13/20
9/20
1/20
7/20
SPHINX
19/20
19/20
16/20
14/20
Figure 6: Distribution Successful Rollouts: Across 4 real-world tasks, we visualize initial state distri- bution for successful trials across methods by overlaying segmented initial images. Notably, SPHINX handles the widest degree of spatial variety, while achieving a much higher task success rate compared to the most com- petitive baseline. For particularly difficult tasks like Train Track and Coffee, the baselines tend to memorize motions for particular object arrangements (i.e. mug behind pods) rather than generalize spatial variations.
Drawer Unseen height
Drawer Novel viewpoint
Cup Stack
Cup Stack Novel viewpoint
Method
Visual distraction
Diffusion Policy
4/10
1/10
1/10
0/10
SPHINX
9/10
9/10
8/10
9/10
We compare SPHINX to DP on Drawer and Cup Stack across various unseen scenar- ios. In Drawer. SPHINX successfully opens the drawer al unseen heights by attending to the handle as salient point, while DP struggles. In Cup Stack, SPHINX maintains strong performance despite visual distractors, its salient points focus on the cups, and the wrist-only dense policy ignores surrounding scene changes. Finally, SPHINX generalizes to unseen camera viewpoints in both tasks, whereas DP's image-based approach suffers.
Table 1: OOD Results:
task of opening articulated drawer (Drawer, 20 demonstrations) and two simulated environments Robomimic (Mandlekar et al., 2021) (Can, demonstrations, and Square, 50 demonstrations).
Baselines: We compare SPHINX against four baselines. The first two are SoTA II. policies in robotics - dense-only image-based Diffusion Policy (Chi et al., 2023) and its point-cloud based extension 3D Diffusion Policy (3D DP) (Ze et al., 2024). We train them on the interpolated data for, ). In addition to RGB images from the same external cameras used by the point cloud-based methods, the image-basee DP also incorporates wrist camera inputs ensure optimal performance.
Additionally, we consider two waypoint baselines that can also be seen as separatet ablations over the importance of salient point and offset prediction. Vanilla Waypoint Transformer backbone as SPHINX but removes both salient point and offset prediction. instead adds Translation Token, similar the Rotation Token, the Transformer and directly predicts the target translation from the output of that token using MSE loss. Vanilla Waypoint + Auxiliary Salient Points predicts the translation the same way as vanilla waypoint but adds the salient point prediction of SPHINX as an auxiliary task. This is essentially SPHINX without offset prediction.
the input and
Results: Fig. 4 (right), we find that SPHINX achieves 1.6x better performance than the best SOTA image-based or 3D-based baseline policy. The vanilla waypoint bascline is only slightly favorable in the simplest Can task, but notably worse than the diffusion policies on the more complex Square and Drawer tasks. This is interesting it suggests that the shorter prediction horizon of waypoint policies alone not necessarily a benefit without predicting salient points. Adding the salient point prediction changing the action representation, especially in Square which requires attending to highly precise object parts such small tool handle. This suggests the utility of salient points for encouraging better action prediction. SPHINX'S 's dominant performance over the two waypoint baselines suggests the effectiveness of anchoring waypoint action prediction relative to salient points.
auxiliary task improves the performance of vanilla waypoint without

---

Published as a conference paper at ICLR 2025
Spatial Arrangements and Visual Distractors
Camera Viewpoint
Execution Speed
Train-Time
Test-Time
2x speedup
Figure 7:
Generalization Capabilittes: SPHINX generalizes across multiple axes on Cup Stack, handling static (e.g.. office supplies) and dynamic (e.g., moving toy train) distractors (see website). Using calibrated point clouds input further allows for generalization unseen camera viewpoints at test time, and the way- point controller enables accelerated execution speeds, completing Cup Stack 9.2 s. on average.
5.3 VISUAL GENERALIZATION
We next evaluate SPHINX'S ability handle visual rather than only spatial generalization on Drawer and Cup Stack. As seen in Table 1, SPHINX demonstrates a promising degree of generalization to visual distractors during execution, and unseen third-person camera viewpoints (Fig. 7), retaining high performance. This is likely enabled by using viexpoint-agnostic point clouds (assuming cal- ibration), salient points encouraging the policy to ignore distractors, and wrist-camera images in dense mode which are unaffected by surrounding changes in the surrounding scene. Diffusion pol- icy heavily overfits to the training scenes and suffers noticeable performance drop.
5.4 EXECUTION SPEED
key advantage of SPHINX over dense-only methods is its decoupled waypoint and dense policies. While dense methods are tied to the speed of actions recorded during data collection, SPHINX uses a waypoint controller that allows flexible execution speeds at test time. We specifically collect all data across all tasks using controller limited to maximum delta of I at 10Hz. After training SPHINX and dense-only diffusion policy on Cup Stack, we perform 10 trials of the task, where we compare SPHINX implemented with 2 sped-up controller (2 maximum delta at test-time) to DP trained on the 1 x data. SPHINX completes the task in an average of 99 seconds, a 1.7x speedup over diffusion policy (15.6 seconds). While further speed increases led controller imprecision, SPHINX has potential for even faster execution on more capable hardware.
6 LIMITATIONS AND CONCLUSION
SPHINX demonstrates strong performance and generalization across a range tasks, but our policy not without failures. The majority of SPHINX's failures stem from the dense policy being slightly imprecise for grasping or manipulation. Although we mitigate this by using the dense policy only for short horizons salient points, performing the "last mile" of precise insertion alignment mains challenging for some tasks. Additionally, our data collection interface uses a linear controller reach waypoints. This currently limits SPHINX to fairly quasistatic tasks without fast, dynamic movements. Finally, the performance of our waypoint policy is limited by the quality of the input point cloud. We currently perform one-time calibration procedure to obtain multi-view extrinsics and point clouds, but sensor noise and calibration error is not completely avoidable.
To summarize, we present SPHINX, visuomotor IL policy which learns perform complex manip- ulation tasks from limited amount of demonstrations while generalizing across many axes: novel spatial arrangements, visual distractors, novel viewpoints, and even customizable execution speeds (Fig. 7). SPHINX achieves this using a hybrid policy architecture that takes point clouds and wrist- images as input, and outputs waypoints and dense actions guided by salient points. SPHINX achieves an average success rate of 86.6% across 2 simulated and 4 real-world high-precision, long-horizon tasks, including making coffee and assembling a train set with several pieces. Our policy outper- forms state-of of the- IL baselines by 41.1% on average across 440 real world robot trials. Avenues for future work include improving dense mode manipulation with additional sensing modalities such tactile sensors, extending our data collection interface support dynamic manipulation tasks, and deploying SPHINX X in the wild across a range of real-world environments.
10

---

Published as a conference paper at ICLR 2025
ACKNOWLEDGMENTS
This work is part supported by funds from NSF Awards 2132847 and 2218760, the Office of Naval Research under ONR K00014-22-1-2293, DARPA TIAMAT, DARPA #W911NF22100 as well as Intrinsic and the Stanford Institute for Homan-Cenneered Artificial Intelligence (HAI). Priya Sundaresan is supported by NSF GRFP. We would like to thank Suvir Mirchandani, Suneel Belkhale and Yuchen Cui for their helpful feedback and suggestions.
REFERENCES
Christopher Agia, Toki Migimatsu, Jiajun Wu, and Jeannette Bohg. Taps: Task-agnostic policy sequencing. arXiv preprint arXiv:2210. 12250, 2022.
Michael Ahn, Anthony Brohan, Noah Brown, Yevgen Chebotar, Omar Cortes, Byron David, Chelsea Finn, Chuyuan Fu, Keerthana Gopalakrishnan, Karol Hausman, Alex Herzog, Daniel Ho, Jasmine Hsu, Julian Ibarz, Brian Ichter, Alex Irpan, Eric Jang, Rosario Jauregui Ruano, Kyle Jeffrey, Sally Jesmonth, Nikhil Joshi, Ryan Julian, Dmitry Kalashnikov, Yuheng Kuang, Kuang-Huei Lee, Sergey Levine, Yao Lu, Linda Luu, Carolina Parada, Peter Pastor, Jornell Quiambao, Kanishka Rao, Jarek Rettinghouse, Diego Reyes, Pierre Sermanet, Nicolas Sievers, Clayton Tan, Alexander Toshev, Vincent Vanhoucke, Fei Xia, Ted Xiao, Peng Xu, Sichun Xu, Mengyuan Yan, and Andy Zeng. Do as i can and not as i say: Grounding language in robotic affordances. In arXiv preprint 111.0, 2022.
Christopher G Atkeson and Stefan Schaal. Robot learning from demonstration. In ICML, volume 97, pp. 12-20, 1997.
Suneel Belkhale, Yuchen Cui, and Dorsa Sadigh. Hydra: Hybrid robot actions for imitation learning. In Conference on Robot Learning, pp. 2113-2133. PMLR, 2023.
Homanga Bharadhwaj, Jay Vakil, Mohit Sharma, Abhinav Gupta, Shubham Tulsiani, and Vikash Kumar. Roboagent: Generalization and efficiency in robot manipulation via semantic augmenta- tions and action chunking. In 2024 IEEE International Conference on Robotics and Automation (ICRA), pp. 4788-4795. IEEE, 2024.
Kevin Black, Noah Brown, Danny Driess, Adnan Esmail, Michael Equi, Chelsea Finn, Niccolo Fusai, Lachy Groom, Karol Hausman, Brian Ichter, Szymon Jakubezak, Tim Jones, Liyiming Ke, Sergey Levine, Adrian Li-Bell, Mohith Mothukuri, Suraj Nair, Karl Pertsch, Lucy Shi, and James Tanner. T0: A vision-language-aaee URL https:ttp///o www. physt cal intellll gence.. company / down oad/ /ttpppi../ 0. pd..
flow model for general robot control physical intelligence.
Jessica Borja-Diaz, Oier Mees, Gabriel Kalweit, Lukas Hermann, Joschka Boedecker, and Wol- fram Burgard. Affordance learning from play for sample-efficient policy learning. In 2022 International Conference on Robotics and Automation (ICRA), pp. 6372-6378. 2022. doi: 10. 10.1109/1CRA46639 2AW.ICCRAAA63392222..556 2022.9811899999
Cheng Chi, Siyuan Feng, Yilun Du, Zhenjia Xu, Eric Cousineau, Benjamin Burchfiel, and Shu- ran Song. Diffusion policy: Visuomotor policy learning via action diffusion. arXiv preprint arXiv:2303.0413 04137, 2023.
Cheng Chi, Zhenjia Xu, Chuer Pan, Eric Cousineau, Benjamin Burchfiel, Siyuan Feng, Russ Tedrake, and Shuran Song. Universal manipulation interface: in-the-wild robot teaching with- out in-the-wild robots. arXiv preprint arXiv:2402.1033 2024.
Murtaza Dalal, Deepak Pathak, and Russ R Salakhutdinov. Accelerating robotic reinforcement learning via parameterized action primitives. Advances in Neural Information Processing Sys- tems, 34:21847-21555555 2021.
Theophile Gervet, Zhou Xian, Nikolaos Gkanatsios, and Katerina Fragkiadaki. Act3d: 3d feature field transformers for multi-task robotic manipulation, 2023. URL https: pestts://axxite wttp/iiivvvoor/.eeeeee........oa abs/2 17817.
11

---

Published as a conference paper at ICLR 2025
Ankit Goyal, Jie Xu, Yijie Guo, Valts Blukis, Wei Chao, and Dieter Fox. Rvt: Robotic view transformer for 3d object manipulation. In Conference on Robot Learning, pp. 694-710. PMLR, 2023.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the IEEE conference computer vision and pattern recognition, pp- 770-778, 2016.
Frederik Ebert Jedrzej Orbik. Oculus reader: Robotic teleoperation interface, 2021. URL https: //github. com/ http/raaabbbbeeeeeeeeee httpe/eeeeeeeeeeclueeeeeeeeeeeee http:llyyyocuuuurreeeeeeee Accessed: 2024-09-20.
Moo Jin Kim, Karl Pertsch, Siddharth Karamcheti, Ted Xiao, Ashwin Balakrishna, Suraj Nair, Rafael Rafailov, Ethan Foster, Grace Lam, Pannag Sanketi, et al. Openvla: An open-source
vision-language-action model. arXiv preprint (tfi:22006009444. 2024.
Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In International Conference on Learning Representations (ICLR), 2015.
Ajay Mandlekar, Danfei Xu, Josiah Wong, Soroush Nasiriany, Chen Wang, Rohun Kulkarni, Li Fei- Fei, Silvio Savarese, Yuke Zhu, and Roberto Martín-Martín What matters in learning from offline human demonstrations for robot manipulation. In Conference Robot Leurning (CoRL), 2021.
Soroush Nasiriany, Huihan Liu, and Yuke Zhu Augmenting reinforcement learning with behavior primitives for diverse manipulation tasks. In 2022 International Conference on Robotics and Automation (ICRA). pp. 7477-7484. IEEE, 2022.
Dean A Pomerleau. Alvinn: An autonomous land vehicle in a neural network. Advances in neural information processing systems, 1, 1988.
Charles R. Qi, Li Yi, Hao Su, and Leonidas J. Guibas. Pointnet++: deep hierarchical feature learning point sets in metric space. In Proceedings of the 31 International Conference on Neural Information Processing Systems, NIPS' 17, pp. 5105-5114, Red Hook, NY, USA, 2017. Curran Associates Inc. ISBN 9781510860964.
Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019. URL https://apl.
Language models are unsupervised multitask learners. semanticschole.
wwwhoola.rrrg/Coor wrgg/Crrpsss.......ooooo 1 160025533.
Moritz Reuss, Maximilian Li, Xiaogang Jia, and Rudolf Lioutikov. Goal conditioned imitation learning using score-based diffusion policies. In Robotics: Science and Systems, 2023.
Ashutosh Saxena, Justin Driemeyer, and Andrew Y. Ng. Robotic grasping of novel objects using vision. The International Journal of Robotics Research, 27(2):1571773, 2008. doi: 10.1177/ 0278364907087172. URL https::/p//...c....t.
Stefan Schaal. Learning from demonstration. Advances in neural information processing systems, 9, 1996.
Lucy Xiaoyang Shi, Archit Sharma, Tony Z Zhao, and Chelsea Finn. Waypoint-based imitation learning for robotic manipulation. arXiv preprint arXiv:2307.14326, 2023.
Mohit Shridhar, Lucas Manuelli, and Dieter Fox. Perceiver-actor: A multi-task transformer for robotic manipulation. In Proceedings of the 6th Conference on Robot Learning (CoRL), 2022.
Mohit Shridhar, Lucas Manuelli, and Dieter Fox. Perceiver-actor: multi-task transformer for robotic manipulation. In Conference on Robot Learning, pp. 785-799. PMLR, 2023.
Priya Sundaresan, Suneel Belkhale, Dorsa Sadigh, and Jeannette Bohg. Kite: Keypoint-condittonee policies for semantic manipulation. arXiv preprint arXiv:2306. 16605, 2023.
Martin Sundermeyer, Arsalan Mousavian. Rudolph Triebel, and Dieter Fox. Contact-graspnet: Ef- ficient 6- grasp generation in cluttered scenes. In 2021 IEEE International Conference on Robotics and Automation (ICRA), pp. 13438-13444 IEFE, 2021.
12

---

Published as a conference paper at ICLR 2025
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, ukasz Kaiser, and Illia Polosukhin. Attention all you need. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Gareett (eds.), Ad- vances in Neural Information Processing Systems, volume 30. Curran Associates, Inc., 2017. URL https: http:s/:////c/eeeeeeooeo Yeatman, Jiuguang Wang, Robin Walters, and Robert Platt. Equivariant diffusion policy. arXiv preprint arXiv:2407.018// 2024.
Guyon, U. Von
Dian Wang, Stephen Hart, David Surovik, Tarik Kelestemur, Haojie Huang, Haibo Zhao, Mark
Jingyun Yang, Zi-ang Cao, Congyue Deng, Rika Antonova, Shuran Song, and Jeannette Bohg. Equibot: Sim e3)vivirrr diffusion policy for generalizable and data efficient learning. arXiv preprint arXiv:2407.014: 04400, 2024a.
Jingyun Yang, Congyue Deng, Jimmy Wu, Rika Antonova, Leonidas Guibas, and Jeannette Bohg. Equivact: Sim (3)-quuvariiant visuomotor policies beyond rigid object manipulation. In 2024 IEEE International Conference on Roboties and Automation (ICRA), pp. 9249-9255 IEEE, 2024b.
Tianhe Yu, Ted Xiao, Austin Stone, Jonathan Tompson, Anthony Brohan, Su Wang, Jaspiar Singh, Clayton Tan, Jodilyn Peralta, Brian Ichter, et al. Scaling robot learning with semantically imag- ined experience. arXiv preprint arXiv:2302. 11550, 2023.
Yanjie Ze, Gu Zhang, Kangning Zhang, Chenyuan Hu, Muhan Wang, and Huazhe Xu. 3d diffusion policy, arXiv preprini M:Xiv:2403.03900 2024.
Tong Zhang, Yingdong Hu, Jiacheng You, and Yang Gao. Leveraging locality to boost sample efficiency in robotic manipulation. arXiv preprint arXiv:2406. 10615, 2024.
Tony Z Zhao, Vikash Kumar, Sergey Levine, and Chelsea Finn. Learning fine-grained bimanual manipulation with low-cost bardware. arXiv preprint arXiv:2304113705 2023.
13

---

Published as a conference paper at ICLR 2025
A ADDITIONAL LOSSES FOR THE WAYPOINT POLICY IN SPHINX
In Section 4.2 we define the salient point prediction loss and offset loss for SPHINX. Here we complete the full loss definitions for the waypoint policy. Recall that the waypoint policy needs to predict translation É. rotation &, binary gripper state 9 and next mode m.. As described in the main paper, the translation prediction is decomposed to first predict salient points (Eq. (2)) and then predict offset & w.r.t. the salient point (Eq. (3)). We represent rotations in Euler angles and loss is mean squared error (MSE) between the prediction & and target a::
Lix -
(6)
Although MSE on Euler angle ignores the wrap-around effect, i.e. -IT and я represent the same rotation but the loss is not O, we choose it for its simplicity and find it to work well in practice. Other representations for rotation and their corresponding losses, such as quaternion, should work similarly. The gripper state is binary with 1 for open and 0 for close, Assuming g is the ground-truth gripper state and ý is the predicted probability of the gripper being open, then the gripper loss is binary cross entropy loss:
L Lyripper = g log 9 + (1 - g) log(1 g)-
The waypoint policy also need predict the next mode, which is three-way classification among candidates {waypt, dense, terminate). We train it via negative logliikeihood:
Imode = log m
(8)
where is the predicted probability for ground-truth mode m. Finally, the full waypoint loss the sum of all terms, and we find that simple unweighted sum works well in practice, eliminating the need of additional hyperparameters:
Layyoint
Lisatinn + Lottser + Liva + Leripper + Lmmode
(9)
B IMPLEMENTATION DETAILS OF SPHINX
B.1 WAYPOINT POLICY
The waypoint policy in SPHINX is using a Transformer predict salient probability and offset per point, as well as to predict rotation, gripper and mode using additional tokens similar to the [CLS] token in visual classification. The Transformer has 6 layers and each layer has 512 embedding dimensions over 8 attention heads. We remove positional embeddings from Transformer as the point cloud input has no ordering. We set dropout 0.1 for all Transformer blocks to avoid overfitting. We optimize the waypoint policy with Adam (Kingma & Ba, 2015) optimizer with base learning rate 1e-4 and cosine learning decay over the entire training process, i.e. decaying to 0 at the end of training. We clip the gradient with maximum norm 1. We set batch size to 64. We also maintain an exponential moving average (EMA) of the policy with the decay rate annealing from to 0.9999. We use the final EMA policy in all evaluations without any further model selection. All waypoint policies are trained for 2000 epochs.
As mentioned in the Section 4.2, for waypoint training, a technique we refer to as temporal augmentation:
observations from interpolated steps data augmentation
{(or, de. waypt, wy, mr) sts t' + +krr
(10)
where t' is the initial step when the demonstrator specified the current waypoint, bt is the total num- ber of steps it takes for the controller C' to execute this waypoint. Essentially we train π *aysr (wije ) on the initial or as well as the intermediate observations 0f+1:'' Here a means no tem- poral augmentation and a = 1 means training on the entire waypoint segment. In practice, we find that setting a = 0.2 strikes a balance between sufficient augmentation while avoiding interpolated observations that occur too late in waypoint segment, which can cause the policy to confuse the current target waypoint with the next one. As a concrete example, the waypoint dataset for the Robomimic Square task contains 50 demonstrations, each containing 6 waypoints. The raw dataset for training the waypoint policy contains 300 examples. With the temporal augmentation, the dataset now contains roughly 1800 examples, increasing the amount of data by times. All the waypoint
14

---

Published as a conference paper at ICLR 2025
Env Waypoint Method
Can (20 demos)
Square (50 demos) Vanilla (+ Aux SP)
Vanilla
Vanilla (+ Aux SP)
SPHINX
Vanilla
SPHINX
w/o Temp. Aug.
68%
70%
77.5%
8%
68%
75.5%
w/ Temp. Aug.
70%
78%
93.5%
13%
65.5%
86.5%
Table Waypoint Policy Ablations
policies. including vanilla waypoint baselines in Section 5.2 are trained with temporal augmentation a = 0.2 as it improves the performance for all of them. The performance of each waypoint policy with and without temporal augmentation is listed in Table 2. We can see that temporal augmentation improves the performance of waypoint policies in five out of six scenarios and achieves similar per- formance in the remaining one. SPHINX benefits the most from the augmentation technique, with 13.5% average improvement across the two environments.
Apart from the temporal augmentation, we also apply random translation augmentation to the point cloud input and target action. The amount of random translation is sampled uniformly from [-5em, 5cm] along 2, y, 2-axes. We follow the common practice to crop the point cloud to remove points outside of the workspace but do not apply any other vision pipelines (i.e. no object detection nor segmentation) to preprocess the point cloud.
B.2 DENSE POLICY
The dense policy in SPHINX is diffusion policy. We closely follow the original implementation of Chi et al. (2023). Specifically, we use ResNet-18 (He et al., 2016) encoder to process the wrist image and append the proprioceptional to the image embedding before feeding it to a I-D convo- lutional UNet for action denoising. The diffusion policy is trained with DDPM to predict the noise given the noisy action as input and observation context. We follow the best practices of training it using Adam (Kingma& & Ba, 2015) with weight decay, cosine learning rate schedule and take the exponential moving average of the policy as final policy for evaluation. Our implementation is able to reproduce the results on Robomimic from the original paper.
C EXTENDED RELATED WORK
Relationship between Salient Point and Affordance. Affordance learning is common concept in robotics manipulation. can refer to classifying the nature of interaction for certain object points (e.g., tool handle is "graspable," a button is "pressable") (Borja-Diaz et al., 2022), or more broad action candidates or success possibilities associated with objects (Ahn et al., 2022). Salient points in SPHINX can be thought of as form of per-point affordance, but must be combined with offsets to specify how the end-effector should interact with a given point. Critically, we demonstrate that the per-point affordances in SPHINX allow the policy to focus task-relevant features and avoid paying attention to arbitrary objects, leading to robust execution in the presence of visual distractors.
Action Representation in Robotics. Several works have also considered using classification instead of regression for action prediction in the imitation learning setting. PerAct (Shridhar et al., 2022) divides the entire 3D workspace into voxels and converts the end-effector pose prediction problem into classification over the fixed set of voxels. The precision of prediction depends on the granularity of the voxel, and the number of action candidates grows cubically in the number of voxels, making it challenging for tasks that require high-precision. Act3D (Gervet et al., 2023) performs coarse- to-fine scoring for "ghost points", which addresses the granularity issue of PerAct, but still lacks any kind of explicit intermediate representation such as salient points. In comparison, SPHINX first predicts the salient point, a point that physically exists in the input point cloud, through classification and then predicts an offset w.r.t. the salient point to recover the full action. This allows SPHINX to be arbitrarily precise without incurring the high cost of having fine- -rained voxels.
SGRv2 (Zhang et al., 2024) is recent work that uses per-point offset prediction for robotic ma- nipulation. It predicts actions as per-point offsets for all points, and uses a weighted average to get the final action output. In contrast, SPHINX utilizes salient points learned from human labels as the anchor for offset prediction. Due to the existence of salient points, SPHINX applies the offset loss only to the proximal points of the "demonstrator speerfffe salient point", i.e. the red points in Fig. and Eq. (3). This an easier task for the neural network since it does not need to allocate capacity to predict offsets for points far from the salient points. Additionally, the salient point learning ob-
15

---

Published as a conference paper at ICLR 2025
jective is one of the main strengths of SPHINX. The ablation in the right panel of Fig. 4 shows that the classification loss is helpful. Even having it as an auxiliary task significantly improves the performance of the "vanilla waypoint" policy. On the implementation side, SPHINX uses GPT-2 style Transformer while SGRv2 uses PointNeXt as the backbone. Although both methods should work well with either architecture choice but think it is interesting see generic architecture not specialized for point clouds to perform quite well in these tasks.
D IMPLEMENTATION AND PERFORMANCE OF BASELINES
In this section we discuss the steps we have taken to ensure that the performance of our baselines is valid and discuss potential reasons for the low performance for baselines like Hydra and OpenVLA..
Diffusion policy (DP): Our DP implementation is able to reproduce the reported results from the original paper and it achieves the reported performance on Robomimic Can and Square with 200 demonstrations from the original Robomimic dataset. It uses the same set of cameras (3rd person and wrist) as SPHINX. In the SPHINX experiment we collected 50 demonstrations for the Square and we have verified that the DP trained on the Sphinx dataset (44%, 50 demonstrations) is similar to the one trained on 50 demonstrations from the Robomimic dataset (45%, 50 demonstrations).
3D diffusion policy (DP3): Our implementation closely follows the one from the original codebase. In the original paper, DP3 is not evaluated on the Robomimic benchmark. We find to perform similarly as DP on the real world Drawer task but perform slightly worse than DP on the Robomimic benchmarks and other more complicated real world tasks. This is reasonable given that DP3 purely conditions point clouds constructed from 3rd person cameras and does not use wrist camera images. The close-up information from the wrist camera images is crucial for Robomimic tasks as well as real world tasks. In comparison, SPHINX switches to wrist-view image based diffusion policy for fine-mand mane-manppultion, resolving the issue of lfck-of-detail in point cloud inputs.
Hydra: We modernize Hydra by using the diffusion policy as its dense policy while keeping the rest of the implementation as close to the original design as possible. The low performance of Hydra seems unreasonable first glance, but it can be explained via close look at their original results and our ablations. Our mafknng task is similar to the one in Hydra. However, the original Hydra paper collected 100 demonstrations with little variation on the location of the coffee machine, and put the cup and coffee pod on shelf to make them easier to pick up. In our case, we randomize the initial location and orientation of the cup. pod and coffee machine, and only use 60 demonstrations. Therefore, it is reasonable to expect much lower performance for Hydra on this task. Hydra's waypoint branch is similar to our "vanilla waypoint", i.e. directly predicting a target pose via regression, but uses images instead of point clouds. From the ablation (Fig. 4, right panel), we see that this vanilla waypoint policy noticeably worse than plain diffusion policy on the hard task (Square) that requires precision. Therefore, considering that Hydra's 's waypoint policy is worse than diffusion policy, it is not surprising to see that the full Hydra policy (vanilla waypoint diffusion policy) performs worse than diffusion policy in the real world tasks.
We follow the instructions and code provided by the OpenVLA authors and apply ev- erything in their fine-tuning script (which does use image augmentation). We believe the poor performance of VLA is not surprising for a few reasons. From Figure 5 (left half) of the VLe paper, we that fine-tuned Open VLA is worse than Diffusion Policy in 2 out of the cases in single-task setting on a Franka Panda robot arm. In general, Open distribution of pre- training data relies heavily on the Bridge Dataset, which has a major embodiment gap compared to our Franka dataset. Open VLA is only compatible with single 3rd person view, and cannot leverage a close-up wrist image view. On all of the real world tasks we did, the wrist camera view is crucial to sensing end-effector object aligmment for precise manipulation. For example, for cup stack (the easiest of the hybrid tasks), even being off by a centimeter can lead to knocking over one or both of the cups. Thus, Open VIA's poor performance is unsurprising given its limited observation space. Other recent work (Black et al.) reports similar trends on Open VLA where a fine-tuned VLA performs poorly, achieving 0% success rate on 4 out of 5 settings and v 35% on setting.
VLA:
16


Reviews:
[{"review_id": "7DnhkEFaV0", "reviewer": "Reviewer_7kbH", "summary": "This paper introduces an imitation learning framework, SPHINX (Salient Point-Based Hybrid ImitatioN and eXecution), designed to improve the generalization and efficiency of robotic control in complex manipulation tasks. Traditional imitation learning approaches often struggle with diverse visual and spatial variations, especially in tasks requiring long-term planning and precision. SPHINX addresses these challenges by utilizing a hybrid approach that combines two modes of execution: waypoints for large, coarse movements and dense actions for precise, close-up interactions. They show extensive empirical results that demonstrate SPHINX's effectiveness across both real-world and simulated tasks, showing a substantial performance improvement over existing IL baselines.", "questions": "1. There seems to be a very basic error in the paper: I believe the EE pose should be 6 DoF, but the paper frequently mentions 7 DoF. Is this due to the use of quaternions for rotation, or does it include the EE’s open/close state as an extra DoF?\n2. Generalization problem. Does the generalization reported in the paper include generalization across objects of different appearances and shapes, or is it limited to generalization over the initial pose?\n3. Mode switching and subtask decomposition. The paper does not seem to address subtask decomposition for long-term tasks, appearing to rely on mode switching. For the two proposed modes, how does the model ensure continuity across temporal sub-steps, and how does it detect the salient point corresponding to the next subtask (distinct from the previous step)?\n\nOverall, this paper is in good shape, with a reasonable method, sufficient experiments, and clear implementation details. If the above concerns and questions can be addressed, I would raise the score.", "limitations": "", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. This paper proposes a two-stage framework. The first stage leverages salient points for generating long-range movements, specifically as waypoints, followed by interpolation for motion planning and control. The second stage employs an image-based action similar to a diffusion policy, where actions are based on EE poses. This framework is reasonable for multi-stage manipulation tasks that require contact.\n2. The paper is relatively complete, covering the proposed method, dataset collection, simulations, and real-world experiments. The baselines are adequately diverse, and the real-world experimental demonstrations are effectively presented.\n3. The writing in this paper is clear, with accurate illustrations that aid understanding of the proposed framework. Explanations of the data collection module and experiments are well-supported with corresponding videos.", "weaknesses": "1. The novelty of the paper could be further clarified. Overall, applying a hierarchical approach to improve generalization and performance in IL is reasonable. This is novel within IL (only my point, if other reviewers can provide references, I will defer it). However, using a two-stage approach (i.e., long-range navigation + fine-grained manipulation) to handle long-term manipulation tasks is a common strategy, and I suggest the authors add a discussion on this.\n2. There are some concerns regarding the visual module. The first stage’s visual module appears to use a transformer for salient point extraction, which resembles affordance learning. It would be beneficial to include a discussion on affordance learning. Additionally, the paper seems to use a limited training set for salient point training, such as 30 samples for cup stacking. Without pre-training, I am skeptical that these datasets can achieve accurate and generalizable visual perception.\n3. Concerns related to the robotics aspect. The paper does not seem to mention object segmentation or part-level pose estimation, which also appear to be integrated within the waypoint transformer. It is unclear if this approach is effective or if additional processing was performed in the real-world experiments.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "g1H3C7YMjs", "reviewer": "Reviewer_4Uf5", "summary": "In this work, the authors consider the task of imitation learning in the context of tabletop robot manipulation tasks. Specifically, they investigate the well-known tension between the need for both long-horizon and short-horizon control: e.g. non-dexterous freespace motion for a large fraction of a trajectory, followed by a short dexterous stretch. They articulate the hierarchical nature of many manipulation tasks, and propose to address the hierarchy with a hybrid policy which can switch between two modes: a high-level waypoint prediction which is grounded in a global observation point cloud, and a low-level fine-grained control which operates on wrist camera inputs. They describe a policy architecture and training procedure to fit this paradigm, as well as a data collection/annotation method for generating datasets with required annotations suitable for training. They then demonstrate the efficacy of their policy on 3 custom real-world manipulation tasks.", "questions": "* Can the authors speak to sample complexity analysis for the different methods considered?\n* How could this be used / modified to not require human annotation of mode/salient points directly?\n* How does this method perform on simulated benchmarks w/ published baselines/metrics?", "limitations": "", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 2, "strengths": "# Originality\n\n* While hierarchical approach has been proposed, for instance in HACMan, NDF, TAX-Pose, etc, and explicit mode-switching between policies has been explored, I haven’t really seen this sort of mode switching in imitation learning for long-ish horizon manipulation.\n* Giving the policy a mechanism to change its inputs is a neat design consideration\n* First time seeing diffusion policy in a hierarchical context.\n\n# Quality\n\n* The real-world experiments are well-designed, and demonstrate good performance on a relatively small number of demos\n* The idea is straightforward and a reasonable solution to this class of quasistatic manipulation problems.\n* Decent ablation coverage.\n* The design of the visual generalization experiments was interesting.\n\n# Clarity\n\n* Very clear and well-written.\n\n# Significance\n\n* Reasonable factorization of the problem, so might be useful for for practitioners looking to solve this class of problems expediently and don’t have needs for generalizing outside of quasi-static pick-and-place tasks.", "weaknesses": "* The model’s hypothesis class (e.g. mode switching between long-distance free space motion, and short horizon fine-grained manipulation) imposes a major constraint on the kinds of problems it can represent effectively. Of course, in the extreme case, either one or the other mode can always be predicted, but in these settings no benefit / insight is offered, and hard attention switching across the inputs limits flexibility.\n* The assumptions about dataset preparation are a major weakness/limitation here, for two reasons:\n    1. Adding a different label for mode is a major annotation requirement, and can be quite noisy/arbitrary.\n    2. The notion of labeled “salient points” may make some sense because humans do attend to salient points reasonably consistently, but generally speaking they are not a consistent/principled thing to expect a human to label - who is to say which specific points should be annotated? Seems quite noisy, and a major source of error.\n* Comparisons were somewhat weak.The authors assert that long-horizon goal prediction is not precise enough for this kind of imitation learning, but evidence from e.g. 3DDA, RPDiff, NDF, TAX-Pose, etc. demonstrate that the quasistatic problems considered in the real-world tasks are probably solvable with goal prediction.\n* Benchmark tasks are also somewhat weak. While the real-world experiments are good, the authors should compare against other methods on simulated benchmarks (including annotating using their system or a synthetic annotation scheme) such as RLBench where some medium-horizon tasks have been examined with a litany of different approaches.\n* There’s not much analysis of sample complexity.\n* Several related works are missing / not discussed:\n    * Spatial grounded policies: Spatial Action Maps, Transporter Nets, HACMan, 3D Diffuser Actor\n    * Relative placement / reasoning: NDF, TAX-Pose, etc.\n    * Keyframing: Many of Stephen James’ works on top of RLBench look at long-horizon gripper positional reasoning and keyframing.\n    * The “soft salience” mechanism has been used before (e.g. TossingBot, many others), so while it’s pretty standard it would be good to compare / reference\n* The variation for initial conditions in the real-world experiments appear somewhat limited, and having the high-level policy run on a global world frame is a bit unfair as a selling point because it doesn’t have to generalize as much by construction (probably could modify the other methods to have similar properties).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "YP1sn433Og", "reviewer": "Reviewer_B6a8", "summary": "This paper builds on the HYDRA approach by splitting motion into two modes: waypoint-reaching and dense manipulation. The key improvements are: (1) using point clouds instead of RGB images for more robust waypoint localization and using separate PointNet, (2) upgrading the dense manipulation module from a simple MLP to a diffusion policy for smoother control, and (3) introducing \"salient points\" that explicitly signal when to switch between modes.\n\nThe method is evaluated on three challenging, long-horizon real-world tasks, with results showing clear performance gains over baseline approaches. The paper is clearly written, and the experiments are thoughtful, including interesting tests on out-of-distribution scenarios.", "questions": "1. **Real-World Experiment Details**: Could the authors share more on how they measure success rates? Do you count broken down points as OpenVLA, or does it only reflect the final task outcome? A failure analysis would also be insightful.\n2. **Point Cloud Format**: What specific format of point cloud data does the waypoint transformer use? Are both RGB and XYZ coordinates involved, or only XYZ? As I am curious how the salient point prediction can distinguish the pink and yellow cup.\n3. **Handling Distant Salient Points**: When a salient point isn’t near a physical point—like opening an oven or a door—how does the method handle this? If there’s a substantial distance to any nearby object, does the model has to use the mode of dense manipulation?\n4. **OpenVLA Training Clarifications**: The performance of OpenVLA seems unexpectedly low in some tasks, like cup stacking. Could the authors clarify their training setup for OpenVLA? For example, was image augmentation applied, and was it trained on a single or multiple tasks?\n5. **HYDRA Performance Insights**: It would be interesting to understand why HYDRA, despite using a similar waypoint and dense manipulation modes, performs way less effectively. Does it struggle more in the waypoint phase or the dense manipulation phase?\n6. **Camera Viewpoint Calibration In Novel Viewpoint**: Were novel camera viewpoints recalibrated during inference? Additionally, are the training and inference data in the same coordinate system, like the robot’s base?\n7. **Waypoint Transformer**: It’s interesting that the waypoint transformer model can effectively handle unordered points with relatively small data. Could you provide more details on the network structure and training? What's more, were any data augmentations applied?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. **Enhanced Localization with Point Clouds**: The use of point clouds bolsters waypoint localization, offering a more resilient approach than RGB data alone.\n2. **Improved Precision in Dense Manipulation**: Replacing the MLP with a diffusion policy for dense manipulation leads to greater accuracy during complex manipulation phases.\n3. **Streamlined Mode Switching**: By introducing salient points, the authors provide a clear and novel mechanism for mode switching, simplifying task segmentation and improving execution flow.\n4. **Comprehensive Real-World Tests**: The real-world experiments are particularly strong, presenting complex tasks that highlight the robustness and versatility of the approach.", "weaknesses": "1. **Reliance on Salient Points**: The approach depends on expert-determined salient points, which may limit scalability, especially if multiple annotators are involved, as this could introduce inconsistencies in training data.\n2. **Lacks Details and Analysis in Experiemnts**: The baseline are too low to tell whether the baselines are correctly run and under fair comparison. It will be better to have some simple tasks which baselines can perform not that bad or provide a failure analysis about how they fail.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "iaV7x422T8", "reviewer": "Reviewer_57XM", "summary": "The paper introduces SPHINX which interleaves 3D saliency-based waypoint prediction with fine-grained policy that uses wrist camera image. The proposed method, SPHINX, is compared against SOTA imitation learning methods for robotic manipulation like Diffusion Policy, 3D Diffusion Policy, HYDRA, and also a fine-tuned openVLA. The authors also compare SPHINX with the ablated waypoint models without saliency points. The experiments were done on 4 real world tasks (cup stack, train track, coffee, open drawer) and two simulated tasks in Robomimic environemnt.", "questions": "1. Is there any positive transfer between the dense policy part and the waypoint prediction part? Or are these just separate models with no synergy in terms of learning?\n\n2. What specific advantage does SPHINX’s waypoint approach offer? I would expect SOTA keypose models like PerAct, Act3D, 3D Diffuser-Actor, and RVT-1/2 to outperform the proposed waypoint transformer. These models are more structurally sophisticated and battle-tested. They also incorporate locality bias, which I believe would provide benefits similar to those of the paper's saliency-based approach. Adding displacement to saliency essentially assigns affordance to specific, localized areas. Furthermore, these models do not require human-annotated saliency, and outputs SE(3) keypose which would be much stronger than R^3 waypoints.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "The proposed method seamlessly integrates multiple concepts in robotic manipulation such as waypoints, affordances, and coarse-to-fine, 2D+3D hybrid approach. These techniques were often explored separately in prior works. Leveraging these techniques, SPHINX effectively solves long-horizon tasks (due to the waypoint approach) and is robust to OoD settings (due to the 3D + saliency structure). Empirical results are strong enough to support these claims. SPHINX show substantial improvements over previous SOTA methods. Overall, the paper is of high quality.", "weaknesses": "The primary weakness of this paper lies in its lack of novelty. While the paper successfully integrates many concepts into one, each concept is not novel. For instance, guiding fine-grained manipulation policies with coarse waypoints is not a new approach. Similarly, using saliency prediction to assist manipulation has been widely addressed in prior works, often categorized as affordance-based approach. It is well known that 3D-based policies are inherently more robust than 2D models in OoD scenarios such as existence of distracting objects.\n\nAdditionally, there are several keypose-based methods (which are similar to waypoints) that use 3D point clouds, including PerAct, Act3D, 3D Diffuser-Actor, RVT-1/2. It is unclear why the waypoints of SPHINX is better than that of these models.\n\nLastly, the saliency-based waypoint approach has clear limitations. While it is indeed a very useful inductive bias for many tasks, it may harm the performance in certain cases, particularly when the grippers are far away from any point cloud in the scene. This isn't true for more general methods like Diffusion Policy. Thus, it’s unsurprising that SPHINX outperforms Diffusion Policy, for its added structure and inductive bias. Furthermore, SPHINX relies on additional human-annotated keypoints. While collecting such data is not particularly difficult, this limits SPHINX being applicable to tasks in which the concept of keypoint doesn't really make sense or is unidentifiable for human.\n\nOverall, while this paper presents a well-executed combination of existing ideas, it appears rather incremental. The experimental results are promising, making it a valuable contribution to a robotics-focused conference. However, the contribution may not be sufficient to recommend acceptance at an ML conference, where significant methodological novelty is typically expected.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ahnP3YHtS5", "reviewer": "Reviewer_gz3J", "summary": "This paper presents SPHINX, a novel framework for imitation learning in robotics. SPHINX integrates a hybrid approach by leveraging multimodal inputs (point clouds and wrist camera images) and using both waypoint-based and dense action predictions. It introduces a method for switching between these modalities and action types depending on the task phase, which enables better spatial and visual generalization. The paper demonstrates how SPHINX can handle long-horizon tasks with high precision, achieving superior results compared to state-of-the-art baselines, with an average performance improvement of 41.1%.", "questions": "For questions and concerns, please refer to the weaknesses section above. Overall, I believe this is a strong paper that effectively leverages the advantages of both waypoint and dense policies. The experimental results significantly outperform the next best state-of-the-art IL baseline, and the writing is clear, making the paper easy to follow and understand.", "limitations": "", "rating": 8, "confidence": 5, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "1. Effective Multimodal Input Usage: The use of point clouds for spatial awareness in long-range movements and wrist-camera images for precision is well-conceived. By switching between input modalities at the appropriate phases of a task, the system exploits the strengths of each modality to improve both efficiency and accuracy.\n\n2. Empirical Performance: The paper presents strong experimental evidence, showing significant performance improvements across multiple real-world and simulated tasks. Specifically, it demonstrates SPHINX’s ability to generalize across novel configurations, camera viewpoints, and visual distractors, which is critical for practical deployments in environments.", "weaknesses": "1. Similarity to SGRv2: I noticed that the waypoint policy in this paper is quite similar to the one proposed in SGRv2 [1]. However, the authors do not discuss this in the related work section, nor do they compare their method against SGRv2 in the experiments. Including a detailed comparison or discussion would help clarify how SPHINX improves upon or differs from existing approaches in waypoint-based policies.\n\n[1] Zhang, Tong, et al. \"Leveraging Locality to Boost Sample Efficiency in Robotic Manipulation.\" arXiv preprint arXiv:2406.10615 (2024).\n\n---\n\n2. Insufficient Visual Generalization Experiments: The experiments on visual generalization are somewhat limited. It remains unclear whether SPHINX’s salient point-based design allows the policy to generalize to different objects within the same category (e.g., cups of varying colors or shapes). Additionally, it would be valuable to test SPHINX’s robustness to changes in environmental factors such as background, lighting, or table textures, which are common in real-world settings. Expanding the generalization tests to cover these variations would provide a more comprehensive evaluation of SPHINX’s capabilities.\n\n3. Limited Exploration of Dynamic Tasks: While SPHINX excels at quasistatic tasks, its performance on dynamic tasks (e.g., tasks requiring fast, reactive control) is not explored. The reliance on a linear controller for waypoints may limit the system’s applicability in environments requiring fast, non-linear movements.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)