Modeling the Real World with High-Density Visual Particle Dynamics
Authors: William F Whitney
Jake Varley
Deepali Jain
Krzysztof Marcin Choromanski
Sumeet Singh
Vikas Sindhwani
Venue: CoRL 2024 (2024)
Keywords: point clouds
particle dynamics
world models for control
Performers
TL;DR: Scalable robotics world models with learned particle dynamics trained from RGB-D

Abstract: We present High-Density Visual Particle Dynamics (HD-VPD), a  learned world model that can emulate the physical dynamics of real scenes by processing massive latent point clouds containing 100K+ particles. To enable efficiency at this scale, we introduce a novel family of Point Cloud Transformers (PCTs) called Interlacers leveraging intertwined linear-attention Performer layers and graph-based neighbour attention layers. We demonstrate the capabilities of HD-VPD by modeling the dynamics of high degree-of-freedom bi-manual robots with two RGB-D cameras. Compared to the previous graph neural network approach, our Interlacer dynamics is twice as fast with the same prediction quality, and can achieve higher quality using 4x as many particles. We illustrate how HD-VPD can evaluate motion plan quality with robotic box pushing and can grasping tasks. See videos and particle dynamics rendered by HD-VPD at https://sites.google.com/view/hd-vpd.

Modeling the Real World with High-Density Visual Particle Dynamics
William F. Whitney Google DeepMind
Jacob Varley* Google DeepMind
Deepali Jain" Google DeepMind
Krzysztof Choromanski Google DeepMind
Sumeet Singh
Vikas Sindhwani
Google DeepMind
Google DeepMind
Abstract: We present High-Density Visual Particle Dynamics (HD-VPD), a learned world model that emulate the physical dynamics of real scenes by processing massive latent point clouds containing 100K+ particles. To enable ef- ficiency this scale, we introduce novel family of Point Cloud Transformers (PCTs) called Interlacers leveraging intertwined linear-attention Performer layers and graph-based neighbour attention layers. We demonstrate the capabilities of freedom bi- bi-manual robots the previous graph neural network ap-
HD-VPD by modeling the dynamics of high degree-of-freedom with two RGB-D cameras. Compared proach, our Interlacer dynamics is twice as fast with the same prediction qual- ity, and can achieve higher quality using 4x as many particles. We illustrate how HD-VPD can evaluate motion plan quality with robotic box pushing and can grasping tasks. See videos and particle dynamics rendered by HD- VDD https://sites http://ossssssssllleeee cww/cic//iaaiie wwon/viiw/// wwwrii/whddyyg hd-vpd.
Keywords: point clouds, particle dynamics, world models for control, Performers
Reconstruction
Prediction
Reconstruction
Prediction
HD-VPD
Ground
truth
Figure 1: HD-VPD accurately predict the dynamics of complex real-world interactions between robots (here, 16-DoF bi-manual Kukas) and objects/tools. Left: push-pedal trash-can opening task and Right: bimanual dustpan sweeping task. Top row: Renders from HD-VPD. The first image each pair is recon- struction of the matching input frame, and the second is prediction several timesteps into the future given a sequence of robot actions. Bottom row: Ground- -ruth test set video frames.
Introduction
Physical simulators are the linchpin of modern robotics, enabling cheap data collection, safe ver- ification of control algorithms, and real-time control via planning. Traditional analytic simulators [1. 2, 3, 4. 5] are fast and convenient to use, but lack the ability to precisely match the complex objects and dynamics of real-world scenes. Learned dynamics models make fewer assumptions on the form of objects and their interactions, but typically come with onerous constraints on their training data, e.g., requiring information such as 3D meshes and poses for all objects or per-object segmentation masks [6, 8].
Eqqua contribution. Correspondence to wwhitney@google.comm
8th Conference on Robot Learning (CoRI. 2024), Munich, Germany.

---

To address this problem, new classes of learned dynamics models have been proposed that can train directly on miew RGB-D observations, with no object-centrality in their data requirements or representations. A state-of-the-arr approach of this flavor is Visual Particle Dynamics (VPD) [9], which represents scenes as a collection of 3D particles whose interaction dynamics is governed by graph neural networks (GNNs) [10, 11] and rendered to images with conditional NeRF [12]. VPD models are trained end-to-end with a video prediction loss, support 3D state editing and multi- material dynamics modeling, and are data-efficient enough to model simple dynamic scenes with < 50 training trajectories. However, VPD has only been applied to simple simulated scenes under passive dynamics (without actuation), and it is unable to scale beyond ~30K particles due to memory and speed limitations. For applications in robotics, VPD needs two extensions: (1) to take robot actions into account, and (2) to be able to model environments with much higher level of detail. This is the focus of this paper.
We propose High-Density Visual Particle Dynamics (HD-VPD) world model which can train on robot interactions in real scenes and model their dynamics with 100K+ particles. To reach this scale, we propose new class of Point Cloud Transformers (PCTs) [13] called Interlacers, which inter- twine linear-attention Performer [14] layers and local neighborhood attention layers. We demon- strate the capabilities of HD-VPD by modeling the dynamics of bi-manual Kuka robots with multi- view depth input data. We find that Interlacer dynamics models are able to equal GNN's prediction quality in half the time using matched point cloud densities, and that they can exceed the best GNN's s quality by leveraging high-density point clouds with 4x as many particles as the GNN can fit in memory. We complement these results with illustrative experiments using HD-VPD for down- stream applications such as planning in manipulation, leveraging HD-VPD's ability to capture a scene from single observation and define goals in 3D space. See videos and particle dynamics predictions from our model at https: /tt////site /sittes.ss...//./ gowgle..ooo..oooooooooooo.oo.... com//iew/hddvpd.
Our main contributions are follows:
1. We present High-Density Visual Particle Dynamics (HD-VPD) world models that train end-to-end on real robotics data. They take RGB-D images and robot kinematic skeletons representing actions as inputs, and they predict future 3D particles and rendered images.
2. To enable HD- VDD to operate on large, detailed scenes, we propose a new class of Trans- formers for point clouds called Interlacers, which scale linearly (vs. quadratically) with point cloud size. Interlacers combine Befforme-PCTT layers [14, 15] and memory efficee-fficieent GNN-inspired local attention layers, enabling efficient global point-to-point attention (with the former) while maintaining high-fieelity local geometric detail (with the latter).
3. We show that HD-VPD can deliver realistic action-conditiones video predictions com- plex scenes of bi-manual robots interacting with various objects. We find that the Interlacer architecture enables fast, memory-efficiene
and high-fidelity video prediction with HD- VPD. Whereas typical NeRF models require tens or hundreds of camera views for training, HD-VPD uses only two, making real-world data collection straightforward. We use these models to illustrate downstream applications of HD-VPD in robotic bi-manual control, where the HD-VPD world model is able to predict the success or failure of candidate plans.
2 HD-VPD: High-density visual particle dynamics world models
HD-VPD 3D dynamics model disguised as a video prediction model. It takes RGB-D images and robot actions as inputs and predicts images of the future, just like video world model. Under the hood, though, represents both the current state of the world and the actions which operate on as particles in a shared 3D space. To make prediction several timesteps into the future, HD-VPD goes through three steps, shown in Figure 2:
1. Encode input images from all views and times into 3D particles that represent the state of the scene.
2. Predict the dynamics of the scene. Combine particles representing the first robot action with those representing the scene state and predict the change state. Use this update the state. Then repeat this procedure for the next robot action, and so forth.
3. Render the particles representing the predicted state of the scene into an image.

---

In this way, HD-VPD can make predictions many steps into the future while remaining in particle space, only rendering to pixel space as needed for training or visualization. The rest of this section describes the HD-VPD model, architectural choices, and training in more detail.
Treecin)
Oudfti trupo
Mial luggr
Thac
Benderer
Prediced Dertoaaleegg
Protital TWYI tragg
Figure 2: Overview of HD-VPD model with learned Encoders, Dynamics and Render. Encoders encode RGB-D images into point cloud representation with latent per-point features. Dynamics predicts the evolution of the conditioned the current scene well kinematic skeleton representing the motion of the robot. Renderer Point-NeRF style model which enables generation images of the predicted future scene. The entire model is trained end-to- end-to-end with a pixel-wise L loss.
Encoding HD- VPD receives RGB-D images from multiple cameras (in our experiments, at timesteps - l.... - for an input window of length throughout this work use | = 2. Each RGB image is first encoded into feature image of per- per-pixel feature vectors using U-Net [16]. Using known camera intrinsics and extrinsics, combined with the depth channel of the RGB-D input, each of these per-pixel features can be unprojected into the global coordinate frame, forming a point cloud where each point corresponds to input pixel. Each point is associated with the predicted feature vector from its corresponding pixel to form a particle represented as a (location, feature) tuple (x, f). The particles from across cameras merged together within one timestep before being subsampled uniformly at random to a fixed total number of particles N.
Action representation Unlike VPD, HD-vPD includes a representation of actions, allowing it to model robotic scenes and be used for planning and other forms controllable generation. We represent actions set of kinematic particles describing the motion of the robot across multiple timesteps: previous steps - and T 1, and next step T which represents where the robot plans to move. The particles from each timestep are located the joints, fingertips, and the base of the grippers of the two arms. Each kinematic particle associated with a feature vector consisting of two concatenated one-hot vectors: indicating which robot joint it is, and indicating which timestep it came from. By representing actions where they occur 3D space, HD-VPD can learn associate them directly the scene particles which they affect. More details and example in Appendix C.
Dynamics We employ the Interlacer architecture, described in detail in Section 3, for the dynamics. To predict the scene state at time T, the dynamics model takes as inputs the scene particles from the encoder corresponding to timesteps T - 2 and T and the kinematic particles corresponding to timesteps T 2, T 1, and T. The Interlacer encodes each timestep's s scene particles and the full set of kinematic particles separately before combining them in one large trunk. Predictions made the form (Ax,, Af,) for each particle from timestep T - 1. The prediction for timestep T can then be constructed as {(x?" + Ax, rf" + Af))
Rendering The renderer uses a ray- ray-based renderer conditioned on point cloud, similar to Point- NeRF [17], to render images. This involves casting rays through the scene and, at each sampled location along the ray, finding set of neighboring particles. Summary statistics describing these particles are computed and then provided as input to a NeRF MLP, which predicts the color and density of that location in the scene. These predictions are composited along the ray to produce rendered pixel. For more details, refer to [9].
Training During training, we encode set of input RGB-D images into point clouds, then recur- sively apply the dynamics model with actions from time T... K make predictions multiple timesteps into the future. For supervision we sample a small set of rays to render at each timestep and compute the 7,2 loss between the predicted and observed RGB values. Unlike typical NeRF models, which require tens or hundreds of views for training, we train HD-VPP with data from just.

---

two cameras at each timestep. Both views are provided as input to the model, and the training loss computed on pixels sampled from both views.
3 Interlacers: when linear-attention Transformers and GNNs meet
PERFURMER-RCT
Tmeet-1
NEICHBOR-ATTENDER
NEICHBOR ATTENDER
Time:t
Kinematic Secleture Times:t- Fine:tt2211-1 2ndd-
Figure 3: Interlacer dynamics. The input point clouds from each timestep are processed by the neighbor- attender layers, followed by the Performer layers (see Appendix separate third channel is reserved for processing kinematic particles describing the actions conducted by the robot, which are preprocessed by regular PCT layer. Then, all of the preprocessed point clouds are merged. The model predicts particles' displacements well as deltas of their corresponding feature vectors, after apply- ing one more neighbor-attende and Performer. See Figure for how Interlacer integrated with the HD-VPD model.
details). In the HD-VPD model,
To make a prediction for time T, the Interlacer takes as input the scene state from each timestep in
given window, where each scene state S, = ( Fr} } = - I,...,
for the window length I)
consists of the set of the 3D positions X, = [x] ...., x E R of given set of particles X in time
t and their corresponding feature vectors at that time: F, {ff .... fyy E RR (for some f > 0)-
For modeling robotics data, the Interlacer also receives kinematic particles timesteps T ..... representing the action the robot will take.
The Interlacer consists of two main types of layers: (1) linear attention Performer-PCT layer [14, 15] layer, leveraging structural inductive priors encoded by local neighborhoods in graphs. that we refer as Neighbor-Attender: The input point cloud for each Neighbor-Attender timestep is first processed by the Neighbor- Attender modules. Their outputs are then processed by Performer-PCT modules, which produce versions of their input point clouds with updated features. These feature point clouds from each timestep are then merged together into one large point cloud along with the robot's skeleton points from all times. The Interlacer then applies one more Neighbor- Attender layer, followed by final block of Performer-PCT layers. This model predicts the change location and features (Ax,, Af;) for each particle i the last input timestep T 1. These deltas are applied to (x"-11, ff--1) to construct a particle prediction at time T, which fed back into the dynamics to predict another step forward into the renderer for image generation.
and (2) graph-based ceature-agglomeran
The Performer-PCT [14, 15] is described in detail in Appendix A. Next, discuss Neighbor-
Attender, a novel mechanism to incorporate geometry into attention.
3.1 Incorporating geometry into attention: the Neighbor-Attende layer
The Hhighbor-Attender illustrated in Figure 4 is designed to provide each particle with information about the geometry and features of its immediate neighborhood efficiently as possible. A simple approach might find the & nearest neighbors each particle and extract summary of those neigh- bors' features and relative positions, but this would involve forward pass &N netticlenniighbor pairs. Inspired by RandLA-Net [18], Neighbor-Attene tures only small, salformly-samplle subset of particles call anchor particles, then uses the anchor particles to update the rest of the particles. This results in bottleneck step of only &N
instead computes such neighborhood fea-
pairs

---

MURK (
Step1
Step
Skep 3
Step
Step
Step
Figure 4: The anatomy the Neighbor- Attender layer. subset anchor particles are sampled uniformly from the full set of particles, and these anchor particles aggregate information from their & nearest neighbors. Then the full set of particles updated, each using the features of the closest anchor particle.
for a subsampling rate allowing us to control memory consumption at will. The computations of the Neighbor-Attenen layer consist of six steps that we explain in detail below; steps 1-4 aggregate neighborhood features onto the anchor particles, then steps 5-6 use those to update the rest.
Choosing anchor particles: Sample uniformly random r = 4 particles from the input set of N particles. We refer to them as anchor particles.
2. Computing neighbors of anchors:
For each anchor particle i, compute the set T2 of its 16 nearest neighbors from the entire N N-element set.
3. Computing attention-vect vectors: For each anchor particle and its neighbor E Ti, compute the relative position feature vector defined as: (x,,,,, Xj, X, ||x; -Xy,
x, (12) and concatenate it with the feature vector f,. We refer to the resulting vector as the (i,j) edge feature f.y.
4. Updating feature vectors for all anchor particles: For each anchor particle i, compute its new feature vector as the weighted sum of its edge features using learnable MLP module:
cxp (MLP(f.y: 01)
Σ
(1)
JET. E exp (MLP(fa:: 01)
5. Finding closest anchors for all the particles): For each particle in the original N- element set, find its closest anchor particle a(i).
6. Computing new feature vectors for all the points: For each particle i in the original N-element set, compute its new feature vector f." = MLP(f;, fa
Experiments
4.1 Datasets and HD-VPD instantiations
Hardware and dataset We train HD-VPD models on dataset of bi-manual Kuka robots interact- ing with a variety of objects and tasks. The dataset consists of episodes collected from 5 different robots over several months. The robots (details in Appendix B, [19]) are equipped with an overhead RealSense and left arm wrist RealSense cameras capable of providing RGB-D images. The dataset an uneven distribution of approximately of at most 8 steps with 500ms passing between consecutive steps. The dataset is 1.6TB.
tasks. We break the episodes into snippets of trajectory
Models We train HD-VPD models with three different dynamics architectures and varying numbers of particles for the experiments below. These architectures are: (1) hierarchical GNN using the architecture from [9] with the addition of kinematic particles for actions; (2) Performer-PCT. which consists of repeated Performer-PCT layers; and (3) Interlacer, which includes both Performer-PCTT and Neighbor- Attenderr layers. Architecture and training details are available in Appendix E.
4.2 Learned bi-manual robot dynamics with HD-VPD
In this section, we present learned dynamics obtained with our HD-VPD method for the bi-manual Kuka robot. Corresponding videos are available at https: //sitts////te.../
Video quality scales with particles We evaluate the quality of next-step image predictions (mea- sured by SSIM) on the test set made with each architecture using varying numbers of particles. In
5

---

2,000
Frediction SK1M
0.61
memory
20
pas
1.200
10
0.00
600
131003
2040
4005
52703
00230 131072
Number of particles
Number of particles
particles
BONK Perfomee-PCT PCT Interlaeer
Ghi Perforse PCT miriertcc
G204
Perfoman-PT I ritttaee
(a) Prediction quality
Dynamics speed
(c) Training memory
Figure 5: Analysis of models' behavior as function of number of particles. Note GNN is unable to be with 65K 131K partieles duc to memory limitations. Interlacer with 131K particles provides the best prediction quality while staying competitive with the GNN baselines for dynamics speed, and with Performer- PCTs for memory requirements. (a) Test set SSIM prediction quality increases with the number of particles, and Interlacer with 131K points does the best. (b) Interlacer is faster than GNN while able to handle many more particles. Perforer-PPTT is faster, but achieves worse results. (c) Performer-PCT and Interlacer use less memory than the GNN baseline, enabling them to scale larger point clouds.
Fig. 5a, we show that the Interlacer-basee HD- -PD model produces comparable-qualle predictions the GNN every number particles. The GNN model is unable be trained 65K and 130K particles due memory consumption, whereas the Interlacer model continues improve with scale. The Performer-Pet model lags behind the GNN and Interlacer at each number particles. We also explore the behavior of these models on multi-step predictions, where the increasing uncertainty associated with long-range prediction might render high-density models' image quality advantage moot. Figure 6 shows that while Interlacers of all sizes are less accurate the further into the future they predict, using more particles continues to produce better predictions.
Computational requirements We evaluate the key computational costs associated with running and training on the three classes of models. Figure 5b shows number of dy- namics steps per second achieved with each ar-
0.86-
Particles 131,072
0.84
0.82
chitecture and number of particles. The In- terlacer is at least as fast as the GNN model at each scale, some cases providing more than 2x speedups. The Performer-PCT is the fastest by significant margin, but unable to match the image quality of the other mod- els. In terms of memory consumption (Fig- ure 5c), the GNN model always requires con- siderably more memory to train than the Inter- lacer Performer-PCT1 models, and is unfeasi- ble to use at scales beyond 32K.
SSIM
0.80
2,048
0.78-
0.76
2
Rollout step
Figure 6: Video quality with longer rollouts and in- creasing particle density.
4.3 Downstream experiments on hardware
125m
Figure 7: Experimental setups. Left: Box pushing task setup. We measure cost as the error between a goal push vector (red) and learned dynamics rollouts of candidate push vectors (blue). Right: Grasping task setup. We measure cost as function of the distance upward travelled by the Coke can particles and compare against real world grasp success.

---

Recaretenn ateie
Predktio:
Figure 8: Top: Box Pushing Dynamies Rollout. HD-VPD prediction of scene dynamics from left push motion. The particles on the box move in the direction consistent with the push.
Prodicted liabout of Stzzneokd Groep (hin oftee?
Predicted Utained Grnap (12 offea)
Figure 9: Grasping Dynamics Rollouts. Left: Red points belong to the can whose height is tracked. Center: shows the motion of successful grasp (Coke can points merge with gripper and lift). Right: shows HD-VPD prediction for failed grasp (Coke can points don't l leave the table).
Box push plan selection This task is to move the box a 12.5 cm to the right as shown in Fig. 7. We have fixed set of motion plans that move between 12.5 to -12.5 cm. These planned motions are rolled out with HD-VPD, By segmenting the box points from the input point cloud. we define the cost of a push plan as the median distance between the predicted and goal box points. Lower cost for a plan implies that the model predicts the box will be closer to the goal. Figure 10a shows that HD- VPD's ss predicted costs are close to ground truth (distance 0.25 for large push the wrong way, 0.0 for push to the right location). Choosing the minimum-cost plan under HD-VPD's cost predictions leads to optimal behavior in this experiment.
Grasp plan selection This task is to lift a Coke can straight up off the table to a position 0.2m in the air. We have fixed set of planned grasps various locations relative to the object, with offset 0.0 corresponding to grasping at the center of the object. We roll out HD-VPD to get predicted outcomes of each plan. By segmenting the points corresponding to the can out of the point cloud, we can define a cost function for each plan. Lower cost is better and implies can particles are closer the goal. We show that the HD-VPD model cost increases as the planned grasp pose moves away from the object and becomes less likely to succeed as shown in Figure 10b. The HD-VPD model is able to infer that the Coke can points will be lifted on the grasp motions that do succeed in the real world (blue points). More details in Appendix D.
0.20-
Predicted cost
0.20-
Plan
Predicted cost
000 000000
0.15-
Best Other
0.15-
.
0.10-
0.10-
Grasp outcome
0.05-
0.05-
Success Failure
0.00
-0.3
-0.06
0.00
0,06
0.13
0.00
0.00
0.05
0.10
Plan push distance
Plan grasp location offset
(a) Box pushing
(b) Coke can grasping
Figure 10: Results on two planning tasks. (a) Results for planning a desired 0. 0.5mmm push. X-axis is planned push distance for different potential push plans, Y-axis is the cost for cach plan as measured via the HD-VPD rollout. The yellow point shows that planned push of 1.15m is the best option when the goal is to move the box 0. 1.55m and the cost of other push plans ordered consistently by this cost function. (b) Grasp offset from object center vs HD-VPP predicted plan cost. With sufficiently large offsets, the grasps miss the can and fail. HD-VPD latent dynamics is able to understand that the points of the coke can should lift up following a grasp trajectory for well planned grasps, and understands that the can should not lift upwards for poorly planned grasps that fail in real world execution.
Dataset inspection After training HD-VPD, we run it on trajectories from the training set and find those with anomalously high loss. This surfaces previously-unnoticn errors and outliers the data, including sessions with incorrect geometric camera calibration, faulty camera exposures, and humans the robot workspace. Such dataset introspection may be useful when preparing a dataset for training models such as policies. Examples are shown in Appendix F.
7

---

5 Related work
Dynamics models for robotics Accurately modeling robot dynamics is essential for tasks such as motion planning, control, and simulation. Traditional approaches rely on rigid body dynamics models [1, 4, 2, 5, 3]. but these models can be inaccurate in complex environments where objects are deformable or interacting in unpredictable ways, and require painstaking authoring of scenes and assets. A significant literature has explored the use of learned models for robot dynamics, which can broadly be divided into two camps: 2D and 3D. 2D models operate directly on pixels and use architectures designed for image generation [20, 21J. While these have found some success by scaling up to modern foundation model sizes and datasets [22, 23I, their learned dynamics can be nonphysical, and they lack 3D representations that can be used for cost functions and scene editing-
More closely related to this work are the 3D learned dynamics models. Various works have used datasets of 3D dynamics to train dynamics models with particle [24, 25], mesh [26, 71, or object [27, 28] representations, and these models can be more accurate than analytic simulators in some cireumstances [8]. However, the requirement for training data with ground-truth 3D particle or object poses limits their applications. Works have attempted to train 3D dynamics models with per- ceptual data such videos and point clouds, many leveraging Neural Radiance Fields (NeRF) [12]. One approach has been to apply point cloud distance functions such as Chamfer distance to define loss functions between predicted points and the next-step point cloud [29] or NeRF [30] observa- tions, though these often require object segmentations and have not been applied for large-scale scenes. Another strategy pre-trains conditional or object-level NeRF renderers, then trains a dynam- ies model on these representations [31, 32]. This split training process and coarse representation is limiting for dealing with complex scenes and results in lower-quality predictions [9].
HD-VPD derives from Visual Particle Dynamics (VPD) [9], extended to support robot actions and larger, more detailed scenes. Relative to the broader literature, HD-VPD has less restrictive data requirements (2 RGB-D cameras) than other NeRe-based works or mesh- or object-based works, and it jointly predicts 3D dynamics and 2D video, unlike pure 2D video or point cloud models.
Point cloud architectures The rise of deep learning models has opened up new possibilities for processing and understanding 3D point cloud data. Models like PointNet++ [33], Point Transformer [34] and Point Cloud Transformers [35] have demonstrated strong performance in tasks such as ob- ject classification, segmentation, and registration. Similar to our work, RandLA-Net [18] focuses on building memory- and time-efficient architectures that scale up large point clouds. However, these models are typically designed for static point clouds and do not directly model the dynamics of the scene. Our work introduces new class of PCTss, called Interlacers, which are specifically designed for modeling the dynamics of large-scale point clouds. Performer-PCT architecture was introduced in [15], but applied in different setting of scalable Roboties Transformers rather than dynamics modeling. In this paper, we show that intertwining Performer-PCT layers with the introduced here Neighbor-AAtender layers is key to achieving scalable and high-precision dynamics models.
Conclusion
We present a new high-density partecle-based world model called HD-VPD and train it on real-world RGB-D and kinematic bi-manual robotics data. To support HD-VPD'S high fidelity predictions we propose Interlacers, combining finear-attentin techniques with GNN-inspired local neighborhood attention methods, can model point clouds of sizes 100K+, infeasible for previous PCTs and GNNs. HD-VPD outperforms previous SOTA particle-based world models both in prediction quality and computational cost. Finally, we show how HD-VPD can be used in robotic planning, as accurate world models for scenes involving objects and robotic agents operating on them.
Limitations
HD-VPD is deterministic dynamics model, meaning that its predictions blur with long horizons and stochastic dynamics. We leave extending the model to probabilistic predictions and long-horize tasks to future work. HD-VPD might be improved via (1) more detailed finger and arm modeling in the kinematic skeleton; (2) data with near misses for robot planning applications: (3) upweighting points most relevant to robot interaction in training loss computations; (4) using loss functions to enforce physically realistic 3D motion rather than only correct RGB values. Planning with visual costs might avoid any mismatch between particle motion and physical outcomes.
8

---

References
[1] E. Todorov, T. Erez, and Y. Tassa. Mujoco: A physics engine for model-based control. In 2012 IEEE/RSJ International Conference on Intelligent Robots and Systems, IROS 2012, Vilamoura, Algarve, Portugal, October 7-12, 2012, pages 5026 5033. IEEE, 2012. doi:10.1 10990.... 2012.6386109. URL httpp:://///. //tp:::////...oe org/10 1109 IROS. 6386109.
[2] E. Coumans and Y. Bai. Pybullet, a python module for physics simulation for games, robotics and machine learning. In http://pybullet.org 2016.
[3] S. Shah, D. Dey, C. Lovett, and A. Kapoor. Airsim: High-fidelity visual and physical sim- ulation for autonomous vehicles. In M. Hutter and R. Siegwart, editors, Field and Service Robotics, Results of the 11th International Conference, FSR 2017, Zurich, Switzerland, 12-15 September 2017, volume 5 of Springer Proceedings Advanced Robotics, pages 621-635. URL https: /toii/oiiio/// ortp.///00/10100
Springer, 2017. 1itt:101107/78---33111
[4] R. Tedrake. Drake: Model-based design and verification for robotics, 2019. URL https:
// //ddaaemiii edu.
[5] V. Makoviychuk, L. Wawrzyniak, Y. Guo, M. Lu, K. Storey, M. Macklin, D. Hoeller, N. Rudin, A. Allshire, A. Handa, and G. State. Isaac gym: High performance gpu-based physics simulation for robot learning. ArXiv, abs/2108. 10547008.10470, 2021. URL https://api. semanticscholar org/ arggCorpusID: :37277992..7777
[6] T. Pfaff, M. Fortunato, A. Sanchez- Gonzalz. and W. Battaglia. Learning mesh-based simulation with graph networks. In 9th International Conference Learning Represen- tations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021. OpenReview.net, 2021. URL http://pprrvieee..eetffrrrtrm ooroonn XP.
[7] K. R. Allen, Y. Rubanova, T. Lopez-Guevara, W. Whitney, A. Ganzaez- P. W. Battaglia, and T. Pfaff. Learning rigid dynamics with face interaction graph networks. CoRR, abs/2212.03574, 2022. doi: 0001004455AAR2112122205777 08550/arXin htt.:/aarXxv.22..33.7...
URL https://doi.org//1
[8] K. R. Allen, T. Lopez-Guevara, Y. Rubanova, K. L. Stachenfeld, A. Sanchez-Gonzalez, P. W. Battaglia, and T. Pfaff. Graph network simulators can learn discontinuous, rigid contact dy- namics. In K. Liu, D. Kulic, and J. Ichnowski. editors. Conference on Robot Learning, CoRL 2022, 14-18 December 2022, Auckland, New Zealand, volume 205 of Proceedings of Machine Learming Research, pages 1157-1167. PMLR, 2022. URL https: //proceedigcs.. mlr. press/v205/ altp//allen2aa..mm html.
[9] W. Whitney, T. Lopez-Guevara, Pfaff, Rubanova, T. Kipf, K. Stachenfeld, and K. R. Allen. Learning 3d particle-based simulators from rgb-d videos. arXiv preprint arXiv:2312.05359, 2023.
[10] Scarselli, M. Gori, A. C. Tsoi, M. Hagenbuchner, and G. Monfardini. The graph neural network model. IEEE Trans. Neural Networks, 20(1): 2009, 11111..11//NNN.20000 2005605. URL https: ///e:i//////.// org/ 10. 1109/TNN. 2008. 2005605.
[11] R. Liu, P. Calafiura, S. Farrell, X. Ju, D. Murnane, and M. Pham. Hierarchical graph neural networks for particle track reconstruction. CoRR, abs/2303.01640, 2023. doi: 10.48550/ ARXIV.2303. 01640. URL https: //aoi. org/10 10. 48550/855555 20.500//Xii.. 2303. 01640.
[12] B. Mildenhall, P. Srinivasan, M. Tancik, T. Barron, R. Ramamoorthi, and R. Ng. Nerf: Representing scenes neural radiance fields for view synthesis. In ECCV, 2020.
[13] M. Guo. J. Cai, Z. Liu, T. Mu, R. R. Martin, and S. Hu. PCT: point cloud transformer. Comput. Vis. Media, 187 2021. 1oi:10. 10900004411555001--0000 10. 1007 000007//411090001--22228
URL https::////... org/

---

[14] K. M. Choromanski, V. Likhosherstov, D. Dohan, X. Song, A. Gane, T. Sarlós, P. Hawkins, J. Q. Davis, A. Mohiuddin, L. Kaiser, D. B. Belanger. J. Colwell, and A. Weller. Re- thinking attention with performers. In 9th International Conference on Learning Represen- tations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021. OpenReview.net, 2021. URL. www.:/nee//eeeeee http://rrmmmiidauuuuuaaaee
[15] I. Leal, K. Choromanski, D. Jain, A. Dubey, J. Varley, M. Ryoo, Y. Lu, F. Liu, V. Sindhwani, Q. Vuong, et al. Sara-rt: Scaling up robotics transformers with self-adaptive robust attention. ICRA 2024, 2024.
[16] O. Ronneberger, Fischer, and T. Brox. U-net: Convolutional networks for biomedical image segmentation, 2015.
[17] Q. Xu, Z. Xu,J. Philip, S. Bi, Z. Shu, K. Sunkavalli, and U. Neumann. Point-nerf: Point-based neural radiance fields. In Proceedings of the IEEE/CVF Conference Computer Vision and Patlern Recognition, pages 5438-5448, 2022.
[18] Q. Hu, Yang. Xie, S. Rosa, Y. Guo, Z. Wang. N. Trigoni, and A. Markham. Randla-net: Efficient semantic segmentation of large-scale point clouds, 2020.
[19] Varley, S. Singh, D. Jain, K. Choromanski, A. Zeng, S. B. R. Chowdhury, A. Dubey, and V. Sindhwani. Embodied ai with two arms: Zero-shot learning, safety and modularity. arXiv preprint arXiv:2404.035570 2024.
[20] C. Finn, Goodfellow, and S. Levine. Unsupervised learning for physical interaction through video prediction. Advances in neural information processing systems, 29, 2016.
[21] D. Ha and Schmidhuber. World models. arXiv preprint 1803v 2018.
[22] M. Yang, Du, K. Ghasemipour, J. Tompson, D. Schuurmans, and P. Abbeel. Learning interactive real-world simulators. arXiv preprint arXivv2310.061114 2023.
[23] Bruce, M. Dennis, A. Edwards, Parker-Holder Y. Shi, Hughes, M. Lai, A. Mavalankar, R. Steigerwald, C. Apps, Y. Aytar, S. Bechtle, F. M. P. Behbahani, S. Chan, N. M. O. Heess, L. Gonzalez, S. Osindero, S. Ozair, Reed, Zhang. K. Zolna, J. Clune, N. de Freitas, S. Singh, and T. Rocktaschel. Genie: Generative interactive environments. ArXiv, abs/2402.1539. 2024. URL attps://apt www.nticcccoollattt..oooomoaaaaa org/CorpusID: 267897982.
[24] Y. Li, J. Wu, R. Tedrake, J. B. Tenenbaum, and A. Torralba. Learning particle dynamics for manipulating rigid bodies, deformable objects, and fluids. arXiv preprint arX1v:1810.01566, 2018.
[25] A. Sanchez-Gonzalez, J. Godwin, T. Pfaff, R. Ying. Leskovee, and P. Battaglia. Learning to simulate complex physics with graph networks. In International Conference Machine Learning, pages 8459-8468. PMLR, 2020.
[26] T. Pfaff, M. Fortunato, A. Sanchez-Gonzalez, and P. Battaglia. Learning mesh-based simula- tion with graph networks. In International Conference on Learming Representations, 2021.
[27] W. Battaglia, R. Pascanu, M. Lai, J. Rezende, and K. Kavukeuoglu. Interaction networks for learning about objects, relations and physics. In Neural Information Processing Systems, 2016. URL httph:tt///// / /api..pp.../ http://pp.ssaaaicccccooosoooo/oe semanticacholaar
prrg/CorpusID: 2200675.
[28] Y. Rubanova, T. LupezaGuevara, K. Allen, W. F. Whitney, K. Stachenfeld, and T. Pfaff. Learning rigid-body simulators over implicit shapes for large-scale scenes and vision. arXiv preprint arXiv:2405. 2404:24405..4045. 2024.
[29] H. Shi, H. Xu, S. Clarke, Y. Li, and J. Wu. Robocook: Long-horizon elasto-plastic ob- ject manipulation with diverse tools. ArXiv, 2e0/2306.14477 2023. URL. aattps://apt: semanticscholar. org/CorpusID: 1255//or 259251806.
10

---

[30] H. Xue, A. Torralba, J. Tenenbaum, D. Yamins, Y. Li, and H.-Y. Tung. 3d-intphys: Towards more generalized 3d-grounded visual intuitive physics under challenging scenes. Proceed- ings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pages 3624- 3634, 2023.
[31] Y. Li, S. Li, V. Sitzmann, Agrawal, and A. Torralba. 3d neural scene representations for visuomotor control. arXiv preprint arXiv:2107.04004 2021.
[32] D. Driess, Z. Huang, Y. Li, R. Tedrake, and M. Toussaint. Learning multi-object dynamics with compositional neural radiance fields. In K. Liu, D. Kulic, and J. Ichnowski, editors, Conference Robot Learning, CoRL 2022, 14-18 December 2022, Auckland, New Zealand, volume 205 of Proceedings of Machine Learning Research, pages 1755-1768. PMLR, 2022. URL http:: www.hhtss//pppoeeeeee http//prreeens..... mlr. prwpprressss.e. caieas/v205//ree 2205/áriess23a... hhtml. html.
[33] C. R. Qi, L. Yi, H. Su, and J. Guibas. Pointnet++: Deep hierarchical feature learning on point sets in metric space. Advances in neural information processing systems, 30, 2017.
[34] H. Zhao, L. Jiang, J. Jia, P. H. Torr, and V. Koltun. Point transformer. In Proceedings of the IEEE/CVF international conference on computer vision, pages 16259-16268, 2021.
[35] M.-H. Guo. J.-X. Cai, Z.-N. Liu, T.-J. Mu, R. R. Martin, and S.-M. Hu. Pet: Point cloud transformer. Computational Visual Media, 7:187-199, 2021.
[36] I. Loshchilov and F. Hutter. Decoupled weight decay regularization. arXiv preprint
171051171..0500,, 2017.

---

Appendix
A Performer-PCTs linear attention: closer look
Let Xin Nxd be an input to the Transformer's attention block, where N stands for the num- ber of points and is the dimensionality of their corresponding latent feature vectors. The out- put Xout E R of the regular attention module can be computed as follows for learnable Wq. WK E Rdxdox Wy E Raxd and query/key dimensionality dok
X Xout D D- Wv), AlN
(Xin W. (Xin (x)
A exp
vdqk
In Eq. 2, IN RR denotes the all-one vector and matrices: X Ww. Xnn Wo. X WK are often referred to as: V (value), Q (query) and K (key) respectively. Furthermore, mapping exp is applied glement-wise Finally, matrix A called the attention matrix and encodes how points attend each other via the so-called fefmmxx-kernd (defined as: K(u, v) exp(uv) for row-vectors u, v).
Time- and space-complexity of the regular attention mechanism is clearly quadratic in the number of points N, since attention-matrix A has N2 entries. This makes it prohibitively expensive for massive point clouds. Thus Interlacers use instead linear attention mechanism introduced in the class of Transformers, called Performers [14]. Performers propose the following computational model of attention, where Rdqk 1m is applied row-wise and is hyper-parameter:
X D D'I (Xin Wv)),
D Q=((K)) 1N). Q' = (QQ). K' (
The brackets indicate the order of computations. Calculations in Eq. 3 can be performed in time linear in N (and in the syper-paramette m that in practice we choose as m < N). System defined by Eq. 3 is obtained from the one given by Eq. 2, by replacing regular attention matrix A with
matrix A' def Q'(K')T Following [15], we use Φ = ReLU applied element-wise (thus m dok).
B Hardware Setup
Overview of the hardware setup is shown in Figure 11
Cameras and Image Processing We use a Realsense D415 overhead camera and Realsense D405 attached to the left wrist. We sample the overhead camera at 480x640 resolution, and the wrist camera captured 480x848 resolution and cropped to 480x640. We inpaint the overhead camera depth images via cv.INPAINT TELEA. We mask both cameras observations to exclude points more than meters from the cameras according to the depth images.
Hardware setup. The robot consists of two kuka iiwa arms extended over table, and 2 realsense cameras (1. overhead, 2. left wrist). Data collected from 5 of these cells with some variation in background and lighting. A representative sample of objects from dataset is shown.
Figure 11:
12

---

C Action Space
We represent actions as 26 particle kinematic skeletons at the desired future state positions. There is an additional particle representing where the fingertips would be if the gripper was closed, particles representing the estimated current position of the fingertips. Also, there is an additional particle on the side of each wrist so that the orientation of the gripper can be disambiguated especially when the fingers are closed. The fingers themselves are somewhat compliant so the skeleton points do not exactly reflect the true pose of the fingertips especially when contact with the environment. The origin of the space is at the corner of the pink mat closest to the base of the right arm (no wrist camera). An example rendering of this skeleton shown in Figure 12.
0.6
0.6
Figure 12: Action Space. Consists of 26 points representing kinematic configuration of the robot.
D Grasp Experiments
For the grasp planning real world evaluations, two threads of fishing wire where attached to the bottom of the can and run through the table and then tied to a small weight. This way whenever the Coke can is dropped, it returns to exactly the same pose for all rollouts.
We start with an initial grasp centered at the center of the can, and then back off the grasps along the y-axis of the table (aligned with pink mat) in 5mm increments. Figure 10b overviews the relationship between these offsets and predicted plan cost. Figure/ 13 shows the observations from the left and overhead cameras snapshot during the real world rollout of these planned grasps.
E
Training and architecture details
We train all models with a batch size of 16 for 1M total steps. We use a learning rate schedule of 3e-4 until step 1000, then 1e-4 until step 100K, then 3e-5 until the end of training. We use the AdamW optimizer [36] with weight decay 1e-3 and clip the gradient norm to 0.01 to prevent outliers in the dataset from destabilizing training.
All models are trained end-to-end with 6-step rollouts during training. At each of the two input timesteps and the 6 predicted timesteps, losses are computed on 128 sampled rays. Rays from input timesteps are rendered conditioned on the particles encoded from their respective input timesteps, while rays from predicted timesteps use the particles predicted by recursively rolling out the dynam- ics model in particle space conditioned on a sequence of actions.
The encoder uses U-Net [16] with [32, 64, 128, 256, 256, 128, 64, 32] channels ap- plied to each input image. outputs 16-d per-pixel feature vectors. Points from the encoder are only included in the set of dynamics particles if they fall within the robot's workspace. Input and target RGB-D images are masked using the depth channel, changing the color of any pixel whose depth 2m to solid white.
13

---

html). This same kernel is used Successful Grasp (No offset)
Time
Failed Grasp (With offset)
Time
Two exemplar rollouts from the grasp experiments. The top grasp has no offset from the center of the can. The bottom grasp has an offset from the center of the can and misses.
Figure 13:
The Interlacer dynamics model uses one Neighbor-Attender block and one Performer-PCT block on each input timestep of scene particles, using separate weights for each input timestep, and one (quadratic) PCT block on the kinematic particles. These three point clouds are then con- catenated. Then this large, multi-timestep. multi-ipput-mddalii point cloud goes through another Neighbor-Attender block, followed by 3 ReLU blocks. All Performer-ReLU layers use a key dimension of 16 and value dimension of 64. This dynamics model outputs (Ax,, Af, for each particle i of timestep T 1. The Ax; values are constrained to be in [-0.15, 0.15 using tanh and scaling, preventing any particle from moving more than 15cm in any single step. The output feature vectors f are 16-d to match the features coming from the encoder. To compute neighbors in the Neighbor- Attender layers, we use CUDA kernel generated by Pallas battps://jas readthedocs wiw.en////// www.aaatttaaaaaeaaaeaaaaaaaaaaee for finding nearby points in the renderer.
The renderer follows the same scheme for constructing input features for the NeRF MLP as Whitney al. [9]. We use four concentric annular kernels with radii [o, 010 0.02, 0.0 and corre- sponding bandwidths [0. 01, 0.01, 0. 0.01, 0. 0.]5 and these kernels are approximated using 16 nearest neighbors. The near plane for ray rendering is set at mm and the far plane at 2m. Rays are composited against solid white background.
14

---

Reconstruction
Prediction
Figure 14: HD-VPD can accurately predict the dynamics real-world interactions between robots (16-dof bimanual kukas) and complex objects, c.g., an articulated trash can whose lid opens upon pushing pedal as shown here. This model trained several hundred examples randomized over trash can pose, images here are from the test set. Top row: Renders from HD-VPD. Timesteps t and are reconstructions the two input timesteps; timesteps t... . are predictions into the future given robot actions. Bottom row: Ground-tre -tuth test set video.
F Dataset inspection
We introspect our training data by computing a histogram, shown in Figure 15, of HD-VPD's loss on sample of the training set and using that to set threshold for unusually high losses. Once have set this threshold, we can search the training data for trajectories with high loss.
300-
Number of examples
30
0.00 0.25 0.50
0.75
1.00
HD-VPD loss
Figure 15: Loss histogram of HD-VPD sample of the training The data has long tail of outliers with losses many times the mean. Note log scale on the axis.
In Figures 16 and 17 present examples of dataset quality issues discovered by inspecting training examples where HD- VPD's ''s loss is anomalously high. These issues with our dataset were not previ- ously known, and might pose problems for other applications of this data, such as policy learning. Discovering them with HD-V DDD also allows us refine our hardware setup, fixing calibration issues and avoiding problematic robot states.
15

---

(a) Person workspace
(b) Camera miscalibration
Figure 16: Two examples of problematic examples training dataset. (a): A person is in the robot workspace. (b): The wrist camera calibration is drastically incorrect. Visualized here are the points from the wrist and overhead cameras together; note the two intersecting planes of the table.
Figure 17: Another example problematic data point. The robot pose almost completely occludes
overhead camera's 's view the scene, and the color calibration of the wrist and overhead cameras is very different.
16


Reviews:
[{"review_id": "bGbpzkF3cU", "reviewer": "Reviewer_FVjU", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": 3, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "This manuscript presents a method to predict visual particle dynamics of the real-world interactions between robots and robots and objects given an input scene and the robot actions. Specifically, a new point cloud transformer called Interlacers is introduced. This architecture intertwines efficient linear-attention Performer layers proposed in [19] and graph-based neighbor attention layers. Experiments were conducted on a bi-manual Kuka arms setting with two cameras. Compared to the previous Graph neural network-based method, the proposed method achieves comparable performance with less computation time or achieves better performance using 4x more particles. Experiments also demonstrate the potential of using this model for downstream tasks, such as rollout evaluation for robot planning and detecting outliers from a training dataset.", "summary_of_recommendation": "The paper presents a new architecture that can model the visual particle dynamics of a robot manipulation scene given future robot actions. Although the new architecture is more efficient compared to prior work VPD, I feel the technical novelty in this architecture is limited and some additional experiments should be conducted on the Deformable Collision dataset. The experiments are missing important details and the generalization ability is unclear. The training is limited to one robotic dataset and the paper can be further improved by adding more experiments on public robotic manipulation datasets such as Droid.", "technical_quality": 3, "clarity_of_presentation": 3, "potential_impact": 2, "robotics_focus": 4, "quality_of_limitations": 2, "questions_for_rebuttal": "I think the following things could further improve the paper:\n* Clarify the training/testing protocol in experiments.\n* Benchmark the model on the Deformable Collision dataset [9]\n* Add experiments for unseen robots, objects, and tasks.\n* Add experiments on public robot manipulation datasets such as droid.\n* Varying the number of cameras.", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "BFFhfqYxUF", "reviewer": "Reviewer_L6Jk", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": 3, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "The paper presents a new model architecture which combines linear attention and GNN for training a 3D particle dynamics model with more particles", "summary_of_recommendation": "Major concern is the quality of the generated videos. This seems worse compared to prior work such as VPD", "technical_quality": 3, "clarity_of_presentation": 3, "potential_impact": 3, "robotics_focus": 4, "quality_of_limitations": 2, "questions_for_rebuttal": "Why are the generated future predictions blurry? The demonstrated case mostly deal with simple rigid body dynamics such as pushing and grasping, which should be fairly deterministic. Additionally, it seems that the generated videos become blurry even before the robot makes any contact.", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "CnhAk3pm9o", "reviewer": "Reviewer_F4kU", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": 3, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "This work proposes HD-VPD, a method combining a particle-based visual representation with an efficient Performer-based Point Cloud Transformer that utilizes GNN-inspired neighborhood aggregation, Interlacers, for dynamics modeling for physics and interaction prediction. The proposed dynamics module is more efficient than previous GNNs used for similar tasks and produces comparable or better performance on dynamics prediction. Notably, the method utilizes only two camera views and actions for training.", "summary_of_recommendation": "Overall, I like the ideas presented in this paper and the focus on efficiency and real-world applications. I appreciate the data-collection effort and I find several key contributions that are novel with moderate impact and significance. I have raised several concerns in the “Weaknesses” and “Questions” parts of my review that I would like the authors to address, but I think the paper is relevant and would be interesting to the CoRL community. As such, my current recommendation is “Weak Accept”, but I’m willing to increase my score given convincing answers to my concerns and the concerns that might be raised by the other reviewers. **Post-rebuttal**: I appreciate the authors' clarification regarding the merging process, I highlt recommend including it in the paper for the curious reader. I'm still not convinced regarding the autoregressive training of the Transforner. The authors claim that \"Performing a few steps of rollout during training helps the model learn to correct its own errors and produce stable long predictions, and is especially helpful for a deterministic model.\", which I think is a very strong claim without any ablation of the rollut steps (which is set to 6 across all experimetns if I understood correctly). I think there is a general consensus that the prediction quality is not great but the method is still applicable for downstream tasks. I thank the authors again, and given the above I'd like to keep my score.", "technical_quality": 3, "clarity_of_presentation": 3, "potential_impact": 3, "robotics_focus": 4, "quality_of_limitations": 3, "questions_for_rebuttal": "* Prediction window: throughout this work, the authors use l=2, meaning T-2, T-1 and T. To my understanding, the method observes 2 frames and performs a one-step prediction? Then, an autoregressive process is applied to predict more time-steps. Is it possible to train the model on one-step predictions but in-parallel? For example, Transformers usually train with a one-step prediction loss, but they are using “teacher-forcing” to train with a larger horizon (without autoregressive training, only at inference).\n* Particle merging from different views: are you using a common method to merge different views? I couldn't find details on that in the appendix.\n* Action representation: I find it interesting that you used one-hot vectors to encode the temporal position of the joints. In Transformers, it is common to learn additive embeddings, have you experimented with this kind of positional embedding?\n* Other loss functions: as I mentioned, the prediction/reconstruction visual quality seems low. Have you considered other losses such as the pixel-wise L1 loss?\n* Anchor particles: have you considered other sampling methods other than uniform, e.g. farthest-point-sampling (FPS)? What happens if uniform sampling yields two particles that are very close, won’t that be “wasteful”?\n* Do the authors plan to release an open-source code for further research?", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: **Post-Rebuttal Guidance**.
Guidance: accept, but borderline. This is clearly a weak accept/borderline paper - If the PCs have limited capacity on accepting papers and would like to reject this paper, I would not argue for its acceptance. Reading the rebuttal between the authors and the reviewers, I suspect there is some hesitancy in accepting the results of this paper. First of all, this approach is only validated on a single robot and single dataset - initial results were provided on the Deformable Collision dataset (but the proposed approach had comparable performance with the baseline). All three reviewers admitted that the prediction results were not high quality. While I do appreciate that the proposed approach is more time/memory efficient, I feel that it is incomplete without demonstrating better performance on more datasets (e.g. Droid) or on more tasks.

**Pre-Rebuttal Summary**

Strengths:
- The proposed approach is more time and memory efficient with comparable or better perf
- Clear writing

Weaknesses:
- Limited novelty over prior work (VPD, Performer-PCT, etc.)
- Evaluation is not as convincing - only 1 setting is shown. Reviewers have suggestions for other datasets (eg. Droid)
- Generated reconstruction quality is still very poor

The reviewers have mixed responses to the paper - authors hope you could check and address them
Decision: Accept