Modeling the Real World with High-Density Visual Particle Dynamics

Authors: william f whitney
google  deepmind
jacob  varley
deepali  jain
krzysztof  choromanski
sumeet  singh
vikas  sindhwani
Venue: robot-learning_org_CoRL_2024_Conference
Published: 2024-06-06 17:12:53

Abstract: We present High-Density Visual Particle Dynamics (HD-VPD), a  learned world model that can emulate the physical dynamics of real scenes by processing massive latent point clouds containing 100K+ particles. To enable efficiency at this scale, we introduce a novel family of Point Cloud Transformers (PCTs) called Interlacers leveraging intertwined linear-attention Performer layers and graph-based neighbour attention layers. We demonstrate the capabilities of HD-VPD by modeling the dynamics of high degree-of-freedom bi-manual robots with two RGB-D cameras. Compared to the previous graph neural network approach, our Interlacer dynamics is twice as fast with the same prediction quality, and can achieve higher quality using 4x as many particles. We illustrate how HD-VPD can evaluate motion plan quality with robotic box pushing and can grasping tasks. See videos and particle dynamics rendered by HD-VPD at https://sites.google.com/view/hd-vpd.

Main paper: 
 introduction: 
 Physical simulators are the linchpin of modern robotics, enabling cheap data collection, safe verification of control algorithms, and real-time control via planning. Traditional analytic simulators [1, 2, 3, 4, 5] are fast and convenient to use, but lack the ability to precisely match the complex objects and dynamics of real-world scenes. Learned dynamics models make fewer assumptions on the form of objects and their interactions, but typically come with onerous constraints on their training data, e.g., requiring information such as 3D meshes and poses for all objects or per-object segmentation masks [6, 7, 8] .To address this problem, new classes of learned dynamics models have been proposed that can train directly on multi-view RGB-D observations, with no object-centrality in their data requirements or representations. A state-of-the-art approach of this flavor is Visual Particle Dynamics (VPD) [9] , which represents scenes as a collection of 3D particles whose interaction dynamics is governed by graph neural networks (GNNs) [10, 11] and rendered to images with a conditional NeRF [12] . VPD models are trained end-to-end with a video prediction loss, support 3D state editing and multimaterial dynamics modeling, and are data-efficient enough to model simple dynamic scenes with < 50 training trajectories. However, VPD has only been applied to simple simulated scenes under passive dynamics (without actuation), and it is unable to scale beyond ∼30K particles due to memory and speed limitations. For applications in robotics, VPD needs two extensions: (1) to take robot actions into account, and (2) to be able to model environments with a much higher level of detail. This is the focus of this paper.We propose a High-Density Visual Particle Dynamics (HD-VPD) world model which can train on robot interactions in real scenes and model their dynamics with 100K+ particles. To reach this scale, we propose a new class of Point Cloud Transformers (PCTs) [13] called Interlacers, which intertwine linear-attention Performer [14] layers and local neighborhood attention layers. We demonstrate the capabilities of HD-VPD by modeling the dynamics of bi-manual Kuka robots with multiview depth input data. We find that Interlacer dynamics models are able to equal GNN's prediction quality in half the time using matched point cloud densities, and that they can exceed the best GNN's quality by leveraging high-density point clouds with 4× as many particles as the GNN can fit in memory. We complement these results with illustrative experiments using HD-VPD for downstream applications such as planning in manipulation, leveraging HD-VPD's ability to capture a scene from a single observation and define goals in 3D space. See videos and particle dynamics predictions from our model at https://sites.google.com/view/hd-vpd.Our main contributions are as follows:1. We present High-Density Visual Particle Dynamics (HD-VPD) world models that train end-to-end on real robotics data. They take RGB-D images and robot kinematic skeletons representing actions as inputs, and they predict future 3D particles and rendered images.2. To enable HD-VPD to operate on large, detailed scenes, we propose a new class of Transformers for point clouds called Interlacers, which scale linearly (vs. quadratically) with point cloud size. Interlacers combine Performer-PCT layers [14, 15] and memory-efficient GNN-inspired local attention layers, enabling efficient global point-to-point attention (with the former) while maintaining high-fidelity local geometric detail (with the latter).3. We show that HD-VPD can deliver realistic action-conditioned video predictions on complex scenes of bi-manual robots interacting with various objects. We find that the Interlacer architecture enables fast, memory-efficient, and high-fidelity video prediction with HD-VPD. Whereas typical NeRF models require tens or hundreds of camera views for training, HD-VPD uses only two, making real-world data collection straightforward. We use these models to illustrate downstream applications of HD-VPD in robotic bi-manual control, where the HD-VPD world model is able to predict the success or failure of candidate plans.2 HD-VPD: High-density visual particle dynamics world models HD-VPD is a 3D dynamics model disguised as a video prediction model. It takes RGB-D images and robot actions as inputs and predicts images of the future, just like a video world model. Under the hood, though, it represents both the current state of the world and the actions which operate on it as particles in a shared 3D space. To make a prediction several timesteps into the future, HD-VPD goes through three steps, shown in Figure 2: 1. Encode input images from all views and times into 3D particles that represent the state of the scene.2. Predict the dynamics of the scene. Combine particles representing the first robot action with those representing the scene state and predict the change in state. Use this to update the state. Then repeat this procedure for the next robot action, and so forth.3. Render the particles representing the predicted state of the scene into an image.In this way, HD-VPD can make predictions many steps into the future while remaining in particle space, only rendering to pixel space as needed for training or visualization. The rest of this section describes the HD-VPD model, architectural choices, and training in more detail. Encoding HD-VPD receives RGB-D images from multiple cameras (in our experiments, 2) at timesteps T -l, . . . , T -1 for an input window of length l; throughout this work we use l = 2. Each RGB image is first encoded into feature image of per-pixel feature vectors using a U-Net [16] . Using known camera intrinsics and extrinsics, combined with the depth channel of the RGB-D input, each of these per-pixel features can be unprojected into the global coordinate frame, forming a point cloud where each point corresponds to an input pixel. Each point is associated with the predicted feature vector from its corresponding pixel to form a particle represented as a (location, feature) tuple (x, f ).The particles from across cameras are merged together within one timestep before being subsampled uniformly at random to a fixed total number of particles N .Action representation Unlike VPD, HD-VPD includes a representation of actions, allowing it to model robotic scenes and be used for planning and other forms of controllable generation. We represent actions as a set of kinematic particles describing the motion of the robot across multiple timesteps: previous steps T -2 and T -1, and next step T which represents where the robot plans to move. The particles from each timestep are located at the joints, fingertips, and the base of the grippers of the two arms. Each kinematic particle is associated with a feature vector consisting of two concatenated one-hot vectors: one indicating which robot joint it is, and one indicating which timestep it came from. By representing actions where they occur in 3D space, HD-VPD can learn to associate them directly to the scene particles which they affect. More details and an example are in Appendix C.Dynamics We employ the Interlacer architecture, described in detail in Section 3, for the dynamics.To predict the scene state at time T , the dynamics model takes as inputs the scene particles from the encoder corresponding to timesteps T -2 and T -1 and the kinematic particles corresponding to timesteps T -2, T -1, and T . The Interlacer encodes each timestep's scene particles and the full set of kinematic particles separately before combining them in one large trunk. Predictions are made in the form (∆x i , ∆f i ) for each particle i from timestep T -1. The prediction for timestep T can then be constructed as{(x T -1 i + ∆x i , f T -1 i + ∆f i )} N i=1 .Rendering The renderer uses a ray-based renderer conditioned on a point cloud, similar to Point-NeRF [17] , to render images. This involves casting rays through the scene and, at each sampled location along the ray, finding a set of neighboring particles. Summary statistics describing these particles are computed and then provided as input to a NeRF MLP, which predicts the color and density of that location in the scene. These predictions are composited along the ray to produce a rendered pixel. For more details, refer to [9] .Training During training, we encode a set of input RGB-D images into point clouds, then recursively apply the dynamics model with actions from time T . . . T + K to make predictions multiple timesteps into the future. For supervision we sample a small set of rays to render at each timestep and compute the L 2 loss between the predicted and observed RGB values. Unlike typical NeRF models, which require tens or hundreds of views for training, we train HD-VPD with data from just two cameras at each timestep. Both views are provided as input to the model, and the training loss is computed on pixels sampled from both views.3 Interlacers: when linear-attention Transformers and GNNs meet To make a prediction for time T , the Interlacer takes as input the scene state from each timestep in a given window, where each scene state S t = {X t , F t } (t = T -l, ..., T -1 for the window length l) consists of the set of the 3D positions X t = {x t 1 , ..., x t N } ∈ R 3 of a given set of particles X in time t and their corresponding feature vectors at that time: F t = {f t 1 , ..., f t N } ∈ R f (for some f > 0). For modeling robotics data, the Interlacer also receives kinematic particles at timesteps T -l, . . . , T representing the action the robot will take.PERFORMER-PCT NEIGHBOR-ATTENDER Time: t -2 M E R G E R NEIGHBOR-ATTENDER PERFORMER-PCT NEIGHBOR-ATTENDER Time: t Time: t -1 P E R F O R M E R P C T Kinematic Skeletons Times: t -2, t -1 and t REGULAR PCTThe Interlacer consists of two main types of layers: (1) linear attention Performer-PCT layer [14, 15] and (2) graph-based feature-agglomeration layer, leveraging structural inductive priors encoded by local neighborhoods in graphs, that we refer to as Neighbor-Attender. The input point cloud for each timestep is first processed by the Neighbor-Attender modules. Their outputs are then processed by Performer-PCT modules, which produce versions of their input point clouds with updated features. These feature point clouds from each timestep are then merged together into one large point cloud along with the robot's skeleton points from all times. The Interlacer then applies one more Neighbor-Attender layer, followed by a final block of Performer-PCT layers. This model predicts the change in location and features (∆x i , ∆f i ) for each particle i in the last input timestep T -1. These deltas are applied to(x T -1 i , f T -1 i) to construct a particle prediction at time T , which is fed back into the dynamics to predict another step forward or into the renderer for image generation.The Performer-PCT [14, 15] is described in detail in Appendix A. Next, we discuss Neighbor-Attender, a novel mechanism to incorporate geometry into attention. 
incorporating geometry into attention: the neighbor-attender layer: 
 The Neighbor-Attender, illustrated in Figure 4 is designed to provide each particle with information about the geometry and features of its immediate neighborhood as efficiently as possible. A simple approach might find the k nearest neighbors of each particle and extract a summary of those neighbors' features and relative positions, but this would involve a forward pass on kN particle-neighbor pairs. Inspired by RandLA-Net [18] , Neighbor-Attender instead computes such neighborhood features only on a small, uniformly-sampled subset of particles we call anchor particles, then uses the anchor particles to update the rest of the particles. This results in a bottleneck step of only kN r pairs for a subsampling rate r, allowing us to control memory consumption at will. The computations of the Neighbor-Attender layer consist of six steps that we explain in detail below; steps 1-4 aggregate neighborhood features onto the anchor particles, then steps 5-6 use those to update the rest.1. Choosing anchor particles: Sample uniformly at random r = N 4 particles from the input set of N particles. We refer to them as anchor particles. 2. Computing neighbors of anchors: For each anchor particle i, compute the set τ i of its s = 16 nearest neighbors from the entire N -element set. 3. Computing attention-vectors: For each anchor particle i and its neighbor j ∈ τ i , compute the relative position feature vector defined as: (x i , x j , x i -x j , x i -x j 2 ) and concatenate it with the feature vector f j . We refer to the resulting vector as the (i, j) edge feature f ij . 4. Updating feature vectors for all anchor particles: For each anchor particle i, compute its new feature vector f i as the weighted sum of its edge features using a learnable MLP module:EQUATION5. Finding closest anchors for all the particles): For each particle i in the original Nelement set, find its closest anchor particle a(i). 6. Computing new feature vectors for all the points: For each particle i in the original N -element set, compute its new feature vectorf i = MLP(f i , f a(i) ; θ 2 ).4 Experiments 
datasets and hd-vpd instantiations: 
 Hardware and dataset We train HD-VPD models on a dataset of bi-manual Kuka robots interacting with a variety of objects and tasks. The dataset consists of episodes collected from 5 different robots over several months. The robots (details in Appendix B, [19] ) are equipped with an overhead RealSense and left arm wrist RealSense cameras capable of providing RGB-D images. The dataset is an uneven distribution of approximately 60 tasks. We break the episodes into snippets of trajectory of at most 8 steps with 500ms passing between consecutive steps. The dataset is 1.6TB.We train HD-VPD models with three different dynamics architectures and varying numbers of particles for the experiments below. These architectures are: (1) a hierarchical GNN using the architecture from [9] with the addition of kinematic particles for actions; (2) Performer-PCT, which consists of repeated Performer-PCT layers; and (3) Interlacer, which includes both Performer-PCT and Neighbor-Attender layers. Architecture and training details are available in Appendix E.In this section, we present learned dynamics obtained with our HD-VPD method for the bi-manual Kuka robot. Corresponding videos are available at https://sites.google.com/view/hd-vpd.We evaluate the quality of next-step image predictions (measured by SSIM) on the test set made with each architecture using varying numbers of particles. In Fig. 5a , we show that the Interlacer-based HD-VPD model produces comparable-quality predictions to the GNN at every number of particles. The GNN model is unable to be trained at 65K and 130K particles due to memory consumption, whereas the Interlacer model continues to improve with scale.The Performer-PCT model lags behind the GNN and Interlacer at each number of particles. We also explore the behavior of these models on multi-step predictions, where the increasing uncertainty associated with long-range prediction might render high-density models' image quality advantage moot. Figure 6 shows that while Interlacers of all sizes are less accurate the further into the future they predict, using more particles continues to produce better predictions. Computational requirements We evaluate the key computational costs associated with running and training on the three classes of models. Figure 5b shows the number of dynamics steps per second achieved with each architecture and number of particles. The Interlacer is at least as fast as the GNN model at each scale, in some cases providing more than 2x speedups. The Performer-PCT is the fastest by a significant margin, but it is unable to match the image quality of the other models. In terms of memory consumption (Figure 5c ), the GNN model always requires considerably more memory to train than the Interlacer or Performer-PCT models, and is unfeasible to use at scales beyond 32K. shows the motion of a successful grasp (Coke can points merge with gripper and lift). Right: shows HD-VPD prediction for a failed grasp (Coke can points don't leave the table ).Box push plan selection This task is to move the box a 12.5 cm to the right as shown in Fig. 7 .We have a fixed set of motion plans that move between 12.5 to -12.5 cm. These planned motions are rolled out with HD-VPD. By segmenting the box points from the input point cloud, we define the cost of a push plan as the median distance between the predicted and goal box points. Lower cost for a plan implies that the model predicts the box will be closer to the goal. Figure 10a shows that HD-VPD's predicted costs are close to ground truth (distance 0.25 for a large push the wrong way, 0.0 for a push to the right location). Choosing the minimum-cost plan under HD-VPD's cost predictions leads to optimal behavior in this experiment.Grasp plan selection This task is to lift a Coke can straight up off the table to a position 0.2m in the air. We have a fixed set of planned grasps at various locations relative to the object, with offset 0.0 corresponding to grasping at the center of the object. We roll out HD-VPD to get predicted outcomes of each plan. By segmenting the points corresponding to the can out of the point cloud, we can define a cost function for each plan. Lower cost is better and implies can particles are closer to the goal. We show that the HD-VPD model cost increases as the planned grasp pose moves away from the object and becomes less likely to succeed as shown in Figure 10b . The HD-VPD model is able to infer that the Coke can points will be lifted on the grasp motions that do succeed in the real world (blue points). More details in Appendix D. Dataset inspection After training HD-VPD, we run it on trajectories from the training set and find those with anomalously high loss. This surfaces previously-unnoticed errors and outliers in the data, including sessions with incorrect geometric camera calibration, faulty camera exposures, and humans in the robot workspace. Such dataset introspection may be useful when preparing a dataset for training models such as policies. Examples are shown in Appendix F. 
related work: 
 Dynamics models for robotics Accurately modeling robot dynamics is essential for tasks such as motion planning, control, and simulation. Traditional approaches rely on rigid body dynamics models [1, 4, 2, 5, 3] , but these models can be inaccurate in complex environments where objects are deformable or interacting in unpredictable ways, and require painstaking authoring of scenes and assets. A significant literature has explored the use of learned models for robot dynamics, which can broadly be divided into two camps: 2D and 3D. 2D models operate directly on pixels and use architectures designed for image generation [20, 21] . While these have found some success by scaling up to modern foundation model sizes and datasets [22, 23] , their learned dynamics can be nonphysical, and they lack 3D representations that can be used for cost functions and scene editing.More closely related to this work are the 3D learned dynamics models. Various works have used datasets of 3D dynamics to train dynamics models with particle [24, 25] , mesh [26, 7] , or object [27, 28] representations, and these models can be more accurate than analytic simulators in some circumstances [8] . However, the requirement for training data with ground-truth 3D particle or object poses limits their applications. Works have attempted to train 3D dynamics models with perceptual data such as videos and point clouds, many leveraging Neural Radiance Fields (NeRF) [12] . One approach has been to apply point cloud distance functions such as Chamfer distance to define loss functions between predicted points and the next-step point cloud [29] or NeRF [30] observations, though these often require object segmentations and have not been applied for large-scale scenes. Another strategy pre-trains conditional or object-level NeRF renderers, then trains a dynamics model on these representations [31, 32] . This split training process and coarse representation is limiting for dealing with complex scenes and results in lower-quality predictions [9] .HD-VPD derives from Visual Particle Dynamics (VPD) [9] , extended to support robot actions and larger, more detailed scenes. Relative to the broader literature, HD-VPD has less restrictive data requirements (2 RGB-D cameras) than other NeRF-based works or mesh-or object-based works, and it jointly predicts 3D dynamics and 2D video, unlike pure 2D video or point cloud models.The rise of deep learning models has opened up new possibilities for processing and understanding 3D point cloud data. Models like PointNet++ [33] , Point Transformer [34] and Point Cloud Transformers [35] have demonstrated strong performance in tasks such as object classification, segmentation, and registration. Similar to our work, RandLA-Net [18] focuses on building memory-and time-efficient architectures that scale up to large point clouds. However, these models are typically designed for static point clouds and do not directly model the dynamics of the scene. Our work introduces a new class of PCTs, called Interlacers, which are specifically designed for modeling the dynamics of large-scale point clouds. Performer-PCT architecture was introduced in [15] , but applied in a different setting of scalable Robotics Transformers rather than dynamics modeling. In this paper, we show that intertwining Performer-PCT layers with the introduced here Neighbor-Attender layers is a key to achieving scalable and high-precision dynamics models. 
conclusion: 
 We present a new high-density particle-based world model called HD-VPD and train it on real-world RGB-D and kinematic bi-manual robotics data. To support HD-VPD's high fidelity predictions we propose Interlacers, combining linear-attention techniques with GNN-inspired local neighborhood attention methods, can model point clouds of sizes 100K+, infeasible for previous PCTs and GNNs. HD-VPD outperforms previous SOTA particle-based world models both in prediction quality and computational cost. Finally, we show how HD-VPD can be used in robotic planning, as accurate world models for scenes involving objects and robotic agents operating on them. 
limitations: 
 HD-VPD is a deterministic dynamics model, meaning that its predictions blur with long horizons and stochastic dynamics. We leave extending the model to probabilistic predictions and long-horizon tasks to future work. HD-VPD might be improved via (1) more detailed finger and arm modeling in the kinematic skeleton; (2) data with near misses for robot planning applications; (3) upweighting points most relevant to robot interaction in training loss computations; (4) using loss functions to enforce physically realistic 3D motion rather than only correct RGB values. Planning with visual costs might avoid any mismatch between particle motion and physical outcomes. 
hd-vpd ground truth: 
 Figure 1 : HD-VPD can accurately predict the dynamics of complex real-world interactions between robots (here, 16-DoF bi-manual Kukas) and objects/tools. Left: a push-pedal trash-can opening task and Right: a bimanual dustpan sweeping task. Top row: Renders from HD-VPD. The first image in each pair is a reconstruction of the matching input frame, and the second is a prediction several timesteps into the future given a sequence of robot actions. Bottom row: Ground-truth test set video frames. 
 
 Appendix: We represent actions as 26 particle kinematic skeletons at the desired future state positions. There is an additional particle representing where the fingertips would be if the gripper was closed, particles representing the estimated current position of the fingertips. Also, there is an additional particle on the side of each wrist so that the orientation of the gripper can be disambiguated especially when the fingers are closed. The fingers themselves are somewhat compliant so the skeleton points do not exactly reflect the true pose of the fingertips especially when in contact with the environment. The origin of the space is at the corner of the pink mat closest to the base of the right arm (no wrist camera). An example rendering of this skeleton is shown in Figure 12 . For the grasp planning real world evaluations, two threads of fishing wire where attached to the bottom of the can and run through the table and then tied to a small weight. This way whenever the Coke can is dropped, it returns to exactly the same pose for all rollouts.We start with an initial grasp centered at the center of the can, and then back off the grasps along the y-axis of the table (aligned with pink mat) in 5mm increments. Figure 10b overviews the relationship between these offsets and predicted plan cost. Figure 13 shows the observations from the left and overhead cameras snapshot during the real world rollout of these planned grasps.We train all models with a batch size of 16 for 1M total steps. We use a learning rate schedule of 3e-4 until step 1000, then 1e-4 until step 100K, then 3e-5 until the end of training. We use the AdamW optimizer [36] with weight decay 1e-3 and clip the gradient norm to 0.01 to prevent outliers in the dataset from destabilizing training.All models are trained end-to-end with 6-step rollouts during training. At each of the two input timesteps and the 6 predicted timesteps, losses are computed on 128 sampled rays. Rays from input timesteps are rendered conditioned on the particles encoded from their respective input timesteps, while rays from predicted timesteps use the particles predicted by recursively rolling out the dynamics model in particle space conditioned on a sequence of actions.The encoder uses a U-Net [16] with [32, 64, 128, 256, 256, 128, 64, 32] The renderer follows the same scheme for constructing input features for the NeRF MLP as Whitney et al. [9] . We use four concentric annular kernels with radii [0, 0.01, 0.02, 0.05] and corresponding bandwidths [0.01, 0.01, 0.01, 0.05], and these kernels are approximated using 16 nearest neighbors. The near plane for ray rendering is set at 0.1m and the far plane at 2m. Rays are composited against a solid white background. We introspect our training data by computing a histogram, shown in Figure 15 , of HD-VPD's loss on a sample of the training set and using that to set a threshold for unusually high losses. Once we have set this threshold, we can search the training data for trajectories with high loss. In Figures 16 and 17 we present examples of dataset quality issues discovered by inspecting training examples where HD-VPD's loss is anomalously high. These issues with our dataset were not previously known, and might pose problems for other applications of this data, such as policy learning.Discovering them with HD-VPD also allows us refine our hardware setup, fixing calibration issues and avoiding problematic robot states. 


Decision: **Post-Rebuttal Guidance**.
Guidance: accept, but borderline. This is clearly a weak accept/borderline paper - If the PCs have limited capacity on accepting papers and would like to reject this paper, I would not argue for its acceptance. Reading the rebuttal between the authors and the reviewers, I suspect there is some hesitancy in accepting the results of this paper. First of all, this approach is only validated on a single robot and single dataset - initial results were provided on the Deformable Collision dataset (but the proposed approach had comparable performance with the baseline). All three reviewers admitted that the prediction results were not high quality. While I do appreciate that the proposed approach is more time/memory efficient, I feel that it is incomplete without demonstrating better performance on more datasets (e.g. Droid) or on more tasks.

**Pre-Rebuttal Summary**

Strengths:
- The proposed approach is more time and memory efficient with comparable or better perf
- Clear writing

Weaknesses:
- Limited novelty over prior work (VPD, Performer-PCT, etc.)
- Evaluation is not as convincing - only 1 setting is shown. Reviewers have suggestions for other datasets (eg. Droid)
- Generated reconstruction quality is still very poor

The reviewers have mixed responses to the paper - authors hope you could check and address them

Reviews: review: title: title: Review
paper_summary: paper_summary: This manuscript presents a method to predict visual particle dynamics of the real-world interactions between robots and robots and objects given an input scene and the robot actions. Specifically, a new point cloud transformer called Interlacers is introduced. This architecture intertwines efficient linear-attention Performer layers proposed in [19] and graph-based neighbor attention layers. Experiments were conducted on a bi-manual Kuka arms setting with two cameras. Compared to the previous Graph neural network-based method, the proposed method achieves comparable performance with less computation time or achieves better performance using 4x more particles. Experiments also demonstrate the potential of using this model for downstream tasks, such as rollout evaluation for robot planning and detecting outliers from a training dataset.
main_review: main_review: ### Strength
* The proposed method is more time and memory-efficient while achieving comparable or better performance compared with a previous GNN-based method for the robot dynamics prediction.
* The proposed model represents robot actions as a set of robot joint particles as input for the dynamics model. This overcomes the limit of VPD [9] which can only work on passive dynamic scenes (without actuation).
* The writing is clear and easy to follow. The Appendix and the website provide additional details and animations to promote an easier understanding of the paper.

### Issues
* The contribution on the technical side is a bit incremental. The overall framework is an improvement upon VPD [9] by replacing the GNN-based dynamics model and adding additional robot joints as input. The presented dynamics model, i.e., the Interlacer, combines a previous work Performer-PCT layer [14,15] and a neighbor-attenter, which is similar to the local feature aggregation in RandLA-Net [18].
* The proposed method is only trained and evaluated in a single setting. Since the method is not limited to bi-manual robots, it would be more convincing if the method could be evaluated in other public robot datasets, such as Droid dataset [a].
* Clarification: It's unclear how the dataset is split for training and testing. Since the dataset includes 5 different robots and 60 tasks,  I would like to see how generalizable the model is with different robots, unseen objects, and unseen tasks.
* What is the performance of the proposed Interlacer architecture on the Deformable Collision dataset [9]?

### Typos
* Line 75: "...the scene state and and..." (duplicated "and")

[a] Khazatsky, A., Pertsch, K., Nair, S., Balakrishna, A., Dasari, S., Karamcheti, S., ... & Finn, C. (2024). Droid: A large-scale in-the-wild robot manipulation dataset. arXiv preprint arXiv:2403.12945.

---
Post-rebuttal updates:

Thanks to the authors for the added evaluation of HD-VPD vs VPD on the deformable collision dataset. It's interesting that the proposed model performs slightly worse than VPD, on both plots and the GIF in the rebuttal material. This is somehow consistent with the observation (also from other reviewers) that the predicted video is a bit blurry. 

It's also great to see the additional robot execution videos, which reduces my (and other reviewer's) concern on how to use it for real robot planning.

I also appreciate that the authors promise to conduct an evaluation on held-out data in the camera-ready version. They also promised to open-source the code.

Regarding DROID: I still think the proposed method can benefit from these modern large-scale robot datasets (e.g., droid, open x, rh20t, etc) and potentially generalize better for downstream applications, since these datasets contain more diverse scenes, robot motions, camera views, etc. 

I changed my rating of `Robotics Focus` and increased the final score.
questions: questions: I think the following things could further improve the paper:
* Clarify the training/testing protocol in experiments.
* Benchmark the model on the Deformable Collision dataset [9]
* Add experiments for unseen robots, objects, and tasks.
* Add experiments on public robot manipulation datasets such as droid.
* Varying the number of cameras.
limitations: limitations: 2
review_summary: review_summary: The paper presents a new architecture that can model the visual particle dynamics of a robot manipulation scene given future robot actions. Although the new architecture is more efficient compared to prior work VPD, I feel the technical novelty in this architecture is limited and some additional experiments should be conducted on the Deformable Collision dataset. The experiments are missing important details and the generalization ability is unclear. The training is limited to one robotic dataset and the paper can be further improved by adding more experiments on public robotic manipulation datasets such as Droid.
title: title: Not super impressive results and better presentation needed
paper_summary: paper_summary: The paper presents a new model architecture which combines linear attention and GNN for training a 3D particle dynamics model with more particles
main_review: main_review: ### Strength
1. Overall pipeline is reasonable. The problem is well motivated and the experiments show that more particles lead to better future video prediction, which is a nice result and justifies the motivation.
2. The experiments clearly show benefits over GNN in terms of the memory requirement.

### Weakness
1. Generated video quality is poor. Future prediction tends to become blurry even without contact. This puts the capability of the model in modeling fine-grained dynamics in question.
2. When comparing to GNN, the performance gain seems to shrink with scale. This puts a doubt on the scalability.
3. The method is somewhat incremental over VPD, with mostly an architecture change. Furthermore, it is unclear how the method compare with VPD from the experiments section. While I understand that VPD does not include the robot actions and may not scale well, it should be necessary to make clear other differences both in text and through experiments.
4. The application to control is not clearly demonstrated. If I understand correctly, the planning is done over simplified action space, such as the grasping offset or the pushing distance. Is the model not fast enough for planning more complex actions? Visualization in figure 8 and 9 are not clear enough; Would be nice to extend the x-axis of figure 10 (a)
questions: questions: Why are the generated future predictions blurry? The demonstrated case mostly deal with simple rigid body dynamics such as pushing and grasping, which should be fairly deterministic. Additionally, it seems that the generated videos become blurry even before the robot makes any contact.
limitations: limitations: 2
review_summary: review_summary: Major concern is the quality of the generated videos. This seems worse compared to prior work such as VPD
title: title: I like the ideas presented in this paper and the focus on efficiency and real-world applications.
paper_summary: paper_summary: This work proposes HD-VPD, a method combining a particle-based visual representation with an efficient Performer-based Point Cloud Transformer that utilizes GNN-inspired neighborhood aggregation, Interlacers, for dynamics modeling for physics and interaction prediction. The proposed dynamics module is more efficient than previous GNNs used for similar tasks and produces comparable or better performance on dynamics prediction. Notably, the method utilizes only two camera views and actions for training.
main_review: main_review: **Strenghts**:
* Novel architecture combining linear attention and neighborhood aggregation ideas from GNNs.
* Significant data collection effort.
* Real-world robotic application.
* Nice visualizations.
* Convincing performance gains over baselines.
* Computational complexity analysis.
* Detailed appendix.

**Weaknesses**:
* Visual reconstruction/prediction quality is low and results in very blurry generated videos.
* Writing and organization (Clarity): I found it quite hard to understand the downstream applications section – the heading is followed by several figures and then jumps to mechanically describe tasks and results without explaining the setup, e.g., “we demonstrate how HD-VPD can be used in real-world downstream tasks by using it to plan..” or something like that. I highly encourage the authors to improve the writing in this part.
* Downstream results: only graphs, no images/videos of successful/failed executions.

**Limitations**: Most of the limitations are clearly addressed. I encourage the authors to further discuss failure cases and an analysis that can provide insights into what parts of the model can be improved.
questions: questions: * Prediction window: throughout this work, the authors use l=2, meaning T-2, T-1 and T. To my understanding, the method observes 2 frames and performs a one-step prediction? Then, an autoregressive process is applied to predict more time-steps. Is it possible to train the model on one-step predictions but in-parallel? For example, Transformers usually train with a one-step prediction loss, but they are using “teacher-forcing” to train with a larger horizon (without autoregressive training, only at inference).
* Particle merging from different views: are you using a common method to merge different views? I couldn't find details on that in the appendix.
* Action representation: I find it interesting that you used one-hot vectors to encode the temporal position of the joints. In Transformers, it is common to learn additive embeddings, have you experimented with this kind of positional embedding?
* Other loss functions: as I mentioned, the prediction/reconstruction visual quality seems low. Have you considered other losses such as the pixel-wise L1 loss?
* Anchor particles: have you considered other sampling methods other than uniform, e.g. farthest-point-sampling (FPS)? What happens if uniform sampling yields two particles that are very close, won’t that be “wasteful”?
* Do the authors plan to release an open-source code for further research?
limitations: limitations: 3
review_summary: review_summary: Overall, I like the ideas presented in this paper and the focus on efficiency and real-world applications. I appreciate the data-collection effort and I find several key contributions that are novel with moderate impact and significance. I have raised several concerns in the “Weaknesses” and “Questions” parts of my review that I would like the authors to address, but I think the paper is relevant and would be interesting to the CoRL community. As such, my current recommendation is “Weak Accept”, but I’m willing to increase my score given convincing answers to my concerns and the concerns that might be raised by the other reviewers. **Post-rebuttal**: I appreciate the authors' clarification regarding the merging process, I highlt recommend including it in the paper for the curious reader. I'm still not convinced regarding the autoregressive training of the Transforner. The authors claim that "Performing a few steps of rollout during training helps the model learn to correct its own errors and produce stable long predictions, and is especially helpful for a deterministic model.", which I think is a very strong claim without any ablation of the rollut steps (which is set to 6 across all experimetns if I understood correctly). I think there is a general consensus that the prediction quality is not great but the method is still applicable for downstream tasks. I thank the authors again, and given the above I'd like to keep my score.