GauDP: Reinventing Multi-Agent Collaboration through Gaussian-Image Synergy in Diffusion Policies
Authors: Ziye Wang
Li Kang
Yiran Qin
Jiahua Ma
zhanglin peng
LEI BAI
Ruimao Zhang
Venue: NeurIPS 2025 poster ()
Keywords: 3D Gaussian Reconstruction
Diffusion Policy
3D Scene Representation
Robot Action Generation
Primary area: applications

Abstract: Despite significant advances in robotic policy generation, effective coordination in embodied multi-agent systems remains a fundamental challenge—particularly in scenarios where agents must balance individual perspectives with global environmental awareness.
Existing approaches often struggle to balance fine-grained local control with comprehensive scene understanding, resulting in limited scalability and compromised collaboration quality.
In this paper, we present GauDP, a novel Gaussian-image synergistic representation that facilitates scalable, perception-aware imitation learning in multi-agent collaborative systems.
Specifically, GauDP reconstructs a globally consistent 3D Gaussian field from local-view RGB images, allowing all agents to dynamically query task-relevant features from a shared scene representation. 
This design facilitates both fine-grained control and globally coherent behavior without requiring additional sensing modalities. 
We evaluate GauDP on the RoboFactory benchmark, which includes diverse multi-arm manipulation tasks. 
Our method achieves superior performance over existing image-based methods and approaches the effectiveness of point-cloud-driven methods, while maintaining strong scalability as the number of agents increases. 
Extensive ablations and visualizations further demonstrate the robustness and efficiency of our unified local-global perception framework for multi-agent embodied learning.

GauDP: Reinventing Multi-Agent Collaboration through Gaussian-Image Synergy in Diffusion Policies
Ziye Wang 12
Li Kang Yiran Qin
Jiahua Ma
Zhanglin Peng*
Lei Bai Ruimao Zhang
Sun Yat-sen University
The University of Hong Kong
Shanghai Jiao Tong University SShangaai Al Laboratory
"The Chinese University of Hong Kong, Shenzhen
Abstract
Recently, effective coordination in embodied multi-agent systems remains funda- mental caclllenee--attcularry. in scenarios where agents must balance individual perspectives with global environmental awareness. Existing approaches often strug- gle to balance fine-grained local control with comprehensive scene understanding, resulting in limited scalability and compromised collaboration quality. this paper, present GauDP, novel Gaussian -image synergistic representation that facilitates scalable, perception-awe aware imitation learning in multi-agent collaborative systems. Specifically, GauDP constructs globally consistent 3D Gaussian field from decentralized RGB observations, then dynamically redistributes 3D Gaussian attributes each agent's 's local perspective. This enables all agents adaptively query task-critical features from the shared scene representation while maintaining their individual viewpoints. This design facilitates both fine-grained control and globally coherent behavior without requiring additional sensing modalities. We evaluate GauDP the RoboFactory benchmark, which includes diverse multi- arm manipulation tasks. Our method achieves superior performance over existing image-based methods and approaches the effectiveness of point-cloud-driven meth- ods, while maintaining strong scalability as the number of agents increases. Codes are available at https: //tp://ezeeeeeeeeeeee wwaeeeeeeeegiiiieeee github. io/ Lo/gaudp. io/
Introduction
Multi-agent embodied collaboration 2 is emerging as a key enabler in a wide range of real- world domains, including industrial assembly [5), surgical robotics [6], and assistive household [7] tasks. Unlike single-agent settings, multi-agent collaboration introduces a unique challenge: each agent must complete its assigned task while remaining synchronized with others avoid catastrophic failures such as collisions or task disruptions.
Existing approaches 8 9 for multi-agent control typically rely two paradigms of observation. The first aggregates local observations from all agents and feeds them into a single shared policy (Fig. [b). While local views offer fine-grained details necessary for precise manipulation, simply concatenating these observations fails to capture the joint collaborative state, often leading to misaligned execution. may attempt to place food into pot before the other has finished lifting the
For instance, lid res-resuiing in failed coordination. The second paradigm employs a global observation of the entire environment (Fig. [b), which provides consistent representation for joint decision-making.
WWrk completed by Ziye Wang as visiting research student at Sun Yat-sen University. "Equal contribution. *Corresponding author: Ruimao Zhang ruimao. wwwnggiieeecooommm
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Local Context
Global Context
Local Context
Global Context
2D-DP
3D-DP
Gmu-DP
Input
2D Image
3D Points
20 Image
Global Coot.
Local Only
Global Only
Global
Diffusioo
Diffiusion Policy
Diffission Policy
Policy
action
action
action
Blased
(8)
(c)
Collision
Localization
Localization
Coordination
Figure 1: Both local and global context are essential in multi-agent collaboration. Comparison of multi-agent decision-making using different types of contextual information. (a) Using only local context leads to miscoordination such as collisions due to lack of global awareness, (b) Using only global context provides a holistic scene view but lacks detailed local features, resulting in inaccurate control, such as biased localization. (c) Our proposed method, GauDP, fuses global context, which reconstructed from 2D local images via shared 3D Gaussian representation, on top of local observations. This integration enables both accurate localization and coordinated execution. Our proposed method, based solely on 2D observations, effectively aggregates global context on top of the local context.
However, this approach often lacks the high-resolution, agent-specific information required for reliable low-level control, such grasping placement, thus reducing individual agent performance.
To address this dilemma, effectively integrating both global and local observations is crucial. However, naive fusion of these signals typically lacks 3D structural constraints, making it difficult for the model to reason about spatial relationships and agent-specific contexts. This motivates the need for a unified representation that can simultancously encode global consistency and local precision.
To this end, we propose GauDP. unified
Gaussian representation for multi-agent embodied collaboration (Fig. D... Our framework first reconstructs 3D Gaussian [10] field from the agents' local-view RGB images captured from arbitrary viewpoints. This allows the system to build globally consistent yet spatially detailed scene representation. Each agent then dynamically queries the shared Gaussian representation to extract task-relevant features for decision- pocksinn-making, enabling coordination while preserving fine-grained control. Importantly, our design naturally scales to more agents without requiring architectural changes, thanks
flexibility of the Gaussian representation.
We evaluate GauDP on the RoboFactory [ benchmark across diverse multi-ane collaboration tasks. Experiments show that our method significantly outperforms image-based imitation learning methods [] and achieves performance comparable to point-cloud- boint-cloud-based methods such as 3D Diffusion Policy (.M) despite using only RGB input. Further ablation studies demonstrate GauDP's robustness and scalability as the number of agents increases. Visualization results confirm its ability to integrate multi-agent observations into high-quality 3D global representation that improves decision accuracy.
Our contributions are summarized as follows: (1) We introduce GauDP, a unified framework that integrates local and global observations via 3D Gaussian fields for multi-agent embodied collaboration. (2) We design a dynamic representation selection mechanism that enables each agent to reason over shared 3D context while maintaining individual precision. (3) We demonstrate the effectiveness and scalability of GauDP on the RoboFactory benchmark, achieving strong performance with only RGB input.
Related Work
2.1 3D Reconstruction from Multi-View Images
The advent of Neural Radiance Fields (NeRF) [12 and 3D Gaussian Splatting (3DGS) 13 has significantly advanced 3D scene reconstruction by representing entire environments unified set

---

of spatially distributed primitives. These methods are capable of not only accurately reconstructing photorealistic visual uppearances but also capturing the underlying 3D geometric structure of a scene from multi-view images. However, achieving high-fidelity reconstructions typically requires densely sampled input views and lengthy optimization time for each scene.
To address this, a growing body of work has focused on adapting NeRF [4 16 17 and 3DGS [T] 19 20 21 22 23 to operate under sparse-view conditions. These approaches typically introduce additional priors, such as semantic information or geometric constraints, to regularize the inherently ill- pol-p problem of reconstruction from limited viewpoints. Besides, they still often rely on accurate camera poses and sufficient overlap among the views, which are difficult to obtain in real-world robotic manipulation scenarios.
Beyond reconstructing high-fidelity 3D scenes, traditional Pruuture-from-Mooiie (SfM) pipelines [24] estimate both 3D structure and camera poses based sparse feature correspon- dences. While SfM remains effective in many cases, its performance significantly degrades under wide baselines or extremely sparse views, where reliable feature matching becomes challenging. Recently, learning-based approaches have emerged that directly infer dense 3D geometry from a small number of images 255
26 27 28). These methods mark a shift toward end-to-end systems that implicitly learn geometric relationships, enabling 3D structure estimation even from as few as two input views.
2.2 Robot Manipulation
Behavioral Cloning (B 30 31 32 33 trains policies using pre-recorded human demonstrations to directly imitate expert behaviors, whereas Offline Reinforcement Learning (3RL) 35 36)
refines action selection through reward maximization over large-scale fixed datasets. While BC directly mimics demonstrated behavior, ORL enables further policy improvement by optimizing over offline rewards. Generative approaches have expanded the landscape of policy learning: Action Chunking with Transformers (ACT) combines Transformer architectures with conditional variational autoencoders to capture temporal dependencies in sequential decision-making [37
38 39 More
recently, diffusion-based frameworks have shown strong potential in robotic imitation learning due to their high-fidelity trajectory generation. Notable examples include Diffusion Policy (40) and its 3D extension [11], which leverages point cloud inputs enhance spatial reasoning. The stochastic generative nature of these models makes them especially effective in capturing multimodal action distributions. Demonstration acquisition primarily relies on human-operated robotic systems across diverse tasks [4] 42 301. while simulator-basee trajectory synthesis has emerged as scalable alternative 43
47 Simulated environments allow for controlled task variation and embodiment flexibility. However, existing systems largely focus on single-agent scenarios. Effective data-driven policy learning for multi-agent robotic manipulation cnmipuultionnpprttlllaaal particularly in settings involving coordination among multi-agent-reman significantly underexplored.
3 Method
3.1
Preliminary
3D Gaussian Splatting (3DGS). 3D Gaussian Splatting [33 represents a 3D scene as a collection of spatially distributed anisotropic Gaussian primitives. Each Gaussian is parameterized by 3D mean position la E R3, a scaling vector s E R a rotation represented by a unit quaternion r E IR4, an opacity value a E R. and color information e E R To model view-dependent effects, the RGB color can be modulated by additional spherical harmonics coefficients h E R Therefore, a group of Gaussians can be represented as (U(M,, 8,, T,, 0i2 C; h.)).
Rendering in 3DGS is performed through a differentiable rasterization process that computes each Gaussian's 's contribution to the image plane. First, all Gaussians are projected into the camera coordinate system. The screen space is then divided into tiles, and Gaussians falling outside the view frustum efficiently culled to reduce computation. Finally, for each pixel, visible Gaussians sorted in view-space depth order, and their contributions are composited via alpha blending.
This rendering pipeline enables accurate and efficient supervision of the underlying 3D structure. During optimization, the parameters of the Gaussians are updated to minimize the discrepancy between the rendered images and the ground-truth observations across multiple camera views.
3

---

Importantly, when the optimized Gaussian representation yields rendered images that are consistent with the ground-truth across views, it can be regarded as a faithful reconstruction of the true scene geometry. Formally, let 9Λ and GB denote two distinct 3D Gaussian configurations, and let R(G, v) denote the rendered image of & under viewpoint v. the renderings of G and GB are identical across all training views V
U1, U2, UN:
Vu E V. R(G = v),,
then GA and GB must be geometrically equivalent, i.e., GA GB. Conversely, if they are not geometrically equivalent, there must exist at least one view v E V such that their renderings differ:
A # 9B
EVE,, R(GA, v) * R).
This implies that mieti-view consistency effectively constrains the optimization to a unique, geomet- rically faithful solution in a self-supervised manner.
Diffusion Policy (DP). Diffusion Policy [88] formulates action generation as conditional denoising diffusion process. Given sequence of past observations O = I2......... INN. the goal is to generate future action sequence a {a1, alt,aa,.....
The target action sequence a is gradually perturbed by Gaussian noise through a forward diffusion process:
q(a" a ()a-))- .V -N(VVVI aa- B.1), k I..,,,,,K
where B% is the noise variance schedule. The reverse process learns to iteratively denoise a random sample a ~N(0., I) back to clean action sequence, conditioned on observations O:
p4(a" k a 0)) = (N (a
,o,d,,kk))) ,0, Eg O,,))
where & denotes the parameters of the conditional denoising network. Through iterative denoising, the learned policy σ generates action trajectories by:
T (a
O)
E ((..)
II po (a k- .0)
3.2 Problem Formulation
We consider the problem of predicting future action sequences for multi-arm embodied agents based on multi-view visual observations. Let O =
In} denote a set of synchronized observations captured from N views. Each image I, captures the scene from a unique perspective, offering complementary geometric information. The prediction target is sequence of future actions {ar, a2...., ar.). where a, IR represents the control signal timestep t.
Rather than directly predicting a from 2D image features, we first reconstruct a compact and differentiable 3D Gaussian representation
global context from O. We define a conditional policy nh parameterized by that generates the action sequence a conditioned on both the raw visual inputs O and the reconstructed 3D representation
T& (a O) n (a O,9).
Where, G = F(O) and F(-) mapping from multi-view observations a of Gaussians. To model the complex conditional distribution over action sequences, we adopt the Diffusion Policy framework. Let a denote the target action sequence, and let a (O, be a sample from an isotropic Gaussian prior. The generative process denoises K through a learned reverse process conditioned on (0,9):
II (a K- -i a K-it1 (K9-
To (a: (0,9) = Ba (((.))
where pa denotes the learned denoising transition at each timestep. This framework allows the policy to generate realistic and context-aware action trajectories without relying on strong priors over the action space.

---

Obseration
Otservation
Local Context
Encoder
Global
Context
Encoder
Encoder
Cross Attatiin
Local Centext
Encoder
Deooder
Decoder
free
Local Context
Encoder
Guussian
Attention
uf
Predicted
Action
Diffusion Policy
Fusion Context
Rendered Image
(a)
(b)
Figure 2: (a) Overview of the proposed GauDP framework for multi-agent imitation learning. Each agent extracts a local context from its 2D observation. A shared 3D Gaussian field is constructed from all views to form the global context, which is fused with the local context and passed through an encoder. The resulting per-agent features are processed by diffusion policy via cross- cross-attention to predict actions. (b) Pipeline for constructing the global Gaussian field. Multi-view images encoded and aggregated via cross-attention, followed by reconstruction loss Lree between rendered and input views to ensure consistency.
3.3 Overview
Mainstream approaches typically train policies to predict future actions directly from either image observations global point clouds. However, these methods face limitations in multi-agent collabora- tive tasks: they either rely solely on local observations for each agent or require access to a centralized global point cloud, which restricts both accurate localization and effective coordination among agents. To address these challenges, we propose a novel framework solely based on image observations that enables collaborative multi-agent manipulation through effective global context integration. Specifically, our method fuses multi-view observations to reconstruct a unified global context and selectively distributes task-relevant features back individual agents as needed. This design not only enhances inter-agent coordination but also improves precise perception and localization. An overview the proposed framework is illustrated in Fig. 2 a).
3.4 Global Context Reconstruction
In this work, we define a global context unified, view-independent representation built within common 3D coordinate space. This representation should not only preserve color information from raw multi-view image observations, but also restore the underlying 3D structure the scene reconstructed from these views. To achieve this, we design framework that reconstructs 3D scenes in a self-supervised manner using only the multi-view 2D observations typically employed for training diffusion policies. Our reconstruction framework is built upon 3D Gaussian Splatting (3DGS). However, conventional 3DGS methods suffer from two major limitations: First, they require densely sampled views with accurate camera poses. Second, they demand scene-specific optimization that can take several minutes per scene. These constraints render them impractical for embodied scenarios, where rapid adaptation and generalization are essential.
To overcome these challenges, we adopt Noposplat [9], a need-forward network capable of directly reconstructing 3D Gaussian representations from sparse and unposed views. We further fine-tune the pretrained Noposplat model using multi-view observations collected from our robotic manipulation scenarios, which are the same data used to train downstream diffusion policy.

---

As illustrated in Fig each RGB image is independently encoded by shared-weight ViT [50] encoder across all views. The resulting per-view features are then passed through a cross-view ViT decoder, which fuses information across different perspectives using cross-attention layers in each transformer block. Finally, a Gaussian parameter prediction head estimates a set of 3D Gaussians for each pixel based on the fused features. This process can be expressed as:
g. =(xi), E
where x; denotes the fused feature at pixel i, F(-) is the mapping network, and G E RCxxHXW represents the estimated parameters of the corresponding 3D Gaussian.
To further improve the fidelity of the reconstructed 3D structure, we introduce an additional depth supervision during finettuning Specifically, in the rendering process, each estimated 3D Gaussian is projected onto the camera coordinate system. Instead of computing the RGB contribution of each Gaussian to the image pixels, we compute the contribution of its projected depth the corresponding pixel. This depth rendering process yields a synthetic depth map D, which can then be supervised using available ground-truth depth D via a reconstruction loss Ldepth. This depth-based supervision provides stronger geometric guidance and encourages the model to recover 3D Gaussians that are more consistent with the actual scene geometry. The overall reconstruction loss is defined as:
Lree Ligh Edepth:
where a is balancing weight that controls the influence of the depth supervision.
It is worth noting that depth maps and camera poses are used only during the fine-tuning stage of Noposplat. During policy training and inference, our framework solely relies on multi-view RGB observations to infer the 3D Gaussians, making it lightweight and pose-free at deployment time.
3.5 Global Context Allocation and Pixel-level Synergy
The reconstructed global context encodes rich multi-view and multi-agent information, capturing both the semantic and geometric structure of the scene. However, directly feeding the entire global context to each agent is suboptimal, as it introduces irrelevant information and may interfere with the agent's s ability to focus on task-relevant cues from its perspective. Moreover, effective synergy between global and local context remains underexplored. Existing approaches typically aggregate global and local information only at a coarse level, which fails capture fine-grained spatial alignment and task-specific dependencies. This coarse fusion strategy may lead to diluted feature representations and impaired action reasoning. especially in densely interactive multi-agent scenarios.
To address these challenges, we introduce a selective global context dispatch mechanism along with pixel-aligned fusion strategy for fine- fine-grained integration of global and local information. Recall that in Section 3.4 we reconstruct 3D Gaussians by aggregating multi-vier image tokens via cross- attention. Each predicted Gaussian encodes both visual appearance and geometry within unified global coordinate system, while remaining naturally aligned with the input image pixels from which derived. Leveraging this alignment, selectively dispatch the predicted 3D Gaussians back to the corresponding agent's observation frame based on their image of origin. Instead of distributing the entire global context indiscriminately, each agent receives only the subset of Gaussians associated with its own view. These Gaussians have already integrated information from other views during reconstruction, thus providing distilled and relevant global summary for that agent.
For synergistic fusion, we transform the selected Gaussians back into a 2D grid that matches the spatial dimensions of the original image. These global context features are then concatenated with the agent's 's local image features and passed through lightweight convolutional fusion module, which leams to combine the complementary strengths of local perception and global understanding.
This design ensures that each agent benefits from a targeted and contextually relevant global repre- sentation, while preserving spatial consistency and enabling pixel-level synergy between local and global cues, both of which are critical for precise and coordinated action planning in multi-agent manipulation tasks.

---

Table 1: Quantitative Comparison of 3D Gaussian Reconstruction. Improved visual quality reflects higher accuracy in 3D reconstruction.
Method
PSNR
SSIM
LPIPS 4
Pretrain
17.918
0.580
0.492
Ours
23.424
0.779
0.148
4 Experiment
4.1 Experiment Setup
Imitation learning in multi-agent collaborative manipulation presents significant chal- lenges due to the high levels of complexity, coordination, synchronization, and symmetry awareness required, making it difficult collect high-quality data real-world scenarios. address this issue, we leverage the RoboFactory benchmark [1], an automated data collection framework specifically designed for embodied multi-agent agent systems. We select 6 tasks from RoboFactory involving collab- orative manipulation using two to four robotic arms. These tasks designed to cover range of coordination complexities and physical interaction patterns. Please refer to the Appendix for detailed descriptions of each task.
Dataset.
Baseline. Existing diffusion- ceciiion-ooiccy---e based approaches predominantly focus either 2D 3D) modalities. To ensure a comprehensive and fair comparison, we evaluate our method against several representative baselines in both domains. For 2D vision-based observations, we adopt Diffusion Policy [48] and 2D Dense Policy [5I]; for 3D input modalities, we include 3D Diffusion Policy (I] and 3D Dense Policy [5] as baselines. For fair comparison, maintain consistent visual backbone across all methods: ResNet- 18 is used for 2D visual inputs following the original Diffusion Policy, and lightweight MLP is used for 3D data in 3D Diffusion Policy.
Experiment setting. We use success rate the primary evaluation metric assess the effectiveness of each policy. Evaluation is performed every 100 training epochs over 100 episodes per policy. All experiments are implemented using the PyTorch framework and conducted on a single NVIDIA A800 GPU. Policies are trained for 100 epochs using a batch size of 32. We adopt the Adam optimizer with an initial learning rate of 10 combined with warm-up phase followed by cosine decay scheduling. To ensure a fair comparison, all baseline and ablation models are trained using the same set of hyperparameters and optimization settings.
For fair comparison, our proposed GauDP and all baseline methods are trained under identical hyper- parameters and optimization settings following standard Diffusion Policy benchmarks. Specifically, we use an action prediction horizon of 8, 3 observation steps, and 6 action execution steps. Both GauDP and DP adopt DDPM with 100 denoising steps, while DP3 employs DDIM with the same number of steps.
4.2 Experiment Results
Reconstruction Results. As discussed in Section 3.1 higher-quality rendered images indicate more accurate reconstruction the underlying 3D scene, including both geometry and color information. To evaluate reconstruction performance, we conduct experiments where the full scene is reconstructed using observations from only two reference viewpoints. Quantitative results are summarized in Table I and qualitative comparisons of the rendered Gaussians from both reference and novel views are shown Figure] 3
Our finetuned model significantly outperforms the pretrained baseline, producing reconstructions that are not only sharper and more detailed, but also more faithful the original scene geometry and appearance. As shown in Figure 3 our method yields consistent and high-fidelity renderings across both reference and novel views, with clearly defined object boundaries. contrast, the pretrained model often produces blurry and distorted results, with noticeable discrepancies between the reconstructions and the original images. Besides, the first two columns, the reconstructed positions of the robot arms differ considcrably from those in the ground truth. Our method consistently
7

---

Align Camera
Stack Cube
GT
Pretrain
Ours
GT
Pretrain
Ours
Ref
View
Ref.
View
Novel
View
View
Figure 3: Visualization of Reconstruction Results. Our method achieves significantly improved reconstruction quality.
Table 2: Success rate comparison across multi multi-agent manipulation tasks with different 2-4 arms setting. Underlines denote the best baseline of point cloud-based diffusion policy(DP); bold highlights best image-based DP. GauDP achieves the highest average performance across all settings.
2 Arms
3 Arms
Arms
Method
Lift Barrier
Place Food
Stack Cube
Align Camera
Stack Cube
Take Photo
Avg
DP3(XYZ) III
30%
21%
1%
3%
0%
9%
10.67%
DP3(XYZ+RGB) [|
%
25%
1%
18%
0%
11%
14.33%
3D Dense Policy [1
28%
18%
0%
0%
0%
7%
8.83%
DP 48
9%
12%
6%
3%
0%
0%
5.00%
2D Dense Policy GauDP
3%
2%
0%
0%
0%
9%
2.33%
72%
15%
2%
26 %
0%
3%
19.67%
maintains high structural fidelity and visual consistency, demonstrating its effectiveness in capturing the accurate 3D structure of the scene.
Point Cloud-based Diffusion Policy. The DP3 model leverages 3D point cloud observations to inform multi-agent manipulation. As shown in the table 2 this policy achieves moderate performance across two-arm tasks, such as 30% on Lift Barrier and 21% on Place Food, indicating that point cloud inputs capture sufficient geometric structure for spatially grounded actions. However, its effectiveness drops sharply in tasks with more agents and higher coordination requirements such as only % on Stack Cubes (2 arms). These results suggest that point cloud-based methods lack the fine-grained local control afforded by geometric inputs like point clouds, which limits their effectiveness in precision vrccisoo-crrttcall tasks such stacking cubes, especially in multi-arm settings.y in tasks with more agents and higher coordination requirements.
Image-based Diffusion Policy. shown in the table 2 our method GaulPP-reffuse significantly
outperforms prior 2D diffusion policies across multiple tasks. Compared to DP 40) and 2D Dense

---

Table 3: Training and inference efficiency on the Lift Barrier task (Training on A100 GPU; inference on NVIDIA RTX 5090 GPU).
Method
Training Time (GPU h)
Inference Speed (FPS)
DP
4.8
1.49
DP3
2.5
1.57
GauDP
6.5
1.28
Table 4: Real-robot performance comparison across three multi-agent collaboration tasks. Each score in the table reported as m/n, where m denotes the number of successful executions and represents the total number of rollouts performed for that task.
Card Box Stacking
Card Box Handover
Grab Roller
Method
Place Suec.
Stack Suce.
Succ.
Place Succ.
Handover Suce.
Suce.
Succ.
DP
19/30 23/30
11/19 17/23
11/30
22/30
14/22 19/24
14/30
22/30
GanDP
17/30
24/30
19/30
27/30
Policy [5], which struggle across the board (mostly below 10%). GauDP achieves a remarkable 72% success rate in the Lift Barrier task, indicating its superior ability to model geometry- aware visual rep- resentations. Additionally, GauDP shows strong performance in tasks requiring semantic alignment, such Align Camera (26%), where other methods fail to generalize. These results demonstrate that integrating geometric priors into image-based pipelines enables better spatial understanding and more effective multi-agent coordination, especially in tasks with complex embodiment and scene variability.
Training and Inference Efficiency. We evaluate the training and inference efficiency of GauDP compared with diffusion-based baselines. As shown in Table [3] GauDP requires slightly longer training time due to its geometry-aware modules, but maintains comparable inference speed while offering superior performance.
Real-World Experiments. To further verify the practicality, conduct real-robot experiments three representative multi-agent collaboration tasks: Card Box Stacking, Card Box Handover, and Grab Roller. As shown in Table GauDP consistently outperforms DP across all tasks, particularly in stacking and handover scenarios that require precise spatial coordination.
Discussion. As shown in Table 2 our method GauDP achieves the highest average success rate of 19.67% across diverse multi- agett manipulation tasks, significantly outperforming all baselines, including those relying on 3D point cloud inputs such as DP3 [T]] and 3D Dense Policy [51]. Notably, while method operates solely on 2D RGB inputs, it surpasses several 3D-based counterparts in tasks that require global coordination (e.g., Align Camera: 26% vs. 18%) and even matches them in fine-grained manipulation tasks (e.g., Stack Cube: 2% 1%). These results highlight that approach's 's strength lies not in the modality itself, but in the design of visual representations that fuse both local visual details and global spatial context. This balance enables agents to perceive geometric structure from images and reason over scene-level relationships, allowing for generalizable cooperation without relying on explicit 3D geometry.
4.3 Ablation Study
Ablation on the Coordinate System of Gaussians. We replace the original camera-coordinate parameterization of Gaussians with unified world-coordinate system aligned to the first observation frame. Results show that using local coordinates performs better, as preserves agent-centric spatial relationships and avoids alignment errors across diverse viewpoints. Ablation on Fusion Strategies for Local and Global Context. We replace the default fine-grained, pixel-level fusion with a coarse feature-level concatenation of independently encoded modalities. Performance declines with this coarse fusion, likely due to the loss of spatial alignment and fine- fine-grained cross-modal reasoning. Ablation on the Role of Image and Gaussian. We remove either the image input the Gaussian

---

Table 5: Ablation study on key components of GauDP across multi-agent manipulation tasks. Bold highlights the best among image-basee based configurations. Our full model achieves the highest average success rate, demonstrating the effectiveness of combining geometric and visual cues.
2 Arms
3 Arms
4 Arms
Method
Lift Barrier
Place Food
Stack Cube
Align Camera
Stack Cube
Take Photo
Avg
w/ unify coor.
30%
1%
8%
26%
0%
0%
10.83%
w/o prefuse
2%
4%
0%
1%
0%
0%
1.17%
w/o Image
32%
7%
0%
28%
0%
0%
11.17%
w/o Gaussian
9%
12%
6%
3%
0%
0%
5,00%
Ours
72%
15 %
2%
26%
0%
3%
19.67%
representation during policy training and inference. Using both inputs achieves the best results, as images provide appearance cues while Gaussians supply global geometric context.
5 Conclusion
In this paper, we present GauDP a novel framework for multi-agent collaboration through Gaussian- image synergy in diffusion policies. Specifically, GauDP globally consistent 3D Gaussian repre- sentation from the local RGB observations of each agent and reallocates the Gaussian information back individual agents. This process significantly enhances each agent's perception of the global task information, thereby boosting the success rate of complex collaborative tasks. We evaluate the performance of GauDP using the RoboFactory benchmark, which features diverse set of multi- arm manipulation tasks. As the number of agents increases, GauDP not only outperforms existing image-based methods but also matches the effectiveness of point-cloud-driven methods. Future directions will focus on: (1) designing Gaussian representations that are more suitable as inputs for the Larioon-nnnuggg-AAee Acttonn model enhance its capabilities multi-agent collaboration; and (2) leveraging Gaussians to improve the representation of dynamic scenes, enabling them to play a role in world models designed for multi-agent environments.
10

---

References
[1] Y. Qin, L.. Kang, X. Song, Z. Yin, X. Liu, X. Liu, R. Zhang, and L. Bai, "Robofactory: Exploring embodied agent collaboration with compositional constraints," arXiv preprint arX1v:2503316440, 2025.
[2] B. Huang, Chen, T. Wang, Y. Qin, Yang, N. Atanasov, and X. Wang, "Dynamic handover: Throw and catch with bimanual hands," arXiv preprint arXiv:2309.05655, 2023.
[3] Z. Mandi, S. Jain, and S. Song, "Roco: Dialectic multi-robot collaboration with large language models," in 2024 IEEE International Conference on Robotics and Automation (ICRA), pp. 286- 299, IEEE, 2024.
[4] M. Chang, G. Chhablani, A. Clegg, M. D. Cote, R. Desai, M. Hlavac, V. Karashchuk, J. Krantz, R. Mottaghi, Parashar, el al., "Partor: A benchmark for planning and reasoning embodied multi-agent tasks," arXiv preprint Stti:24111.00055 2024.
[5] D. Jiang, H. Wang, and Y. Lu, "Mastering the complex assembly task with a dual-arm robot: novel reinforcement learning method," IEEE Robotics & Automaaion Magazine, vol. 30, no. pp. 57-66, 2023.
[6] J. W. Kim, T. Z. Zhao, S. Schmidgall, A. Deguet, M. Kobilarov, C. Finn, and A. Krieger, "Surgical robot transformer (srt): Imitation learning for surgical tasks," arXiv preprint arXiv:2407. 12998, 2024.
[7] T. Zhang, D. Li, Y. Li, Z. Zeng, L.. Zhao, L. Sun, Y. Chen, X. Wei, Y. Zhan, L. Li, et al., "Empowering embodied manipulation: A bimanual-mobile robot manipulation dataset for household tasks," arXiv preprint arXiv:2405. 2024.
[8] J.-J. Jiang, X.-M. Wu, Y-X. He, L.-A. Zeng, Y.-L. Wei, D. Zhang, and W.-S. Zheng. "Rethinking bimanual robotic manipulation: Learning with decoupled interaction framework," arXiv preprint arXiv:2503.00 086 2025.
[9] Lv, H. Li, X. Deng, R. Shao, Y. J. Hao, L. Gao, M. Y. Wang, and L. Nie, "Spatial-temporal graph diffusion policy with kinematic modeling for bimanual robotic manipulation." arXiv preprint arXiv:2503. 10743, 2025.
[10] H. Xiong, S. Muttukuru, R. Upadhyay, P. Chari, and A. Kadambi, "Sparsegs: Real-time 360deg sparse view synthesis using gaussian splatting," arXiv preprint arXiv:2312.00000 2023.
[11] Y. Ze, G. Zhang, K. Zhang, C. Hu, M. Wang, and H. Xu, "3d diffusion policy: Generalizable visuomotor policy learning via simple 3d representations," in Proceedings of Robotics: Science and Systems (RSS), 2024.
[12] B. Mildenhall, P. P. Srinivasan, M. Tancik, J. T. Barron, R. Ramamoorthi, and R. Ng. "Nerf: Representing scenes as neural radiance fields for view synthesis," Communications of the ACM, vol. 65, no. 1, pp. 99-106, 2021.
[13] B. Kerbl, G. Kopanas, T. Leimküüher, and G. Drettakis, "3d gaussian splatting for real-time radiance field rendering.." " ACM Trans. Graph., vol. 42, no. 4, pp. 139-1, 2023.
[14] A. Jain, M. Tancik, and P. Abbeel, "Putting nerf on diet: Semantically consistent few-shot view synthesis." " in Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 1885-58994 2021.
[15] A. Yu, V. Ye, M. Tancik, and A. Kanazawa, "pixelnerf: Neural radiance fields from one or few images," in Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 4578 3578-4587, 2021.
[16] W. Yang, G. Chen, C. Chen, Z. Chen, and K.-Y. K. Wong, "S3-nerf: Neural reflectance field from shading and shadow under a single viewpoint," Advances Neural Information Processing 2022.
Systems, vol. 35, pp.

---

[17] B. G. Gerats, J. M. Wolterink, and I. A. Broeders, "Nerf-or: neural radiance fields for operating room scene reconstruction from sparse-view rgb-d videos," International journal of computer assisted radiology and surgery, pp. 1-10, 2024.
[18) H. Xu, A. Chen, Y. Chen, C. Sakaridis, Y. Zhang, M. Pollefeys, A. Geiger, and F. Yu, "Murf: radiance fields," " in Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recognition, pp.
2024.
[19] Q. Zhao and S. Tulsiani, "Sparse-view pose estimation and reconstruction via analysis by generative synthesis," Advances in Neural Information Processing Systems, vol. 37, pp. 111899- 111922. 2024.
[20] D. Charatan, S. Li, A. Tagliasacchi, and V. Sitzmann, "pixelsplat: 3d gaussian splats from image pairs for scalable generalizable 3d reconstruction," in Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 19457-19467, 2024.
[21] Y. Chen, H. Xu, C. Zheng, B. Zhuang, M. Pollefeys, A. Geiger, T.-J. Cham, and Cai, "Mvsplat: Efficient 3d gaussian splatting from sparse mueti-view images," in European Conference Computer Vision, pp. 370-386, Springer, 2024.
[22] Y. Wan, M. Shao, Y. Cheng, and W. Zuo, "'Sggaussinn: Sparse-view super-resolution 3d gaussian splatting," " arXiv preprint (eXii:2503.043/4 2025.
[23] H. Huang. Y. Wu, C. Deng, G. Gao, M. Gu, and Y.-S. Liu, "Fatesgs: Fast and accurate sparse- view surface reconstruction using gaussian splatting with depth-feature consistency," arXiv preprint arXiv:2501.00 046288 2025.
[24] J. L.. Schonberger and J.-M. Frahm, "Structure-from-mmee revisited," in Proceedings of the IEEE conference computer vision and patlern recognition, pp. 4104-4113, 2016.
[25] S. Wang, V. Leroy, Y. Cabon, B. Chidlovskii, and J. Revaud, "Dust3r: Geometric 3d vision made easy," in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 20697-20709, 2024.
[26] J. Edstedt, Q. Sun, G. Bökman, M. Wadenbäck, and M. Felsberg, "Roma: Robust dense feature matching," in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 19790-19800 2024.
[27] V. Leroy, Y. Cabon, and J. Revaud, "Grounding image matching in 3d with mast3r," in European Conference Computer Vision, pp. 71-91, Springer, 2024.
[28] Y. Cabon, L. Stoffl, L. Antsfeld, G. Csurka, B. Chidlovskii, J. Revaud, and V. Leroy, "Must3r: Multi-view network for stereo 3d reconstruction." " arXiv preprint arXiv:2503.0166 2025.
[29] M. Dalal, A. Mandlekar, C. R. Garrett, A. Handa, R. Salakhutdinov, and D. Fox, "Imitating task and motion planning with visuomotor transformers." in Conference on Robot Learning, 2023.
[30] E. Jang, A. Irpan, M. Khansari, D. Kappler, Ebert, C. Lynch, S. Levine, and C. Finn, "Be-z: Zero-shot task generalization with robotic imitation learning," in Conference on Robot Learning. pp. 991-1002, PMLR, 2022.
[31] Y. Jiang, A. Gupta, Z. Zhang, G. Wang, Y. Dou, Y. Chen, L. Fei-Fei, A. Anandkumar, Y. Zhu, and L. Fan, "Vima: General robot manipulation with multimodal prompts," in Fortieth International Conference on Machine Learning, 2023.
[32] A. Mandlekar, D. Xu, R. Martín-Martin S. Savarese, and L. Fei-Fei, "Learning to generalize across long- horizoorizon tasks from human demonstrations." arXiv preprint arXiv:2003.06085, 2020.
[33] T. Ma. J. Zhou, Z. Wang, R. Qiu, and Liang. "Contrastive imitation learning for language- guided multi-task robotic manipulation," arXiv preprint arXiv:2406.09733, 2024.
[34] D. Kalashnikov, J. Varley, Y. Chebotar, B. Swanson, R. Jonschkowski,, C. Finn, S. Levine, and K. Hausman, "Mt-opt: Continuous multi-task robotic reinforcement learning at scale," arXiv preprint 210.:22004.02.12,, 2021.
12

---

[35] A. Kumar, A. Singh, F. Ebert, M. Nakamoto, Y. Yang, C. Finn, and S. Levine, "Pre-training for robots: Offline rl enables learning new tasks from handful of trials." arXiv preprint arXiv:2210.05178 2022.
[36] Y. Chebotar, Q. Vuong, K. Hausman, F. Xia, Y. Lu, A. Irpan, A. Kumar, T. Yu, A. Herzog,
K. Pertsch, et al., "Q-transformer: Scalable offline reinforcement learning via autoregressive q-functions," " in Conference on Robot Learning, pp. 3909-3928, PMLR, 2023.
[37] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. Kaiser, and I. Polosukhin, "Attention is all you need," Advances in neural information processing systems, vol. 30, 2017.
[38] T. Z. Zhao, V. Kumar, S. Levine, and C. Finn, "Learning fine-grained bimanual manipulation with low-cost hardware," arXiv preprint arXiv:2304. 13705, 2023.
[39] T. Buamanee, M. Kobayashi, Y. Uranishi, and H. Takemura, "Bi-act: Bilateral control control-based based imitation learning via action chunking with transformer," in 2024 JEEE International Conference on Advanced Intelligent Mechatronics (AIM), pp. 410-415, IEEE, 2024.
[40] C. Chi, Z. Xu, Feng, E. Cousineau, Y. Du, B. Burchfiel, R. Tedrake, and Song. "Diffusion policy: Visuomotor policy learning via action diffusion," The International Journal of Robotics Research, p. 02783649241273668, 2023.
[41] F. Ebert, Y. Yang. K. Schmeckpeper, B. Bucher, G. Georgakis, K. Daniilidis, C. Finn, and S. Levine, "Bridge data: Boosting generalization of robotic skills with cross cross-domain datasets," " arXiv preprint 131.29119..3366, 2021.
[42] A. Mandlekar, Y. Zhu, A. Garg, J. Booher, M. Spero, A. Tung. J. Gao, J. Emmons, A. Gupta, E. Orbay, et al., "Roboturk: A crowdsoureing platform for robotic skill learning through imitation," " in Conference on Robot Learning, pp. 879-893, PMLR, 2018.
[43] Y. Mu, I. Chen, S. Peng, Z. Chen, Z. Gao, Y. Zou, L. Lin, Z. Xie, and P. Luo, "Robotwin: Dual-arm robot benchmark with generative digital twins (early version)," arXiv preprint Sttiv:2409.029900 2024.
[44] S. James, Z. Ma, D. R. Arrojo, and J. Davison, "Rlbench: The robot learning benchmark & learning environment," IEEE Robotics and Automation Letters, vol. 5, no. 2, pp. 3019-3026, 2020.
[45] S. Tao, F. Xiang, A. Shukla, Y. Qin, X. Hinrichsen, X. Yuan, C. Bao, X. Lin, Y. Liu, T. kai Chan, Y. Gao, X. Li, T. Mu, N. Xiao, A. Gurha, Z. Huang, R. Calandra, R. Chen, S. Luo, and H. Su, "Maniskill3: Gpu parallelized robotics simulation and rendering for generalizable embodied ai," arXiv preprint arXiv:2410.00425, 2024.
[46] S. Nambiar, M. Jonsson, and M. Tarkian, "Automation in unstructured production environments using isaac sim: A flexible framework for dynamic robot adaptability," Procedia CIRP, vol. 130, pp. 837-846, 2024.
[47] S. Nasiriany, A. Maddukuri, L. Zhang, A. Parikh, A. Lo, A. Joshi, A. Mandlekar, and Y. Zhu, "Robocasa: Large- scale simulation of everyday tasks for generalist robots," in Robotics: Science and Systems, 2024.
[48] C. Chi, 7. Xu, S. Feng, F. Cousineau, Y. Du, B. Burchfiel, R. Tedraae, and S. Song, "Diffusion policy: Visuomotor policy learning via action diffusion," The International Journal of Robotics Research, p. 02783649241273666 2023.
[49] B. Ye, S. Liu, H. Xu, X. Li, M. Pollefeys, M.-H. Yang, and S. Peng, "No pose, no prob- lem: Surprisingly simple 3d gaussian splats from sparse unposed images," arXiv preprint arXiv:2410.24207. 2024.
[50] A. Dosovitskiy, L. Beyer, A. Kolesnikov, D. Weissenbom, X. Zhai, T. Unterthiner, M. Dehghani, M. Minderer, G. Heigold, S. Gelly, et al., "An image is worth 16x16 words: Transformers for image recognition at scale," arXiv preprint arXiv:2010.1 /9299 2020.
[51] Y. Su, X. Zhan, H. Fang, H. Xue, H.-S. Fang, Y.-L. Li, C. Lu, and L. Yang. "Dense policy: Bidirectional autoregressive learning of actions," arXiv preprint arXiv:2503.132111 2025.
13

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
The main claims made in the abstract and introduction accurately reflect the paper's contributions and scope.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
We give the limitation of work in the supplementary material.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
14

---

Justification:
The paper provide the full set of assumptions and a complete (and correct)
proof.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: The paper fully disclose all the information needed reproduce the main experimental results of the paper.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, it may be necessary either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, or other means that appropriate to the research performed.
While NeurlPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome deseribe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
15

---

Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [No]
Justification: For reason, haven't provided the open access code now. But will release our source code once the paper is accepted.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nnis.cc
aublic/guudees/Codeebbbisss catassionoooyyyyy for more details.
While encourage the release of code and data, understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https
/ //nips. wttps/cccppppclgguiuaeaeaeccuuee
wtt.eeeeeeoSSSeeesSSSeoeee.. www.sssoiccgyyy for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: The paper specify all the training and test details necessary understand the results.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented the core the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: The results are accompanied by eттor bars, confidence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by emror bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
16

---

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative еπτог rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification:
The paper provide sufficient information on the computer resources.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported the paper (e.g., preliminary or failed experiments that didn't make it into the paper).
9. Code of ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics https::/neuripss http:://wwurrppsccpppbebeeeeeeee ctcc/uublc//bhhiccuueeeeeeee
Answer: [Yes]
Justification: The research conducted in the paper conforms, in every respect, with the NeurlPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurlPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: The paper discuss both potential positive societal impacts in supplementary materials.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
17

---

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
The paper poses no such risks.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g.. code, data, models), used in the paper, properly credited and the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification:
We explicitly mentioned and properly respected the license and terms of assets.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URI.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
18

---

If assets are released, the license, copyright information, and terms of use in the thcode has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
package should be provided. For popular datasets, paperswi
latasets.paperssit
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information not available online, the authors are encouraged to reach out to the asset's s creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: The paper does not release new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/moode as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country institution) were obtained?
Answer: [NA]
Justification: The paper does not involve crowdsoureing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
19

---

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.
Question:
Answer: [NA]
Justification: The core method development in this research does involve LLMs as any important, original, or non-standard components.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non-standard components.
Please refer to our LLM for what should or should not be described.
20


Reviews:
[{"review_id": "yKQar5Uuj6", "reviewer": "Reviewer_zMRB", "summary": "The paper introduces GauDP, where a 3D gaussian splat representation generated using a (fine-tuned) MLP is used in conjunction with a ViT to generate a unified representation of the scene that is conducive to multi-agent robotics scenarios. They show that their method performs better than previous baselines in a simulated multi-robot environment, and explain some failure modes that other sensing methods such as local image observations and point clouds exhibit.", "questions": "What are the exact setup parameters for your diffusion policy (and baselines)? I would like to see all relevant hyperparameters such as action prediction horizon, execution horizon, scheduler parameters, etc. \nHow much time does your method require to train and infer to recreate the results in the paper, and how does this compare to your baselines? \nThe final version of this paper should have details and background on diffusion policy in the preliminaries. I believe you can shorten the ablations section significantly while conveying the same information to fit this in. \nI would like to see a search over relevant hyperparameters, such as the action (prediction, execution) horizon, and possibly normalize based on training flops if training times are long. This could be very beneficial to your paper: intuitively, I can imagine that GauDP would scale well with longer prediction horizons due to increased precision and coordination. I disagree with the notion that identical hyperparameters allow for a fair comparison (especially since the architectures used seem to be different). In fact, it would be the exact opposite as there is a chance the hyperparameters were tuned to your method (e.g., long action horizon), and these choices might have been incompatible with others. What if using half the action horizon reduced the occurrences of collision without the need for the 3D representation? It is possible that some errors would not happen if given the time to react (perhaps even using a smaller execution horizon might show this). \nHow many parameters is your ViT and splatting MLP (combined)? Additionally, how many parameters are in each baseline? I see in Sec. 4.1 that the 2D baselines were ResNet 18, this is quite a small model. If you were to flop or parameter-normalize each method, how do the results change?\n\nIf these questions are addressed (and the relative performance of GauDP holds), I would be inclined to increase my score as the results seem otherwise strong and intuitive.", "limitations": "See above", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\nMethodology is simple and achieves strong results in the domain they test. Substantially outperforms baselines.\nMethod is explained clearly, and it is intuitive why performance should be better than the baselines.\nAblations show that parts of the pipeline are not redundant.\n\nWeaknesses: \nIn general, the paper is devoid of any important hyperparameters required to reproduce the material. For instance, it is completely missing any details on their diffusion policy setup, such as action prediction horizon, action execution horizon, denoising steps/schedule, etc., none of these details are in the appendix. This should be added to the main paper for clarity.\nThere is no information about runtimes of different methods, training times/flop estimates, inference times, etc. I would like to see information about the throughput of each system. For example, how much slower is GauDP (if it is, this method seems computationally expensive) compared to the point cloud and local context only methods?\nIn section 4.1, they claim that they use identical hyperparameters for “fair comparison”. I disagree with this, and explain why in question 3.\nComparisons are potentially unfair, 2D representations seem to be using a smaller model, though details on exact model sizes or parameters for GauDP are not provided in either the supplementary or main paper.", "was_revised": true, "final_justification": "The authors have added crucial training details (majority of the reason for the original rating), and added experiments regarding model size normalization. All of the results check out, and the downsides are now clear (lower throughput and longer training time). However, the downsides are mitigated by stronger empirical performance.", "rebuttal": "We sincerely thank the reviewer for the detailed and constructive feedback. We appreciate the suggestions regarding reproducibility, fairness of comparison, and computational analysis. **All of our hyperparameters were aligned with the configurations used in prior work**. We will include additional hyperparameter details in the main paper and release all related code and configuration files to ensure full reproducibility. We have carefully addressed each point as follows:\n\n**[A1] Hyperparameter Details for Reproducibility**\n\nWe agree with the reviewer that providing complete hyperparameter settings is essential for reproducibility. We will include these details in the revised version of the main paper. Specifically, our proposed GauDP and all baseline methods adopt identical hyperparameters following prior diffusion policy works and existing benchmarks:\n- Action prediction horizon: 8\n- Number of observation steps: 3\n- Number of action execution steps: 6\n- Denoising schedule: Our experiments follow the optimal configurations of DP and DP3 to ensure fair comparison. Specifically, both GauDP and DP adopt DDPM with 100 denoising steps for both training and inference, while DP3 uses DDIM, also with 100 steps.\n\n**[A2] Training/Inference Time, Parameter Count**\n\nWe have conducted a thorough analysis of training and inference runtimes, as well as model size. \n\nTraining time depends on the trajectory length of each task.  Take the Lift Barrier task as an example, the number of elapsed steps is approximately 100:\n\n- **Training time** (100 epochs on A100 GPU):\n  - DP: 4.8 GPU hours\n  - DP3: 2.5 GPU hours\n  - GauDP: 6.5 GPU hours\n\n  **GauDP has comparable training time to DP**, thanks to two factors:\n  - The Gaussian estimation module is **frozen** during the whole policy training phase;\n  - We implemented extensive optimizations on data I/O and training efficiency for all these methods. (The relevant codes will be released after the review process.)\n- **Inference speed of action chunk** (Simulation evaluation on NVIDIA 5090 GPU):\n  - DP: 1.49 FPS\n  - DP3: 1.57 FPS\n  - GauDP: 1.28 FPS\n- **Parameter counts** (depend on the number of agents):\n\n*Table 1*\n  |         | DP  | **GauDP-policy** | LargeDP | GauDP-full |\n  |:-----:|:-----:|:-----:|:-----:|:-----:|\n  | 2 Agents | 129.949M | 129.959M | 721.286M |750.505M |\n  | 3 Agents | 163.584M |  163.594M |  956.335M | 784.140M |\n  | 4 Agents | 197.130M | 197.140M | 1.191G | 817.686M |\n\nIn the table above, **GauDP-full** refers to the complete model including the Gaussian estimation module, while **GauDP-policy** only includes the policy learning component. To ensure a fair comparison in terms of parameter count, we introduce **LargeDP**, which scales up the original DP model to match the parameter size of *GauDP-full*. The corresponding performance is reported in *Table 2*. It is also important to note that the majority of parameters in *GauDP* are frozen during policy learning, whereas all parameters in *LargeDP* are learnable. This comparison highlights that *GauDP* introduces minimal overhead within the policy module, yet achieves a significant improvement in success rate, demonstrating the effectiveness of our framework in enhancing multi-arm coordination.\n\n**[A3] Fairness of Comparison**\n\nWe appreciate the reviewer’s concern regarding fair comparison. To clarify, our policy architecture is **strictly aligned with DP**: we **use the same observation encoder (Resnet18) and the same 1D conditional diffusion module as DP**. The only additions in GauDP are:\n- A fine-tuned NoPoSplat module with local depth constraints for estimating the Gaussian representation;\n- Two 3-layer CNNs: one for channel adjustment and one for pre-fusion of the Gaussian image.\n\nThis design choice allows us to isolate and evaluate the benefits of our proposed **Gaussian-Image Synergy framework** under controlled conditions. By keeping the core policy module unchanged, we ensure that observed improvements result from the proposed representation rather than from architectural scaling. We will further clarify the model architecture details in the supplementary material to avoid any potential confusion.\n\n**[A4] Model Size Normalization**\n\nWe further scaled up the baseline DP model into **LargeDP** to match or even exceed the parameter count of GauDP, as shown in *Table 1*. GauDP still achieves the highest success rate, as shown in *Table 2*, reinforcing that the performance gain stems from our proposed framework rather than increased model capacity. LargeDP still lacks an effective representation for global perception and fine-grained understanding of 3D structure and spatial relationships. This further supports the effectiveness of our method in facilitating multi-agent cooperation.\n\n*Table 2*\n| Model   | Lift Barrier | Place Food | Stack 2 Cubes | Align Camera | Stack 3 Cubes | Take Photo | Average |\n|:-----:|:-----:|:-----:|:-----:|:-----:|:-----:|:-----:|:-----:|\n| DP      | 9            | 12         | 6              | 3             | 0              | 0           | 5       |\n| DP3     | 30           | 21         | 1              | 3             | 0              | 9           | 10.67   |\n| LargeDP | 60           | 12         | 4              | 29            | 0              | 0           | 17.5    |\n| **GauDP**   | 72           | 15         | 2              | 26            | 0              | 3           | **19.67**   |\n\n**[A5] Revisions to Main Text and Supplementary Materials**\n\nWe will include additional hyperparameter details in the main paper and provide background information on diffusion policy in the preliminaries. Additionally, we will shorten the ablation study section in the main paper and provide more detailed descriptions of the model architecture in the supplementary materials accordingly in the revised version."}, {"review_id": "OU2z4NAq8v", "reviewer": "Reviewer_px8R", "summary": "The paper studies the problem of multi-robot collaboration for manipulation tasks in the robotFactory benchmark. The paper introduces GauDP, a framework for training multi-agent coordination policies that integrate local and global observations using 3D Gaussian fields as a global representation for multi-agent embodied collaboration. The proposed method first reconstructs global 3D gaussian fields using each agents local view, next each agent dynamically queries shared gaussian representation to include additional information aggregated from multiple views. This is achieved by projecting the global 3D gaussian field back to agents original coordinate space. Next, the authors demonstrate the effectiveness and scalability of GauDP on the RoboFactory benchmark, achieving strong performance with only RGB input compared to other methods that use global point cloud representation and 2D image based policy learning.", "questions": "None", "limitations": "The approach proposed in the paper is quite simple and effective what is lacking in the current paper is breadth of tasks in the experiments. This is understandable as the benchmarks for this setting do not have a lot of diverse tasks. I would recommend authors to add results for missing tasks from RoboFactory benchmark to make the current paper stronger given the benchmarks that exists already for completeness.", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n\n1. The paper is well-written and easy to follow\n2. The approach proposed in the paper is novel, intuitive, simple and effective.\n3. The proposed method outperforms other baselines that use only 2D images or builds global point cloud representations on majority of tasks from RoboFactory. The tasks where it underperforms 2D image based methods seems like those could be solved using a single agent more effectively and hence no improvement from effective collaboration.\n4. The ablations in the paper are thorough and clearly show effectiveness of the global aggregation technique used by the authors. These experiments are quite insightful  \n\nWeaknesses:\n\n1. It looks like the paper only includes results for a subset of tasks from RoboFactory. I believe the paper has a nice method that is effective so authors should include results on missing tasks in the main paper (ex: pass shoe in 2 agent setup, long pipeline delivery in 4 agent).\n2. It is unclear why the task stack cube has such low or 0 success rates. Additional analysis highlighting failure modes would help build insights about what is missing in the current methods. It seems like these tasks should be solvable by a single agent itself so it seems counterintuitive that it is not working in multi-agent setting.", "was_revised": false, "final_justification": "", "rebuttal": "**[W1] missing tasks of RoboFactory in the main paper**\n\n**[A1]** Thank you for your kind recognition and valuable suggestion. We agree that including more diverse tasks is important for comprehensive evaluation. However, the tasks \"pass shoe (2-agent)\" and \"long pipeline delivery (4-agent)\" are relatively weakly cooperative in nature. These tasks can be decomposed into sequential pick-and-place subtasks that each agent can complete independently with minimal coordination. Since our framework is designed to emphasize and promote **tight and synergistic collaboration** among agents, we focused on tasks that require strong inter-agent dependency and coordinated decision-making. \n\n**[W2] \"Stack cubes\" tasks have a low or 0 success rate**\n\n**[A2]** We appreciate your insightful comment. Indeed, the “stack cubes” tasks are challenging despite appearing simple at first glance. There are two major contributing factors to their low success rates:\n- **Accumulated manipulation difficulty:** Stacking cubes requires precise pick-and-place actions for each individual cube. Even a small deviation in position can cause failure in stacking. Given that several such precise actions must be executed in sequence, the overall success rate decreases approximately exponentially with the number of cubes.\n- **High task complexity due to spatial and sequential constraints:** Each cube has a unique color and must be stacked in a specific order. Agents must select the appropriate cube based on color and determine which agent is closest and most suitable to pick it up. Since cube positions are randomly initialized, the spatial layout and action sequence vary significantly across demonstrations. This introduces substantial variability and demands that the policy generalize across many permutations—much more than in other tasks.\n\nTo further investigate, we fine-tuned Pi0 (a strong VLA-based method) on the “stack cube three” task. The success rate remained extremely low (~1%), suggesting that the difficulty arises not from our specific method, but from the fundamental complexity of the task.\n\nIn summary, the “stack cubes” tasks uniquely combine the need for high-precision manipulation, spatial reasoning, and decision-making under dynamic layouts. These factors make them the most difficult among our experiments, and we hope our analysis helps clarify the observed performance gap."}, {"review_id": "aZthHvOOWP", "reviewer": "Reviewer_o7bR", "summary": "This paper presents a novel way of encoding multi-agent spatial information with 3D Gaussian representations that fuses both local observation and global context. It achieves superior results compared to other image-based representations on selective tasks, while comparable to point cloud 3D representations. The method is especially effective in tasks that require visual alignment rather than fixed coordinate alignments.", "questions": "1. How is your method's per-step latency compared to other 2D/3D based methods? Though GauDP is performed with 2D inputs, it is possible that 3D Gaussian reconstruction in each environment step will be significantly more expensive than even 3D point cloud based methods.\n2. Is your method robust against real world disturbance in 2D observations like different lighting conditions / errors in execution?\n3. What is the core difference in representation that results in the significant success rate increase of Lift Barrier, even comparing to 3D diffusion policy?\n\nI will consider increasing my score if the author provide additional details addressing my questions above.", "limitations": "yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 4, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n1. It is a novel approach to use multi-view 2D images as 3D prior in embodied cooperation.\n2. The method performs well in Lift-barrier and Align-camera tasks, both considered difficult for image-based diffusion policies.\n\nWeaknesses:\n1. The use of 3D reconstruction potentially makes latency high in completing tasks.\n2. Experiments done only in simulated environments where 3D reconstruction works really well by fine-tuning.", "was_revised": true, "final_justification": "The authors addressed my concerns properly. Admittedly, the method achieves higher success rates at the cost of a 2D Gaussian overhead. This is acceptable but not enough to qualify for a score of 5 in my opinion. Therefore, my score stays 4.", "rebuttal": "We sincerely thank the reviewer for the thoughtful and insightful questions.\n\n**[Q1] Per-step latency and computational expense**\n\n**[A1]** We evaluated the inference latency per action chunk during simulation on an NVIDIA RTX 5090 GPU. The results are shown in the table below:\n\n|     | DP  | DP3 | GauDP |\n|-----|-----|-----|-------|\n| FPS | 1.49 | 1.57 | 1.28   |\n\nThe inference latency of GauDP can be decomposed into two parts:\n- **Gaussian Estimation:** This component relies on a Transformer to compute self-attention and cross-attention. Thanks to extensive prior work on CUDA optimizations for Transformer computations, this part achieves high inference efficiency.\n- **Action Generation:** This part is identical to that of DP, except for a minor additional CNN module. Consequently, its latency is the same as DP, with the main latency arising from the iterative denoising process.\n\nTherefore, our GauDP method introduces only a marginal increase in inference latency compared to DP and DP3.\n\nWhile DP3 achieves the highest FPS in simulation, it's important to note that the point clouds used during DP3 simulation have been preprocessed to remove background and table clutter. This is feasible in simulation due to access to privileged semantics and optimized rendering. However, in real-world scenarios, acquiring high-quality point clouds and performing such preprocessing (e.g., filtering irrelevant points and applying farthest point sampling) is both time-consuming and computationally expensive. Therefore, simulation efficiency of point cloud-based policies may not translate directly to real-world deployments. In contrast, GauDP leverages 2D observations and reconstructs 3D Gaussians efficiently.\n\n**[Q2] Robustness to real-world visual disturbances and execution errors**\n\n**[A2]** Our Gaussian estimation module has been fine-tuned on large-scale datasets that already encompass diverse conditions, such as varying viewpoints and illumination. Additionally, we enhance the accuracy and robustness of Gaussian estimation by incorporating local depth constraints from multiple viewpoints. This design endows our method with inherent robustness against visual disturbances.\n\nTo further evaluate robustness, we conducted experiments under novel settings that introduce challenging lighting variations—including random changes in light color, direction, and intensity—as well as random distractor objects (10 task-irrelevant objects randomly placed beside the manipulated objects on the table), following RoboTwin 2.0. In these settings, our method still achieved the highest task success rate:\n\n|             | DP  | DP3 | GauDP |\n|-------------|-----|-----|-------|\n| Grab Roller | 20% | 46% | 50%   |\n\n  **Robustness to execution errors**\n  - **Gaussian Estimation:** Regarding robustness to execution errors, our method reconstructs the 3D scene from two or more 2D observations. Since execution noise and actuation errors have limited impact on the 2D observations, they do not significantly affect the quality of the estimated Gaussians.\n  - **Action Generation:** The core contribution of our work is the proposal of a novel framework that leverages 3D Gaussian representations to coordinate 2D image inputs. Our policy learning technique remains consistent with that of DP and DP3, so the robustness of action generation to execution errors shares the same inherent limitations as diffusion policies. However, an interesting observation from our experiments is that policies based on GauDP tend to attempt re-execution of the current subtask more readily when encountering mid-task failures.\n\n**[Q3] Significant performance improvement on Lift Barrier task**\n\n**[A3]** Thank you for this insightful question. We believe that Lift Barrier and Grab Roller are particularly suitable for evaluating multi-agent coordination, as success requires tightly synchronized manipulation from both arms. Any premature or delayed movement by one agent typically results in failure.\n\nThe Barrier object is made of thin white metal wires and appears as subtle lines in 2D observations, making it difficult to perceive using standard 2D encoders. In contrast, it is much more distinguishable in 3D point cloud representations, especially processed by FPS (Farthest Point Sampling), which explains the higher success rate of DP3 compared to DP.\n\nOur method bridges this gap by reconstructing 3D Gaussians from multiple 2D views in a self-supervised manner. This allows us to recover the spatial structure of challenging objects like the Barrier without requiring explicit 3D supervision or point cloud input. The improved success rate of GauDP on Lift Barrier demonstrates the effectiveness of our learned 3D representations for precise and coordinated manipulation."}, {"review_id": "Ba8HSS8Doz", "reviewer": "Reviewer_CMJx", "summary": "This paper tackles the multi-agent collaboration challenge for robotic manipulation tasks. It proposes a framework, named GauDP, that integrates local and global observations from different agents via 3D Gaussian fields so that each agent can reason over its own perception results enhanced by the shared 3D context. GauDP is shown to outperform existing work (other diffusion policy based frameworks) that relies only on 2d or 3d observations (but not both) on the RoboFactory benchmark.", "questions": "See \"weaknesses\"", "limitations": "yes", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n\n+ The work is well-motivated as the shared global and local observation helps collaboration between agents for robotic tasks.\n+ The algorithmic design is clean and reasonable with a good amount of ablation studies (and experimental comparisons) in a relatively fair setup.\n+ The results seem convincing to me.\n\nWeaknesses:\n\n+ The proposed approach relies on noposplat. What if the generated 3DGS results are not ideal? How much does it matter? In case it is relatively robust, does it mean there might be better \"latent\" representation of the shared 3D scene representation (say, affordance-aware and not an explicit repr) that benefits downstream robotic controls.", "was_revised": true, "final_justification": "I maintain my original rating.", "rebuttal": "**[Q1] What if the generated 3DGS results are not ideal?**\n\n**[A1]** Our 3D Gaussian estimation module has been pretrained and finetuned on large-scale datasets that encompass a wide range of variations, including diverse viewpoints, lighting conditions, and occlusions. To further assess its robustness, we conducted additional experiments that introduce challenging lighting variations—including random changes in light color, direction, and intensity—as well as random distractor objects (10 task-irrelevant objects randomly placed beside the manipulated objects on the table), following RoboTwin 2.0.. Despite these perturbations, our method maintained SOTA performance, achieving the highest task success rate:\n\n|             | DP  | DP3 | GauDP |\n|-------------|-----|-----|-------|\n| Grab Roller | 20% | 46% | 50%   |\n\nThis demonstrates that our Gaussian estimation is robust and reliable even in complex environments.\n\n**[Q2] How much do the 3DGS results matter?**\n\n**[A2]** The 3D Gaussian representations serve two critical roles in our framework. First, they act as a shared global scene representation that facilitates coordination among multiple manipulators in collaborative settings. Second, they serve as a self-supervised structural prior that helps each local agent infer fine-grained spatial cues from 2D observations. In both respects, the quality of the 3DGS representation significantly contributes to the effectiveness of our manipulation pipeline.\n\n**[Q3] Does it mean there might be a better \"latent\" representation of the shared 3D scene representation?**\n\n**[A3]** Yes! We appreciate the reviewer for raising this insightful point. This line of thought aligns well with our ongoing research. The current work represents our initial attempt to incorporate 3D Gaussian representations into manipulation tasks, where we explicitly use Gaussians to represent low-level geometry and appearance. However, we believe there is considerable potential in developing a more expressive and compact latent representation.\n\nIn particular, we are currently exploring a Gaussian tokenization framework that integrates high-level semantic cues and latent action information into the 3D representation. This would enable the representation to not only reflect the spatial structure of the scene but also better support reasoning and action generation. We envision such a latent representation as a more efficient interface for downstream tasks, including integration with VLMs or VLAs."}]

Author rebuttal:

Meta-review: This paper presents GauDP, a framework that uses 3D Gaussian field representations reconstructed from multiple agents' RGB observations to enable effective multi-agent robotic collaboration. The technical approach bridges 2D image-based and 3D point cloud methods by allowing agents to query a shared 3D scene representation for enhanced spatial understanding during manipulation tasks. Initial reviewer concerns regarding reproducibility, computational efficiency, and experimental fairness were comprehensively addressed through detailed rebuttals that provided complete hyperparameters, thorough computational analysis showing acceptable overhead, and fair model size comparisons. The experimental validation on RoboFactory demonstrates clear performance improvements over baselines. The work makes a solid technical contribution to multi-agent robotics with practical value, and all reviewers ultimately recommended acceptance after their concerns were adequately resolved.
Decision: Accept (poster)