Title: GauDP: Reinventing Multi-Agent Collaboration through Gaussian-Image Synergy in Diffusion Policies
Venue: NeurIPS 2025
Accepted: yes
Keywords: 3D Gaussian Reconstruction

Diffusion Policy

3D Scene Representation

Robot Action Generation
Abstract: Despite significant advances in robotic policy generation, effective coordination in embodied multi-agent systems remains a fundamental challenge—particularly in scenarios where agents must balance individual perspectives with global environmental awareness.
Existing approaches often struggle to balance fine-grained local control with comprehensive scene understanding, resulting in limited scalability and compromised collaboration quality.
In this paper, we present GauDP, a novel Gaussian-image synergistic representation that facilitates scalable, perception-aware imitation learning in multi-agent collaborative systems.
Specifically, GauDP reconstructs a globally consistent 3D Gaussian field from local-view RGB images, allowing all agents to dynamically query task-relevant features from a shared scene representation. 
This design facilitates both fine-grained control and globally coherent behavior without requiring additional sensing modalities. 
We evaluate GauDP on the RoboFactory benchmark, which includes diverse multi-arm manipulation tasks. 
Our method achieves superior performance over existing image-based methods and approaches the effectiveness of point-cloud-driven methods, while maintaining strong scalability as the number of agents increases. 
Extensive ablations and visualizations further demonstrate the robustness and efficiency of our unified local-global perception framework for multi-agent embodied learning.


Peer reviews:
comments: SUMMARY:
This paper tackles the multi-agent collaboration challenge for robotic manipulation tasks. It proposes a framework, named GauDP, that integrates local and global observations from different agents via 3D Gaussian fields so that each agent can reason over its own perception results enhanced by the shared 3D context. GauDP is shown to outperform existing work (other diffusion policy based frameworks) that relies only on 2d or 3d observations (but not both) on the RoboFactory benchmark.

STRENGTHS AND WEAKNESSES:
Strengths:

+ The work is well-motivated as the shared global and local observation helps collaboration between agents for robotic tasks.
+ The algorithmic design is clean and reasonable with a good amount of ablation studies (and experimental comparisons) in a relatively fair setup.
+ The results seem convincing to me.

Weaknesses:

+ The proposed approach relies on noposplat. What if the generated 3DGS results are not ideal? How much does it matter? In case it is relatively robust, does it mean there might be better "latent" representation of the shared 3D scene representation (say, affordance-aware and not an explicit repr) that benefits downstream robotic controls.

QUESTIONS:
See "weaknesses"

LIMITATIONS:
yes
TITLE: Official Review
CLARITY: 3
IMPACT: 3
ORIGINALITY: 3
DATE: 2025-06-27

comments: SUMMARY:
This paper presents a novel way of encoding multi-agent spatial information with 3D Gaussian representations that fuses both local observation and global context. It achieves superior results compared to other image-based representations on selective tasks, while comparable to point cloud 3D representations. The method is especially effective in tasks that require visual alignment rather than fixed coordinate alignments.

STRENGTHS AND WEAKNESSES:
Strengths:
1. It is a novel approach to use multi-view 2D images as 3D prior in embodied cooperation.
2. The method performs well in Lift-barrier and Align-camera tasks, both considered difficult for image-based diffusion policies.

Weaknesses:
1. The use of 3D reconstruction potentially makes latency high in completing tasks.
2. Experiments done only in simulated environments where 3D reconstruction works really well by fine-tuning.

QUESTIONS:
1. How is your method's per-step latency compared to other 2D/3D based methods? Though GauDP is performed with 2D inputs, it is possible that 3D Gaussian reconstruction in each environment step will be significantly more expensive than even 3D point cloud based methods.
2. Is your method robust against real world disturbance in 2D observations like different lighting conditions / errors in execution?
3. What is the core difference in representation that results in the significant success rate increase of Lift Barrier, even comparing to 3D diffusion policy?

I will consider increasing my score if the author provide additional details addressing my questions above.

LIMITATIONS:
yes
TITLE: Official Review
CLARITY: 4
IMPACT: 3
ORIGINALITY: 3
DATE: 2025-06-30

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper studies the problem of multi-robot collaboration for manipulation tasks in the robotFactory benchmark. The paper introduces GauDP, a framework for training multi-agent coordination policies that integrate local and global observations using 3D Gaussian fields as a global representation for multi-agent embodied collaboration. The proposed method first reconstructs global 3D gaussian fields using each agents local view, next each agent dynamically queries shared gaussian representation to include additional information aggregated from multiple views. This is achieved by projecting the global 3D gaussian field back to agents original coordinate space. Next, the authors demonstrate the effectiveness and scalability of GauDP on the RoboFactory benchmark, achieving strong performance with only RGB input compared to other methods that use global point cloud representation and 2D image based policy learning.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n\n1. The paper is well-written and easy to follow\n2. The approach proposed in the paper is novel, intuitive, simple and effective.\n3. The proposed method outperforms other baselines that use only 2D images or builds global point cloud representations on majority of tasks from RoboFactory. The tasks where it underperforms 2D image based methods seems like those could be solved using a single agent more effectively and hence no improvement from effective collaboration.\n4. The ablations in the paper are thorough and clearly show effectiveness of the global aggregation technique used by the authors. These experiments are quite insightful  \n\nWeaknesses:\n\n1. It looks like the paper only includes results for a subset of tasks from RoboFactory. I believe the paper has a nice method that is effective so authors should include results on missing tasks in the main paper (ex: pass shoe in 2 agent setup, long pipeline delivery in 4 agent).\n2. It is unclear why the task stack cube has such low or 0 success rates. Additional analysis highlighting failure modes would help build insights about what is missing in the current methods. It seems like these tasks should be solvable by a single agent itself so it seems counterintuitive that it is not working in multi-agent setting.\n\nQUESTIONS:\nNone\n\nLIMITATIONS:\nThe approach proposed in the paper is quite simple and effective what is lacking in the current paper is breadth of tasks in the experiments. This is understandable as the benchmarks for this setting do not have a lot of diverse tasks. I would recommend authors to add results for missing tasks from RoboFactory benchmark to make the current paper stronger given the benchmarks that exists already for completeness.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper introduces GauDP, where a 3D gaussian splat representation generated using a (fine-tuned) MLP is used in conjunction with a ViT to generate a unified representation of the scene that is conducive to multi-agent robotics scenarios. They show that their method performs better than previous baselines in a simulated multi-robot environment, and explain some failure modes that other sensing methods such as local image observations and point clouds exhibit.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\nMethodology is simple and achieves strong results in the domain they test. Substantially outperforms baselines.\nMethod is explained clearly, and it is intuitive why performance should be better than the baselines.\nAblations show that parts of the pipeline are not redundant.\n\nWeaknesses: \nIn general, the paper is devoid of any important hyperparameters required to reproduce the material. For instance, it is completely missing any details on their diffusion policy setup, such as action prediction horizon, action execution horizon, denoising steps/schedule, etc., none of these details are in the appendix. This should be added to the main paper for clarity.\nThere is no information about runtimes of different methods, training times/flop estimates, inference times, etc. I would like to see information about the throughput of each system. For example, how much slower is GauDP (if it is, this method seems computationally expensive) compared to the point cloud and local context only methods?\nIn section 4.1, they claim that they use identical hyperparameters for “fair comparison”. I disagree with this, and explain why in question 3.\nComparisons are potentially unfair, 2D representations seem to be using a smaller model, though details on exact model sizes or parameters for GauDP are not provided in either the supplementary or main paper.\n\nQUESTIONS:\nWhat are the exact setup parameters for your diffusion policy (and baselines)? I would like to see all relevant hyperparameters such as action prediction horizon, execution horizon, scheduler parameters, etc. \nHow much time does your method require to train and infer to recreate the results in the paper, and how does this compare to your baselines? \nThe final version of this paper should have details and background on diffusion policy in the preliminaries. I believe you can shorten the ablations section significantly while conveying the same information to fit this in. \nI would like to see a search over relevant hyperparameters, such as the action (prediction, execution) horizon, and possibly normalize based on training flops if training times are long. This could be very beneficial to your paper: intuitively, I can imagine that GauDP would scale well with longer prediction horizons due to increased precision and coordination. I disagree with the notion that identical hyperparameters allow for a fair comparison (especially since the architectures used seem to be different). In fact, it would be the exact opposite as there is a chance the hyperparameters were tuned to your method (e.g., long action horizon), and these choices might have been incompatible with others. What if using half the action horizon reduced the occurrences of collision without the need for the 3D representation? It is possible that some errors would not happen if given the time to react (perhaps even using a smaller execution horizon might show this). \nHow many parameters is your ViT and splatting MLP (combined)? Additionally, how many parameters are in each baseline? I see in Sec. 4.1 that the 2D baselines were ResNet 18, this is quite a small model. If you were to flop or parameter-normalize each method, how do the results change?\n\nIf these questions are addressed (and the relative performance of GauDP holds), I would be inclined to increase my score as the results seem otherwise strong and intuitive.\n\nLIMITATIONS:\nSee above", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-08"}