Title: RayFusion: Ray Fusion Enhanced Collaborative Visual Perception
Venue: NeurIPS 2025
Accepted: yes
Keywords: Collaborative visual perception; 3D object detection; ray occupancy information.
Abstract: Collaborative visual perception methods have gained widespread attention in the autonomous driving community in recent years due to their ability to address sensor limitation problems. However, the absence of explicit depth information often makes it difficult for camera-based perception systems, e.g., 3D object detection, to generate accurate predictions. To alleviate the ambiguity in depth estimation, we propose RayFusion, a ray-based fusion method for collaborative visual  perception. Using ray occupancy information from collaborators, RayFusion reduces redundancy and false positive predictions along camera rays, enhancing the detection performance of purely camera-based collaborative perception systems. Comprehensive experiments show that our method consistently outperforms existing state-of-the-art models, substantially advancing the performance of collaborative visual perception. Our code will be made publicly available.


Peer reviews:
comments: SUMMARY:
This paper proposes RayFusion, a method that uses ray occupancy to improve 3D object localization in multi-agent perception. It includes spatial-temporal alignment, depth-aware ray encoding, and multi-scale feature aggregation to enhance collaboration and accuracy.

STRENGTHS AND WEAKNESSES:
**Strengths**
- Good presentation and clear methodology;
- Significant performance gain over baselines;
- The experiments are thorough;
- The code is promised to be released, which will benefit the community.

**Weaknesses**  
This is a solid piece of work. To be honest, I found it difficult to identify any major weaknesses, and the following are just some minor suggestions.
- It would be helpful to see an ablation where the predicted depth distribution is replaced with a one-hot vector (e.g., peak bin only) or even a uniform distribution.
- The detection performance is affected by the communication bandwidth between agents. It would be helpful if Table 1 could report the communication cost of all methods.
- There are some minor typos (e.g., line 584: "Additionally, We plan").

QUESTIONS:
Please refer to the **weaknesses** above.

LIMITATIONS:
yes
TITLE: Official Review
CLARITY: 4
IMPACT: 4
ORIGINALITY: 3
DATE: 2025-06-06

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes RayFusion, a novel cooperative perception framework that leverages ray information from multiple agents to resolve depth ambiguity in camera-only 3D detection. There are several significant concerns about the experimental design that weaken the paper's conclusions. These primarily relate to the fairness of baseline comparisons, the choice of datasets, the reliance on LiDAR supervision, and the limited perception range used in the experiments.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n1. The paper provides comprehensive robustness evaluations under delay and noise conditions, as well as communication and computation efficiency analysis.\n2. The supplementary video provides a compelling demonstration of the model's performance.\n\nWeaknesses:\n1. Table 1 compares methods using different detection backbones: Sparse4D for the proposed method vs. BEVFormer for baselines cited from IFTR. This undermines the fairness of the comparison, as performance gains may stem from the stronger backbone rather than the proposed fusion strategy. A controlled comparison with a shared backbone is necessary for a new cooperative perception model.\n2. The authors acknowledge the limitations of the DAIR-V2X dataset. Why the paper does not benchmark its method against more recent and comprehensive real-world V2X datasets, e.g., TUMTrafV2X (V2I) and V2X-Real or V2XPnP (V2V, V2I, I2I). Relying heavily on simulated data and a limited real-world dataset makes it difficult to assess the method's practical effectiveness and its ability to bridge the gap between simulation and real-world data.\n3. Although presented as “camera-only,” the method uses LiDAR supervision during training for depth estimation. This contradicts the claim. An ablation study removing LiDAR supervision is needed to assess true independence.\n4. The perception range is approximately 50m in each direction, which is more characteristic of single-agent perception. To provide a more compelling demonstration of the benefits of the proposed cooperative approach, experiments should be conducted over a much larger range.\n\nQUESTIONS:\nSee the weaknesses.\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 2, "REVIEWER_CONFIDENCE": 5, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-06-09"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes RayFusion, a novel method for collaborative camera-only perception. The key idea behind RayFusion is to reduce ambiguity through depth estimation by modeling occupancy along the rays of different agent cameras. To do so, object predictions and depth estimates are first made from the individual camera images which are then transformed to the ego coordinate system. Next, the camera rays are encoded using an MLP before being passed through a self-attention layer to generate a single ray encoding for each object instance. Lastly, the instance features undergo pyramid window self-attention, before being passed to the detection head. The authors evaluate their method on three collaborative perception datasets and demonstrate consistent performance improvement over the state-of-the-art.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n- The paper is well-written with clear figures.\n- The method is clever and well-motivated (leveraging multiple camera views to reduce depth ambiguity is natural), and the explanation is easy to follow\n- The experiments are thorough and show good performance across multiple datasets.\n- RayFusion demonstrates superior performance in settings with localization errors and communication delays over competing collaborative perception methods.\nWeaknesses:\n- The depth prediction head is supervised using LiDAR point clouds, meaning extra sensor data is required during training time.\n- According to ablation studies, the performance gain from solely the ray occupancy encoding (the key contribution of this method) is only moderate, suggesting that more can be done to reduce depth ambiguity.\n\nQUESTIONS:\n1. Given the reliance of this method on transformation of the instance features to a unified camera coordinate system, have the authors examined the effect of errors in the camera projection matrices?\n2. How exactly is the LiDAR point cloud used to supervise the dense depth prediction? This particular detail is a little under-explained in the paper.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-30"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes RayFusion, a collaborative perception framework for camera-only autonomous driving systems. It introduces a ray-based fusion strategy leveraging ray occupancy information across agents to resolve depth ambiguity in monocular 3D detection. The system includes spatial-temporal alignment, ray occupancy encoding, and multi-scale instance feature aggregation. Experiments on real and simulated datasets show improved accuracy and robustness under localization noise and communication delays.\n\nSTRENGTHS AND WEAKNESSES:\n# Strengths\n\n1. Introducing ray occupancy encoding into collaborative perception is original and conceptually insightful. It addresses the well-known ambiguity in depth estimation from monocular views.\n2. The modular system design (STA, ROE, MIFA) is clean and well-motivated.\n3. Experiments demonstrate solid performance gains over prior methods in simulated environments.\n\n# Weakness\n\n1. While the proposed RayFusion framework shows promising performance under general settings, it does not explicitly address the challenges posed by crowded scenes, where overlapping instances and dense object distributions may significantly increase false positives and degrade the effectiveness of ray-based cross-view validation. Existing experiments do not isolate this regime for targeted analysis, and no specific design is introduced to mitigate perceptual conflicts in such cases. As a result, it remains unclear whether the proposed ray occupancy encoding can robustly distinguish between true and spurious detections under crowded traffic conditions.\n\n2. The proposed RayFusion framework relies on structured information (pose, velocity, detection outputs) from collaborating agents, assuming they are sufficiently accurate after spatial-temporal alignment (STA). However, in realistic scenarios, errors from other agents — such as inaccurate localization, erroneous depth estimation, or false positives in their predictions — can easily propagate through the collaboration pipeline and directly degrade the ego vehicle's performance.\n\nWhile the authors mention future work on \"unknown-pose settings\", the current method lacks mechanisms to assess or suppress unreliable information from noisy collaborators. This is especially concerning in safety-critical scenarios, where faulty inputs from a single vehicle could severely mislead perception for the ego vehicle. More discussion or empirical evaluation in such adversarial or noisy-agent settings would strengthen the paper.\n\nQUESTIONS:\nHow sensitive is RayFusion to inaccurate or noisy inputs from collaborating vehicles? Have the authors considered simulating such failure modes (e.g., corrupted pose, false positives) to analyze the impact on ego vehicle perception accuracy?\n\nLIMITATIONS:\nYES", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-02"}