RayFusion: Ray Fusion Enhanced Collaborative Visual Perception
Authors: Shaohong Wang
Bin Lu
Xinyu Xiao
Hanzhi Zhong
Bowen Pang
Tong Wang
Zhiyu Xiang
Hangguan Shan
Eryun Liu
Venue: NeurIPS 2025 poster ()
Keywords: Collaborative visual perception; 3D object detection; ray occupancy information.
Primary area: deep_learning
TL;DR: RayFusion is a superior camera-only collaborative perception framework based on ray occupancy information fusion.

Abstract: Collaborative visual perception methods have gained widespread attention in the autonomous driving community in recent years due to their ability to address sensor limitation problems. However, the absence of explicit depth information often makes it difficult for camera-based perception systems, e.g., 3D object detection, to generate accurate predictions. To alleviate the ambiguity in depth estimation, we propose RayFusion, a ray-based fusion method for collaborative visual  perception. Using ray occupancy information from collaborators, RayFusion reduces redundancy and false positive predictions along camera rays, enhancing the detection performance of purely camera-based collaborative perception systems. Comprehensive experiments show that our method consistently outperforms existing state-of-the-art models, substantially advancing the performance of collaborative visual perception. Our code will be made publicly available.

Ray Fusion: Ray Fusion Enhanced Collaborative Visual Perception
Shaohong Wang', Bin Lu', Xinyu Xiao², Hanzhi Zhong', Bowen Pang', Tong Wang', Zhiyu Xiang', Hangguan Shan', Eryun Liu'
'Zhejiang University IInstitute of Automation of Chinese Academy of Sciences hetp:/1112zzuedu.ccco eryunliudzju.eda
Abstract
Collaborative visual perception methods have gained widespread attention in the autonomous driving community in recent years due their ability address sensor limitation problems. However, the absence of explicit depth information often makes it difficult for camera-based perception systems, c.g., 3D object detection, to generate accurate predictions. To alleviate the ambiguity in depth estimation, we propose RayFusion, a ray-based fusion method for collaborative visual perception. Using ray occupancy information from collaborators, RayFusion reduces redun- daney and false positive predictions along camera rays, enhancing the detection performance of purely camera- based collaborative perception systems. Compre- hensive experiments show that our method consistently outperforms existing state- of-the-art models, substantially advancing the performance of collaborative visual perception. The code is available https://github.. htt.ggghhhh..cowanngggeee wwwpw/agwwwwwww1./omo www.aaaggn///g//aaaaaaaaa 1/RayFusion,
Introduction
V2X-based collaborative perception, enabled by information sharing among agents, has been proven to be beneficial for enhancing percep- tion performance in autonomous driving sys- tems. It effectively alleviates occlusions, miti- gates depth estimation ambiguity in individual agent systems, extends perception range, and improves the performance of camera-based 3D
Redundant Predictions along the Camera Ray
Collaborator
Collaborator's
Image
Ego's Image
object detection [35] 23 6 33 22 29]. Ex-
Ray occupancy Information
isting mainstream camera- based collaborative
perception methods 33 22
29
typically con-
struct unified and dense bird s-eye view (BEV) representation through collaborative communi- cation (e.g. sharing BEV features or image fea- tures from individual agents) and then perform object detection in BEV space. Some query- based approaches | 3 achieve collaboration by conducting attention-batt based interactions among multi-agent query features. However, existing camera-based based collaborative perception methods often do not explicitly model the camera imaging pro- cess of the same object from different agent viewpoints. This limitation hinders accurate cross-view
Figure 1: Due to the ambiguity in depth estimation, individual agent typically predicts multiple targets along the camera ray. However, when an instance observed by multiple agents, they record the target's 's occupancy information along the ray and cross-validate its true 3D position.
*Coresponding author.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

validation of an object's true 3D location, reduces the network's 's ability to distinguish hard negative samples along the camera rays, and ultimately leads suboptimal perception performance.
In this work, we propose RayFusion, a ray-based fusion method for collaborative visual perception. As illustrated in Figure [ individual agent typically tends to predict multiple targets along the camera rays, However, when an instance is observed by multiple agents, the ray occupancy information from different perspectives enables precise localization of the instance in 3D space. Here, ray occupancy information refers to whether certain points along camera rays in 3D space are occupied by an object, representing the occupancy state along the ray's is trajectory. Based on this insight, our key idea is to leverage the ray occupancy information of instances to reduce redundancy and false positive predictions along camera rays, thereby enhancing the network's 's ability to distinguish hard negative samples and accurately localize true positive samples. Specifically, RayFusion consists of three key designs: i) Spatial-temporal alignment, which achieves spatial alignment of agent information using a unified coordinate system and improves robustness to communication delays by modeling motion. ii) Ray oceupancy information encoding, obtaining an explicit representation of ray occupancy for each instance by incorporating depth information into camera rays. This explicitly accounts for the 3D structure of the scene and the potential real 3D positions of instances, which helps improve the network's is depth estimation capability and consequently enhances its ability to accurately localize instances. iii) Multi-scale instance feature aggregation, which enables effective multi-agent interaction by capturing both local and global spatial features in parallel, enhancing robustness to localization noise and ultimately achieving superior collaborative perception performance.
To evaluate RayFusion, we conduct extensive experiments on real-world dataset, DAIR-V2X 38 and two simulated datasets, V2XSet (35) and OPV2V [36). The experimental results demonstrate that RayFusion significantly outperforms previous works in terms of both performance and robustness across multiple datasets. In summary, our contributions are as follows:
1. We introduce RayFusion, a superior camera-only collaborative perception framework based on ray occupancy information fusion. This framework enhances the network's s ability distinguish hard negative samples and accurately localize true positive samples, thereby improving the detection performance of collaborative visual perception systems.
2. We design spatial-temporal alignment module that uses a unified coordinate system to achieve spatial alignment of agent information while modeling motion to enhance the system's 's robustness to communication delays.
3. We propose a ray occupancy information encoding module that explicitly accounts for the 3D structure of the scene and the potential precise 3D positions of instances. This reduces redundancy and false positive predictions while simultaneously enhancing the network's 's ability to localize instances along camera rays.
4. We develope a multi-scale instance feature aggregation module, enabling effective interaction with multi-agen instance features to achieve superior collaborative perception performance. Our method, RayFusion, achieves state-of- oftthe-art performance on real-world dataset, DAIR-V2X. and two simulation datasets, V2XSet and OPV2V, with AP70 improvements of 3.64, 3.47, and 8.21 over the previous ttateof-thh-ar art methods, respectively.
2 Related work
2.1 Collaborative perception
Collahorative perception holds significant potential for enhancing the safety of autonomous driving, with intermediate fusion strategies gaining widespread attention due to their potential to achieve optimal performance under limited and controlled communication resources. V2VNet [31] leverages spatially aware graph neural network to aggregate shared feature representations among multiple agents. V2X-VIT VIT [35] introduces a heterogeneous multi-agent attention module to enable adaptive information fusion between heterogeneous agents. Where2comm [5] transmits essential perceptual information via a sparse spatial confidence map to reduce communication bandwidth consumption. TransIFF I further reduces communication bandwidth consumption by transmitting object queries. CoAlign 23 enhances robustness in collaborative perception by correcting pose errors. CoCa3D 66] leverages multi-view information from collaborators to alleviate the depth estimation ambiguity of single agent. However, its potential for further enhancement is limited as it does not explicitly

---

consider the 3D structure of the scene. IFTR [29] interacts with multi-view images from multiple agents using a predefined BEV grid, facilitating the advancement of budget-constrained collaborative systems. HM-ViT 33 and HEAL I. [22] explore multi-agent heterogeneous modality collaborative perception, further expanding the scale of collaboration.
However, existing collaborative visual perception methods often do not fully exploit the 3D structural information of the scene and the multi-view imaging process of the same object from different agent perspectives, As a result, they struggle to resolve depth estimation ambiguities, making it difficult for the network to distinguish true positive and false positive samples along camera rays, which ultimately leads to suboptimal perception performance. In this paper, we propose RayFusion, novel framework that explicitly models the 3D scene structure and the camera imaging process from multiple agent viewpoints. By performing cross-vier view validation, RayFusion effectively distinguishes true positives and false positives, thereby achieving more efficient and practical collaborative perception.
2.2 Camera-based 3D object detection
Generally speaking, there are two research directions on camera-based 3D object detection, i.e., dense feature based algorithms and sparse query based algorithms. Dense algorithms represent the main research direction in camera-bass based 3D detection, leveraging dense feature vectors for view transformation, feature fusion, and box prediction. Currently, BEV-based approaches form the core of dense algorithms. LSS [25] and CaDDN (27] utilize view transformation modules to convert dense 2D image features into BEV space through forward projection. BEVDet ( and BEVDepth [0] employs a lift-splat operation for view transformation and further encodes BEV features using BEV encoder. BEVFormer [ 37 generates BEV features through deformable attention in backward projection manner, circumventing reliance on explicit depth information. FB-BEV [12] and DualBEV
[9] effectively combine the advantages of forward and backward projection methods to generate higher-quality BEV features, leading to enhanced perception performance.
Compared to dense algorithms, the sparse query-based paradigm have recently gained significant attention in the community due to their low computational complexity. DETR3D [32 is represen-
tative sparse approach that utilizes a set of sparse 3D query vectors to sample and fuse multi-view image features. The PETR series [18] 19 30 introduce 3D position encoding, leveraging global attention for direct multi-view feature fusion and conducting temporal optimization. SparseBEV [I7] and Sparse4D [14] [1]] [6] enhances DETR3D through multi-point feature sampling and temporal fusion, thereby improving perception performance.
3 RayFusion
As depicted in Figure RayFusion comprises a single-agent detector, a collaborative message generation module, apatial-tempora alignment module, a ray occupancy information encoding module, a multi-scale instance feature aggregation module, and a detection head. The single-agent detector and the collaborative message generation module generate instance information from multi- view input images for communication and collaboration. The spatial-temporal alignment module performs spatial-temporal alignment of instance information across agents by employing unified coordinate system and modeling motion. The ray occupancy information encoding module leverages multi-view information from multiple agents to mitigate depth estimation ambiguity. The multi-scale instance feature aggregation module enables effective interaction among instance features through multi-scale feature fusion. Finally, the updated instance features are fed into the detection head to predict the target's 's category and location information.
3.1 Collaborative message generation
Single-agent camera-only 3D object detector. RayFusion adopts Sparse4D [6 as the single-agent detector, which consists of an image encoder, a depth prediction head, a decoder, and a detection head. The image encoder is a standard 2D backbone (ResNet50 [4) used to extract semantic features from multi-view images of agent j. The depth prediction head estimates dense depth distribution D for the k-th camera view and is supervised using LiDAR point clouds to accelerate convergence. The decoder then initializes N learnable object queries Qo E NxC in 3D space, which interact with multi-view image features to iteratively update instance representations. Finally, the decoder's output
3

---

Ray occupancy encoding y MIPCE plly
Weightd
mestgg manii
Single
Coltsbomtive
Top
Pe--i & M(Y)
da.s
Pynena windoe Weftstenten
Matti images
agent y timestamp
NZ.
&
V.X
Colishorntive mixrua poneration
Singleaget
D ....
Ray occupancy
I,PB
Multi-ecale instanee
Detection
information encoding
feature aggregation
head
Merging of instancee
Mali view images of ego
M,
A window perform multi head self antenionn
Concationate
Detection outgut
Figure 2: Overall architecture of RayFusion. i) The single-agent detector and the collaborative message generation module generate instance information for communication and collaboration; ii) The spatial-temporal alignment module enhances system robustness to latency by modeling motion; ii) The ray occupancy information encoding module leverages multi-view information mitigate depth estimation ambiguity; iv) The multi-scale instance feature aggregation facilitates effective interaction among instance features, promoting comprehensive and precise collaborative perception.
instance features Ij processed by detection head (a multi-layer perceptron, MLP) to predict the target's category and anchor predictions Aj = [r, y: 2, In we In h, In l, sin yom, cos yam, Vai, By Uz]-
Collaborative message generation. Given the outputs of the single-agent detector (i.e., Dj, Pj IjjAAAA) Ay), the collaborative message shared via communication is M. the ray occupancy information. Specifically, R, = {oA: uf. aj k 0, 10.,, nJ,, denotes the coordinate of the camera origin, u² represents the direction vector of the camera ray (i.c., the ray from the camera origin the center of the instance anchor), a is the angle between the camera ray and the optical axis, pf indicates the occupancy state along the camera ray, and the number of surround-vier cameras. To obtain pý. we project the center of the instance anchor onto the corresponding depth map Dj_ and apply bilinear sampling. This process is formulated as:
(I,, Rý}, where R, denotes k
where of
p = -Bilinear(Dj (D), (5, U, 2)) E R
where (ar, y, =) represents the pixel coordinates of the instance anchor center projected onto the k-th image view, denotes the number of discrete depth bins. During the collaborative perception process, agent j selects the top M instances with the highest confidence scores and broadcasts their corresponding instance information for communication and collaboration.
3.2 Ray-driven position encoding
Position encoding is crucial for instance-level fusion, as it introduces location information during feature fusion to compensate for the limitations of the attention mechanisms (28). However, previous works typically adopt learnable or sinusoidal position encoding, which are inadequate for effectively modeling the camera imaging process of instances in 3D space. our proposed RayFusion, the position encoding consists of two components: vanilla position encoding and ray occupancy informa- tion encoding. The vanilla position encoding maps the anchor into high-dimensional space using lightweight anchor encoder & (MLP). Meanwhile, the ray occupancy information encoding explicitly represents the ray occupancy state the corresponding instance, as detailed Section 3.2.2
3.2.1 Spatial-temporal alignment and merging of instances
Spatial-temporal alignment. After receiving the shared message M,, from agent j. the first step is align instance information with respect to ego. Since different agents represent the 3D space using distinct coordinate systems, discrepancies naturally arise in the structured information descriptions (i.e., Aj. R2)) of the same instanee across agents. Therefore, we choose the ego coordinate system as the unified coordinate system. Moreover, given that instance image features I, are decoupled from

---

their structured information, we only need to align the structured information, leaving the instance image features I intact to achieve handle the structured information of instances by unifying the coordinate systems of all agents and modeling motion, thereby achieving spatial-temporal alignment among agents, facilitating effective multi-agent collaboration and enhancing the system's robustness to communication delays.
alignment across agents. Thus, our key idea is to
In autonomous driving, there are two types of motion: instance motion and ego motion. Instance motion refers to the movement of other instance in the environment around the ego, while ego motion refers to the motion of the ego itself within the environment. For simplicity, we model the instance motion as uniform over short periods. Given the received anchor Aj. we first use the instance s velocity attribute to warp its position to the current frame, thereby compensating for instance motion:
[xc y, z|c = [zz y, z], + [Ux, Byy, Uslr (t (t--7] 7)
(2)
where and represent the timestamp of the ego and the collaborator, respectively.
Next, we warp the position, velocity, and rotation attributes of collaborator instances to the unified coordinate system in the current frame, based on the ego pose changes. This compensates for ego motion and achieves spatial alignment:
[sr, y, =] R( (r 2/ + T ((r-t))
(3)
(Uay Uy, U2] R((r-1) UUz, Vy Unlr + T(r-))
(4)
[cos yau, sin yow, 0] = (()) jcos yaw, sin yaw, 0],
(5)
where R(r-e) and T(r-o) are the rotation and translation matrices from the collaborator's frame to the unified coordinate system in the current frame. When there is no communication delay (i.c.,
t = T), the above operations reduce to a coordinate transformation from the collaborator's frame into the unified coordinate system, achieving spatial alignment of instance anchors. Notably, the alignment of ray occupancy information differs from that of anchors. Specifically, only the camera center o and the camera ray direction vector u are transformed into the unified coordinate system, without performing motion compensation, as this would disrupt the original scene imaging process.
Merging of instances. We concatenate the aligned instance information from collaborators with all instances extracted by the ego along the sequence dimension, resulting in the merged instance information MI = {I, A, RJ, where I, A, and R denote the aligned instance features, anchors, and ray occupancy information, respectively.
3.2.2 Ray occupancy information encoding
Given the aligned ray occupancy information R = (ox, uk, Ok. k: 1,.,,,,,,,,,
k), we propose the ray occupancy information encoding module explicitly incorporate the camera imaging process of each instance from multiple agent perspectives. Camera imaging is based on light ray projection, where each ray traveling through 3D space may be blocked by objects or pass through different media. As shown in Figure M single agent perceives the object from single viewpoint, typically allowing it estimate the object's presence along the direction of the light ray but not its precise depth position along the ray. When multiple agents observe the same object, each camera records the occupancy information of the object along the direction of its respective light rays. Since the ray directions from different cameras vary, cross-verification among multiple agents enables accurate 3D localization of the object by identifying the intersection of multiple rays. This improves the network's ability to distinguish hard negative samples and improves its localization accuracy for true positive samples (see Figure 4.. Based on this observation, we propose a ray occupancy information encoding module, as illustrated in Figure 2 This mechanism consists of three components: ray encoding, occupancy information encoding, and multi-camera ray occupancy information fusion. For simplicity, we omit the camera index in the ray encoding and occupancy information encoding.
Ray encoding. A camera ray can be uniquely represented by the camera's optical center coordinate E R and the direction vector of the ray u E IR,, formulated as r(A) = o X u(> 2 0). However, as noted in [4 26), neural networks tend to favor low-frequency function learning. Directly learning the optical center coordinates and direction vectors to represent the ray would lead to suboptimal performance in capturing high-frequency variations. Therefore, applying a high-frequency function to map the input into a high- high-dimensional space before feeding it into the network can improve data
5

---

fitting. Here, we first define a mapping function fL to transform the real number from I to dimensionll embedding space IR 122 defined as follows:
[t(a) = (a,sinn (2,ssi((mm)) cos(22 ma),...................... sin (2nn221mmmm ma), cos((2 E TR
(6)
The function f.(()) is applied separately to each of the three element in both the camera's 'ss optical center and the ray's direction vector u. The resulting embeddings are concatenated along the channel dimension and then processed through an MLP to obtain the ray encoding for the given instance. This process can be formalized as follows:
= MLP(fr(o) (u)) E R
(7)
where: denotes concatenation along the channel dimension. our experiments, we set L 10 for (o) and L = 4 for (u).
Occupancy information encoding. Encoding only the camera rays corresponding to instances cannot fully represent the potential distribution of instances along the rays. Therefore, we incorporate the classification depth distribution predicted by the depth prediction head to reveal the occupancy state along the camera ray. First, we project the predefined uniformly distributed discrete depth bins onto the ray direction as follows:
= MLP ((do, dp....,d--1). cos cos a) E R
(8)
where de represents the real depth corresponding to the i-th depth bin in the depth prediction. Next, we use MLP to obtain the single camera ray occupancy encoding, formalized as follows:
MLP(X : P) : Y R
Multi-camera ray occupancy information fusion. When agent has camera inputs, the same instance receives different ray occupancy encodings from each camera, denoted as Y = (%o, f1..., 9--1) E R*x2C, However, since instance may only appear in the field of view of subset of cameras, we adopt an attention network V to fuse the multi-camera ray occupancy encodings into unified representation. This process be formalized follows:
(Q.K) KT
W = Softmax
(Mean
dim = =-1) + M) E R
(10)
VC
R = M MLP (w + winI +... WR- 1ix-1ix-1) E R
(11)
where Q, K E R are high-dimensional vectors obtained by applying projection matrices T, wy represents the i-th component of W. The mask matrix M E Rº indicates whether an instance in the field of view of camera, where indicates the instance is within the camera's 's field of view, and -oo indicates it is outside. Notably, in order to account for communication delays without disrupting the original scene imaging process. we follow the approach of (35) to incorporate the delay information by encoding it with sinusoidal embeddings Pt-+ E R In summary, the ray-driven positional encoding in RayFusion can be formalized as:
PE = (AA) + + P RC
(12)
3.3 Multi-scale instance feature aggregation
Given the instance features I and the corresponding ray-driven positional encodings PE, we first concatenate them along the channel dimension to form the input / E RNN-LM)22C for the multi-scale instance feature aggregation module, where denotes the number of collaborators. After fusing the ray occupancy information, the instance features explicitly incorporate the 3D) structure of the scene and the potential actual 3D position of the instance. Through effective interaction of instance features, the ambiguity in instance depth estimation can be significantly alleviated, reducing redundancy and erroneous detections along the camera rays. As shown Figure 2] to achieve efficient interaction between multi-agent instance features, we propose pyramid window self-attention, which enhances the robustness of the system against localization noise by enabling multi-scale interactions between multi-agent instance features, thereby capturing both local and global spatial features in parallel. The process of b-th branch is formulated as follows:
Q KT
+ D(D(ss) < <ra) .VERN+LMYXC
f, = Softmax
(13)
Vd

---

Table 1: 3D detection performance comparison the DAIR-V2 V2X 138, V2XSet 335 and OPV2V
36 datasets under perfect settings. CoSparse4D aligns multi- instances (see Section [32.1]
and applies detection head (MLP) to produce the perception results. f indicates using Sparse4D
16 as the single-agent detector.
DAIR
V2X
V2XSet
OPV2V
Method
AP50
AP70
AP50
AP70
AP50
AP70
No Collaboration
10.74
1.64
30.37
13.79
45.94
25.56
Late Fusion
18.57
5.16
51.41
25.59
77.62
51.92
V2VNet (BCCV/20)
15.26
2.97
59.54
39.00
79.06
57.59
V2X-Vit (ECCV122)
15.84
3.07
59.14
41.23
78.41
58.38
Where2comm (2)
16.03
3.67
61.69
43.96
77.14
58.60
CoBEVT (CoR1:22)
15.92
3.18
58.84
40.81
80.26
59.34
CoAlign (ICRA233
16.55
3.23
64.79
39.64
80.21
60.46
IFTR (FCCV'24)
20.51
7.90
71.73
49.67
85.56
66.04
CoSparse4D (Baseline)
23.38
9.17
65.69
49.33
79.38
67.43
RayFusion f(Ours)
26.29
11.54
70.32
53.14
86.59
74.25
where Q, K, (NN+LM) are obtained by applying an ML.P to 7, D(s.) represents the distance between the positional attributes of the i-th query instance and the j-th key instance, and ry defines the receptive field threshold for the b-th branch. The indicator function g(-) ensures that only instances within the defined receptive field contribute to the attention computation. Next, the results from multiple branches are combined through a weighted summation:
W Softmax(MLP(CC f :.... iB1)) E B
(14)
i= wo + w + -in-in (N
(15)
where v., represents the b-th component of W. and B denotes the number of parallel branches. The pyramid window self-attention significantly improves the robustness of instance interactions (see Figure 3): in branches with smaller Th. attention is confined to smaller receptive field, preserving local contextual information; while branches with a larger Tb, attention operates over larger receptive field, enabling the model to capture distant visual cues and compensate for larger localization errors. Finally, the output features of pyramid window self-attention module î is followed by feed forward network (FFN) and an Add&Norm module to get the output I of multi-scale instance feature aggregation module.
3.4 Detection head
RayFusion employs an MLP- based detection head, taking instance features i E (N++MM) as input, and producing detection results O E RN We utilize the & loss for regression and the focal loss [13] for classification. Additionally, we incorporate cross-entropy loss leverage LiDAR point clouds for supervising the depth estimation task, which accelerates network convergence.
4 Experiments
4.1 Experimental setup
Datasets. We evaluate proposed method three multi-agent datasets: DAIR-V21 38), V2XSet [35, and OPV2V (36]. DAIR-V2X (38) is an open-source real-world collaborative perception dataset with images, containing 18,000 data samples with an image resolution of 1080 x 1920. V2XSet (35] and OPV2V 36) are simulated datasets supporting collaborative perception with 2 to connected agents, co-simulated by Carla 2 and OpenCDA (4], with an image resolution of 600 x 800.
Implementation details. We implement RayFusion following Section 3 and train it for 72 epochs using the Adam W (20) optimizer. The initial learning rate is set to 1 x 10-4 and follows cosine annealing decay schedule, The instance feature dimension C' is set 256, and the number discrete depth bins D is set to 80, while the number of ego instances N and shared instances M are set to
7

---

V2VNet
Where2comm
CoAlign
RayFusion
V2XVIT
COBEVT
1.0
FTR Royfurion
3.5
Buyhhssnn
3.40
Prectsion
65-
AP50 on V2X
AP5D on V2XSet
2.00
2.73
50-
50-
3.6
0.0
04
25
5.8
0.
40
Recall
25
Figure 4: Precision-recall curves eval- uated the OPV2V dataset with an IoU threshold of 0.70. RayFusion* represents the removal of the ray occu- pancy information encoding module.
0.0
0.1
0.4 0.5
100
300
400 s00
(a) Localization error (m)
(b) Communication delay (ms)
Figure 3: Robustness to localization errors and communica-
tion delays on the V2XSet dataset.
600 and 200, respectively. We utilize three pyramid windows with receptive field thresholds of 4m, 8m, and 16m. The perception range x, E [-511.20 5..mm, 51.2m], and the communication range is set to 70m. For BEV-based collaborative methods, we follow the implementation in IFTR (29) which adopts BEVFormer [T]] as the single-agent detector. We evaluate detection performance using the Average Precision (AP) metric with Intersection over Union (IoU) thresholds of 0.50 and 0.70.
4.2 Main results
Benchmark comparison. Table compares the proposed RayFusion with previous collaborative methods. We observe that the proposed RayFusion outperforms previous methods both real-world and simulated datasets, validating the superiority of our model and robustness to various real- world noises. Specifically, RayFusion improves upon the previous state-of-the-art method, IFTR, by 3.64, 3.47, and 8.21 in AP70 on the DAIR- V2X, V2XSet, and OPV2V datasets, respectively. Compared to previous collaborative methods, our method alleviates the ambiguity in instance depth estimation by leveraging multi-view information, enhancing the network's 's discriminative ability for hard negative samples along the camera rays and improving localization accuracy for true positive samples. Additionally, multi-scale instance feature aggregation facilitates effective instance interaction, promoting comprehensive and precise collaborative perception.
Robustness to localization noise. We follow the localization noise settings in 35
29
(Gaussian noise with a mean of Om and a variance ranging from Om to 0.6m) and validate RayFusion Fusion's 's robustness to realistic localization noise in Figure (a). We observe that as localization errors increase, the performance of all intermediate collaborative methods deteriorates due to the mismatch in spatial feature information. However, RayFusion significantly outperforms previous SOTAs in terms of the performance degradation slope. This is because the pyramid window self-attention in the multi-scale instance feature aggregation module captures local and global spatial features in parallel, enhancing the system's robustness to localization noise.
Robustness communication delays. In autonomous driving, both instance motion and ego motion can lead to feature fusion errors under communication delays, impairing collaborative perception performance. Figure 3 (b) analyzes the detection performance of RayFusion under varying communication delays (ranging from 0 to 500ms). Notably, as communication delays increase, all other intermediate fusion methods inevitably suffer significant performance degradation due to incorrect feature matching. However, RayFusion significantly outperforms previous SOTAs under communication latency and even surpasses some fusion methods without latency (such as V2VNet and CoBEVT) under severe latency conditions (500ms). This robustness is achieved by modeling motion to handle the structured information of instances, achieving temporal alignment of instance features across agents and enhancing the system's 's resilience to communication delays.
Precision-Recall Analysis. We assess the ability of RayFusion to reduce false positive predictions along the camera rays and improve localization accuracy for true positive samples by plotting precision-recall curves. Figure f shows that the ray occupancy information encoding module consistently improves precision across all recall levels, and enhances recall across all precision levels
8

---

under an IoU threshold of 0.70. These results validate the effectiveness of RayFusion in reducing false positive predictions and improving the localization accuracy for true positive samples.
4.3 Ablation studies
Effectiveness of different modules in RayFu- sion. Here we investigate the effectiveness of various components in RayFusion, with the base model being simply merging multi-agent instance features and applying an MLP to pro- duce 3D detection results. As shown in Table 2 i) STA achieves AP70 improvement of 28.99 on OPV2V and 4.19 on DAIR- V2X by aligning the structured instance information from multi- ple agents, thereby enabling effective instance feature fusion; ii) MIFA achieves improvements 4.61 and 1.03 in AP70 on OPV2V and DAIR- V2X, respectively. by fusing multi-scale infor- mation and enhancing the semantti understand- ing of instance features; iii) ROE further con- tributes performance gains of 2.21 and 1.34 in AP70 on OPV2V and DAIR-V2L respectively. This achieved by leveraging multi-view informa- tion from multiple agents to cross-verify the true 3D positions of instances, which mitigates ambiguity in depth estimation and strengthens the model's ability to localize objects along camera rays. Ad- ditionally, replacing the pyramid window self-attention (PWA) in MIFA with a vanilla multi-head self-attention mechanism leads to suboptimal results, as the pyramid window design in PWA enables parallel capture of both local and global spatial features, enriching feature representations with more comprehensive contextual information.
Table 2: Ablation study results on the OPV2V and DAIR-V2X datasets. STA, MIFA, ROE represent: i) spatial-temporal alignment, ii) multi-scale in- stance feature aggregation, and iii) ray occupancy information encoding, respectively. IFA replaces the pyramid window self-attention in MIFA with vanilla multi-head self-attention.
OPV2V
DAIR-V2X
STA MIFA ROE IFA
AP50
AP70
AP50
AP70
57.80
38.44
15.11
4.98
79.38
67.43
23.38
9.17
85.40
72.04
24.86
10.20
86.59
74.25
26.29
11.54
V
86.40
73.04
26.31
10.41
Analysis of different components in ROE. In Table 3 we analyze the effectiveness of each component in ROF. The results indicate that: i) the ray encoding based on optical center coordi- nates and direction vectors is crucial for the ef- fectiveness of ROE, as it explicitly represents the imaging path of instances and provides potential true position information in 3D space; ii) oc- cupancy information encoding further enhances the performance of ROE, as the occupancy states along the ray reflect the possible distribution of instances, offering a stronger prior for multi- view validaaii of instance positions; iii)
Table Analysis of different components in ROE on the OPV2V and DAIR-V2X. RE, OE represent: i) ray encoding and ii) occupancy information en- coding, respectively. WH represents the removal of high- high-dimensional mapping in ray encoding.
OPV2V
DAIR-V2X
RE OB WH
AP50
AP70
AP50
AP70
85.40
72.04
24.86
10.20
86.43
72.87
25.52
10.63
86.59
74.25
26.29
11.54
/ v
86.18
73.72
25.87
11.02
removing the high-dimensional mapping in ray encoding, i.e., encoding optical center coordinates and direction vectors solely through MLP leads to performance degradation, as the high-dimensional mapping projects the high-free frequency information of camera rays into a higher- dimber--imensional space, thereby reducing the learning complexity of the network.
4.4 Qualitative evaluation
We conduct a qualitative analysis of the RayFusion's performance using representative samples from the V2XSet dataset, as shown Figure 3 Overall, RayFusion produces more precise results that closely align with the ground truth, while reducing redundant predictions along the camera rays. This improvement can be attributed to: i) ROE, which effectively mitigates ambiguity instance depth estimation, thereby enhancing localization accuracy along the camera rays; and ii) the pyramid window design in multi-scale instance feature aggregation module, which captures both local and global spatial features in parallel, leading to more robust semantic representation of instance features. Please refer to the supplementary materials for video visualization results on the DAIR- -2X, V2XSet, and OPV2V datasets.

---

(a) No Collaboration
(b) IFTR
(c) RayFusion
Figure 5: Visualization of predictions from (a) No Collaboration. (b) IFTR and (c) RayFusion on the V2XSet test Green and red 3D bounding boxes represent the ground truth and prediction, respectively.
Conclusion
RayFusion is a ray-based fusion method designed to enhance the detection performance of camera- only collaborative perception systems by leveraging ray occupancy information from collaborators. It consists of three key components: spatial-temporal alignment module, ray occupancy information encoding module, and multi-scale instance feature aggregation module. The spatial-tempon align- module achieve spatial-temporal alignment across agents, improving the system's robustness communication delays. The ray occupancy information encoding module mitigates depth estimation ambiguities through multi-view information, enhancing the network's ability distinguish hard negative samples along camera rays and improving the localization of true positive samples. The multi-scale instance feature aggregation module captures both local and global spatial features in par- allel using pyramid windows, enabling effective instance feature interaction. Extensive experiments demonstrate the superiority of RayFusion and the effectiveness of its key components.
Limitation and future work
In this work, utilize the pose information of each agent to compute transformation matrices for effective multi-agent information interaction. future work, we will explore collaborative perception under unknown poses to better protect the privacy of participating agents. Additionally, We plan to improve the system's 's robustness to communication delays by developing more effective occupancy representations for instance, predicting future occupancy representations by propagating historical information through feature flow.
Acknowledgements
This work supported in part by Zhejiang Province Key R&D programs, China (Grant No. 2025C01039, 2024C01017, 2024C01010), in part by the National Natural Science Foundation of China under Grants U21B2029, and in part by Zhejiang Provincial Natural Science Foundation of China under Grant LR23F010006.
10

---

References
[1] Ziming Chen, Yifeng Shi, and Jinrang Jia. Transiff: An instance-level feature fusion framework for vehicle-infraastrueture cooperative 3d detection with transformers. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 18205-18214, 2023.
[2] Alexey Dosovitskiy, German Ros, Felipe Codevilla, Antonio Lopez, and Vladlen Koltun. Carla: An open urban driving simulator. In Conference on robot learning, pages 1-16. PMLR, 2017.
[3] Siqi Fan, Haíbao Yu, Wenxian Yang. Jirui Yuan, and Zaiqing Nie. Quest: Query stream for practical cooperative perception. In 2024 IEEE International Conference on Robotics and Automation (ICRA), pages 18436-18442, 2024.
[4] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference computer vision and pattern recognition, pages 770-778, 2016.
[5] Yue Hu, Shaoheng Fang. Zixing Lei, Yiqi Zhong, and Siheng Chen. Where2comm: Communication teaeumicationn-ffciientt efficient collaborative perception via spatial confidence maps. Advances in neural information processing systems, 35:4874-4886, 2022.
[6] Yue Hu, Yifan Lu, Runsheng Xu, Weidi Xie, Siheng Chen, and Yanfeng Wang. Collaboration helps camera overtake lidar 3d detection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 9243-9252, 2023.
[7] Junjie Huang and Guan Huang. Bevdet4d: Exploit temporal cues in multi-camera 3d object detection. arXiv preprint arXiv:2203. 17054, 2022.
[8] Junjie Huang, Guan Huang, Zheng Zhu, Yun Ye, and Dalong Du. Bevdet: figh-ppeffrmmacc multi-camera 3d object detection in bird-eye-view. arXiv preprint 1rXi:22112.11790 2021.
(9] Peidong Li, Wancheng Shen, Qihao Huang, and Dixiao Cui. Dualbev: Con is all you need in view transformation. arXiv preprint 24X3::2403.05402.. 2024.
[10] Yinhao Li, Zheng Ge, Guanyi Yu, Jinrong Yang. Zengran Wang. Yukang Shi, Jianjian Sun, and Zeming Li. Bevdepth: Acquisition of reliable depth for multi-view 3d object detection. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 37, pages 1477-1485, 2023.
[11] Zhiqi Li. Wenhai Wang, Hongyang Li, Enze Xie, Chonghao Sima, Tong Lu, Yu Qiao, and Jifeng Dai. Bevformer: Learning bird's-eye-view representation from multi-camera images via spatiotemporal transformers. In European conference on computer vision, pages 1-18. Springer, 2022.
[12] Zhiqi Li, Zhiding Yu, Wenhai Wang, Anima Anandkumar, Tong Lu, and Jose M Alvarez. Fb-bev: Bev representation from forward-backwart view transformations. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 6919-6979999 6928, 2023.
[13] Tsung-Yi Yi Lin, Priya Goyal, Ross Girshick, Kaiming He, and Piotr Dollár. Focal loss for dense object detection. Proceedings of the IEEE international conference on computer vision, pages 9980-2988, 2017.
[14] Xuewu Lin, Tianwei Lin, Zixiang Pei, Lichao Huang, and Zhizhong Su. Sparse4d: Multi-view 3d object detection with sparse spatial-temporal fusion. arXiv preprint arXiv:2211. 2022.
[15] Xuewu Lin, Tianwei Lin, Zixiang Pei, Lichao Huang, and Zhizhong Su. Sparse4d v2: Recurrent temporal fusion with sparse model. arXiv preprint 230.: 2023.
[16] Xuewu Lin, Zixiang Pei, Tianwei Lin, Lichao Huang, and Zhizhong Su. Sparse4d v3: Advancing end-to-end 3d detection and tracking. arXiv preprint arXiv:2311.11722 2023.
[17] Haisong Liu, Yao Teng, Tao Lu, Haiguang Wang, and Limin Wang. Sparsebev: High- performance sparse 3d object detection from multi-camera videos. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 18580-18590, 2023.

---

[18] Yingfei Liu, Tiancai Wang. Xiangyu Zhang, and Jian Sun. Petr: Position embedding transfor- mation for multi-view 3d object detection. European Conference on Computer Vision, pages 531-548. Springer, 2022.
[19] Yingfei Liu, Junjie Yan, Fan Jia, Shuailin Li, Aqi Gao, Tiancai Wang, and Xiangyu Zhang. Petrv2: A unified framework for 3d perception from multi-camera images. In Proceedings of the IEEE/CVP International Conference on Computer Vision, pages 3262-3272777 2023.
[20] I Loshchilov. Decoupled weight decay regularization. arXiv preprint 1eXiv:/711.05111 2017.
[21] José Manuel Lozano Domínguez and Tomás Jesús Mateo Sanguino. Review on v2x, i2x, and p2x communications and their applications: comprehensive analysis over time. Sensors, 19(12):275,, 2019.
[22] Yifan Lu, Yue Hu, Yiqi Zhong, Dequan Wang. Siheng Chen, and Yanfeng Wang. An extensible framework for open heterogeneous collaborative perception. arXiv preprint arXiv:2401. 13364, 2024.
[23] Yifan Lu, Quanhao Li, Baoan Liu, Mehrdad Dianati, Chen Feng, Siheng Chen, and Yanfeng Wang. Robust collaborative 3d object detection in presence of pose errors. In 2023 IEEE International Conference on Robotics and Automation (ICRA), pages 4812-4818. IEEE, 2023.
[24] Ben Mildenhall, Pratul P Srinivasan, Matthew Tancik, Jonathan T Barron, Ravi Ramamoor- thi, and Ren Ng. Nerf: Representing scenes as neural radiance fields for view synthesis. Communications of the ACM, 65(1):99-106, 2021.
[25] Jonah Philion and Sanja Fidler. Lift, splat, shoot: Encoding images from arbitrary camera rigs by implicitly unprojecting to 3d. In Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 28, 2020, Proceedings, Part XIV 16, pages 194-210. Springer, 2020.
[26] Nasim Rahaman, Aristide Baratin, Devansh Arpit, Felix Draxler, Min Lin, Fred Hamprecht, Yoshua Bengio, and Aaron Courville. On the spectral bias of neural networks. International conference on machine learning, pages
PMLR, 2019.
[27] Cody Reading, Ali Harakeh, Julia Chae, and Steven L Waslander. Categorical depth distribution network for monocular 3d object detection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 8555-8564, 2021.
[28] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems, 30, 2017.
[29] Shaohong Wang, Lu Bin, Xinyu Xiao, Zhiyu Xiang, Hangguan Shan, and Eryun Liu. Iftr: An instance-level fusion transformer for visual collaborative perception. In European Conference on Computer Vision, pages 141. Springer, 2024.
[30] Shihao Wang, Yingfei Liu, Tiancai Wang, Ying Li, and Xiangyu Zhang. Exploring object- centric temporal modeling for efficient milti-view 3d object detection. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 3621-3631, 2023.
[31] Tsun-Hsuan Wang, Sivabalan Manivasagam, Ming Liang, Bin Yang, Wenyuan Zeng, and Raquel Urtasun. V2vnet: Computer Mision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part 11 16, pages 605-621. Springer, 2020.
communication for joint perception and prediction. In
[32] Yue Wang, Vitor Campagnolo Guizilini, Tianyuan Zhang, Yilun Wang, Hang Zhao, and Justin Solomon. Detr3d: 3d object detection from multi-view images via 3d-to-2d queries. In Conference on Robot Learning, pages 180-191. PMLR, 2022.
[33] Hao Xiang, Runsheng Xu, and Jiaqi Ma. Hm-vit: Hetero-modal vehicle-to-vehiclee cooperative perception with vision transformer. arXiv preprint arXiv:2304. /0628, 2023.
[34] Runsheng Xu, Yi Guo, Xu Han, Xin Xia, Hao Xiang, and Jiaqi Ma. Opencda: an open coopera- tive driving automation framework integrated with co-simulation. In 202/ IEEE International Intelligent Transportation Systems Conference (ITSC), pages 1155-1162. IEEE, 2021.
12

---

[35] Runsheng Xu, Hao Xiang, Zhengzhong Tu, Xin Xia, Ming-Hsuan Yang, and Jiaqi Ma. V2x-vit: eeverytitt cooperative perception with vision transformer. In European conference on computer vision, pages 107-124. Springer, 2022.
[36] Runsheng Xu, Hao Xiang, Xin Xia, Xu Han, Jinlong Li, and Jiaqi Ma. Opv2v: An open benchmark dataset and fusion pipeline for perception with saaicle-t--veichee communication. In 2022 International Conference on Robotics and Automation (ICRA), pages 2583-2589, IEEE, 2022.
[37] Chenyu Yang, Yuntao Chen, Hao Tian, Chenxin Tao, Xizhou Zhu, Zhaoxiang Zhang, Gao Huang, Hongyang Li, Yu Qiao, Lewei Lu, et al. Bevformer v2: Adapting modern image backbones to bird' s-eee-view recognition via perspective supervision. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 17830-17839, 2023.
[38] Haibao Yu, Yizhen Luo, Mao Shu, Yiyi Huo, Zebang Yang, Yifeng Shi, Zhenglong Guo, Hanyu Li, Xing Hu, Jirui Yuan, et al. Dair-v2x: A large-scale dataset for vehicle- infrastrttfa cooperative 3d object detection. In Proceedings of the IEEE/CVE Conference on Computer Vision and Puttern Recognition, pages
2022.
13

---

A The efficiency of RayFusion
In RayFusion, we employ Sparse4D as the single-agent detector, which projects sparse 3D reference points onto the image plane for feature sampling. This design eliminates the need for complex view transformations and dense feature extraction, significantly reducing computational overhead. Table compares RayFusion with the previous ofateof--he-arr method IFTR the OPV2V dataset in terms of detection per- formance and inference speed. IFTR builds dense and unified BEV representation through communication and then performs object detec- tion in the BEV space using BEVFormer as the single-agent detector. As shown in the Table 4 RayFusion achieves superior performance with faster inference speed. These gains are attributed to the adoption of a aparse-paradigm single-agent detector and a streamlined multi-agent information fusion strategy. We advocate for the broader use of aparse-paradigge detector achieve better balance between performance and efficiency, facilitating the deployment of collaborative perception systems on resource-constrained edge devices.
Table 4: 3D detection performance and speed parison on the OPV2V dataset, reporting AP met- rics by default under perfect settings. The image input size for both methods is set to 640 x 480, and FPS is measured a single GeForce RTX 3090 using the PyTorch fp32 backend.
Method
AP50 AP70 FPS
IFTR
85.56
66.04 3.36
RayFusion
86.59 74.25 8.79
B Trade-off between detection performance and communication cost
We evaluate communication using the num- ber of bytes in the shared messages. Under the OPV2V experimental settings in Table [ sin- gle agent communication cost per frame for Ray- Fusion, IFTR, and late fusion are 276.6 KB, 18.8 MB, and 2.7 KB, respectively. Other intermedi- fusion methods incur a communication cost of approximately 64.0 MB per frame. Benefiting from interpretable instance-level collaboration, RayFusion can adapt to different bandwidth con- ditions by adjusting the number of shared in- stances M among collaborators. As shown in 5 we explore the trade-off between detection per- formance and communication bandwidth of RayFusion on the OPV2V datasets. The results indicate that RayFusion achieves excellent trade-off between detection performance and communication cost. When the number of shared instances among collaborators M is set to 5, it requires only 6.9 KB of communication data to enable collaborative perception.
Table 5: The trade-off between detection perfor- mance and communication cost of RayFusion on the OPV2V datasets. M is the number of shared instances among collaborators.
M
AP50
AP70
Bytes
200
86.59
74.25
276.6KB
100
86.50
74.17
138.3KB
50
86.47
73.85
69.1KB
25
86.34
73.74
34.6KB
10
84.66
72.00
13,8KB
5
77.73
63.87
6.9KB
C The reasonableness of modeling instance motion as uniform
Following the default settings Section 4 (M 200), each agent transmits 276.6 KB of information
frame. A commonly used V2X communication technology, IEEE 802. 102.11P-ber based DSRC, provides a transmission bandwidth of 27 Mbps (2I). Assuming five agents are collaborating, each agent is allocated an average bandwidth of 5.4 Mbps, resulting in transmission delay of 400.2 ms for the data. As shown in Figure 3 the performance degradation of RayFusion under this latency is not significant, which demonstrates the feasibility of modeling instance motion uniform motion. Moreover, the transmission delay can be reduced by decreasing the number of shared instances (e.g., 20.0 ms when transmitting 10 instances per frame, i.e., M 10), making it more feasible to model instance motion as uniform motion under such settings.
D Occupancy representation without additional supervision
The purpose of the ray occupancy information encoding module to alleviate the ambiguity in instance-level depth estimation, thereby enhancing the network's ability distinguish hard nega-
14

---

tive samples along the camera ray direction. A natural form of supervision is instance-level depth supervision, which can be applied guide the ray occupancy information encoding module. Specifi- cally, after incorporating the ray occupancy information into the instance features, the output Í of multi-scale instance feature aggregation module can be passed through depth estimation head to predict the depth distribution of the instance, which can then be supervised for instances that are successfully matched with ground truth targets. However, when using regression-based loss (e.g.. & loss) for depth supervision, this supervision becomes essentially a sub-task of the 3D object detection supervision task. Therefore, we do not apply explicit additional supervision for the occupancy representation.
E
Challenges in real-world datasets
As shown in Table [ the performance the real-world DAIR-V22 dataset is significantly lower than on the simulated V2XSet and OPV2V datasets. This performance gap is primarily attributed three factors: i) DAIR-V2L V2X includes at most two collaborative agents, whereas V2XSet and OPV2V support up to five; ii) DAIR-V2X provides only front-view camera images, while V2XSet and OPV2V offer four surround- vieww camera inputs; and iii) real-world scenes are generally more complex than simulated ones, making depth estimation substantially more challenging.
F
Broader impacts
The positive impact of this work is that RayFusion leverages V2X communication technology to share instance information, alleviating the ambiguity in depth estimation, reducing redundancy and false positive predictions along camera rays. This improves the perceptual capabilities of budget-constrained vehicles regarding their surrounding environment, thereby reducing the occurrence of traffic accidents.
In addition, RayFusion achieves spatial alignment of multi-agent instance information through unified coordinate system and improves robustness to communication delays by modeling motion. Its multi-scale feature interaction design further enhances resilience to localization noise.
RayFusion also reduces computational cost by adopting a sparse-paradigms single-agent detector and reduces communication overhead through instance-level information sharing, making it well-suited for real-world deployment.
Morcover, RayFusion is inherently scalable, supporting the integration additional agents for collaboration without requiring retraining.
In summary, this work promotes the practical application of camera-only collaborative perception systems in autonomous driving.
15

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
Abstract and Section | |cccrattlly reflect the paper's contributions and scope.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results
be expected to generalize other settings.
is fine to include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: Section 5a discuss the limitations of this work.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification. asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low images are taken in low lighting. Or speech-to-text system might be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
16

---

Justification:
This paper does not include theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proof' can either appear the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification:
Section fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, it may be necessary either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, or other means that appropriate to the research performed.
While NeurlPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome deseribe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
17

---

Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Question:
Answer: [No]
Justification: Our code will be open-sourced to the public after the paper is officially published.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips... cc/
oliccy for more details.
While encourage the release of code and data, understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https
//nips.ccpublle pwwppppguuuiiieeeep httpc///uuudess/deeuuuubiiis
(CodeSubiisss wwtplubmmsssionii ssioopolccy for more details.
The authors should provide instructions data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: Section [specify all the training and test details.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [No]
Justification: Error bars are not reported because it would be too computationally expensive.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
18

---

The method for calculating the error bars should be explained (closed form formula, call a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful figures symmetric error bars that would yield results that are out of range (e.g. negative rates).
show in tables or
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: We provide the execution time and computing device the proposed method in Section
A
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code of ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics http:///wwrrpp.cccooo/ollcccccee
Answer: [Yes]
Justification: This work conforms the NeurIPS Code of Ethics in all respects.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive socictal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
We discuss the broader impacts in Section
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
19

---

Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), faireess considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
This paper poses such risks.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g.. code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Yes, all external assets used in the paper (such as code, data, and models) properly credited, with their licenses and terms of use clearly stated and strictly respected.
Justification:
Guidelines:
The answer NA means that the paper does use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
20

---

If assets are released, the license, copyright information, and terms of use in the thcode. has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
package should be provided. For popular datasets, paperswi
batasets.papersstt
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. If this information is available online, the authors encouraged to reach out the asset's s creators.
13. New assets
Question: Are assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: We will release the code and related documentation upon the official publica- tion of the paper.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
This paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then much detail as possible should be included the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
This paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does involve crowdsourcing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
21

---

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, formatting purposes and does not impact the core methodology. scientific rigorousness, or originality of the research, declaration is not required.
Question:
Answer: [NA]
Justification: LLM is used only for writing, editing, formatting purposes.
Guidelines:
The answer NA means that the method development in this research does not involve LLMs as any important, original, or non-standard components.
Please refer to our LLM policy pettps://neurre httpa//rps.cc/oooefeeeee/e/////p cc/CConfereeccce/2225LLLL. for what should or should be described.
22


Reviews:
[{"review_id": "Ogs9Del9sH", "reviewer": "Reviewer_kx1e", "summary": "This paper proposes RayFusion, a collaborative perception framework for camera-only autonomous driving systems. It introduces a ray-based fusion strategy leveraging ray occupancy information across agents to resolve depth ambiguity in monocular 3D detection. The system includes spatial-temporal alignment, ray occupancy encoding, and multi-scale instance feature aggregation. Experiments on real and simulated datasets show improved accuracy and robustness under localization noise and communication delays.", "questions": "How sensitive is RayFusion to inaccurate or noisy inputs from collaborating vehicles? Have the authors considered simulating such failure modes (e.g., corrupted pose, false positives) to analyze the impact on ego vehicle perception accuracy?", "limitations": "YES", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "# Strengths\n\n1. Introducing ray occupancy encoding into collaborative perception is original and conceptually insightful. It addresses the well-known ambiguity in depth estimation from monocular views.\n2. The modular system design (STA, ROE, MIFA) is clean and well-motivated.\n3. Experiments demonstrate solid performance gains over prior methods in simulated environments.\n\n# Weakness\n\n1. While the proposed RayFusion framework shows promising performance under general settings, it does not explicitly address the challenges posed by crowded scenes, where overlapping instances and dense object distributions may significantly increase false positives and degrade the effectiveness of ray-based cross-view validation. Existing experiments do not isolate this regime for targeted analysis, and no specific design is introduced to mitigate perceptual conflicts in such cases. As a result, it remains unclear whether the proposed ray occupancy encoding can robustly distinguish between true and spurious detections under crowded traffic conditions.\n\n2. The proposed RayFusion framework relies on structured information (pose, velocity, detection outputs) from collaborating agents, assuming they are sufficiently accurate after spatial-temporal alignment (STA). However, in realistic scenarios, errors from other agents — such as inaccurate localization, erroneous depth estimation, or false positives in their predictions — can easily propagate through the collaboration pipeline and directly degrade the ego vehicle's performance.\n\nWhile the authors mention future work on \"unknown-pose settings\", the current method lacks mechanisms to assess or suppress unreliable information from noisy collaborators. This is especially concerning in safety-critical scenarios, where faulty inputs from a single vehicle could severely mislead perception for the ego vehicle. More discussion or empirical evaluation in such adversarial or noisy-agent settings would strengthen the paper.", "was_revised": true, "final_justification": "The rebuttal provides additional crowded-scene evaluations (>15 and >20 objects) showing consistent AP70 gains with ROE, partially resolving my concern about performance in dense environments. The analysis of corrupted poses and injected false positives is valuable and confirms improved robustness over baselines, though the method still suffers substantial degradation under severe collaborator errors and lacks explicit suppression mechanisms for unreliable inputs. While these clarifications strengthen the paper, my main reservations about vulnerability to noisy or adversarial collaborators remain. Therefore, my score is unchanged.", "rebuttal": "$\\textbf{W1: } $\n\nThank you for the insightful comment. We agree that crowded scenes present additional challenges for ray-based cross-view validation due to increased instance overlap and dense object distributions. However, the proposed ROE in our RayFusion framework is specifically designed to capture the probabilistic distribution of instance presence along each viewing ray, which helps mitigate such challenges. Concretely:\n\n1) For a real instance $obj_1$ in 3D space, false positive predictions generated by different agents typically lie along their respective observation rays that intersect at $obj_1$. This property allows effective filtering of multiple false positives related to $obj_1$.\n\n2) Suppose agent $i$ observes another real instance $obj_2$ along the observation ray of $obj_1$. However, the observation rays of $obj_2$ from other agents do not overlap with agent $i$’s observation ray of $obj_1$. Typically, there are two distinct intersection points corresponding to $obj_1$ and $obj_2$, so the true object $obj_2$ will not be mistakenly filtered out.\n\nTo support this, we conduct additional targeted evaluations on crowded scenes from the OPV2V dataset:\n\n1) We select 467 test frames in which more than 15 objects appear within the detection range. The AP70 scores with and without ROE are 74.32 and 71.24, respectively.\n\n2) We further select 50 congested frames with over 20 objects within the detection range. In this setting, the AP70 scores with and without ROE are 66.98 and 61.36, respectively.\n\nThese results demonstrate that RayFusion can effectively distinguish true positives from spurious detections even in highly crowded scenarios, confirming the robustness of the proposed occupancy-based encoding.\n\n\n$\\textbf{W2: } $\n\nRayFusion performs spatial-temporal alignment on instance information from multiple agents, aiming to obtain spatial-temporal aligned and accurate instances. However, due to the inherent ambiguity of monocular depth estimation, a single agent may suffer from false positives and radial localization errors. To address this, we propose ROE, which leverages cross-view validation to effectively mitigate depth ambiguity, reduce false positive predictions, and enhance the ego’s perception performance.\nIn addition, localization noise in real-world scenarios can significantly degrade the performance of collaborative systems. To tackle this issue, we introduce MIFA, which employs multi-window attention to capture both local and global spatial features in parallel. As shown in Figure 3(a), RayFusion exhibits strong robustness to localization noise.\n\n\n$\\textbf{W3, Q1: } $\n\nThank you for the valuable suggestion. A comprehensive evaluation of collaborative system robustness is crucial for deployment in real-world scenarios. Benefiting from instance features enriched with image semantics, MIFA uses multi-window attention to compensate for localization noise. In Figure 3, we analyze RayFusion’s detection performance under localization noise and communication delays, demonstrating its strong robustness against these factors.\nHowever, if agent $i$ predicts an instance $obj_3$ that cannot be seen by all other collaborators or the ego vehicle (e.g., due to severe occlusion), the model theoretically lacks the ability to identify or filter this instance. When $obj_3$ is visible to other collaborators or the ego vehicle but no corresponding prediction exists, we expect the model to learn to filter out such false predictions.\nHere, we further analyze the impact of corrupted poses and false positive predictions on perception accuracy:\n\n1) Corrupted poses: We simulate corrupted poses with localization noise drawn from a Gaussian distribution with mean 0 and variance 10 m, and heading noise with mean 0 and variance 10°. Under these conditions, RayFusion’s AP70 drops from 74.25 to 33.44 (–54.96\\%), while IFTR’s AP70 drops from 66.04 to 8.45 (–87.20\\%).\n\n2) False positives: We simulate false positive instances by replacing the anchor boxes of the top $N_{\\text{fp}}$ high-confidence predictions from collaborators with those from low-confidence predictions. These modified instances are then merged into the collaborative instances. As $N_{\\text{fp}}$ increases from 0 to 10, 50, and 100, RayFusion’s performance drops from 74.25 to 68.81, 67.68, and 66.42, respectively. This demonstrates that erroneous false positives from collaborators can indeed negatively impact model performance. In future work, we plan to further enhance robustness by integrating observations from multiple agents to suppress such unreliable predictions."}, {"review_id": "HzYweNxYnC", "reviewer": "Reviewer_4JMZ", "summary": "This paper proposes RayFusion, a novel method for collaborative camera-only perception. The key idea behind RayFusion is to reduce ambiguity through depth estimation by modeling occupancy along the rays of different agent cameras. To do so, object predictions and depth estimates are first made from the individual camera images which are then transformed to the ego coordinate system. Next, the camera rays are encoded using an MLP before being passed through a self-attention layer to generate a single ray encoding for each object instance. Lastly, the instance features undergo pyramid window self-attention, before being passed to the detection head. The authors evaluate their method on three collaborative perception datasets and demonstrate consistent performance improvement over the state-of-the-art.", "questions": "1. Given the reliance of this method on transformation of the instance features to a unified camera coordinate system, have the authors examined the effect of errors in the camera projection matrices?\n2. How exactly is the LiDAR point cloud used to supervise the dense depth prediction? This particular detail is a little under-explained in the paper.", "limitations": "yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n- The paper is well-written with clear figures.\n- The method is clever and well-motivated (leveraging multiple camera views to reduce depth ambiguity is natural), and the explanation is easy to follow\n- The experiments are thorough and show good performance across multiple datasets.\n- RayFusion demonstrates superior performance in settings with localization errors and communication delays over competing collaborative perception methods.\nWeaknesses:\n- The depth prediction head is supervised using LiDAR point clouds, meaning extra sensor data is required during training time.\n- According to ablation studies, the performance gain from solely the ray occupancy encoding (the key contribution of this method) is only moderate, suggesting that more can be done to reduce depth ambiguity.", "was_revised": true, "final_justification": "Authors responded to my comments and I maintain my accept score.", "rebuttal": "$\\textbf{W1: } $\n\nYour observation is absolutely correct — the depth prediction head is supervised using LiDAR point clouds, which indeed requires additional sensor data during training. However, the use of LiDAR supervision leads to significant performance gains [r1, r2]. In fact, camera-based models following the back-projection paradigm typically fail to converge in the absence of both a pre-trained image backbone and LiDAR point cloud supervision [r3].\n\n\n$\\textbf{W2: } $\n\nAs shown in Table 2, ROE improves AP70 by 2.21 (+3.07\\%) on OPV2V and 1.34 (+13.14\\%) on DAIR-V2X. For an already strong perception model, such performance gains are considered substantial. That said, we acknowledge that there is still room for further improvement in reducing depth ambiguity. For example, leveraging multi-frame information and depth hypothesis sampling has shown promise in addressing depth uncertainty [r4]. We plan to explore more effective strategies along these lines in future work. \n\n\n$\\textbf{Q1: } $\n\nIn camera-based perception systems, nearly all methods rely on complex projection operations based on intrinsic and extrinsic camera parameters, and inaccuracies in these projection matrices can significantly affect model performance. To provide a reference, we analyze the impact of errors in the camera extrinsic matrices on perception accuracy. Specifically, we simulate extrinsic perturbations using Gaussian noise with a mean of 0 and a variance of 0.02. Under these conditions, experimental results show that RayFusion’s AP70 drops from 74.25 to 24.11 (–67.53\\%), while IFTR’s AP70 drops from 66.04 to 20.89 (–68.37\\%).\n\n\n$\\textbf{Q2: } $\n\nThank you for pointing out the need for a more detailed explanation of how LiDAR point clouds are used to supervise dense depth prediction. We adopt a predefined depth range $d \\in [2\\mathrm{m}, 52\\mathrm{m}]$, uniformly divided into 80 bins. Our implementation follows similar principles to BEVDepth and FB-BEV. The process is as follows:\n\n1) We project the LiDAR point cloud into the pixel coordinate system using the camera intrinsics and extrinsics, resulting in a dense depth map $\\bar{\\mathcal{D}}_j^k$ and a depth mask $\\bar{\\mathcal{M}}_j^k$ for the $k$-th camera view of agent $j$. The mask $\\bar{\\mathcal{M}}_j^k$ is set to 1 at pixels where valid LiDAR points are projected within the predefined depth range, and 0 elsewhere.\n\n2) Based on the predefined bins and a downsampling ratio, we convert $\\bar{\\mathcal{D}}_j^k$ into per-pixel depth classification targets $\\tilde{\\mathcal{D}}_j^k$ and get a updated depth mask $\\tilde{\\mathcal{M}}_j^k$.\n\n3) The depth head predicts a per-pixel depth probability distribution $\\mathcal{D}_j^k$, which is supervised using cross-entropy loss only on the valid pixels where the updated depth mask is equal to 1. The final depth supervision loss is given by: $\\mathcal{L}\\_{\\text{depth}} = \\text{CrossEntropy}(\\mathcal{D}\\_j^k, \\tilde{\\mathcal{D}}\\_j^k, \\tilde{\\mathcal{M}}\\_j^k).$ \n\nFinally, the overall training loss is defined as: $\\mathcal{L} = \\lambda\\_{\\text{depth}} \\cdot \\mathcal{L}\\_{\\text{depth}} + \\mathcal{L}\\_{\\text{obj}} $, where $\\mathcal{L}\\_{\\text{depth}}$ and $\\mathcal{L}\\_{\\text{obj}}$ denote the depth supervision and object detection losses, respectively, and $\\lambda\\_{\\text{depth}}$ is a hyperparameter set to 0.5 by default.\n\n[r1] BEVDepth: Acquisition of Reliable Depth for Multi-view 3D Object Detection\n\n[r2] FB-BEV: BEV Representation from Forward-Backward View Transformations\n\n[r3] Sparse4D v2: Recurrent Temporal Fusion with Sparse Model\n\n[r4] Time Will Tell: New Outlooks and A Baseline for Temporal Multi-View 3D Object Detection"}, {"review_id": "wbbg54IQY2", "reviewer": "Reviewer_sx6F", "summary": "The paper proposes RayFusion, a novel cooperative perception framework that leverages ray information from multiple agents to resolve depth ambiguity in camera-only 3D detection. There are several significant concerns about the experimental design that weaken the paper's conclusions. These primarily relate to the fairness of baseline comparisons, the choice of datasets, the reliance on LiDAR supervision, and the limited perception range used in the experiments.", "questions": "See the weaknesses.", "limitations": "Yes.", "rating": 2, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 2, "originality": 3, "strengths_and_weaknesses": "Strengths:\n1. The paper provides comprehensive robustness evaluations under delay and noise conditions, as well as communication and computation efficiency analysis.\n2. The supplementary video provides a compelling demonstration of the model's performance.\n\nWeaknesses:\n1. Table 1 compares methods using different detection backbones: Sparse4D for the proposed method vs. BEVFormer for baselines cited from IFTR. This undermines the fairness of the comparison, as performance gains may stem from the stronger backbone rather than the proposed fusion strategy. A controlled comparison with a shared backbone is necessary for a new cooperative perception model.\n2. The authors acknowledge the limitations of the DAIR-V2X dataset. Why the paper does not benchmark its method against more recent and comprehensive real-world V2X datasets, e.g., TUMTrafV2X (V2I) and V2X-Real or V2XPnP (V2V, V2I, I2I). Relying heavily on simulated data and a limited real-world dataset makes it difficult to assess the method's practical effectiveness and its ability to bridge the gap between simulation and real-world data.\n3. Although presented as “camera-only,” the method uses LiDAR supervision during training for depth estimation. This contradicts the claim. An ablation study removing LiDAR supervision is needed to assess true independence.\n4. The perception range is approximately 50m in each direction, which is more characteristic of single-agent perception. To provide a more compelling demonstration of the benefits of the proposed cooperative approach, experiments should be conducted over a much larger range.", "was_revised": true, "final_justification": "Thank you for the authors' responses and clarifications. However, my concerns remain unresolved:\n\n### 1. **Core Contribution Evaluation**\n\nSince the primary contribution of this work lies in *RayFusion*, a multi-agent fusion strategy, the experimental evaluation should primarily focus on comparing RayFusion with other fusion approaches (denoted as *X*). There are two standard configurations for validating RayFusion’s effectiveness:\n\n- **BEVFormer + RayFusion vs. BEVFormer + X**\n- **Sparse4D + RayFusion vs. Sparse4D + X**\n\n**However, the current experimental setup compares**:\n\n- **BEVFormer + X vs. Sparse4D + simple fusion vs. Sparse4D + RayFusion.**\n\nWhile these results demonstrate the improvement of RayFusion over a simple fusion baseline within the Sparse4D framework, it remains unclear:\n\n- Whether **BEVFormer + RayFusion** suffers a performance drop compared to other baselines.\n- Whether replacing the backbone in existing baselines with **Sparse4D** significantly boosts their performance.\n- Whether **RayFusion** can generalize across **different detection backbones**.\n\nThese missing evaluations limit the strength of the claims regarding the superiority and generality of RayFusion.\n\n\n### 2. **Dataset Limitations**\n\nThe paper solely relies on **DAIR-V2X**, a dataset from 2022 with notable limitations:\n\n- Only includes one vehicle and one infrastructure agent.\n- Limited data volume and scene diversity.\n\nThese limitations have already motivated the development of more comprehensive datasets, such as:\n\n- **TUMTrafV2X** (CVPR 2024)\n- **V2X-Real** (ECCV 2024)\n\nThe paper actually acknowledges the limitations of DAIR-V2X, but it does not address them in practice. Evaluation on these **more diverse and modern datasets** is necessary to make the paper complete and demonstrate the performance and robustness of the proposed approach.\n\n---\n\n### **Conclusion**\n\nThese limitations significantly reduce the **persuasiveness** and **effectiveness** of the proposed method in the cooperative perception domain. In the future, addressing them will greatly enhance the quality of this work.", "rebuttal": "$\\textbf{W1: } $\n\nIn Table 1, we adopt BEVFormer as the single-agent detector for baseline methods, following IFTR, since most existing collaborative perception approaches are BEV-based. In contrast, RayFusion uses Sparse4D as the single-agent detector. The rationale is twofold:\n\n1) Resource constraints: BEV-based methods often demand high computational resources in camera-based collaborative perception tasks. For instance, training IFTR requires more than 40 GB of GPU memory on the OPV2V dataset when using the standard IFTR configuration. Due to limited training resources, we use the sparse detector Sparse4D as the single-agent detector to implement RayFusion, which only requires 12GB of GPU memory to complete training. Moreover, it offers significantly lower inference latency. As shown in Table 4, RayFusion improves AP70 by 8.21 (+12.43\\%) on OPV2V while reducing inference time from 298 ms to 114 ms.\n\n2) Higher performance potential under sparse paradigms: In BEV-based methods, the information flow for instance-level perception typically follows: image features → BEV features → instance features. In contrast, sparse detectors directly derive instance-level information from image features: image features → instance features. From an information-theoretic perspective, our instance-level fusion strategy is better suited for sparse detectors, as it incurs less information loss.\n\nTo further ensure that the performance gain does not stem solely from the stronger single-agent detector, we include an additional baseline in Table 1 named CoSparse4D, which uses Sparse4D as the single-agent detector without our proposed fusion strategy. RayFusion consistently outperforms this baseline across all datasets, supporting the effectiveness of our method. Furthermore, extensive ablation and robustness experiments also validate the contribution of our proposed methods. \nWe hope that the aforementioned rationale for adopting Sparse4D as the single-agent detector, together with the CoSparse4D baseline using the identical single-agent setup and our extensive ablation experiments, may help address your concerns.\n\n\n$\\textbf{W2: } $\n\nThank you for your valuable feedback. We acknowledge that the real-world DAIR-V2X dataset has limitations in terms of the number of collaborators and camera views. However, it remains the most widely used real-world dataset for camera-based collaborative perception [r1, r2], which makes it a standard benchmark for fair comparison with prior works.\n\nTo further evaluate the generalization capability of our approach, we have attempted to benchmark RayFusion on the V2X-Real dataset. However, due to the time-consuming data download process and inherent issues in the dataset — for example, some scenes are missing camera images (e.g., train/2023-03-17-15-53-02\\_1\\_0), and there are inconsistencies in camera parameter keys across frames (e.g., cam1\\_left in 000142.yaml vs. cam1 in 000001.yaml of the same sequence train/2023-04-04-15-58-18\\_30\\_0/2/). These issues caused initial training failures. We have since excluded the problematic samples and restarted training. Results from this evaluation can be included in the camera-ready version for reference.\n\n\n$\\textbf{W3: } $\n\nWe sincerely appreciate your valuable suggestion. However, camera-based models following the back-projection paradigm (e.g., BEVFormer and Sparse4D) typically fail to converge in the absence of both a pre-trained image backbone and LiDAR point cloud supervision [r3].  Moreover, applying LiDAR supervision during training incurs minimal additional cost, as point cloud data is still required during annotation even when LiDAR is not used for supervision. In addition, jointly collecting LiDAR and image data for depth supervision has become a standard practice and the community primarily emphasizes camera-only settings during the inference stage [r2, r4, r5]. \nTo eliminate ambiguity, we will revise the statement in the camera-ready version to specify that the inference phase operates under a camera-only configuration.\n\n\n$\\textbf{W4: } $\n\nWe adopt a perception range of $51.2\\mathrm{m}$ in each direction, following the setup in IFTR, to ensure a fair comparison. To more convincingly demonstrate the advantages of RayFusion, we conducte additional experiments with an extended range of $102.4\\mathrm{m}$ in each direction. Under this setting, RayFusion achieves an AP70 score of 48.10, outperforming CoSparse4D (41.73) and IFTR (38.41).\n\n[r1] IFTR: An Instance-Level Fusion Transformer for Visual Collaborative Perception\n\n[r2] Collaboration Helps Camera Overtake LiDAR in 3D Detection\n\n[r3] Sparse4D v2: Recurrent Temporal Fusion with Sparse Model\n\n[r4] Time Will Tell: New Outlooks and A Baseline for Temporal Multi-View 3D Object Detection\n\n[r5] Temporal Enhanced Training of Multi-view 3D Object Detector via Historical Object Prediction"}, {"review_id": "ZUYCpNxSlA", "reviewer": "Reviewer_E2kt", "summary": "This paper proposes RayFusion, a method that uses ray occupancy to improve 3D object localization in multi-agent perception. It includes spatial-temporal alignment, depth-aware ray encoding, and multi-scale feature aggregation to enhance collaboration and accuracy.", "questions": "Please refer to the **weaknesses** above.", "limitations": "yes", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 4, "significance": 4, "originality": 3, "strengths_and_weaknesses": "**Strengths**\n- Good presentation and clear methodology;\n- Significant performance gain over baselines;\n- The experiments are thorough;\n- The code is promised to be released, which will benefit the community.\n\n**Weaknesses**  \nThis is a solid piece of work. To be honest, I found it difficult to identify any major weaknesses, and the following are just some minor suggestions.\n- It would be helpful to see an ablation where the predicted depth distribution is replaced with a one-hot vector (e.g., peak bin only) or even a uniform distribution.\n- The detection performance is affected by the communication bandwidth between agents. It would be helpful if Table 1 could report the communication cost of all methods.\n- There are some minor typos (e.g., line 584: \"Additionally, We plan\").", "was_revised": true, "final_justification": "The rebuttal addresses most of my concerns, so I maintain the original rating.", "rebuttal": "$\\textbf{W1: } $\n\nThank you for the constructive suggestion. In ROE, the occupancy information encoding leverages the depth distribution predicted by the depth head to represent the probabilistic distribution of an object along the camera ray. Indeed, replacing the depth distribution with a one-hot vector helps further analyze the contribution of depth modeling to overall performance. We conduct this ablation study and find that replacing the depth distribution with a one-hot encoding (i.e., using only the peak bin) leads to a drop in AP70 on the OPV2V dataset from 74.25 to 73.12. This performance degradation suggests that modeling the full depth distribution provides richer information about the object’s potential 3D location along the ray, which enhances the network’s ability to accurately localize instances via cross-view validation.\n\n\n$\\textbf{W2: } $\n\nOn the OPV2V dataset, a single agent communication cost per frame for RayFusion, IFTR, and late fusion are 276.6 KB, 18.8 MB, and 2.7 KB, respectively. Other intermediate fusion methods incur a communication cost of approximately 64.0 MB per frame. We will include the communication cost of all methods in Table 1 of the camera-ready version. Thank you for the valuable suggestion.\n\n\n$\\textbf{W3: } $\n\nThank you for the careful suggestion. We will thoroughly proofread the paper in the camera-ready version to eliminate any grammatical or typographical errors."}]

Author rebuttal:

Meta-review: This paper proposes RayFusion, a collaborative camera-only perception framework that introduces ray occupancy encoding to mitigate depth ambiguity in monocular 3D detection. The approach is conceptually novel, cleanly designed, and shows consistent performance gains across OPV2V, DAIR-V2X, and related benchmarks. Reviewers highlighted that the method is well-motivated (4JMZ, E2kt) and empirically validated, with robustness demonstrated under localization noise and communication delays. Reviewer kx1e raised concerns about vulnerability to faulty collaborator inputs, but additional rebuttal experiments in crowded and noisy settings partially alleviated this concern. Reviewer sx6F noted limitations in dataset scope and baseline fairness, yet acknowledged the promise of the approach. While reliance on LiDAR supervision during training slightly blunts the “camera-only” claim, the inference pipeline remains camera-based and practically relevant. This paper has received mixed scores: 2,3,5,5. The AC has gone through all the discussions in detail, and overall, the introduction of ray occupancy fusion represents a meaningful advance for collaborative visual perception. The AC finally recommends acceptance.
Decision: Accept (poster)