RayFormer: Improving Query-Based Multi-Camera 3D Object Detection via Ray-Centric Strategies

Authors: xiaomeng  chu
yanyong  zhang
Venue: acmmm_org_ACMMM_2024_Conference
Published: 2024-04-06 10:06:55

Abstract: The recent advances in query-based multi-camera 3D object detection are featured by initializing object queries in the 3D space, and then sampling features from perspective-view images to perform multi-round query refinement. In such a framework, query points near the same camera ray are likely to sample similar features from very close pixels, resulting in ambiguous query features and degraded detection accuracy. To this end, we introduce RayFormer, a camera-ray-inspired query-based 3D object detector that aligns the initialization and feature extraction of object queries with the optical characteristics of cameras. Specifically, RayFormer transforms perspective-view image features into bird’s eye view (BEV) via the lift-splat-shoot method and segments the BEV map to sectors based on the camera rays. Object queries are uniformly and sparsely initialized along each camera ray, facilitating the projection of different queries onto different areas in the image to extract distinct features. Besides, we leverage the instance information of images to supplement the uniformly initialized object queries by further involving additional queries along the ray from 2D object detection boxes. To extract unique object-level features that cater to distinct queries, we design a ray sampling method that suitably organizes the distribution of feature sampling points on both images and bird’s eye view. Extensive experiments are conducted on the nuScenes dataset to validate our proposed ray-inspired model design. The proposed RayFormer achieves 55.5% mAP and 63.4% NDS, respectively.

Main paper: 
 introduction: 
 Multi-camera perception, with its omnidirectional view and lowcost deployment, has emerged as a promising solution for autonomous vehicles (AVs). As a critical component of AVs' cameracentric perception systems, multi-camera 3D object detection has garnered significant research interest in recent years [1, 28, 33, 39, 42] . Within the exploration of multi-camera 3D object detectors, query-based approaches [4, 11, 20, 23, 34, 38] achieve inspiring accuracy, securing a pivotal role in contemporary methods.Typically, a query-based multi-camera 3D object detector begins by initialing object queries as learnable random parameters or as grids uniformly distributed in Cartesian coordinates. The query features are then iteratively refined either by abstracting multiview image features with a global attention layer or by projecting the 3D query onto the image view to sample the corresponding image features. In this work, we follow the projection and sampling paradigm to perform query feature refinement, as it avoids the large computation overhead introduced by global attention and convergence faster at the training stage.However, due to the optical characteristics of the camera, 3D positions in an instance frustum tend to project into closely positioned image areas [32, 37] . This presents a dilemma: while we aim for each query to represent an independent object, during the backward projection for feature sampling, multiple independent queries in a grid-like distribution within an instance frustum may project onto the same object, leading to similar features, as depicted in Fig. 1(a) . Additionally, the farther away from the camera, the more queries are included within the instance frustum, resulting in a large number of location-distinct queries with semantic-similar features, thus affecting the accuracy of 3D object detection. In this work, we explore how to alleviate the issues outlined above. We find that these problems originate from the practice of grid-like query initialization (i.e., initializing the location of object queries within the Cartesian coordinate system). Such initialization overlooks the actual correspondence between the image and 3D space, where each pixel aligns with a frustum along the camera ray. To this end, we re-formulate the initial location of the object query by well considering the optical characteristics of cameras. As shown in Fig. 1 (b), the radial distribution simulates the camera's imaging rays, effectively mitigating the problem of multiple queries projecting onto the same object in the images. Additionally, the number of queries within the frustum does not markedly increase as the distance extends. Moreover, despite that the location of object queries is improved from the radial distribution, queries at different distances along the same camera ray may still sample similar features. To further address this problem, constructing and sampling features from a second perspective, e.g., BEV, help these queries extract distinctive features. Furthermore, each independent query should extract object-level features from a nearby area in the images, thus the sampling points for the query should be placed along the camera ray, aligning with the query's distribution traits.Formally, we introduce RayFormer, a query-based multi-camera 3D object detection approach that initializes sparse queries in a radial distribution and organizes sampling points along a ray segment for each query to extract both image and BEV features. Specifically, RayFormer leverages image features and predicted depth distribution to generate BEV features via the lift-splat-shoot [31] method.Using the ego car as the center, the perceptual field is divided by dense camera rays, whereupon the base query points are uniformly and sparsely initiated. Besides, we exploit a 2D prior knowledge to supplement the base queries. This is achieved by categorically setting ray densities and projecting their midpoints onto the images. Subsequently, we select rays intersecting the predicted 2D bounding boxes, procuring foreground queries from these rays. Moreover, the feature sampling scheme should align with the query distribution, ensuring each query extracts object-level features. Given the optical properties of the camera, we no longer select sampling points around queries' locations. Instead, we delineate a segment on the ray at each query's position as sampling units. Our design offers the following benefits. It aligns the initialization with the feature sampling pattern of queries following the optical characteristics of the sensors. Meanwhile, queries along a camera ray embed distinct 3D positional features by sampling the BEV features.To demonstrate the effectiveness of our proposed RayFormer, we evaluate its performance on the challenging nuScenes benchmark [2] . Without bells and whistles, our approach achieves 55.5% mAP and 63.3% NDS on the test set, which improves the baseline SparseBEV by 1.2% and 0.6%, respectively.In summary, our main contributions are as follows:• We propose RayFormer, a novel multi-camera 3D object detection paradigm with a ray-like query initialization and the feature sampling which takes ray segments as units, mirroring the inherent optical characteristics of cameras. • We construct dense features from both the image and bird'seye views for feature sampling. Furthermore, we incorporate queries on the foreground rays selected by 2D object detection, without the reliance on inaccurate depth estimations. • We conduct experiments on the challenging nuScenes dataset.Under the same backbone and training configuration as other studies, RayFormer achieves state-of-the-art performance on the test set, with mAP at 55.5% and NDS at 63.3%. 
related work: 
 In this section, we give a brief review of polar representation for 3D perception and visual 3D object detection, including BEV-based 3D object detection and query-based 3D object detection.BEV-based 3D Object Detection: BEV-based 3D Object Detection has seen various innovative approaches, mainly categorized by their view transformation techniques-either through forward projection [44, 46] , which pioneers the use of the LSS method for feature projection onto BEV, or backward projection techniques [3, 41] . BEVDet [10] is distinguished by its unique data augmentation and refined non-maximum suppression. BEVDepth [16] extends this foundation by incorporating depth estimation supervision and a depth refinement module. BEVFormer [17] further advances the field by dividing the BEV into grids, employing deformable crossattention [49] to construct BEV features with temporal integration for enhanced detection capabilities [9, 14, 30] . FB-BEV [18] enhances these techniques by introducing a forward-backward view transformation module, while HOP [50] innovates with temporal decoders and object prediction using pseudo BEV features, capturing both spatial and temporal aspects of object motion. 2 : Overall architecture of RayFormer. Upon inputting multiple frames of multi-camera images into the image encoder, we extract multi-scale image features. These features are processed by a 2D detection head and a depth head to obtain 2D bounding boxes (bboxes) and depth distributions 𝐷 ′ , respectively. The image features and depth distributions are fed into the lift-splat module for forward projection to generate BEV features. We expand the height of the detected 2D bboxes and use them to select foreground rays. On these rays, a specific number of foreground queries are selected. Along with the radially distributed base queries, all queries are fed into the transformer decoder and refined 𝐿 times. The core module of the decoder, ray sampling, sets the sampling points along the camera ray, extracting both image and BEV features. Finally, queries are decoded by the classification head and the regression head for accurate predictions.Query-based 3D Object Detection: DETR3D [38] and PETR [23] utilize transformer decoders to process image features, where the former projects 3D object queries onto 2D features, and the latter integrates the positional embedding of 3D coordinates into image features, yielding 3D position-aware features. StreamPETR [36] builds on PETR with an object-centric temporal mechanism for long-sequence modeling, utilizing historical data through object query propagation. MV2D [40] improves multi-camera detection by generating queries from 2D object detections based on image semantics. Sparse4D [20] meticulously refines anchor boxes through sparse amalgamation and sampling of spatial-temporal features, assigning multiple 4D keypoints to each 3D anchor and fusing these features hierarchically across different views and scales. Sparse-BEV [22] introduces a fully sparse 3D object detection framework that fuses scale-adaptive attention and integrates adaptive spatiotemporal sampling and mixing.Polar Representation for 3D Perception: In the realm of 3D perception, studies leveraging polar representation have shifted from Cartesian to polar coordinates for BEV feature construction. PolarNet [43] pioneers a segmentation approach using a polar BEV representation, optimizing point distribution across grid cells. Polar-BEV [25] enhances processing efficiency through polar embedding decomposition and height-based feature transformation on a hypothetical plane. PolarFormer [12] addresses irregular polar BEV grids and varying object scales with a cross-attention-based detection head and multi-scale learning strategy. TaDe [45] presents a two-step approach that uses an autoencoder to reconstruct noisy BEV maps, optimizes feature correlations, and converts BEV maps into polar coordinates for enhanced alignment. Contrary to these methods, we construct an implicit query-based polar representation, instead of the BEV feature, through sparse radial query initialization, complemented with a ray sampling method that aligns with the radial query distribution. 
approach 3.1 overall framework: 
 As depicted in Fig. 2 , RayFormer introduces a novel approach to multi-camera 3D object detection by employing sparse queries to sample features from dual perspectives. The architecture encompasses an image encoder, a 2D detection head, a depth head, and a transformer decoder [5, 35] . The core of the decoder comprises scale-adaptive self attention [22] , BEV ray-sampling, image raysampling, and adaptive mixing [7] , with the add & norm layers and feed-forward network not shown in Fig. 2 for brevity. The process begins with the input of multi-frame, multi-camera images into the image encoder, which consists of a 2D backbone and an FPN, to extract multi-scale image features. These features are then processed by the 2D detection head and depth head to yield 2D bounding boxes and depth distributions. Employing the Lift-Splat technique, we transform the image features and depth distributions from a forward perspective into BEV, resulting in the generation of BEV features. We initiate 𝑁 𝑏 base queries with a radial distribution. In parallel, the height of the 2D detection boxes is extended to the full image height, facilitating the selection of corresponding foreground rays on BEV. On these rays, 𝑁 𝑓 queries are chosen. These 𝑁 𝑏 + 𝑁 𝑓 queries are then fed into the transformer decoder. Through scaleadaptive self attention, the queries are encoded with dynamically adjustable receptive fields. Following this, each query, via linear layers, generates multiple sampling points along a specified ray segment, which are then employed to sequentially sample features from BEV and image plane. The image features are aggregated by adaptive mixing and fused with BEV features. In the final step, the refined properties of the queries, as decoded by the classification and regression heads, enable precise object detection.We employ the lift-splat-shoot (LSS) method, with depth supervision from the point cloud, as outlined in BEVDepth [16] . This allows us to convert image features into BEV features, presented in Cartesian coordinates. The LSS method innovatively lifts input images into a pseudo point cloud via depth estimation. Subsequently, it aggregates the points within a BEV grid by sum pooling. This procedure is mathematically represented as 𝐵𝐸𝑉 (𝑥, 𝑦) = 𝑖 𝑓 𝑖 • 𝛿 (𝑥 𝑖 -𝑥, 𝑦 𝑖 -𝑦), where each point 𝑖 contributes its features 𝑓 𝑖 to the grid according to its position.To boost depth estimation precision, we use a linearly increasing discretization method to subdivide the depth range [𝑑 𝑚𝑖𝑛 , 𝑑 𝑚𝑎𝑥 ] into varying intervals, each representing a unique category. This approach transforms depth prediction into a more manageable classification challenge within an ordinal regression framework [6] , using a bin size of 𝜎:EQUATIONwhere 𝐾 is the number of depth bins, and l is the bin index.Owing to the optical properties of the camera, 3D points along a camera ray map onto a closely adjacent area within the image. Centered on the ego vehicle, we use polar coordinates to represent the BEV perception space, that is, first divide [0, 𝑅] into 𝑁 𝑟 segments, where 𝑅 is the maximum radian value of FoV (field of view), and then divide evenly [0, 𝐷] into 𝑁 𝑑 segments, where 𝐷 is the maximum value of detection depth range. Consequently, we define the BEV perception area as a circular region with a radius of 𝐷 and partition the BEV perception field using rays and circles instead of conventional grids. Moreover, the number of queries set along the rays should be small to inhibit the independent queries from sampling similar features. Specifically, we establish a relatively large 𝑁 𝑟 to densely segment the radian interval, and a relatively small 𝑁 𝑑 to partition the depth range sparsely. Queries are represented by a combination of C-dimensional features and 3D bounding boxes (bboxes), each of the latter includes four attributes: translation (𝑟, 𝑑, 𝑧), dimensions (𝑤, 𝑙, ℎ), rotation 𝛼, and velocity (𝑣 𝑥 , 𝑣 𝑦 ). For each frame, we uniformly place 𝑁 𝑑 queries {𝑄 1 , 𝑄 2 , ..., 𝑄 𝑑 } along the rays, where 𝑁 𝑑 is significantly smaller than the number of rays. Given that we set only a limited number of queries on each ray to extract more comprehensive features of the object, the sampling range for each query is set to be a ray segment. This segment's endpoints are the midpoints between the location of the query and the locations of the two adjacent queries. Along the ray segment, we adaptively select several ray sampling points to extract features from both perspectives, namely the image view and the BEV. In addition to the current timestamp, 𝑇 -1 historical frames are input to obtain temporal features. We employ a constant velocity model to simulate the movement of objects and dynamically wrap the ray sampling points back to prior timestamps utilizing the velocity vector [𝑣 𝑥 , 𝑣 𝑦 ] from the queries [22] :EQUATIONIt is worth noting that the polar coordinate system is our default choice. For feature sampling and temporal wrapping, we transform to Cartesian coordinates. Subsequently, queries are converted back to polar coordinates for query refinement. The specific ray sampling process is illustrated in Fig. 3 . For each query, we select 𝐾 equally spaced points between the endpoints of the defined ray segment and use the query feature to output the offsets for each ray point through a linear layer. By adding the position of K-equal division points and the ray-point offsets, we obtain the adaptive ray points, which are then wrapped to the 𝑇 -1 previous frames to extract temporal features. Based on the size and orientation attributes of the query bounding boxes, the coordinates of each adaptive ray point are converted into Cartesian coordinates. This conversion facilitates the generation of 𝑃 sampling offsets {(Δ𝑥 𝑖 , Δ𝑦 𝑖 , Δ𝑧 𝑖 )} through a linear layer, resulting in 𝑁 × 𝑇 × 𝐾 × 𝑃 ray sampling points. These points are used for sampling image and BEV features, where 𝑁 represents the number of queries. The sampled multi-frame, multi-point image features are aggregated to 𝑁 ×𝐶 through adaptive mixing. The aggregated image features and BEV features are finally concatenated and fused by a linear layer.BEV Ray Sampling: The historical BEV features with the dimension of 𝐻 𝐵 × 𝑊 𝐵 × 𝐶 are wrapped into the ego coordinate system, forming the features 𝐹 𝐵 = {𝐹 𝑏 𝑡 |𝑡 ∈ {1, 2, ...,𝑇 }}. We employ deformable attention [48] to sample weighted multi-frame BEV features by 𝑄 𝑝 . The expression is as follows:EQUATIONwhere 𝐹 𝑏 𝑡,𝑗 | 𝑗 ∈ {1, 2, ..., 𝑁 𝑠 }} denotes the BEV feature point at one of the 𝑁 𝑠 sampling locations of the t-th frame with the weight 𝑤 𝑡 .Image Ray Sampling: Taking multi-camera images from multiple timestamps as input, the image encoder outputs 𝑇 -frame, 𝐿-scale features of these |Γ|-view images. The sampling point 𝑝 𝑗 along the query 𝑄 𝑝 's ray segment projects onto the |Γ|-view, 𝐿-scale image features 𝐹 𝑚 at the t-th timestamp to extract the corresponding pixel's image feature 𝑓 𝑡,𝑗 :EQUATIONwhere M is the set of camera's projection matrix. 𝑤 𝑖,𝑙 is the weight for the l-th scale in the i-th view.For each query, we follow the AdaMixer [7] approach by aggregating the 𝑇 × 𝑁 𝑠 feature values through Adaptive Mixing. For the specific implementation, our adaptive mixing is based on Sparse-BEV [22] , which elevates the method from 2D to 3D perception. Adaptive mixing comprises two modules: channel mixing and point mixing, which thoroughly decode and aggregate spatio-temporal features across feature channels and point sets.Radian Cost for 3D Assigner: Considering the detailed segmentation of the angular coordinate by rays on the BEV, we aim to primarily refine queries along the depth axis. This effectively redefines the challenge of 3D object localization as a depth estimation task. Therefore, during each transformer layer's query refinement process, we confine the object's positional adjustments within a specified frustum. When aligning predicted 3D objects with their actual counterparts, the angular disparity between normalized i-th predicted radian 𝜃 𝑖 and j-th ground-truth 𝜃 ′ 𝑗 is factored into the matching cost Φ, which is expressed as follows:EQUATIONwhere 𝑤 𝑐 , 𝑤 𝑏 and 𝑤 𝑟 are the weights of matching cost for classification Φ 𝑐 , box attributs Φ 𝑏 , and radian Φ 𝑟 , respectively.In addition to the uniformly distributed base queries described in Sec 3.3, we use 2D object detection to guide the initialization of foreground queries, which are then merged with the base queries for prediction. As shown in Fig. 4 (a), by utilizing camera intrinsics, we observe that 2D projections from 3D positions on a BEV ray form a near-vertical line, facilitating the detection of 3D objects by checking if any point on the ray aligns with the object's vertical image region. To do this, we extend the detected 2D bounding boxes to the full height of the image to define a broad foreground region. As a result, on the BEV map, we pre-define uniformly distributed rays of varying densities according to the prior dimensions of different categories. We select rays whose midpoints can be projected into the foreground region of each category as the foreground rays. Thus, without relying on the accuracy of depth estimation, the results of 2D object detection alone can guide the initialization of additional queries on these foreground rays. This increases the density of queries around real objects. Specifically, for category 𝐶, we divide the BEV map into 𝑁 𝑐 rays based on its prior dimensions. As shown in Fig. 4 (b), in the nuScenes [2] dataset, cars have relatively large prior dimensions, resulting in relatively sparse 𝑁 𝑐𝑎𝑟 rays, while pedestrians, with relatively small sizes, have a larger number of 𝑁 𝑝𝑒𝑑 rays. Next, we project the midpoint 𝑄 ′ 𝑚 of each ray onto the image. On each ray whose midpoint falls in the foreground region, we set 𝑁 ′ 𝑑 queries, which are merged with the base ray queries. Finally, all queries are input into the transformer decoder for subsequent prediction. We conducted experiments on the expansive nuScenes autonomous driving dataset [2] , a resource rich in perception challenges such as object detection, motion tracking, and LiDAR-based segmentation. This dataset is organized into a total of 1,000 instances, systematically divided across training (700 instances), validation (150 instances), and testing (150 instances). Each instance captures 20 seconds of detailed sensory observations, marked with key-frames twice per second. The sensor suite mounted on the data collection vehicle includes a single LiDAR unit, a hexad of cameras providing complete environmental coverage, and a quintet of radars. For evaluating the precision of object detection, the nuScenes benchmark has established a suite of true positive metrics: Average Translation Error (ATE), Average Scale Error (ASE), Average Orientation Error (AOE), Average Velocity Error (AVE), and Average Attribute Error (AAE), targeting assessment of positional, dimensional, angular, motional, and categorical accuracy, respectively. Furthermore, the Mean Average Precision (mAP) and the holistic nuScenes Detection Score (NDS) serve as key indicators of detection efficacy, with higher values signifying more accurate performance. 
implementation details: 
 The perception area for BEV is confined to a circle with a 65𝑚 radius, segmented by 135 base rays in polar coordinates. On each ray, which ranges from 0 to 65, we uniformly select 6 queries. Additionally, 30 foreground rays are selected by 2D object detection with 3 points on each ray. This results in a total of 900 queries, which align with previous methods [20, 22, 36] . The architecture of the transformer decoder is designed with 6 layers with shared weights across these layers for efficiency. During ray sampling, we select K points dynamically between midpoints of consecutive queries on the ray. Each ray point, processed through a linear layer, generates 4 sampling offsets. We set K to 5 for BEV feature sampling and 3 for image feature sampling. The Hungarian algorithm is employed to assign predictions to ground truths by utilizing a focal loss [19] as the cost for classification, L1 loss as the cost for both box and angular regression, with the coefficients set at 1, 0.25, and 1, respectively. We generate a depth distribution that is downsampled by a factor of 16 relative to the input. Subsequently, the LSS method is utilized to produce BEV features in Cartesian coordinates. The size of the BEV features is determined by the input image resolution; for a resolution of 1600 × 640, the feature size is 256 × 256, and for other resolutions, it is 128 × 128. In the comparison with other studies, we use T = 8 frames, with an approximate interval of 0.5 seconds between adjacent frames. For ablation studies, we default to conduct experiments with T = 1 frame to validate the function of each module. Futhermore, we train SparseBEV [22] as the baseline under our settings. For a fair comparison, we set the number of sampling points per query point in SparseBEV to 12, aligning it with the total number of sampling points selected along ray segment for each query in our RayFormer. To achieve rapid convergence, we incorporate the query denoising strategy from PETRv2 [24] . RayFormer is trained using the AdamW [27] optimizer with a global batch size of 8. The initial learning rates for the backbone and other parameters are set at 2e-5 and 2e-4, respectively, with a cosine annealing [26] decay strategy. We utilize ResNet [8] and VoVNet-99 (V2-99) [13] as our backbone networks for feature encoding. ResNet parameters are pretrained on nuImages [2] , and V2-99 parameters are initialized from DD3D [29] . All models are trained for 24 epochs without the use of CBGS [47] or test time augmentation.Comparison with the State-of-the-art Methods: Among all the non-temporal models, RayFormer achieves the highest scores in mAP and NDS on the nuScenes validation set, showing significant improvements of 2.1% and 0.4% respectively over SparseBEV, as illustrated in Tab. 3. The comparison results with multi-frame input (T=8) on the val set are presented in Tab. Visualization Results: Fig. 5 showcases a system visualization with an input resolution of 512 × 1408, using ResNet101 as the backbone. The BEV layout on the right captures the full scene, with blue and green boxes indicating predictions and ground truths. The left side shows 3D bounding box projections onto the image plane, with yellow, blue, and red coding for cars, pedestrians, and bicycles, respectively. Missed detections are outlined in dashed red, matching the BEV view. Fig. 6 illustrates the ray sampling method, marking high-scoring query sampling points on the BEV plane and images with red dots. These points exhibit a radial pattern on the BEV and a vertical alignment in images, focusing on object areas due to the radial sampling strategy.Image Ray Sampling: Tab. 4 initially evaluates the impact of altering the initialization of the baseline SparseBEV from grid-like (A) to radial (B), while retaining the common practice that generates several sampling points around each query. We find that radial initialization, without changing the sampling method, impairs efficient object-level feature extraction due to the sparse query Foreground Query Supplement: We present the results of 2D guided foreground query supplement in Tab. 4. To avoid ambiguity caused by the gains from the simultaneous 2D perception learning, we first compare the accuracy changes when solely using 2D object detection as an auxiliary learning task in (D) and (E), observing an increase of 0.2% in mAP and 0.8% in NDS. With the addition of foreground queries selected on the rays that intersect the predicted 2D bounding boxes (F), mAP and NDS further increase by 0.3% and 0.6%, respectively, while mATE decreases by 0.7%.Initialization and Sampling Configuration: In Tab. 6, we simplify our RayFormer by excluding the 2D auxiliary task and 2Dguided query enhancement, focusing on the core performance. The Increasing the number of rays improves performance, as seen in the progression from (A) to (C). Specifically, using 6 queries per ray (D) over 3 queries (B) increases mAP and NDS by 0.9% and 1.0%, respectively. Adjusting the adaptive ray points per query, as in (B) versus (E), has minimal impact. However, increasing the sampling offsets per ray point from 1 to 4, as in (B) versus (F), significantly enhances mAP and NDS by 1.1%. We select configuration (B) as the default for its optimal balance of precision and efficiency.The impact of multi-frame input on detection performance is evident in Tab. 7. Adding a single historical frame to the key-frame input results in a notable increase in mAP by 4.1% and a significant boost in NDS by 7.4%, primarily due to a marked reduction in mAVE. Continued increase in the number of historical frames further reduces mAVE and mATE, highlighting the crucial contribution of sequential inputs to velocity estimation and the overall improvement in object detection accuracy. 
conclusion: 
 In this paper, we fully consider the principles of camera optics and introduce RayFormer, which enhances multi-camera 3D object detection via ray-centric strategies. It initializes queries radially and selects sampling points on the associated radial segments to sample image features along with BEV features generated by the lift-splatshoot method. Moreover, RayFormer presets varying ray densities on the BEV based on the categories and extends the predicted 2D bounding boxes to select foreground rays, providing additional queries close to the locations of actual objects. Extensive experiments on the nuScenes dataset with RayFormer show significant improvements over the baseline SparseBEV, highlighting the efficacy of ray-centric query initialization and feature sampling in enhancing multi-camera 3D object detection performance. 
 
 Appendix:  


Decision: The innovative approach uses the polar coordinate system to select and define queries and obtain keys and values from BEV features, which might inspire future research. Thorough experiments have been conducted, with the main results demonstrating the effectiveness of the method. The ablation study highlights the effectiveness of all modules involved in the approach. Despite these advantages, the approach itself is complicated in initialization, where computational overheads should be added to the final version. 

Overall, the pros outweigh the cons. All reviewers also agree to accept this paper.  Thus, the AC suggests accepting this paper.

Reviews: review: paper_summary: paper_summary: RayFormer is a novel query-based multi-camera 3D object detector that improves accuracy by aligning query initialization and feature extraction with camera optical characteristics. It transforms image features into bird’s eye view (BEV) and segments the BEV map based on camera rays, enhancing object detection by extracting distinct features for each query. Evaluated on the nuScenes dataset, RayFormer achieves state-of-the-art performance, significantly surpassing previous methods.
strength_weakness: strength_weakness: 1. Novel idea: The approach is innovative by selecting and defining queries from the perspective of the polar coordinate system and obtaining keys and values from BEV features.
2. Thorough experiments: The main experimental results clearly demonstrate the effectiveness of the method, while the ablation study highlights the effectiveness of all modules.
limitations: limitations: 1. Writing clarity: The principles behind Query Initialization and Ray Sampling are not effectively illustrated with images, making the article difficult to understand.
2. Experimental results: Although the experimental results have shown significant improvement, they still lag behind the current leaderboard on nuScenes, including methods like BEVFormer v2 and VideoBEV.
paper_summary: paper_summary: The paper introduces RayFormer, a 3D object detection framework that leverages ray-centric strategies to address feature ambiguity in multi-camera setups. The proposed method aligns query initialization and feature extraction with the optical characteristics of cameras, transforming perspective-view image features into bird's eye view (BEV) and segmenting the BEV map based on camera rays. The RayFormer is validated on the nuScenes dataset.
strength_weakness: strength_weakness: The paper proposes a novel approach by using a ray-centric query initialization and sampling strategy, which is well-aligned with the optical characteristics of cameras.

 The authors provide extensive experimental results on the nuScenes dataset, demonstrating the effectiveness of their method.

The paper is well-structured, and the methodology is clearly explained, making it easier for readers to follow.
limitations: limitations: While the ray-centric approach is innovative, the overall improvement in performance metrics (mAP and NDS) over the baseline SparseBEV is relatively modest (1.2% and 0.7%, respectively). The practical impact of these improvements should be more critically assessed.

The proposed method introduces significant complexity in the initialization and sampling process. The paper should provide a more detailed analysis of the computational overhead and its implications for real-time applications.
paper_summary: paper_summary: RayFormer is a novel 3D object detection model that improves the accuracy of multi-camera detection systems by aligning object queries with the optical characteristics of cameras. It introduces a unique approach by initializing queries along camera rays. This method reduces feature overlap and enhances distinctiveness, leading to improved detection performance as demonstrated on the nuScenes dataset, where RayFormer achieves significant improvements over existing benchmarks.
strength_weakness: strength_weakness: 1. RayFormer is built on a solid rationale that addresses a key challenge in 3D object detection — the ambiguity of features among overlapping object queries in multi-camera systems. The model targets a fundamental problem that can significantly enhance detection accuracy by focusing on this issue.
2. The approach of RayFormer is logically structured, aligning the initialization of object queries with the optical characteristics of cameras.
limitations: limitations: 1. While RayFormer introduces a ray-based query initialization as its core innovation, the performance improvements reported significantly rely on BEV sampling and initialization from 2D detection. This raises concerns about whether the primary contribution of ray-based querying is robust enough. By simply changing the distribution of queries from a Cartesian coordinate system to a polar coordinate system, and then querying image features for 3D detection like BEVFormer, how much improvement can be achieved?
2. RayFormer's method of initializing 3D detection queries from 2D detection results has been widely used in many methods[1]. This reduces the innovation aspect of the approach.

[1] Object as Query: Lifting Any 2D Object Detector to 3D Detection