RadarOcc: Robust 3D Occupancy Prediction with 4D Imaging Radar

Authors: fangqiang  ding
xiangyu  wen
yunzhou  zhu
yiming  li
chris xiaoxuan lu
Venue: NeurIPS_cc_2024_Conference
Published: 2024-04-23 16:56:30

Abstract: 3D occupancy-based perception pipeline has significantly advanced autonomous driving by capturing detailed scene descriptions and demonstrating strong generalizability across various object categories and shapes. Current methods predominantly rely on LiDAR or camera inputs for 3D occupancy prediction. These methods are susceptible to adverse weather conditions, limiting the all-weather deployment of self-driving cars. To improve perception robustness, we leverage the recent advances in automotive radars and introduce a novel approach that utilizes 4D imaging radar sensors for 3D occupancy prediction. Our method, RadarOcc, circumvents the limitations of sparse radar point clouds by directly processing the 4D radar tensor, thus preserving essential scene details. RadarOcc innovatively addresses the challenges associated with the voluminous and noisy 4D radar data by employing Doppler bins descriptors, sidelobe-aware spatial sparsification, and range-wise self-attention mechanisms. To minimize the interpolation errors associated with direct coordinate transformations, we also devise a spherical-based feature encoding followed by spherical-to-Cartesian feature aggregation. We benchmark various baseline methods based on distinct modalities on the public K-Radar dataset. The results demonstrate RadarOcc's state-of-the-art performance in radar-based 3D occupancy prediction and promising results even when compared with LiDAR- or camera-based methods. Additionally, we present qualitative evidence of the superior performance of 4D radar in adverse weather conditions and explore the impact of key pipeline components through ablation studies.

Main paper: 
 introduction: 
 The safety of autonomous vehicles navigating in the wild hinges on a thorough understanding of the environment's 3D structure. As a unified scene representation built from grid-based volumetric elements known as voxels, 3D occupancy has gained increasing attention within the autonomous driving community [1] [2] [3] [4] [5] . Its rising popularity stems from its comprehensive scene depiction, capturing both geometric and semantic aspects. Crucially, it transcends the limitations of foregroundonly representations (vs. 3D object detection [6] [7] [8] ) and sparse data formats (vs. point cloud segmentation [9] [10] [11] ). Furthermore, 3D occupancy offers a detailed open-set depiction of scene geometry, effectively handling out-of-vocabulary items (e.g., animals) and irregular shapes (e.g., cranes). This capability allows it to address a broader range of corner cases than previous object-based perception approaches [12] [13] [14] .Previous research has predominantly utilized either LiDAR point clouds [2, [15] [16] [17] [18] [19] [20] [21] [22] , RGB images [4, 5, [23] [24] [25] [26] [27] [28] [29] [30] [31] [32] [33] , or a combination of both [3] for 3D occupancy prediction. However, the potential of 4D imaging radar [34, 35] -a critical sensor in autonomous driving-has been largely untapped in this area. Evolving from traditional 3D mmWave radars, this emerging sensor type enhances elevation resolution, enabling detection and resolution of targets across both horizontal and vertical planes, which results in detailed imaging outputs. Meanwhile, 4D radar inherits the traditional advantages of mmWave radar, such as capability in all lighting and weather conditions, object velocity measurement, and cost-effectiveness compared to LiDAR systems. These attributes, particularly its resilience in adverse weather conditions like fog and rain, position 4D radar as an essential component in achieving mobile autonomy.In this work, we explore the potential of 4D imaging radar to enhance 3D occupancy prediction. Previous research in radar perception has largely relied on 4D radar point clouds as input, a method inspired by LiDAR techniques. This 'LiDAR-inspired' framework has demonstrated effectiveness in tasks such as 3D object detection and tracking . However, this approach primarily enhances the detection of foreground objects such as cars, pedestrians, and trucks. In contrast, 3D occupancy prediction requires the detection of signal reflections from all occupied spaces, encompassing both foreground and background elements like roads, barriers, and buildings. The traditional reliance on sparse radar point clouds, therefore, is not optimal for 3D occupancy prediction, as critical environmental signals are often lost during the point cloud generation process [59, 60] . For instance, the surface of highways, typically made of low-reflectivity materials such as asphalt, often yields weak signals back to the radar receiver.To avoid the loss of negligible signal returns, we propose utilizing the 4D radar tensor (4DRT) for 3D occupancy prediction. This raw data format preserves the entirety of radar measurements, offering a comprehensive dataset for analysis. However, employing such volumetric data introduces significant challenges. For instance, the substantial size of 4DRTs-potentially up to 500MB-poses processing inefficiencies that could compromise real-time neural network performance. Additionally, raw radar data is inherently noisy due to the multi-path effect and is stored in spherical coordinates, which diverges from the preferred 3D Cartesian occupancy grid used in our applications.Motivated by the outlined challenges, we introduce a novel approach, RadarOcc, specifically tailored for 4DRT-based 3D occupancy prediction. To address the computational and memory demands, our method initially reduces the data volume of 4DRTs through the encoding of Doppler bins descriptors and implementing spatial sparsification in the preprocessing stages. Our technique features sidelobeaware spatial sparsification to minimize the interference scattered across azimuth and elevation axes, which is further refined through range-wise self-attention mechanisms. Importantly, we observed the typical conversion of spherical RTs to Cartesian data volumes, which often incurs nonnegligible interpolation errors. Instead, we directly encode spatial features in spherical coordinates and seamlessly aggregate them using learnable voxel queries defined in Cartesian coordinates. Our approach further employs 3D sparse convolutions and deformable attention [61] for efficient feature encoding and aggregation. RadarOcc is benchmarked on the K-Radar dataset [42] against state-ofthe-art methods across various modalities, demonstrating the promising performance in radar-based 3D occupancy prediction. Comprehensive experiment results validate its comparable performance to the camera and LiDAR solutions. A qualitative assessment further validates the superior robustness of 4D radar data under adverse weather conditions, establishing its capability for all-weather 3D occupancy prediction. The contributions of this work are three-fold:• Introduction of the first-of-it-kind method, RadarOcc, for 4D radar-based 3D occupancy prediction in autonomous driving. We recognize the limitation of radar point clouds in reserving critical raw signals and advocate the usage of 4DRT for occupancy perception. • Development of a novel pipeline with techniques to cope with challenges accompanying 4DRTs, including reducing large data volume, mitigating sidelobes measurements and interpolation-free feature encoding and aggregation. • Extensive experiments on the K-Radar dataset, benchmarking state-of-the-art methods based on different modalities, and validating the competitive performance of RadarOcc and its robustness against adverse weather. We release our code and model at https: //github.com/Toytiny/RadarOcc.Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?Answer: [Yes] Justification: Our contributions are elaborated as a list in the introduction and briefly introduced in the abstract.Guidelines:• The answer NA means that the abstract and introduction do not include the claims made in the paper. • The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers. • The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings. • It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper. 
related work: 
 3D occupancy prediction. Early attempts on 3D occupancy prediction, aka. semantic scene completion (SSC) [62] , are mainly limited to the small-scale interior scenes [62] [63] [64] [65] [66] [67] [68] [69] [70] [71] . The introduction of SemanticKITTI [72] expands the study of SSC to large-scale outdoor scenes, based on which some works validate the feasibility of outdoor SSC with LiDAR input [15] [16] [17] [18] [19] . In contrast, MonoScene [23] is the seminal work for SCC using only a single monocular RGB image. Since Tesla's disclosure of their occupancy network for Full Self-Driving (FSD) [1] , there has been a recent surge of research on 3D occupancy prediction for autonomous vehicles. While a few works leverage LiDAR point clouds [2, 3, [20] [21] [22] for scene completion, the majority of existing approaches rely on a vision-only pipeline that learns to lift 2D features into the 3D space [3] [4] [5] [24] [25] [26] [27] [28] [29] [30] [31] [32] [33] . Despite these prevalent solutions based on LiDAR and camera, 4D radar sensors are still under-explored for 3D occupancy prediction.4D radar for autonomous driving. As an emerging automotive sensor, 4D mmWave radar prevails over LiDAR and camera in adverse weather (e.g., fog, rain and snow), offering all-weather sensing capabilities for mobile autonomy. In recent years, increasing endeavours have been witnessed to unveil the potential of 4D radar for autonomous driving applications, encompassing 3D object detection and tracking [56] [57] [58] , scene flow estimation [73] [74] [75] , odometry [73, [76] [77] [78] [79] [80] [81] [82] and mapping [80] [81] [82] . Apart from these works, we are the pioneering study for 4D radar-based 3D occupancy prediction, further exploring this unique sensor for the untouched topic.Radar tensor for perception Besides the post-processing radar point cloud, another data type of mmWave radar is the radar tensor (RT), which is the product of applying FFT along the corresponding dimensions to the raw ADC samples (c.f. Sec. 3.1). Unlike the sparse radar point cloud, dense RTs contain rich and complete measurements of the environment, refraining from information loss during point cloud generation (e.g., CFAR [59, 60] ). Consequently, some works attempt to use 2D [37, [83] [84] [85] [86] , 3D [87] [88] [89] or 4D [42, 46, 52] RTs for object detection, yielding satisfactory performance. In this work, we develop a tailored approach to 4D radar-based 3D occupancy prediction based on 4DRTs.3 Preliminary [90] . By compiling ADC samples from all chirps and Tx-Rx antenna pairs, the FMCW radar system constructs a 3D complex data cube for each frame. This data cube is organized into three dimensions: fast time, slow time, and channel, which correspond to range, range rate, and angle, respectively [91] .Radar tensor. Utilizing ADC samples, Fast Fourier Transforms (FFTs) are applied across relevant dimensions to extract detailed information. The first FFT, known as range-FFT, is performed across the sample (fast time) axis to separate objects at different distances into distinct frequency responses within range bins defined by hardware specifications. Subsequently, a Doppler-FFT along the chirp (slow time) axis decodes phase variances-Doppler bins-to derive relative radial velocities, producing a range-Doppler heatmap. For configurations with multiple Rx-Tx antenna pairs, termed virtual antenna elements, additional FFTs (angle-FFT) are executed across the spatial dimensions of the virtual antenna array to determine Angles of Arrival (AoA) for azimuth and elevation angles. This series of transformations results in a comprehensive 4D radar tensor (4DRT), characterized by power measurements across range, Doppler velocity, azimuth, and elevation dimensions.Radar point cloud. Beyond analyzing radar tensors, most FMCW radar sensors further refine their output to identify salient targets, which typically represent less than 1% of the data. Target detection algorithms such as CA-CFAR [59] and OS-CFAR [92] are commonly applied to the range-Doppler heatmap [91, 93] or directly on the 3D/4D radar tensors [42, 46] to isolate peak measurements. This process generates a sparse radar point cloud, with each point characterized by 3D coordinates and attributes such as Doppler velocity, power intensity, or radar cross-section (RCS). While this step significantly reduces data volume and mitigates noise, it also eliminates a substantial amount of potentially valuable information.Question: Does the paper discuss the limitations of the work performed by the authors?Answer: [Yes] Justification: We discuss several limitations of the work as part of the conclusion section where we reflect on our method design, experiments and plans for the future research.Guidelines:• The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper. • The authors are encouraged to create a separate "Limitations" section in their paper.• The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be. • The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated. • The authors should reflect on the factors that influence the performance of the approach.For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon. • The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size. • If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness. • While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations. 
4drt for 3d occupancy prediction: 
 Rationale of using 4DRT. 4D radar tensors (4DRTs) serve as raw sensor data that amalgamate the strengths of LiDAR/radar point clouds and RGB images, providing direct 3D measurements in a continuous data format. These tensors comprehensively capture information from raw radar measurements, effectively addressing the shortcomings associated with the sparseness of radar point clouds caused by the signal post-processing. For instance, low-reflectivity surfaces like asphalt, common on highways, typically do not reflect enough radar signals for detection. By using 4DRTs, these minimal signal returns can be detected, significantly bolstering occupancy prediction capabilities. Furthermore, the volumetric structure of 4DRTs aligns well with 3D occupancy grids, making them ideally suited for advancing 3D occupancy prediction techniques.Challenges. Despite their significant advantages, using 4D radar tensors (4DRTs) for 3D occupancy prediction presents substantial challenges. First, the large data size of 4DRTs (e.g., 500MB per frame in the K-Radar dataset [42] ) hinders computational efficiency, necessitating data volume reduction before processing. Second, the inherent noise in radar data, exacerbated by the multi-path effect of mmWave, requires careful filtering to preserve essential signals while eliminating noise. Third, the discrepancy between the spherical coordinates of 4DRT data and the Cartesian coordinates required for 3D occupancy outputs calls for a tailored network design. This design must effectively translate spatial interactions from spherical to Cartesian dimensions to ensure accurate occupancy predictions.Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material? Answer: [Yes] Justification: We release our code at https://github.com/Toytiny/RadarOcc. We also provide detailed instructions on the GitHub repository to guide others to reproduce our main experimental results on our used public dataset. Guidelines:• The answer NA means that paper does not include experiments requiring code. Justification: There is no societal impact of this work.Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact. • Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations. • The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
task definition: 
 In this work, we consider the task of 3D occupancy prediction with single-frame 4DRT output from 4D imaging radar. Given a 4DRT captured in the current frame denoted as V ∈ R R×A×E×D , our task aims to predict a 3D volume O = {o i } H×W ×L i=1 , of which each voxel element o i ∈ {c 0 , c 1 , . . . , c C } is represented as either free (i.e., c 0 ) or occupied with a certain semantics c i (i > 0) out of C classes. Here, R, A, E, and D denote the number of bins along the range, azimuth, elevation and Doppler axis, respectively, and each scalar of the 4DRT is the power measurement mapped to a location within the space defined by these four axes. H, W and L represent the volumetric size of the predefined region of interest (RoI) in the height, width and length dimensions.RadarOcc consists of four components in tandem (c.f. Fig. 1 ). Before loading heavy 4DRTs to the neural network, we reduce their data volume as the preprocessing steps via encoding the Doppler bins descriptor and performing sidelobe-aware spatial sparsifying to improve the efficiency without losing the key information (c.f. Sec. 4.3). To refrain from the interpolation error, we encode spatial features directly on the spherical RTs without transforming them into Cartesian volumes (c.f. Sec. 4.4) and aggregate the spherical features with 3D volume queries defined in the Cartesian coordinates (c.f. Sec. 4.5). Specifically, range-wise self-attention is used to alleviate the sidelobes, and sparse convolution and deformable attention are leveraged for fast feature encoding and aggregation. The occupancy probabilities are predicted in the 3D occupancy decoding step, which is supervised via our training loss (c.f. Sec. 4.6).Direct processing of raw 4DRTs with neural networks is impractical due to its substantial data size (e.g., 500MB per frame) which leads to heavy computation cost and memory usage. Moreover, the slow data transfer between the sensor, storage device and processing unit (CPU/GPU) of large-volume raw 4DRTs not only hinders the onboard runtime efficiency but also increases the training duration which demands repetitive data loading. For efficiency, we propose to reduce the data volume of 4DRTs through encoding the Doppler bins descriptor and sidelobe-aware spatial sparsifying as the preprocessing steps (see Fig. 1 ). Post reduction, the loading of 4DRTs into the processing unit for runtime inference can be more feasible and the network training can be more efficient. Doppler bins descriptor. Unlike the three spatial axes, which are intuitively critical for spatial perception, the Doppler axis in 4DRTs has often been considered redundant in 3D object detection. Previous studies [42, 46, 52] have employed average-pooling to minimize this axis, aiming to reduce computational overhead. However, we argue that this ostensibly 'redundant' axis contains vital cues for geometric and semantic analysis in 3D occupancy prediction. Specifically, the Doppler axis provides essential information on object speed via peak locations, aiding in differentiating dynamic objects from static backgrounds. Moreover, the power distribution within the Doppler bins offers insights into the confidence levels of true targets-essentially, indicating their likelihood of occupancy.To preserve and utilize this crucial information, we have developed a method to encode the Doppler bins into a descriptor that captures specific statistics for each spatial location within the 4DRTs. This descriptor incorporates the top-three power values along with their indices, the mean power value, and the standard deviation, as depicted in Fig 1 . Note that the number of preserved top values is determined empirically. Consequently, this approach enables us to reduce the data volume of raw 4DRTs by a factor of D 8 , while retaining key information from the Doppler axis. Sidelobe-aware spatial sparsifying. By encoding the Doppler bins into light-weight descriptors, we transform the raw 4DRT into 3D spatial data volume with the original Doppler axis as the 8-channel feature dimension. Nevertheless, it remains costly for neural networks to encode features from 3D dense data volume with operations like 3D convolution [94, 95] . To accelerate the computation, prior arts [42, 46] transfer the dense RT into a sparse format by retraining only the top-percentile elements based on power measurements. However, this approach tends to be biased towards specific ranges that exhibit exceptionally high measurements. It can be observed in Fig. 2 that after percentile-based sparsifying, a significant number of the reserved elements are concentrated within the same ranges spread across the azimuth and elevation axes. These elements manifest as artifacts of sidelobes, which can an be viewed as the diffraction pattern of the antenna [96, 97] . Consequently, this results in the loss of important measurements from other ranges and introduces lots of noise into the sparse tensor. To mitigate this issue, we propose to select the top-N r elements for each individual range instead of on the whole dense RT for spatial sparsifying (see Fig. 1 ). In this way, the dominance of certain ranges can be avoided while the sidelobe level is reduced, as exhibited in Fig. 2 . Note that our spatial element selection is based on the mean power value across the Doppler axis. The final sparse tensor is denoted asT = {t i ∈ R Nr×(8+2) } Ri=1 with the extra two feature channels storing the azimuth and elevation indices of reserved N r elements for each range.Given the sparse RT, we aim to encode representative features for accurate 3D occupancy prediction. As the sparse RTs are inherently in the spherical coordinates, previous works [42, 46] transfer them into the Cartesian coordinates before feature encoding. However, such a transfer would undermine their uniform density distribution and often incur interpolation errors. Inspired by the polar representation of point clouds [10, 98, 99] , we propose to take the elements in RT as voxels rasterized in the spherical coordinates and apply the spherical-based feature encoding directly. The spherical voxel representation naturally matches the spherical-uniform distribution of RTs and can refrain from inducing interpolation errors. In practice, the 3D convolutions can be used to extract grid-based representations by only replacing the X-Y -Z axis with the range-azimuth-elevation axis. In what follows, we illustrate our spherical-based feature encoding process. Range-wise self-attention. In Section 4.3, we address the issue of sidelobes by selecting elements based on range-wise percentiles during the preprocessing phase. To further mitigate sidelobe interference, we introduce a range-wise self-attention mechanism [100] (see Fig. 1 ) as the initial step in our feature encoding process. Specifically, within each range component t i ∈ T, which includes N r RT tokens, we utilize the Doppler bin descriptors as token features. Additionally, two index channels are employed for positional embeddings to enhance the specificity of our spatial encoding.Sequential sparse convolution. For efficiency, we apply a series of 3D sparse convolutions [101] onto the sparse RT for spatial feature encoding in the spherical voxel space. This produces a 3D dense feature volumeF ∈ R R S × A S × E S ×C f (N f = R S × A S × E S )with a reduce spatial dimension characterized by the stride S, where C f denotes the feature dimension. Note that F inherently aligns with the spherical space with each feature element's indices corresponding to a spherical coordinate.Deformable self-attention. Following the consecutive 3D sparse convolution, we use the 3D deformable attention [61] to further refine and augment our feature volume F by enforcing spatial interaction. As a definition, for a query feature z corresponding to a reference point p in the input feature X, its feature can be updated by deformable attention in the following equation:EQUATIONwhere W m and W ′ m are the learnable weights for the m-th attention head, while A mk and ∆p mk is the attention weight and sampling offset calculated with z for its k-th sampling point and the m-th head. X(p + ∆p mk ) is the key features at the sample location (p + ∆p mk ). By applying self-attention toF = {f q } N f q=1 , the refined feature volume F r = {f q r } N fq=1 can be derived by:EQUATIONDecoding 3D Cartesian occupancy from a spherical feature volume is inherently challenging due to misalignments in spatial axes and discrepancies in the regions they represent. An intuitive approach would be to transform the spherical feature volume into a Cartesian one and then decode the 3D Cartesian occupancy. However, this method can introduce feature-level interpolation errors, which we aim to avoid as discussed in Section 4.4.To avoid conducting interpolation, we propose to aggregate the spherical features in a learnable way, with 3D volume queries defined in the Cartesian coordinates attending to the feature samples in F r , as shown in Fig. 1 . First, we build learnable grid-based voxel queriesH = {h q ∈ R C f } H×W ×L q=1which has the same volumetric size as our desired output O and the same feature dimension as the spherical feature volume F r . Each voxel query h q corresponds to a 3D point p q in the Cartesian coordinate. Second, the 3D point p q of each query is transformed from the Cartesian to the spherical coordinate, which is then mapped to a index position in F r denoted as Φ(p q ). We take Φ(p q ) as a 3D reference point in the spherical space and sample key elements in its vicinity from the feature volume F r . Lastly, we leverage deformable cross-attention [61] to aggregate the key samples for each reference point and the output G = {g q ∈ R C f } H×W ×L q=1 can be calculated by:EQUATIONWith the aggregated voxel features G, we leverage consecutive 3D convolutions [94, 95] with skip connection [102] to decode hierarchical feature volumes at N s scales with a scaling step of 2. Multiscale feature volumes are then merged in a top-down way [103] via upsampling features by a factor 2 and concatenated along the feature dimension, resulting in G d ∈ R H×W ×L×NsC f . Finally, the occupancy head equipped with the softmax function is employed to output the normalized occupancy probabilities Õ ∈ {0, 1} H×W ×L×(C+1) for all voxels on C semantic classes and one free class.Our network is trained in a supervised way with the ground truth occupancy. Following [3] , we use the cross-entropy loss as the primary loss to optimize the training and incorporate the lovasz-softmax loss [104] to handle the class imbalances. Moreover, we utilize the scene-and class-wise affinity loss proposed in [23] to enhance the optimization of geometry and semantic IoU metrics. Competing methods. We benchmark RadarOcc against state-of-the-art methods employing different modalities. Given that recent studies do not use radar data for 3D occupancy prediction, we adapt the OpenOccupancy LiDAR-based baseline and CONet [3] to accommodate radar point cloud (RPC) inputs for our comparative analysis. Furthermore, we convert 4DRTs to Cartesian coordinates [42] with a voxel size of 0.4m, referred to as 4DRT-XYZ, and integrate them into the LiDAR-based OpenOccupancy framework [3] . Following best practices from [42, 46] , we process 4DRT-XYZ into a sparser format. For a comprehensive inter-modality evaluation, we also replicate the OpenOccupancy LiDAR-based baseline [3] and both monocular and stereo camera-based SurroundOcc [25] configurations to fit our experimental setup. Notably, we enrich our comparisons by generating 16-beam and 32-beam LiDAR point clouds from the standard 64-beam configurations through elevation-wise downsampling. The evaluation focuses on the overlap area between the horizontal field of view (FoV) of all sensors and our defined RoI to minimize potential data discrepancies beyond the FoV. For implementation, we train all evaluated models on our K-Radar well-condition training set.  
comparison against radar-based methods: 
 We first compare RadarOcc with state-of-the-art baseline methods using radar data for 3D occupancy prediction in Tab. 1. As can be seen, RadarOcc outperforms other approaches in every metric, demonstrating its state-of-the-art performance in radar-based 3D occupancy prediction. Specifically, our 4DRT-based RadarOcc largely improves the performance over RPC-based methods: the mIoU of L-CONet [3] is relatively improved by 39.4%, 40.4% and 56.9% for different volumes (12.8m, 25.6m, 51.2m). Such a significant improvement mainly stems from the dense data format of 4DRT, which retains critical information from low-reflectivity objects, enabling effective occupancy prediction for the whole scene. 4DRT-XYZ based L-baseline [3] also outperforms RPC-based methods but inferior to RadarOcc, especially in long-range FG IoU. We credit this to the interpolation errors led to small and far foreground objects when we converting 4DRT to Cartesian coordinates.To validate the effectiveness of our key designs, we ablate them alone from our 4DRT-based pipeline RadarOcc and show the evaluation results on K-Radar well-condition test split in Tab. 2.Doppler bins descriptor. By replacing the Doppler bins descriptor with the average-pooling result, the performance of RadarOcc is degraded in most metrics (row (a) vs. (b) in Tab. 2), demonstrating the usefulness of preserving the information encoded by the Doppler axis (c.f. Sec. 4.3). However, the improvement is somehow marginal due to the limited Doppler measurement range of the radar used in K-Radar [42] , which wraps around the overflow values, causing ambiguity in Doppler velocity.Sidelobe-aware spatial sparsifying. We conduct this experiment (row (c) in Tab. 2) by changing our sidelobe-aware spatial sparsifying (c.f. Sec. 4.3) to the percentile-based spatial sparsifying used in [42, 46] . Our sidelobe-aware approach leads to a remarkable advancement in performance, especially in mIoU metrics. This is attributed to its ability to preserve more valid elements from diverse ranges and suppress sidelobes for sparse RTs, allowing for more accurate prediciton.Spherical-based feature encoding. For row (d) in Tab. 2, we transform sparse RT to Cartesian coordinates before feature encoding (c.f. Sec. 4.4) and omit the spherical-to-Cartesian feature aggregation (c.f. Sec. 4.5). We can see that our spherical-based feature encoding gains the performance for each metric as our strategy preserves the original data distribution, avoiding incurring interpolation errors. This also validates the effectiveness of our learnable spherical-to-Cartesian feature aggregation.To assess the runtime efficiency of RadarOcc, we conducted our model inference on a single Nvidia GTX 3090 GPU. The results shows an average inference speed of approximately there is still a gap between the real-time application (i.e., 10fps), our inference speed has surpassed that of many camera-based methods as reported in [25] . Further improvements in inference speed can be achieved by reducing network complexity and applying precision reduction techniques, such as converting model precision from Float32 (FP32) to Float16 (FP16).To validate this, we simplified the feature encoding (c.f. Sec. 4.4) and aggregation (c.f. Sec. 4.5) modules by reducing some redundancy layer (e.g., number of layers in deformable attention) for efficiency, and converted the computationally intensive 3D occupancy decoding module (c.f. Sec. 4.6) from FP32 to FP16 via the quantization in PyTorch. These optimizations resulted in a 126% increase in inference speed, reaching approximately 7.46 fps, with only a minimal impact on performance. Please refer to Tab. 3 and Tab. 4 for detailed changes in runtime for each module and performance. Given the increasing computational power of modern embedded GPUs, such as the Nvidia Jetson Orin, which can almost rival desktop GPUs like the Nvidia GTX 2090, we believe this enhanced inference speed demonstrates the potential for real-time application of our method in future vehicle systems, especially if further model quantization is applied.To enrich our benchmark results and provide insights into the performance comparison between different modalities, we also evaluate state-of-the-art baseline methods [3, 25] on LiDAR and camera input. Quantitative results on K-Radar well-condition test split are reported in Tab. 5, while examples of qualitative results on K-Radar adverse-weather testing splits are exhibited in Fig. 3 .Quantitative results under normal weathers. As seen in Tab. 5, not surprisingly, LiDAR-based L-baselines [3] rank the top three in most metrics thanks to LiDAR's low-noise and high-resolution measurements (vs. radar) and direct depth measurement (vs. camera). Due to the inherently lower resolution and considerable noise of radar data, radar-based methods exhibit inferior to LiDAR-based methods in normal weather. However, RadarOcc still shows comparable performance to 16-beam LiDAR, and surpasses monocular and stereo camera-based method in most metrics. Notably, RadarOcc outperforms state-of-the-art SurroundOcc [25] relatively by 39.5%/19.7% and 53.7%/26.1% in mIoU/IoU@51.2m for stereo and monocular input, respectively. Stereo camerabased SurroundOcc [25] ranks third on FG IoU and mIoU@12.8m because of stereo vision's ability to infer accurate depth at short ranges, where the disparity between the two images is more pronounced. Qualitative results under adverse weathers. While we have demonstrated the competitive performance of RadarOcc under normal weather, the key reason behind using radar for perception comes from its unique robustness against adverse weather where LiDAR and cameras fall short. To showcase such an inherent advantage, we provide some examples of qualitative results from different modalities in Fig. 3 . As can be seen, RadarOcc provide robust 3D occupancy prediction under heavy rain and snow. In contrast, the camera lens are covered by the rain/snow and LiDAR measurements of some objects ahead are missing as water droplets or snowflakes can scatter and absorb the laser beams, leading to worse results. Please see our supplementary materials for more qualitative results. 
conclusion: 
 In this work, we propose RadarOcc, a novel 3D occupancy prediction approach based on 4DRTs output from 4D imaging radar, enabling robust all-weather perception for autonomous vehicles. We analyse the rationale and challenges of using 4DRTs for 3D occupancy prediction and present tailored solutions to cope with the large, noisy and spherical 4DRTs. Experiments on the K-Radar dataset show RadarOcc's state-of-the-art performance in radar-based 3D occupancy prediction and comparable results to other modalities in normal weathers. Through qualitative analysis, we also exhibit its unique robustness against various adverse weathers. We believe our work could endorse the potential of 4D imaging radar to be an alternative to LiDAR and setup an effective baseline for further research and development of 4D radar-based occupancy perception.Limitations. As an initial investigation into 4D radar-based 3D occupancy prediction, this work has several limitations as follows. First, our method maps single-frame 4D radar data to single-frame 3D occupancy prediction without modeling the temporal information and performing occupancy forecasting. Second, due to the lack of point-wise annotation, our task is limited to two general semantics, i.e., foreground and background. Future work will aim to address these issues.Range-wise self-attention. In our spherical-based feature encoding, the range-wise self-attention is performed on the non-empty elements per range, i.e., t i ∈ R Nr×(8+2) (i = 1, 2, . . . , R), where N r = 250. The 8-channel Doppler descriptors are considered as the input features while the azimuth and elevation indices are converted to positional embeddings with lookup tables [100] . Specifically, we use two layers of multi-head attention with the embedding dimension set as 32, number of heads as 4 and dropout probability to be 0.1 The output is re-organized to a sparse tensor with a dimension of RN r × (32 + 3), where the range, azimuth and elevation index is stored for each non-empty element.3D sparse convolution. We utilize the spconv library [106] to implement the sparse convolution layers for our spherical-based feature encoding. This encoding process has two types of operation: 3D Submanifold Convolution and 3D Sparse Convolution. 3D submanifold convolution only convolves the active spatial locations without altering the sparsity pattern and spatial resolution, while 3D sparse convolution performs convolution on all active locations, expanding the sparsify pattern and allows for spatial resolution change. We leverage the submanifold convolution as the first and last layer and apply three sparse convolution layers in-between. We set the stride as 2 for the last two of 3D sparse convolution to reduce the spatial dimension. As a result, we obtain a 3D dense feature volumeF ∈ R 64×27×10×C f (C f = 192),where the spatial dimension is decreased by ×4.Deformable self-attention. Given feature volume F, we use 3D deformable self-attention [61] to augment its spatial features. Two attention layers are used and the number of sampling points of each query is set to 8. Each attention layer has 8 heads and apply a dropout of a rate of 0.1 to the output features. The refined feature volume F r has the same dimension as F, i.e., 64 × 27 × 10 × C f .Spherical-to-Cartesian feature aggregation. To aggregate features extracted in the spherical coordinates, we build learnable voxel queries H = {h q } q with a dimension of 14 × 128 × 128 × C f defined in the LiDAR Cartesian coordinates system. By transforming the 3D points p q corresponding to our voxel queries h q into the radar spherical coordinates, we construct a list of 3D reference points Φ(p q ). Then, the deformable cross attention is used to aggregate the spherical features to Cartesian by considering the spherical volume F r as the key and value of attention and the voxel queries H as the query of the attention. Just as the self-attention module, we use the 3D version of the deformable attention [61] , with the same network settings. The dimension of the output Cartesian feature G have the same size as the learnable queries H, which is 14 × 128 × 128 × C f . 3D occupancy decoding. Given the Cartesian voxel features G, we firstly apply the 3D version of ResNet-18 [102] for decoding, resulting in 4 level of feature maps, with 1 2 , 1 4 , 1 8 , 1 16 of the voxel spatial shape and 80, 160, 320, 640 for feature dimension respectively. These multi-level features are then upsampled back to the target spatial space H × W × L using 3D FPN [103] , leading to the final features G d with a dimension of 14 × 128 × 128 × 4C f . Lastly, we use an MLP with the hidden dimension of [64, 64] to reduce the feature channel and predict the occupancy probabilities which are normalized by a softmax layer. The output is denoted as Õ ∈ {0, 1} H×W ×L×(C+1) .Training loss. The overall loss function L used to train our network can be written as:EQUATIONGiven the ground truth denoted asÔ = {ô i ∈ {c 0 , c 1 , . . . , c C }} No i=1 (N o = H × W × L)and the output Õ, the cross-entropy loss L CE can be calculated as:EQUATIONwhere N o is the number of voxels, c and i indexes classes and voxels, õi,c is the predicted logit for i-th voxel on the class c. ôi,c = 1 if ôi = c; else, ôi,c = 0. To balance different classes, we use w c for each class calculated as the inverse of the class frequency in K-Radar [42] . Please refer to [104] and [23] for more details on the lovasz-softmax loss L LS and scene-class affinity loss L geo scal and L sem scal we used in Eq. 4. Training details. We train RadarOcc with 10 epochs using Adam optimizer with a learning rate of 3e-4. The batch size is 1 for each GPU. We follow [3] to use loss normalization to balance the weight of the 4 different losses, and cosine annealing [107] with  
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?Answer: [NA] Justification: The release of our models and code has no such risks.Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?Answer: [Yes]Justification: We have properly cited the original papers of our used dataset and mentioned their license in the appendix.Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL. • The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided. • If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. • For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. • If this information is not available online, the authors are encouraged to reach out to the asset's creators. • We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution. • For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review. 
 
 Appendix: C.1 Impact of the number of reserved top elements N rIn Sec. 4.3, we propose a sidelobe-aware spatial sparsification technique that selects the top-N r elements for each individual range rather than the entire dense radar tensor (RT). There is indeed a trade-off between preserving critical measurements and filtering noise/compressing the radar tensor in this process. Excessive compression/filtering may result in the loss of weak reflections, while insufficient compression/filtering increases computational costs and retains some level of noise.To identify the optimal balance, we conducted a series of experiments varying the number of selected top elements for each range, i.e., N r , and assessed performance and inference speed on the validation set. The results, presented in Table 6 , indicate that RadarOcc achieves the best results in half of all metrics on our validation set when N r = 250. Both higher and lower values of lead to suboptimal results, suggesting that N r = 250 strikes the best balance between retaining critical signals and filtering noise. Additionally, the inference speed at N r = 250 is relatively higher compared to configurations with larger values. Therefore, we select N r = 250 for RadarOcc's evaluation on our testing set.To investigate the effect of the number of preserved top values (i.e., N d ) among Doppler bins for each spatial location, we conducted a series of experiments by varying N d . As shown in Table 7 , the change in N d does not significantly impact our results. For both efficiency and performance, we chose N d = 3 for our method based on the validation set performance.This can be explained by the fact that K-Radar [42] wraps around overflow values in Doppler measurements due to the limited Doppler measurement range. For example, Doppler speeds of 3.0 m/s and 6.0 m/s are measured within the range of -1.92 to 1.92 m/s as 3.0 -3.84 = -0.84m/s and 6.0 -3.84×2 = -1.68m/s, respectively. This ambiguity means the information from the Doppler axis only marginally improves our model. Consequently, changing hardly affects our performance. Table . 2 in our paper also shows that our baseline without Doppler bin descriptor (w/o DBD), which only uses mean power, reflects this minimal impact. However, we believe our Doppler bin encoding method could bring more improvement with other radar sensors that have a larger measurement range.To verify the effectiveness of the range-wise self-attention used in our sphercial-based feature encoding (c.f. Sec. 4.4), we experiment by removing it from the network and show the results in Tab. range-wise self-attention. In particular on FG IoU, the relative gain is 14.4%, 14.7% and 11.3% for the 3D volume of 12.8m, 25.6m and 51.2m, respectively. We credit this to the ability of range-wise self-attention to further suppress the sidelobe noises appearing around the foreground objects.To better show the qualitative results of RadarOcc and baseline methods based on other modalities, we make some video demos under different weather conditions and submit them as a supplementary material. We recommend our audience to watch the video for a better understanding of our work.We observed some failure cases of RadarOcc due to some reasons, such as insufficient resolution and decreased Signal-to-Noise Ratio (SNR) at far distances. An example of such failure cases is exhibited in Fig. 4 . We hope this could shed the light on future research in this field.In our method, we address objects with low radar cross-section (RCS) from two key perspectives:Input perspective. We utilize 4D radar tensor (4DRT) data instead of radar point clouds for 3D occupancy prediction. This approach avoids the loss of weak signal returns that can occur during the point cloud generation process, e.g., those filtered out by the CFAR detection, preserving more measurements from low RCS objects compared to radar point clouds.Method perspective. Our sidelobe-aware spatial sparsifying technique selects the top-elements for each individual range rather than the entire dense RT. As shown in Fig. 2 , this method retains critical measurements scattered across different ranges, including both low and high RCS objects. This contrasts with percentile-based methods, which often concentrate on elements corresponding to high RCS objects, thereby missing important data from low RCS objects.As a result, our method is effective in recognizing objects with low RCS, such as pedestrians, when predicting 3D occupancy. Figure 5 shows an example where RadarOcc successfully handles low-RCS objects while 32-line LiDAR not.NeurIPS Paper Checklisthttps://github.com/kaist-avelab/K-Radar 


Decision: This paper explores the use of 4D imaging radar to predict a 3D occupancy map with applications to autonomous vehicles. The authors point out that this modality could be complementary to vision and lidar based approaches particularly in low-light or adverse weather conditions. The paper was reviewed by a panel of experts who felt that the manuscript did a good job of explaining the method and describing its merits. Experimental validation was limited since there is only a single publicly available dataset that has the requisite 4D radar tensor data. This limits the generality of the inferences that can be drawn from the experiments. Nonetheless the consensus opinion of the reviewers was that the manuscript described an interesting new algorithm and that it would be valuable for people in the field to be aware of what could be achieved with this sensor modality.

Reviews: review: paper_summary: paper_summary: The paper presents RadarOcc, a method that enhances 3D occupancy prediction for autonomous vehicles using 4D imaging radar. It directly processes the 4D radar tensor, aiming to overcome the limitations of sparsity and noise associated with conventional radar processing. The methodology introduces techniques such as Doppler bins descriptors and sidelobe-aware spatial sparsification to improve data integrity and scene detail retention. The evaluation is conducted on the K-Radar dataset, with RadarOcc demonstrating superior performance compared to traditional LiDAR and camera-based methods, particularly under adverse weather conditions.
strength_weakness: strength_weakness: 1. The processing of dense radar tensors can be computationally expensive, potentially limiting real-time application in less powerful onboard systems.

2. The current framework processes single-frame data, which may not fully capture the dynamic nature of driving environments, possibly affecting prediction reliability in highly dynamic scenarios.

3. As this study focuses on solving real problems in autonomous driving, it is important to thoroughly justify why 4D imaging radar has advantages over LiDAR in addressing issues like occupancy prediction. Can methods related to LiDAR really not solve these problems in various environments? In fact, many autonomous vehicles not only refrain from using mm-wave radar but also do not use LiDAR, managing to solve most issues using only vision.
questions: questions: 1. How does the RadarOcc system perform in terms of real-time processing, and what are the computational requirements?

2. Could the method be integrated with temporal data to predict dynamic changes in the environment, and if so, what modifications would be necessary?

3. Are there specific scenarios or environments where RadarOcc's performance might be significantly reduced, such as urban canyons or areas with high radar interference?

4. How does the system handle objects with low radar cross-section, such as pedestrians or animals?
limitations: limitations: 1. The method's high computational demand may not scale well to lower-end processors commonly used in commercial vehicles.

2. The lack of temporal modeling could limit the system's predictive capabilities in highly dynamic environments.

3. It remains unclear how well the method generalizes across different radar systems or configurations not represented in the K-Radar dataset.
paper_summary: paper_summary: It is proposed to utilize 4D imaging 8 radar sensors for 3D occupancy prediction by directly processing the 4D radar tensor, thus preserving essential scene details. RadarOcc innovatively addresses the challenges associated with the voluminous and noisy 4D radar data by employing Doppler bins descriptors, sidelobe-aware spatial sparsification, and range-wise self-attention mechanisms. The demonstration of the RadarOcc’s state-of-the-art performance in radar-based 3D occupancy prediction was carried out on K-Radar dataset.
strength_weakness: strength_weakness: It is not the first paper to utilize the 4D imaging radar raw measurement for perception. As mentioned by the authors, they focus on semantics, rather than typical road users detection and classification. 
For the reference to 4D imaging radar, please consider cite the following paper 
S. Sun and Y. D. Zhang, "4D Automotive Radar Sensing for Autonomous Vehicles: A Sparsity-Oriented Approach," in IEEE Journal of Selected Topics in Signal Processing, vol. 15, no. 4, pp. 879-891, June 2021.

For general automotive radar contribution to autonomous driving, please consider cite the following paper 
S. Sun, A. P. Petropulu and H. V. Poor, "MIMO Radar for Advanced Driver-Assistance Systems and Autonomous Driving: Advantages and Challenges," in IEEE Signal Processing Magazine, vol. 37, no. 4, pp. 98-117, July 2020.
questions: questions: It seems to be a trade-off in the sidelobe-aware spatial sparsification process. Is it interesting to know whether important feature information is lost due to this sparsification. For example, targets with weak reflections, such as the information of road curbs might be lost.
limitations: limitations: As mentioned before, the proposed work does not apply to the general road users detection and classification.
paper_summary: paper_summary: This paper introduces a 3D occupancy prediction method that, unlike previous radar-based approaches, utilizes 4D imaging radar to leverage additional information. To harness the potential of this under-explored 4D data, the paper tackles challenges such as the large size of raw 4D radar data, inherent noise, and discrepancies in coordinate systems. The proposed method results in improved reconstruction accuracy compared with other baselines.
strength_weakness: strength_weakness: Comparing the proposed method to only one published baseline and relying on a single dataset raises concerns about the persuasiveness of the findings.

As a learning-based model, the paper misses an evaluation of the generalization of the learning formulation to datasets out of the domain. This oversight is crucial for assessing the model's applicability across various real-world scenarios.
questions: questions: Can the proposed learning formulation be extended to incorporate radar methods with other modalities, such as images and lidar?

What is the trade-off relationship between compressing the input data (originally 500MB) and its impact on performance? Providing this information could assist readers in balancing efficiency and accuracy.

MISC

Line 110: The line distance is unusual.
limitations: limitations: The experiments are conducted solely on the K-Radar dataset, which is the only autonomous driving dataset containing 4D radar data. This situation highlights a significant issue: the scarcity of 4D data compared to other perception methods (e.g., images, normal radar data), which limits its applicability in broader contexts.
paper_summary: paper_summary: This paper leverages recent advancements in automotive radar technology and introduces a novel approach that utilizes 4D imaging radar sensors for 3D occupancy prediction. The proposed method incorporates Doppler bin descriptors, sidelobe-aware spatial sparsification, and range-wise self-attention mechanisms to effectively manage and mitigate the noise present in 4D radar data.
strength_weakness: strength_weakness: Please consult the question section for further information.
questions: questions: 1. Dataset Limitation: The experimental analysis is thorough, but it is conducted solely on the K-Radar dataset. There is a concern that the method may be overfitted to this specific dataset. Could the authors provide additional examples using other datasets to demonstrate the generalizability of the method?

2. Computational Complexity: The methodology includes both an attention mechanism and multi-scale 3D convolutional blocks. A potential issue is whether this structure significantly increases the computational requirements. Could the authors provide a comparison of model size and computational time to address this concern?

3. Selection Criteria: The method selects the Top-3 power values and their corresponding indices. How does this choice affect performance? Would increasing or decreasing the number of selected values significantly impact the results?

4. Failure Cases: Are there any documented failure cases of the method? Considering that Doppler effects are most pronounced when the relative velocity is aligned with the wave direction, how does the method perform in scenarios where this condition is not met?

5. Would the code be published together with the paper?
limitations: limitations: Please consult the question section for further information.
paper_summary: paper_summary: This paper introduces a novel approach for 3D occupancy prediction in autonomous driving using 4D imaging radar sensors. Traditional methods rely heavily on LiDAR or camera inputs, which are vulnerable to adverse weather conditions. RadarOcc leverages the robustness of 4D radar data, which provides comprehensive scene details even in challenging weather. 

The key contributions of the paper include:

- **Utilization of 4D Radar Tensors (4DRT):** Unlike previous methods that use sparse radar point clouds, RadarOcc processes raw 4DRTs, preserving essential scene details and capturing comprehensive environmental data.
- **Novel Data Processing Techniques:** The paper introduces Doppler bins descriptors, sidelobe-aware spatial sparsification, and range-wise self-attention mechanisms to handle the large and noisy 4DRT data. It also uses spherical-based feature encoding followed by spherical-to-Cartesian feature aggregation to minimize interpolation errors.
- **Benchmarking and Evaluation:** RadarOcc is benchmarked on the K-Radar dataset against state-of-the-art methods using various modalities. The results show that RadarOcc outperforms in radar-based 3D occupancy prediction and demonstrates promising results compared to LiDAR and camera-based methods. The approach also shows superior performance in adverse weather conditions.
strength_weakness: strength_weakness: 1. maybe high computational demand, lack of this part description
2. Dependence on a Single Dataset
3. The experiment was limited because of the data set
questions: questions: 1. In my understanding, the advantage of radar compared with lidar is that it can achieve good perception even in bad weather. However, because of GT procurability,   author just use well-condition sequences, which i think may be unfair to lidar-based methods or vision-based methods, because the domain of radar changes little while others change a lot. It would have been better if the authors had added data on nuscenes to the baseline for comparison. Time may not be enough to complete, but this is really a point of concern to me
2. Could you provide the overlap ratio between different sensors and ROI?
3. Could you provide the overall inference speed and the speed of each module? Additionally, how much does the Doppler bins descriptor reduce the overall computational load, and do the range-wise self-attention and spherical-to-Cartesian feature aggregation introduce significant computational overhead?
4. Since Occ gt is difficult to obtain, is there any way to improve training data and generalization

**I'm willing to raise the grade if address my concerns**
limitations: limitations: As explained by the authors in the limitations section:

1. The model does not fully utilize the 4D radar information.
2. Due to the lack of point-wise annotations in the dataset, the semantic information is limited.

Comments: title: Rebuttal follow up
Rebuttal follow up
Comment
Rebuttal Response Follow-Up
comment: Dear reviewer kdzm, 

As the discussion period is approaching its end, we kindly invite you to review our detailed rebuttal. We believe it addresses the concerns you raised in your review. We also welcome any further comments you may have.

Thank you again for your time in reviewing our paper.
Dear reviewer v56z,
As the discussion period is approaching its end, we kindly invite you to review our detailed rebuttal. We believe it addresses the concerns you raised in your review. We also welcome any further comments you may have.

Thank you again for your time in reviewing our paper.
Dear reviewer f9yn,
As the discussion period is approaching its end, we kindly invite you to review our detailed rebuttal. We believe it addresses the concerns you raised in your review. If you find our work satisfactory, we would greatly appreciate it if you could consider raising your score to a positive one. We also welcome any further comments you may have.

Thank you again for your time in reviewing our paper.
Thanks for the authors' response to my concerns. I have no further questions and feel I can keep my positive ratings for the paper.
Thank you to the authors for their response. I have carefully reviewed the feedback from the other reviewers as well as the corresponding rebuttal. While the rebuttal has addressed most of my concerns, the limited experimental results prevent me from raising my score. I believe the score I have given is appropriate for this submission.

It is a commendable piece of work, and I look forward to seeing more research in this area.
Dear our reviewers,

I am writing to follow up on the rebuttal we submitted for our paper. We understand the review process can be time-consuming but as the discussion deadline is approaching, we would appreciate any updates or feedback at your earliest convenience so that we could do some last-minute clarification if needed.
We have not yet received any feedback and want to ensure that our reviewers have had the opportunity to read our rebuttal, does our rebuttal answer all your questions?

Best regard, 
Submission391 authors
Dear Reviewer XRVn:

We appreciate your detailed summary and positive comments regarding the presentation, novelty, creativity, and the experimental results of our research. Thanks a lot for providing the valuable feedback and raising insightful questions about our work. We address and answer your questions below:


**Q1: Could you provide the overall inference speed and the speed of each module? Additionally, how much does the Doppler bins descriptor reduce the overall computational load, and do the range-wise self-attention and spherical-to-Cartesian feature aggregation introduce significant computational overhead?**

A1: Thank you for raising this concern. Please first refer to our general answer to the **‘Computation complexity and requirement’** question in the global rebuttal.

Specifically, the range-wise self-attention module is highly efficient, with a runtime of only 2.5ms. The spherical-to-Cartesian feature aggregation initially costs 72ms per frame. However, this can be optimized to 29.7ms by reducing the number of layers. This module not only converts the spherical feature volume to a Cartesian one without interpolation errors but also enhances the encoding of voxel feature relationships.

The Doppler bins descriptor significantly reduces the data volume of raw 4D radar tensors (4DRTs) by a factor of D/8, where D equals 64 in our case. This reduction decreases the input data volume to 1/8 of its original size. This reduction not only shortens loading times (from disk/sensor to processors) but also lessens the overall computational burden during spherical-based feature encoding.



**Q2: Dependence on a Single Dataset. The experiment was limited because of the data set**

A2: Thanks for highlighting this concern. Please refer to our general answer to the **‘Dependence on a single dataset’** answer in the global rebuttal.


**Q3: Author just use well-condition sequences, which I think may be unfair to lidar-based methods or vision-based method**

A3: Thanks for this thoughtful feedback. We are not sure if we understand this question exactly. If you were concerned that the training data used to train baselines method does not include sequences collected from adverse weathers, which causes worse performance due to the domain change of LiDAR and camera data, we are happy to answer this question as follows. 

Currently, it is hard to resolve this problem as the LiDAR data is inevitably affected by the adverse weather. For example, water droplets or snowflakes can scatter or absorb LiDAR beams, reducing the effective range of LiDAR and inducing noise in the data. As a result, the occupancy labels obtained from accumulated LiDAR data under adverse weather have low fidelity. The nuScenes dataset also faces a similar issue where occupancy labels are affected by adverse weather, despite having annotated bounding boxes. Adding new nuScenes data collected under adverse weather to train our LiDAR and vision-based baselines might not be feasible. This approach could potentially hinder network training because of noisy labels rather than improve adaptation to adverse weather. Moreover, as nuScenes does not provide 4D radar tensor data,  it’s unfair for our radar-based methods because they cannot benefit from the new data samples.

This situation highlights a significant research question: how can we generate reliable 3D occupancy labels under adverse weather conditions when LiDAR measurements are noisy? We acknowledge the importance of this question and plan to address it in our future work.


**Q4: Could you provide the overlap ratio between different sensors and ROI?**

A4: Thanks for this comment. Our evaluation focuses on the overlap area between the horizontal field of view (FoV) of all sensors and our defined RoI to minimize potential data discrepancies beyond the FoV. Specifically, the overleap hFoV of K-Radar sensor suite is 107◦, symmetrically distributed around the front axis. The ratio of between the final evaluation area and our RoI can be calculated as: 

$$ 1 - \frac{\cot\left(\frac{107^\circ}{2}\right)}{4} \approx 0.812 $$


which means 81.2% area of our RoI is taken into account for our final evaluation area.



**Q5: Any way to improve training data and generalization**

A5: Thanks for this insightful question. It aligns with our research interest in efficiently training 3D occupancy prediction models. Some recent works are exploring self-supervised methods in occupancy prediction. One notable example is EmerNeRF (https://emernerf.github.io/), which can represent highly dynamic scenes in a self-sufficient manner. This approach has the potential to label occupancy ground truth using RGB cameras without requiring human annotation efforts.