FusionOcc: Multi-Modal Fusion for 3D Occupancy Prediction

Venue: acmmm_org_ACMMM_2024_Conference
Published: 2024-04-07 12:33:11

Abstract: 3D occupancy prediction (OCC) aims to estimate and predict the semantic occupancy state of the surrounding environment, which is crucial for scene understanding and reconstruction in the real world. However, existing methods for 3D OCC mainly rely on surround-view camera images, whose performance is still insufficient in some challenging scenarios, such as low-light conditions. To this end, we propose a new multi-modal fusion network for 3D occupancy prediction by fusing features of LiDAR point clouds and surround-view images, called FusionOcc. Our model fuses features of these two modals in 2D and 3D space, respectively. By integrating the depth information from point clouds, a cross-modal fusion module is designed to predict a 2D dense depth map, enabling an accurate depth estimation and a better transition of 2D image features into 3D space. In addition, features of voxelized point clouds are aligned and merged with image features converted by a view-transformer in 3D space. Experiments show that FusionOcc establishes the new state of the art on Occ3D-nuScenes dataset, achieving a mIoU score of 35.94% (without visibility mask) and 56.62% (with visibility mask), showing an average improvement of 3.42% compared to the best previous method. Our work provides a new baseline for further research in multi-modal fusion for 3D occupancy prediction.

Main paper: 
 introduction: 
 Perceiving and modeling the surrounding scenes accurately and comprehensively plays an important role in robotics navigation, autonomous driving and so on. Modern researches for 3D perception mainly focus on 3D object detection and semantic map reconstruction, where objects are described by 3D bounding boxes and generated maps are used for path planning.While the 3D bounding box usually erases the geometric details of objects [3, 37] , 3D occupancy prediction (OCC) offers a meticulous partitioning of the 3D space into semantically labeled voxels, which provides richer semantic and geometric expressiveness. 3D OCC is now widely used in the field of autonomous driving, providing drivers with detailed road conditions and a better driving experience.Compared with 3D object detection [40] , LiDAR segmentation [47] , and semantic map reconstruction, estimating the detailed occupancy states and semantics of a scene is more complicated [2, 21] . Besides, the immense number of voxels brings new challenges to computation, necessitating the employment of techniques such as sparse convolution.Some previous works have focused on vision-based occupancy prediction, aiming to avoid additional economic costs from LiDAR sensors. Typically, forward projection (LSS [33] ) and backward projection (BEVFormer [18] ) is widely used among their models, where 2D features of images are transformed into 3D features in BEV space by 2D-3D projection. The core of LSS is depth estimation, which is tough only using monocular images, especially for distant and small objects. Incorrect estimations of depth could lead to the wrong projection of 2D-3D, resulting in poor prediction [24] . Besides, encoding the image features directly into flattened BEV features is not feasible, causing the loss of geometry information.Other works using LiDAR or fusing LiDAR with camera may easily introduce LiDAR-to-camera's distortion and camera-to-LiDAR's sparsity, which have been illustrated by BEVFusion [24] . Besides, geometry distortion by projecting LiDAR to camera becomes severer when the scene is dynamic [15] , and point clouds are much more sparse when compared with dense voxels in OCC task. Thus, it is not feasible to model the occupancy state of the scene solely by temporally accumulating the point clouds.In this paper, we propose a new network named FusionOcc to unify camera and LiDAR features in the shared 3D representation space for 3D occupancy prediction. The model fuses features of these two modals in 2D and 3D space, respectively. We design a cross-modal fusion module supervised by both sparse point clouds and image semantics, which provide an accurate depth estimation for the transition of 2D image features into 3D space. In addition, features of LiDAR point clouds are aligned and merged with image features converted by a view-transformer in 3D space. Finally, the fused features are fed into a specific occupancy head to predict the occupancy state of 3D space. Validated by extensive experiments, our model has great ability to reconstruct smaller objects and distant scenes. Besides, the model is less vulnerable to harsh weather conditions and insufficient lighting, achieving higher scores than all previous methods.In summary, our main contributions are: 1. A new multi-modal fusion model is proposed for 3D occupancy prediction.2. Our model, named FusionOcc, realizes a cross-modal fusion module that incorporates camera images and LiDAR point clouds to obtain an accurate depth estimation, enabling a better transition of 2D features into the 3D space.3. FusionOcc achieves state-of-the-art performance on the OCC3D-nuScenes dataset and sets up a new baseline for 3D occupancy prediction based on multi-modal fusion. 
related work 2.1 vision-based occupancy prediction: 
 Occupancy Networks were originally proposed by Mescheder et al. [26, 32] , focusing on continuously representing objects in 3D space. Recent variations in occupancy networks have shifted their focus to reconstructing 3D space and predicting voxel-level semantic information from image inputs. SurroundOcc [42] proposes a coarse-to-fine architecture for occupancy prediction. OccNet [39] exploits applying universal occupancy features to various downstream tasks and introduces the OpenOcc benchmark.With the release of OCC-3d nuScenes [38] dataset, many excellent methods emerged in the CVPR 2023 occupancy challenge. FB-OCC [19] adopts a unified design that leverages both forward projection (represented by List-Splat-Shoot [33] ) and backward projection (represented by BEVFormer [18] ), promoting the benefits from each method with improved perception results while overcoming their limitations. UniOCC [31] approached the 3D occupancy task as a rendering problem, attempting to solve it using NeRF's [41] methodology. They also employed a Teacher-Student training approach to train the model.Another related task is Semantic Scene Completion (SSC) [8, 21, 43] , and the main difference between SSC and 3D Occupancy Prediction (OCC) is that SSC usually deals with static scenes while OCC focus on dynamic ones. Though these two tasks differ a little, there are still many excellent works on SSC worthy of reference.SSCNet [36] was the first to address SSC using both image and depth map data, utilizing a 3D convolutional network to generate occupancy and labels within a voxel grid. MonoScene [5] designed a unified camera-based approach for indoor and outdoor scenarios, leveraging line-of-sight projection and an innovative frustum proportion loss. OccDepth [27] utilizes stereo images and corresponding depth supervision. Other approaches incorporate the bird-eye-view (BEV) and temporal information for predicting 3D occupancy. This concept has been further expanded with the Tri-Perspective View [35] .Notably, LiDAR-based methods have shown superior performance compared to camera-based approaches in outdoor scenes. Christoph et al. [34] propose a scene segmentation model for scene completion. A global scene completion function is subsequently assembled from the localized function patches. S3cnet [7] utilizes a sparse convolution based neural network to fuse 2D and 3D point features. Apart from these methods, datasets such as SSCBench [16] and SemanticKITTI [21] , providing annotated ground truth for SSC evaluation, facilitating a more comprehensive assessment of SSC methods.Recently, multi-modal fusion has aroused increased interest in the 3D perception and scene understanding community. Early approaches can be classified into proposal-level fusion and point-level fusion. Proposal-level fusion methods are object-centric and cannot trivially generalize to other tasks such as BEV map segmentation. Point-level fusion methods, on the other hand, usually paint image semantic features onto foreground LiDAR points and then perform LiDAR-based detection on the decorated point cloud inputs, such as Deep Continuous Fusion [20] , DeepFusion [17] and so on.Besides, BEVFusion [24] unifies multi-modal features in a shared bird's-eye view representation space for task-agnostic learning, where both geometric structure and semantic density are maintained. ObjectFusion [4] learns three kinds of modality-specific feature maps (i.e. voxel, BEV, and image features) from the LiDAR point cloud and its BEV projections for 3D object detection, where features of three modalities are further fused at the object level and finally fed into the detection head. 
methodology: 
 The overview of our model is in Fig. 1 . FusionOcc is composed of three main components: (1) A points branch that generates 3D point clouds features through a voxelization of point clouds and a voxel encoder. (2) A images branch that generates 3D image features through a cross-modal fusion of 2D semantics and depth maps, and a 2D-to-3D transformation. (3) An occupancy head that merges the 3D features of images and point clouds, then encodes the fused features to produce occupancy prediction. Specifically, the input of multiple frames of point clouds are merged to obtain relatively dense point clouds. After voxelization, sparse convolution is employed to generate the 3D features of point clouds. Along with the LiDAR branch, the camera branch utilizes images from camera and depth information from LiDAR. A cross-modal fusion module is adopted to fuse features of these two modals. Then, a viewtransformation is adopted to obtain image features that are aligned with features of point clouds in common 3D space. Finally, features of two branches are merged and encoded by a 3D convolution network and then produce a prediction for the 3D occupancy state.Given the multi-modal inputs of point clouds and image data, two modality-specific encoders extract point cloud features and image features respectively. Formally, for the points branch, the input point clouds consist of a set of points and each point is represented as a 5-dimensional vector 𝑝 = (𝑥, 𝑦, 𝑧, 𝑖, 𝑟 ), where 𝑥, 𝑦, 𝑧 is the coordinates in 3D space, 𝑖 is the reflection intensity and 𝑟 is the ring index.The voxel encoder encodes the input LiDAR point cloud. Due to the sparsity of the original point cloud, the voxel encoder, correspondingly, utilizes sparse convolution and sparse down-sampling module rather than common 3d convolution to reduce computation. We use VoxelNet [6] as our backbone. As the final occupancy task requires the voxel feature map to be consistent with the size and range of feature maps of images in the shared 3D space, we voxelize the LiDAR point cloud with 0.05m and apply a 8x downsampling in the voxel encoder. The feature of point cloud in 3D space 𝐵 𝑙 ∈ 𝑅 𝐶 ×𝐷 ×𝐻 ×𝑊 is generated by a dense convert operation followed sparse convolution.Along with the points branch, the images branch is set to extract 3D features of multi-view images. The architecture of the camera branch is more complex compared with the LiDAR branch, and could be divided into three sub-parts.Image Encoder. The input multi-view images of the branch represents a set of RGB images 𝑅 𝐻 ×𝑊 ×3 , where 𝐻 and 𝑊 is the height and width of the image. To exploit the power of multi-resolution features, the image encoder includes a backbone for high-level feature extraction and a neck for multi-resolution feature fusion. Besides, we utilize the temporal-fusion of multiple camera frames proposed by BEVDet4D [12] . Specifically, SwinTransformer [23] is utilized to extract 2D image feature maps and FPN-LSS [13] is adopted to fuse all feature maps of different scales into a single feature map. The module down-samples the feature to 1/16 input size.Cross-modal Fusion. Modern methods unify multi-modal feature maps via BEV-based fusion where image features are projected into BEV space through camera-to-BEV transformation. However, the camera-to-BEV transformation hinges on pre-learned depth estimator to obtain the inherently ambiguous depth estimation of each pixel. Any inaccurate depth estimation can potentially result in spatial misalignment between the image feature maps and point feature maps. Besides, it is worth noting that semantic-level supervision from 2D images could enhance the prediction for 3D occupancy.Based on that, a cross-modal fusion module is designed to unify images and sparse depth maps generated by corresponding point cloud at various levels. Dense depth map of RGB images is predicted by the module. Besides, a segmentation branch is adopted to predict the 2D semantic masks alongside the depth prediction task and two branches' features are interacted with each other via a crossattention mechanism. Details are shown in Fig. 2 , the inputs of the cross-modal fusion module are the feature map of RGB images 𝐹 𝑐 ∈ 𝑅 𝐻 ×𝑊 ×𝐶 and the features 𝐹 𝑑 ∈ 𝑅 𝐻 ×𝑊 ×𝐶 of 2D depth map from point clouds. The 2D depth map is down-sampled x16 to keep the same resolution as image feature map and encoded using onehot. Then, global average pooling is applied to 𝐹 𝑐 and 𝐹 𝑑 along the channel to obtain F𝑑 and F𝑐 ∈ 𝑅 𝐶 . An MLP layer with a sigmoid function is followed to obtain 𝑊 𝑐 and 𝑊 𝑑 :EQUATIONSimilar to SE-Net [11] , we use𝑊 𝑐 and𝑊 𝑑 as the excitation module and a fused feature is obtained by element-wise multiplication along channel:EQUATIONFollowed by concatenating, fusing and splitting 𝐹 𝑐2𝑑 and 𝐹 𝑑2𝑐 . Another average pooling along spatial level is applied to obtain F 𝑑 and F𝑐 ∈ 𝑅 𝐻 ×𝑊 . Then, an 1 × 1 convolution layer with Relu function is followed to obtain 𝑍 𝑐 and 𝑍 𝑑 : Likewise, we use 𝑍 𝑐 and 𝑍 𝑑 to obtain fused features by the element-wise multiplication and the original feature is added up to it:EQUATIONEQUATION𝛼 is a hyper-parameter and the default value of them is set as 1 in Section 4. Features 𝐹 𝐶 𝑐2𝑑 and 𝐹 𝐶 𝑑2𝑐 are further concatenated for depth estimation and 2D semantic segmentation task, and the feature of segmentation branch is fed into the view transformation module to be projected into 3D space. The training strategy is designed to let the module to reconstruct the original depth maps generated by point clouds via randomly masked ones, then it could produce a dense depth map during the inference phase (see Fig. 3 ).Also, dataset such as Occ3D-nuScenes [38] does not provide semantic segmentation labels for 2D images. To ensure consistency in semantic label categories and do not leverage other pre-trained models to generate pseudo labels, we use nuSences-lidarSeg [9] and 3D voxel labels to get the 2D semantic label via projection and up-sampling (see Fig. 4 ). We quantize the depth values of the point clouds to a fixed range, and classification loss is used to train the depth estimation and semantics segmentation branch. The total loss 𝐿 𝑓 𝑢𝑠𝑒 is the sum of depth loss 𝐿 𝑑𝑒𝑝𝑡ℎ and segmentation loss 𝐿 𝑠𝑒𝑔 : where 𝐻 and 𝑊 are the height and width of the feature map which is down-sampled x16 compared to the input. 𝑦 denotes the label and 𝑝 denotes the prediction.EQUATION2D-to-3D. To obtain 3D image features, a view transformer [33] is utilized to transform the feature from image view to 3D space. The dense depth map predicted by cross-modal fusion module could assist in projecting the image feature into 3D space to obtain feature 𝐵 𝑐 ∈ 𝑅 𝐶 ×𝐷 ×𝐻 ×𝑊 .Finally, 3D features of images and point clouds are merged along channel 𝐶 to produce a unified 3D feature. The unified feature is furtherly encoded in 3D space and fed into an occupancy head to predict the 3D occupancy state. We utilize ResNet [10] with classical residual block to construct the module and combine the features with different resolutions by applying FPN-LSS [13] . A crossentropy loss funtion 𝐿 𝑜𝑐𝑐 is used to train the occupancy prediction and the final loss is a weighted sum of 𝐿 𝑜𝑐𝑐 and 𝐿 𝑓 𝑢𝑠𝑒 :EQUATION)where 𝜆 is a hyper-parameter and is set as 0.1 in our model. 
experiments: 
 Our model is evaluated on the Occ3D-nuScenes [38] dataset. Occ3D-nuScenes is based on the nuScenes [3] dataset, which consists of large-scale multi-modal data collected from 6 surrounding cameras, 1 top LiDAR and 5 radars. The dataset is split into training/validation parts including 700/150 clips. The occupancy scope in the dataset is defined as -40m to 40m for X and Y axis, -1m to 5.4m for the Z axis, and 0.4m for the voxel size. Thus, the shape of the occupancy block is [200, 200, 16] . For 3D occupancy prediction task, the official evaluation metric uses mean Intersection over Union (mIoU) to evaluate models' performance:𝑚𝐼𝑜𝑈 = 1 𝑁 𝑁 ∑︁ 𝑖=1𝑇 𝑃 𝑖 𝑇 𝑃 𝑖 + 𝐹 𝑃 𝑖 + 𝐹 𝑁 𝑖 (7) where 𝑇 𝑃 𝑖 , 𝐹 𝑃 𝑖 , 𝐹 𝑁 𝑖 represent the number of voxels that are predicted as true-positive, false-positive and false-negative of the class 𝑖, 𝑁 is the number of classes.We implement the proposed FunsionOcc in the PyTorch framework, based on the MMdetection3D and BEVDet [12] codebases. For the image encoder, we use the SwinTransformer [23] network as the backbone and FPN-LSS [13] to fuse multi-scale feature maps. Input images are cropped to remove invalid parts. The image backbone is pretrained on the nuImage [3] dataset for the 2D detection task. For the voxel encoder, we use VoxelNet [6] as the backbone to obtain point clouds features and down-sample the original point clouds by 8x. The voxel size of lidar is set as [0.05m, 0.05m, 0.05m], and the point clouds range is [-40m, -40m, -1m, 40m, 40m, 5.4m] in X, Y, and Z-axis, respectively.During training, we utilize random flip, and random sample to augment the LiDAR data. Following common practice [1] , we align the previous nine LiDAR sweeps into the current frame for a denser point cloud. For images, we additionally use random flip, random rotation in and random resizing in to augment the images. AdamW [25] is used as the optimizer to train the model with a global batch size of 8. The initial learning rate is set as 1e-4 and is decayed with cosine annealing policy. For all experiments, we train our models for 20 epochs on 8 NVIDIA Tesla V100 GPUs.In Table 1 , we compare FusionOcc with previous state-of-the-art methods on the validation split of Occ3D-nuScenes dataset. In order to make a fair comparison of the performance of these methods, we divide these methods into two groups based on whether they use the visibility mask or not, and the group where the visibility mask is used are marked with * .Methods that do not use the visibility mask include OccFormer (camera-based method), VoxelNet (LiDAR-based method), BEV-Fusion (multi-sensor method) and so on. The input of VoxelNet is a fused point cloud of 8 adjacent frames and the output is encoded by the same encoder used in FusionOcc. 2DPASS-acc denotes the outstanding point cloud segmentation method, we accumulate the segmentation results of multi-frame point clouds to generate dense occupancy blocks. It is shown that although 2DPASS-acc has achieved good performance in point cloud segmentation tasks, the occupancy prediction based on the accumulated segmentation of multi-frame point clouds is relatively poor. Also, BEVFusion shares the same backbone, input size and OCC head with FusionOcc to keep fair. FusionOcc achieves the highest mIoU score of 35.94% among methods that do not use the visibility mask. As for methods that use the visibility mask including FB-OCC, UniOcc and so on, which have all achieved top scores in CVPR 2023 OCC competition, our model still outperforms them with a mIoU score of 56.62%.Effect of Cross-modal Fusion. We employ a cross-modal fusion module to incorporate dense depth estimation and 2D image semantic segmentation. To testify its effectiveness, we conduct experiments in Table 2 . Column 1 represents whether the model utilizes an additional learning branch where 4 different settings are adopted. Column 2 represents the fusion method used by the model.From the table, it is observed that both the semantic branch and the depth branch have a positive impact on the accuracy of the model. The model with the two learning branches achieves the highest mIoU of 56.62%, which is a 2.56% improvement compared to the method without fusion and any additional learning branches. Besides, the designed cross-modal module has a more significant improvement (0.79%) in model accuracy compared to simply concatenating the features of the two modalities.Effect of Merging Multi-frames. Since the point cloud at a single timestamp is sparse, we attempt to merge multiple frames to obtain a relatively dense point cloud. In Table 3 , the mIoU, mAcc and F1 score of our model(with the visibility mask) is observed along different numbers of frames to be merged. From rows 1-4 of the table, it can be proved that the merging of point clouds of multiple frames is very effective. The performance of our model steadily improves as the number of frames increases, and the model Table 1 : 3D occupancy prediction performance on the Occ3D-nuScenes validation set. * denotes using the visibility mask. Our method achieves a higher performance than previous methods. be a significant change in the performance of the model (less than 1% point fluctuation).In addition, the number of parameters of our model is compared.Results are still divided into two groups (with and without visibility mask) for a fair comparison. We can see that in Tab. 4, even with a relatively small model size, FusionOcc can achieve a higher score on mIoU. To lighten the model, we adopt MobileSAM [45] as our backbone in the image branch(marked as tiny). MobileSAM is the ultra-lightweight version of SAM(Segment Anything Model) [14] , which has been pre-trained on various open-domain datasets. As shown in the table, the model's size could be reduced to half without much accuracy loss.Performance in Different Scenarios. The accuracy of scene understanding and modeling is often greatly influenced by environmental conditions, such as rainy days and at night. Multi-modal fusion helps to improve the model's robustness in different scenarios. As seen in Fig. 5 , our model FusionOcc achieves the top scores in three different scenarios. Even in rainy days when LiDAR is likely to be interfered, the accuracy of our model remains relatively stable with little change.Performance Varies with Different Perception Distances. Multi-modal fusion not only improves the final model's robustness to different illumination and weather conditions but also extends the model's perception range. We take the vehicle as the center and define 𝑅 as the distance from the center. By setting different 𝑅 = [15𝑚, 20𝑚, 25𝑚, 30𝑚, 35𝑚, 40𝑚] (with the maximum value of R is 40), we study the performances of different models and modal fusion methods under different perception ranges.As seen in Fig. 6 , by fusing modals of cameras and LiDAR point clouds, the model's ability to perceive objects at extended distances is significantly improved. This phenomenon can be attributed to Li-DAR data having a longer detection range and providing the modal of images with accurate depth information. Although features of point clouds are sparse, they can be combined to nearby features from other modalities, thereby enhancing the ability of the final merged feature to model the 3D scene.Performance Varies with Random Cameras Loss. We demonstrate the robustness of our proposed modal against inferior camera conditions on the dataset. As shown in Fig. 7 , we can see that even with a camera input lost, the model could still achieve a 53.66% mIoU score. With 3 camera inputs lost, the model achieves a 47.40% mIoU score. As expected, when the number of lost cameras increases, the gap between the performance of models that uses multi-modal fusion and single visual modality gradually widens. The reason is that when information from one modality is lost, the other modality can compensate for it.In addition, we visualize the occupancy prediction of FusionOcc compared with VoxelNet, BEVdet4d-occ, and BEVFusion (see Fig 8) . BEVFusion is a multi-sensor fusion method designed for the 3D object detection task and map segmentation, we modify it to be suitable for occupancy prediction task based on its architecture. VoxelNet and BEVdet4d-occ are chosen to represent the OCC model that only uses LiDAR or cameras.In Fig. 8 , three different scenarios are chosen (daytime, rainy day, and nighttime from the top to the bottom). In the daytime scene, we can see that BEVFusion misclassified the road surface and fence in the distance, while our model still predicts them correctly. In the rainy day scene, VoxelNet which uses only LiDAR as the input is the worst, and the prediction is badly influenced by motion blur of the vehicle. FusionOcc is the best compared to BEVDet4d-occ and BEVFusion, proved by its modeling of fine fences and bushes in close proximity. In the nighttime scene, models misclassify small subsections of the scene while our FusionOcc is still the one closest to the GT.  
conclusion: 
 This paper proposes a new 3D occupancy network, named Fu-sionOcc, which is based on the fusion of camera and LiDAR features. FusionOcc realizes a cross-modal fusion module that incorporates images and point clouds to obtain an accurate depth estimation, enabling a better transition of 2D features into 3D space. Additionally, a new baseline for 3D occupancy prediction based on multi-modal fusion is established. Experiments show that FusionOcc achieves the state-of-the-art performance on Occ3D-nuScenes for mIoU. We 
 
 Appendix:  


Decision: 3D occupancy prediction is an important problem, and the authors propose a multi-modal fusion method that achieves encouraging performance. All the reviewers agree to accept this manuscript. The authors should carefully address the concerns in the final version, such as the writing of the paper, ablation studies, and some experimental comparisons.

Reviews: review: paper_summary: paper_summary: This paper proposes a new multi-modal fusion network for 3D occ prediction by fusing features of LiDAR point clouds and surroundview images. By integrating the depth information from point clouds, a cross-modal fusion module is designed to predict a 2D dense depth map, enabling an accurate depth estimation and a better transition of 2D image features into 3D space. In addition, features of voxelized point clouds are aligned and merged with image features converted by a view-transformer in 3D space. Experimental results show its effectiveness and superiority among these SOTAs in occ prediction.
strength_weakness: strength_weakness: 1. Promise performance in occ prediction. The proposed model overpasses the current SOTAs.
2. Easy to understand and interesting video in supplementary material.
3. Simple but effective motivation for integrating LiDAR point clouds for occ prediction.
4. Sufficient about proposed techniques.
limitations: limitations: My rating is borderline and will be adjusted based on the author's answer.

1. Some current works about converting 2D images into 3D voxels should be discussed and compared, especially in the introduction and experiments like PanoOcc, Occformer or RenderOcc. Row 69-79 can't not accurately summarize the latest progress of the current occ model by utilizing the deformable attention (PanoOcc or TPVFormer), especially TPVFormer designs TPV representation for balancing the  geometry information and memory cost. 

2. Poor writing skills. What's the meaning of the "forward/backward projection" like row 71-72 ?  What's the difference between the previous works by utilizing Lidar-camera fusion and FusionOcc? How does the proposed model overcome the LiDAR-to-camera’s distortion problem? 

3. Massive  logic in methods. What's the depth encoder?  How to exact the sparse depth from the LiDAR points? 
"Dense depth map of RGB images is predicted by the module." What's "the module" ? Too many questions about the how to utilize depth information.

4. Unclear model setting and unfair compare. Mentioned that the proposed model utilizes the SwinT as backbone (MobileSAM) and temporal-fusion tricks (how to use it?), I can't see any ablation study about this, espeically some works utilize the R101-DCN as backbone like RenderOcc or BEVFormer. I think this work may mainly develop on the Bevdet codebase. For accurate evaluating these methods, authors should give more clear explanation about the model setting. 

5. Authors should add more ablation experiments about $\alpha$ in Eq.4.
paper_summary: paper_summary: This paper proposes a network for fusing LiDAR features and image features for 3D occupation prediction. The key idea is to use LiDAR point clouds to get depth information and generate depth images. And depth images are first fused with color images by a cross-modal fusion module and the fused images features are further fused with LiDAR features. Occupancy prediction loss, image segmentation loss, and depth prediction loss are jointly used to supervise learning. The proposed method achieves SOTA results.
strength_weakness: strength_weakness: 1. Writing and the motivation is clear and reasonable.
2. The proposed method achieves SOTA results.
limitations: limitations: 1. The key contribution of the proposed method is multi-modal fusion. However, the paper primarily demonstrates the fusion process without providing substantial insights into the design of the cross-modal fusion module. Consequently, the new knowledge introduced in this paper is somewhat limited.
2. From Table 2, the use of the cross-modal fusion module shows only marginal improvement over simple linear layers. This raises doubts about the effectiveness and necessity of the cross-modal fusion module, which is purportedly the most significant contribution of the paper.
paper_summary: paper_summary: This paper aims to integrate the features of LiDAR point clouds and surround-view images for 3D occupancy prediction task.
The main contributions include a cross-modal fusion module that combines camera iamges with LiDAR point clouds.
It achieves the state-of-the-art performance on the OCC3D-nuScenes dataset.
strength_weakness: strength_weakness: The proposed multi-modal fusion model incorporates camera images and LiDAR point clouds with a cross-modal fusion module.
And obtain accurate depth estimation to realize better transformation of 2D features into 3D space.
The method achieves the state-of-the-art performance on the OCC3D-nuScenes dataset.
limitations: limitations: 1. The proposed method actually uses multiple frames of LiDAR point clouds and camera images, but in Figure 1, only one frame is displayed, and the sparse depth only has one view. It is better to display multi-frames of data and multi-view sparse depth.
2. In line 471, the authors claim that the dense depth map could assist in projecting the image feature into 3D space. Does this mean the dense depth can be used or not used in the 2D to 3D projection process. What is the impact of using dense depth on the final results?
3. The proposed method used 2 frames of camera images and 8 frames of LiDAR point clouds. It is better to point out the number of frames used by the comparison methods in Table 1. Because the number of frames used has a significant impact on the mIoU results.
4. In line 521, nine previous LiDAR sweeps are aligned into the current frame, which means that 10 frames of LiDAR point clouds are used during training, but according to Table 1 and Table 3, only 8 frames are used.
5. In line 725, camera input lost means that the network input does not include the camera image and corresponding sparse depth, or just set the RGB value of the camera image to a specific value.
paper_summary: paper_summary: This paper proposes a new 3D occupancy network, named FusionOcc, which is based on the fusion of camera and LiDAR features.
FusionOcc realizes a cross-modal fusion module that incorporates images and point clouds to obtain an accurate depth estimation, enabling a better transition of 2D features into 3D space. Additionally, a new baseline for 3D occupancy prediction based on multi-modal
fusion is established. Experiments show that FusionOcc achieves the state-of-the-art performance on Occ3D-nuScenes for mIoU.
strength_weakness: strength_weakness: 1. This paper proposes a new multi-modal fusion model is proposed for 3D occupancy prediction FusionOcc.
2. FusionOcc realizes a cross-modal fusion module that incorporates camera images and LiDAR point clouds to obtain an accurate depth estimation, enabling a better transition of 2D features into the 3D space.
3. FusionOcc achieves state-of-the-art performance on the OCC3DnuScenes dataset and sets up a new baseline for 3D occupancy prediction based on multi-modal fusion
limitations: limitations: 1.In fact, FusionOCC is BEVFusion combined with SENet for occupancy prediction. Please specifically explain the differences between them and the advantages of FusionOCC

2.In Figure 1, FusionOCC uses a weakly supervised method (similar to MAE) to supervise the training of the depth prediction module, aiming to use sparse or dense depth maps to obtain accurate 3D features. However, this method of obtaining depth is inaccurate and can lead to errors between the generated 3D features and the voxel features generated by LiDAR. Does this inaccuracy significantly impact occupancy prediction?

3.In Table 1, it is unreasonable to compare FusionOCC, which uses multimodal data, with other occupancy predictors that use only single modalities. It is recommended to compare FusionOCC with more methods that also use multimodal data

4.In Figure 8, the visualization shows the prediction results of FusionOCC and other methods. It is recommended to highlight the advantages of FusionOCC in the predictions. Currently, it is difficult to see the benefits of FusionOCC from the paper.