GaussianFusion: Unified 3D Gaussian Representation for Multi-Modal Fusion Perception
Authors: Xiao Zhao
Chang Liu
Mingxu Zhu
Zheyuan Zhang
Linna Song
Qingliang Luo
Chufan Guo
Kuifeng Su
Venue: ICLR 2026 Poster (2026)
Keywords: Gaussian Representation
BEV Representation
Detection
Occupancy
Primary area: applications to robotics, autonomy, planning

Abstract: The bird’s-eye view (BEV) representation enables multi-sensor features to be fused within a unified space, serving as the primary approach for achieving comprehensive multi-task perception. However, the discrete grid representation of BEV leads to significant detail loss and limits feature alignment and cross-modal information interaction in multimodal fusion perception. In this work, we break from the conventional BEV paradigm and propose a new universal framework for multi-task multi-modal fusion based on 3D Gaussian representation. This approach naturally unifies multi-modal features within a shared and continuous 3D Gaussian space, effectively preserving edge and fine texture details. To achieve this, we design a novel forward-projection-based multi-modal Gaussian initialization module and a shared cross-modal Gaussian encoder that iteratively updates Gaussian properties based on an attention mechanism. GaussianFusion is inherently a task-agnostic model, with its unified Gaussian representation naturally supporting various 3D perception tasks. Extensive experiments demonstrate the generality and robustness of GaussianFusion. On the nuScenes dataset, it outperforms the 3D object detection baseline BEVFusion by 2.6 NDS. Its variant surpasses GaussFormer on 3D semantic occupancy with 1.55 mIoU improvement while using only 30% of the Gaussians and achieving a 450% speedup.

Published as a conference paper at ICLR 2026
GAUSSIANFUSION: UNIFIED 3D GAUSSIAN REPRE- SENTATION FOR MULTI-MODAL FUSION PERCEPTION
Xiao Zhao; Chang Liu, Mingxu Zhu, Zheyuan Zhang, Linna Song, Qingliang Luo, Chufan Guo Kuifeng Su Tencent, Autonomous Driving Lab Beijing, 100000, CN
(shawzhao, changeliu atnncnn com
ABSTRACT
The bird's-eye view (BEV) representation enables multi-sensor features to be fused within unified space, serving the primary approach for achieving com- prehensive 3D perception. However, the discrete grid representation of BEV leads significant detail loss and limits feature alignment and cross-modal information interaction in multimodal fusion perception. this work, we break from the conventional BEV paradigm and propose
universal framework for multi- modal fusion based on 3D Gaussian representation. This approach naturally uni- fies multi-modal features within shared and continuous 3D Gaussian space, ef- fectively preserving edge and fine texture details. To achieve this, we design a novel forward wwwaard-rooeccton-- shared cross moddll Gaussian encoder that iteratively updates Gaussian properties based on an attention mechanism. GaussianFusion is inherently a task-agnostic model, with its unified Gaussian representation naturally supporting various 3D perception tasks. Extensive experiments demonstrate the generality and robust- ness of GaussianFusion. On the nuScenes dataset, it outperforms the 3D object detection baseline BEVFusion by 2.6 NDS. variant surpasses GaussFormer on 3D semantic occupancy with 1.55 mloU improvement while using only 30% of the Gaussians and achieving a 450% speedup.
multi-modal Gaussian initialization module and a
INTRODUCTION
Fusing complementary signals captured by different sensors is essential for autonomous driving perception systems. Leveraging the distinct characteristics of each sensor helps reduce prediction uncertainty, leading to more accurate and robust perception outcomes (Liu et al., 2023b; Bai al., 2022: Yan et al., 2023). Since different sensors present data in varying formats, such cameras providing perspective semantic data and Lidar capturing 3D spatial information, multi-modal fusion faces significant challenges due to these view discrepancies. To address this, some methods (Vora al., 2020; Bai et al., 2022; Li et al., 2024; Wang et al., 2024b) achieve multi-modal object detection through point-level fusion. However, point-level fusion strategies are generally unsuitable for semantics-oriee orienntd 3D perception tasks like 3D semantic occupancy prediction. Consequently, recent approaches aim to construct unified representations for multi-modal feature fusion.
Recently, the shared Bird's Eye View (BEV) space has emerged as promis- ing direction for fusing cross-modal features enable task-agnostic leamn ing. Several existing fusion methods (Liu et al., 2023b: Wang et al., 2023a), such as BEVFusion (Liu et al., 2023b), integrate multimodal information via CNNs and feature concatenation, while MetaBEV (Ge al., 2023) mitigates
Table 1: Impact of BEV size model performance
Method
BEV size
Grid size
Memory!
NDS
100x100
1.008m
3228 M
70.5
BEVFusion
200x200
0.504m
M
71.4
400x400
0.252m
20560
72.7
100x100
1008mm
3576 M
73.1
GaussianFusion
200x200
0,504m
5418 M
74.0
400x400
0.252m
6151 M
74.4
"Equal contribution.

---

Published as a conference paper at ICLR 2026
cross-modal feature misalignment by in- troducing meta-BEY queries. However, despite its widespread adoption in multimodal 3D percep- tion, the BEV representation inherently suffers from limitations in information expression.
BEV directly discretizes and quantizes data, leading to inevitable information loss. Dur- ing feature extraction, perception data are pro- jected onto fixed-resolution BEV grid, which compresses spatial information. This issue be- comes particularly severe when the BEV res- olution is low, as it directly impacts model performance by failing to adequately preserve fine-grained scene structures. While increas- ing the BEV resolution will bring unacceptable computational overhead, as shown in Table 1. Additionally, BEV fusion strategies often rely on simple feature concatenation summation, which are insufficient for effective cross-modal feature interaction and alignment, ultimately leading to suboptimal fusion perfor- mance, as illustrated in Fig. I(a).
Voxel
Gaussian
Camera
Lidar
Camenn
Lidar
9C
GL
Bc
B
weighted
Gaussian Encodder
Encoder
Se
BF
To address these challenges, we introduce a fu- sion approach based on 3D Gaussian Splatting (3DGS) (Kerbl et al., 2023) to achieve more ime-grained information modeling and more natural multimodal alignment. As shown in Fig. 1(b), 3DGS employs continuous Gaussian distributions to represent the scene, preserv- ing rich geometric and semantic information in the Gaussian stage and preventing the early quantization-ina induucd information loss seen in BEV-based methods. Unlike direct BEV quantization, 3DGS aggregates information before its final projection onto the BEV grid, allowing cross-modal features to interact at a higher-dimensions level and capturing finer spatial structures prior quantization, Table shows the effectiveness of this strategy. Moreover, the covariance matrices of Gaussians enable adaptive modeling of uncertainty, enhancing the representation of object shapes and boundaries.
(a) BEV-based Fusion
(b) Gaussian-bases Fusion
Figure 1: sentation fusion paradigm (Liu al., 2023b) and our proposed continuous Gaussian representation fusion paradigm. B, 9, C, L,, and F denote BEV, Gaussian, Camera, Lidar, and Fusion.
Comparison of the discrete BEV repre-
Specifically, inspired by (Philion & Fidler, 2020), we propose a forward projection Gaussian initial- ization strategy better initialize camera Gaussian representations in 3D space rather than using random initialization (H et al., 2024). To further achieve continuous alignment and cross-modal fea- ture enhancement, we construct a shared Gaussian encoder. The shared Gaussian encoder supports cross-feature learning of 3D Gaussian features from different modalities, where the covariance ma- trix of each 3D Gaussian adaptively captures feature differences between modalities and iteratively updates the Gaussian parameters. Camera and LiDAR Gaussians are naturally fused via a Gaussian mixture model, and a high-performance Gaussian-tor to-voell fusion module aggregates surrounding Gaussians to generate voxel features, enabling task-agnostic 3D perception. We conduct extensive experiments on BEV object detection and 3D occupancy prediction tasks to validate the generality of the GaussianFusion. Main contributions
follows:
We propose the first unified 3D Gaussian representation multi-modal fusion framework, where cross-view and crosl-modal Gaussian representations are naturally aggregated through the Gaussian mixture model.
A progressive update strategy designed to optimize the multi-modal 3D Gaussian prop- erties iteratively.
The shared 3D Gaussian encoder enables alignment and complementary enhancement of cross-modal features, allowing Gaussian representations from both modalities to achieve consistent uncertainty within unified space.
2

---

Published as a conference paper at ICLR 2026
Our GaussianFusion achieves state-of-the-ar benchmarks in task-agnostic methods on var- ious 3D perception tasks within the nuScenes dataset.
2 RELATED WORK
2.1 MULTIMODAL 3D PERCEPTION
We categorize current multimodal fusion methods into object-centric methods and dense BEV meth- ods. Object-centric methods (Vora et al., 2020; Chen et al., 2023; Zhou & T, 2023; Yin et al.. 2024; Li et al., 2024; Wang et al., 2024b) are specifically designed for tasks such as 3D object detection or tracking. Advanced object-centric (Li et al., 2024; Wang et al., 2024b) methods typically use 2D detection results on camera to enhance multi-modal fusion 3D detection. Additionally, some works (Yang al., 2022; Yan et al., 2023) query based 3D detection decoders learn features from perspective images and Lidar BEV features directly. However, these object-centric methods cannot easily generalize to dense semantic tasks such BEV map segmentation and 3D occupancy predic- tion. Dense BEV methods (Li et al.. 2022a; Liu et al., 2023b; Liang et al., 2022: Chen et al., 2022; Zhao et al., 2024b; Ge al., 2023; Jiao et al., 2023: Wang al., 2023a) naturally adapt to various tasks. Both BEVFusion (Liu et al., 2023b) and UniTR et al., 2023a) achieve multi-modal BEV fusion perception through CNN and feature concatenation in the BEV space. In addition, MetaBEV (Ge et al., 2023) proposed learnable cross-attention mechanism to generate unified BEV features. BEV or 3D voxel also provides a unified representation for 3D occupancy prediction. Some meth- ods design fusion modules (Pan et al., 2024; Wang et al., 2023c; Ming et al., 2024) based on voxel, such as adaptive fusion (Wang et al., 2023c), etc. for mullii-mddll 3D occupancy prediction. There also many camera-only methods (Zhao et al., 2024a; Wang al., 2024a: Li et al., 2023; Lu et al., 2023; Tian et al., 2024; Li et al., 2025; Ma et al., 2024b; Cao et al., 2024; Liu et al., 2024; Ma et al., 2024a) for 3D semantic occupancy prediction based voxel representation. However, discrete voxel representations may result in significant detail loss and hinder effective multimodal complementary fusion.
2.2 3D GAUSSIAN SPLATTING
3D Gaussian Splatting (3DGS) (Kerbl al., 2023) combines the advantages of implicit neural radi- ance fields (Mildenhall et al., 2021) and voxel-based explicit radiance fields (Fridovich-Ked et al., 2022; Müller al., 2022) and is widely applied in 3D reconstruction of 2D image. 3DGS uses set of Gaussian functions to capture the geometric shapes and semantic of different objects or regions within scene, effectively representing the scene. Based on this, some works (Ye et al., 2025; Hu et al., 2024) leverage DGG'ss multi-view synthesis capabilities to achieve 3D scene segmentation.
Recent studies (Gan et al., 2024; Chabot al., 2024: H et al., 2024; Zuo al., 2024; Liu al., 2025) have applied 3DGS to vision-only 3D semantic occupancy prediction, BEV segmentation, and end- to-end autonomous driving. However, these methods rely on randomly initialized Gaussians. For example, GaussianFormer (H et al., 2024) randomly initializes and re- pr-eddictts Gaussian parameters each iteration, making fine-tuning difficult and limiting accurate scene representation. Moreover, these methods do not fully exploit the advantages of Gaussian Mixture Models (GMM) for seamless multimodal Gaussian fusion. In contrast, we propose a forward wwwaard-rojection---e initialization and shared optimization model, leveraging GMM to fuse multi-view camera and LiDAR features within a shared space, ultimately enabling dense semantic understanding and object- centrie multitask perception.
3DGS parameter
METHODS
3.1 OVERALL ARCHITECTURE
The overall pipeline of GaussianFusion is illustrated in Fig. 2, with the goal of fusing multimodal features through 3D Gaussian representations, which naturally preserve both geometric and semantic information. We first initialize separate 3D Gaussian representations for camera and Lidar, denoted as G. ~ Qu and G. ~ Qt., within a unified space. Then, the multimodal Gaussian sets are processed through shared Gaussian Encoder, enabling the integration of semantic and geometric information
3

---

Published as a conference paper at ICLR 2026
Moltimedal backbone
Gaussian
Task-Sprcific Perception
Deformsble Attettonn Causson
Fasion
Calpodan Upykring
Mulli-Camers Featurcs
Cuny Nis
Guueizn-Io- Vesel Pooling
Lica BEV Featuess
St. Q
Sy Qz
5L &
Occupancc
6d) Gumviee
(b) Defermahle
(c) Gaussian to Vosel Poollnng
Initalzzrion
Properties
Mean
(CS2)
Cameras
Context
Projection
Overview of the GaussianFusion framework. Initial Gaussians are refined by shared encoder and fused in Gaussian space, followed by task-specific heads for 3D perception.
Figure 2:
from both modalities. Finally, the learned 3D Gaussian sets Ś. Q. and G.. Q are fused within the unified Gaussian space and fed into task-specific heads to perform 3D perception.
3.2 GAUSSIAN INITIALIZATION
BEVFusion (Liu al., 2023b) projects multimodal features into discrete BEV space for fusion. In contrast, inspired by the Gaussian mixture model paradigm of 3DGS (Kerbl et al., 2023) for modeling scene geometry and semantics, we utilize two Gaussian sets, Gc and GL. within shared space to represent surround-view camera and Lidar information, respectively, to achieve seamless multimodal fusion.
Camera Gaussian Initialization with Forward Projection. The properties of every single 3D Gaussian function are defined by mean μ E 3 scale s E RR and rotation vectors r E R Given N surround camera view, each camera's 's view can be represented by set of 3D Gaussian distributions Ga,i DD xWeli = 2, N, where Dg = μ + s r, D. denotes the number of discrete depths for camera, shown in Fig. 2(a), Gaussian function is assigned to a depth point. It is worth noting that this is completely different from GausianFormer (H et al., 2024), which randomly initializes a set of Gaussians in space, which makes model optimization more difficult. Specifically, inspired by (Huang & Huang. 2022b;a; Liu et al., 2023b; Philion & Fidler, 2020), given surround camera input features Fe,, HxCHH.WW H.xWe, WW, C, H. and W, represent the channel, height, and width of the camera features. Fe,; is fed to LSS (Philion & Fidler, 2020) to obtain the depth distribution D; E RP, i 1.2. .N, N, D; then used as the initial mean of the Gaussian, which is the location of every single Gaussian center. For s and r, we initialize them randomly. And G. = (G.,), E x
1,2, ..... .N, where
The image features Fa, are processed through context network composed of multiple convolu- tional layers to obtain the semantic features F",, Next, an inner product is computed between D, and F to derive the features at each depth point in 3D space, denoted as initial query fea- tures Qe (Qu.i E DC xC x.. 2, ... .N.. Then, Qe are associated with Gaussian (G. ~ Qc). For a given Gaussian set (ge ~ qc: Ic E Ge, qe E Qe), the feature at a point (z, y, 2) within its elliptical space are:
Je (c: pp.,,,))) = exp(
2 (p - (p-1))))
1)))
where Σ RT. S = diag(s), and = q2r(r). E, diag(-), and q2r(-) represent the covariance matrix, the function that constructs a diagonal matrix from a vector, and the function that transforms quaternion into rotation matrix, respectively.
4

---

Published as a conference paper at ICLR 2026
Lidar Gaussian initialization. Lidar's BEV space naturally provides an initialization for the Gaus- sian mean μ. And, S and r are initialized randomly. The Gaussian initialization for Lidar for- mulated 9t. E RCxM,xW, Since directly extracting information from the massive raw lidar point clouds to construct 3D Gaussian representations is both difficult and computationally inten- sive, grid-hased representations offer effective approach alleviate these challenges. For query, given the Lidar features Br E RCCM, where C, Hi. and Wi. represent the channel, height, and width of the BEV features, respectively. Then, we fed the BEV feature Bt, into multilayer perceptron (MLP) to obtain Lidar query QL, which is associated with the initial query features for each Gaussian function (9L QL).
Each voxel in Lidar BEV grid is centered at a 3D location μ E R. and a Gaussian is initialized at I with learnable scale s and rotation r. For Gaussian (9L. ~ Qt) associated with LiDAR feature Qr. E R3, the feature response at 3D point p (x, within its support is computed as:
9L (p, exp(
(p- -) (p- (1)pp)))00 g.
3.3 GAUSSIAN ENCODER
The 3D Gaussian distribution effectively represents the seene, and we have designed a Gaussian encoder to optimize both the properties and query features. This encoder includes deformable attention with Gaussian module and a Gaussian Updating module. The Gaussian encoder is stacked multiple times to update the Gaussian properties in an iterative refinement paradigm. Additionally, to better fuse multi-modal information, we employ a shared Gaussian encoder to simultancously pro- cess the Gaussian distributions G. and Gr, the two modalities are ultimately intended converge towards similar Gaussian distributions. Specifically, we merge Ge and G1 into the batch dimension.
Deformable Attention with Gaussian. As shown in Fig. 2(b), after obtaining the Gaussian distri- bution sets and corresponding query features (G. ~ Qe. GL ~ QL. we first encode Gi ~ Qi, into a new query Q. Specifically, to capture the position and geometric information of the Gaussian functions within the feature maps, we employ an MLP to encode the Gaussian properties. The new query Q is then obtained by adding the encoded properties Pa to the original query Q;:
L,
Q MLP(G) + Q i c, L,
(3)
where the MLP as the position embedding (PE) transforms the dimensions from I Ro to R.. We obtain sets G; ~ Q
Furthermore, we adopt deformable attention with Gaussian (Zhu et al., 2020; H et al., 2024) to extract fea- tures, as shown in Fig.3. Vanilla de- formable attention (Fig.3(b)) initial- izes sampling locations with an ap- proximately "square/kernel-like gion and learns offsets to cover the re- gions of interest. However, this initial distribution lacks inherent geometric priors about object shape. In con- trast, our deformable attention with Gaussian (Fig.3(a)) directly inherits and leverages the shape properties of Gaussians: by projecting the 3D Gaussian distributions onto the BEV feature map (projection in Figure 2b), we obtain prior sampling distribu- tion that encodes orientation, scale, and covariance structure. In other words, the initial sampling points are not uniformly spaced grid, but
Mcan Offset points
Projection
(a) Deformble Attention w. Gaussian
(b) Vamilla Deformable Attention
Figure 3: Comparison of the vanilla deformable attention (Zhu al., 2020) and our proposed deformable attention with Gaussian.
5

---

Published as a conference paper at ICLR 2026
instead follow Gaussian distribution aligned with the underlying object geometry such as aspect ratio, orientation, and spatial uncertainty. This Gaussian prior enables better alignment of cross- modal features to the "likely object extent," thereby enhancing fusion effectiveness capability absent in conventional square-shaped initialization.
Evidently, the properties of the Gaussian functions effectively describe the shape of the potential objects or regions. For each Gaussian function g. we calculate a set of offsets Aμ = (Ar, Ay, A=) based on the covariance matrix. These offsets, combined with the mean J4, yield the corresponding reference points +++ Au.. We then project the 3D reference points onto the BEV feature map, where each Gaussian query q; Q2 is updated through deformable attention, expressed as:
DeformAtt(q,, B.)
Σ A WaB,(H + Ap):
(4)
where B& is the camera BEV feature Be or Lidar BEV feature Br, Wi the weights obtained by linear layer, Aş is the attention weights and As E [0, 1I, K is the number of sampling points, B(u + Au) are the sampled features.
Gaussian Updating. To update the Gaussian properties, we propose an iterative optimization strat- egy of predicting offsets instead of predicting set of new Gaussian distributions as adopted in GaussianFormer (H et al., 2024). In Lidar and camera fusion perception, incrementally updating the Gaussian parameters allows for better handling of the discrepancies caused by different modalities when perceiving the same object. This approach is particularly effective in handling fusion uncer- tainties caused by such as signal attenuation, depth prediction uncertainties, or multi-modal signal discrepancies, demonstrated by the ablation experiments. Incremental updates across layers allow the model to gradually reduce the disparity between modalities, improving fusion accuracy. Specif- ically, by predicting the offsets Aμe, As, and Ar for the Gaussian mean μl, scale s, and rotation using an MLP, we refine the Gaussian distribution without having to predict a completely new set of properties. The updated Gaussian G, as follows:
G. = MLP(() + G,
s, rr).
(5)
3.4
FUSION
By the Gaussian Encoder model, we obtain the multimodal 3D Gaussian representations G. and GL within the shared 3D space, respectively. We merge Ge and St. into a unified set J. and we can easily fuse them. Although the 3D Gaussian distributions irregular distribution of Gaussian points, we need to voxelize these Gaussian distributions to achieve iask-nndependent 3D perception.
effectively represent the scene, to handle the
Specifically, given the unified Gaussian sets 9 ~ Q we divide the Gaussian space into voxel grid H x W. For a non-empty voxel V that contains M Gaussian means μ4, the Gaussian set for that voxel is V ((),, Ø0. ... .9M) ~ (Q1,92, ... qM)) }. To ensure real-time performance and the receptive field of the voxel, we use Mea (Zhou & Tuzel, 2018) to downsample the Gaussian within the voxel, as illustrated in Fig. 2(c). After Gaussian pooling, each voxel contains one Gaussian distribution (g ~ Q):
g
1 M Σ pm. Σ Sma Σ Emm]qq=
1 M
Σ 9m.
(6)
Furthermore, the Gaussian mixture model (Kerbl et al., 2023) naturally aggregate multiple Gaus- sian distributions into a finer-grained distribution, unifying multi-modal Gaussian representations and elegantly capturing the complexity of autonomous driving scenes. Thus, if the total number of Gaussian distributions covering point p in the entire scene is J, the feature fp) at point p is composed of the cumulative contributions of each individual Gaussian:
f(p) = Σ (p: μ. s.
Since each voxel may be associated with multiple 3D Gaussian distributions, following the strategy in (H et al., 2024), we calculate the neighborhood radius based on the scale property of each Gaus- sian. The indices of the Gaussians and the voxels within their neighborhood are paired as tuples

---

Published as a conference paper at ICLR 2026
and appended to list. This list is then sorted by voxel indices, determining which 3D Gaussians each voxel should focus on. Furthermore, for each voxel using 7, we can get the fused feature By. Finally, simple convolutional network is used to further optimize Bp.
3.5 PERCEPTION TASK SETUP
Without loss of generality, we follow BEVFusion (Liu et al., 2023b), GaussianFusion can be applied most 3D perception tasks based on Br. We evaluate the performance of GaussianFusion on 3D object detection and 3D semantic occupancy prediction tasks. We adopt the same Transformer- based detection head as BEVFusion (Bai et al., 2022; Liu et al., 2023b) and the occupancy head consistent with BEVDet (Huang & Huang, 2022a). We also constructed a camera-only version, G. containing only the camera branch shown Fig. 2.
EXPERIMENTS
4.1
DATASET
The nuScenes dataset (Caesar et al., 2020) provides annotation data for tasks such as semantic seg- mentation, object detection, and 3D occupancy (Oce) prediction. It is large-scale multimodal dataset officially split into 700/150/150 scenes for training, validation, and testing, respectively. Each scene includes annotated Lidar point cloud data captured by a 32-beam scanner, along with 6 perspective camera views, offering comprehensive 360-degree coverage at each timestamp. We evaluate our method on the 3D object detection and Occ. In our task, we down-sample the input camera images to 704 x 256 and voxelize the point cloud to 0.075m for detection, following BEV- Fusion (Liu et al., 2023b) and UniTR (Wang et al., 2023a). For 3D object detection, the perception range of the point cloud is set to [-51.2m, 51.2m) along the X and Y axes, and [-5m, 3m] along the axis. For 3D Occ, we evaluate within the region of [-50m, 50m] x [-50m, 50m] around the ego vehicle, following (Liu al., 2023b; Wang al., 2023a; Wei et al., 2023).
4.2 IMPLEMENTATION DETAILS
We adopt VoxelNet (Zhou & Tuzel, 2018) and Swin-T (Liu et al., 2021) as the Lidar and camera backbone to extract the multimodal features following BEVFusion (Liu al., 2023b). The dimen- sions of Lidar features, image features, and 3D Gaussian query features are all set to 128. Depth of image D,=41. Image feature F',, dimensions: 8x22x6. We Gaussian Encoder blocks to 4, see the Appendix for experiments. The BEV size H W is set to 200 x 200.
During training, we follow BEVFusion to adapt the aligned multimodal data augmentation strategy and the bllasebl sampling strategy from CBGS (Zhu et al., 2019). GaussianFusion is trained on NVIDIA A800 GPUs We use AdamW (Loshchilov & Hutter, 2017) optimizer with a weight decay 0.01. We adopt the one-cycle learning rate policy (Smith, 2017) with a maximum learning rate of -4, Both BEV object detection and 3D semantic occupancy prediction are trained for 20 epochs, following the same settings as BEVFusion and GaussianFormer (H et al., 2024), respec- tively.
4.3 3D OBJECT DETECTION
Setting. We utilize the official evaluation metric nuScenes Detection Score (NDS) and mean Aver- age Precision (mAP) for 3D detection.
Results. To highlight the effect of Gaussian representation, only compare the BEV-based method. As shown in Table 2, GaussianFusion achieves SOTA results compared to previous dis- crete BEV representation multimodal fusion methods(Liu al., 2023b; Ge al., 2023; Wang al., 2023a; Hu et al., 2023b) on nuScenes dataset, achieving 74.0 NDS and 71.7 mAP on the val split. Specifically, compared with BEVFusion (Liu et al., 2023b), our GaussianFusion achieves +2.6 NDS and +3.2 mAP on val split by exploring a more natural continuous Gaussian cross-modal comple- mentary fusion. In addition, compared with recent SOTA fusion works, such as UniTR (Wang al., 2023a), EA-LSS (Hu et al., 2023b), and FusionFormer-S (Hu et al., 2023a), GaussianFusion shows superior performance, outperforming them by 1.2, 0.5, and 1.7 respectively in mAP.
7

---

Published as a conference paper at ICLR 2026
Table 2: Comparisons with state-of-the-art 3D object detection methods nuScenes dataset. C
denote Camera, L denote Lidar. All methods construct BEV-based feature maps instead of object- centric fusion based on proposals, which means these methods can also be naturally used for seman- tic tasks. UniTR uses unified backbone for both the camera and Lidar.
Backhone
validation
Methods
Modality
Resolution
Camera
Ladar
NDS
MAP
NDS
mAP
BEVFormer (Li al., 2022b)
C
1600x900
ResNet-101
51.7
41.6
56.9
48.1
PETRy2 (Liu et ul., 2023a)
C
1600x640
VoV-99
59.1
50.8
FB-BEV (Li ct al., 2023)
C
1600x640
VoV-99
62.4
53.7
AutoAlign'2 (Chen et al., 2022)
C+I.
1600x640
CSPNet
VoxelNet
71.2
67.1
72.4
68.4
BEVFusion(M) (Liu et al., 2023b)
C+L
704x256
Swin-T
VoxeINet
71.4
68.5
72.9
70.2
MetaBEV et al., 2023)
C+L
x04
Swin-T
VoxeINet
71.5
68.0
MSMDFusion (Jiao al., 2023)
C+L
800x448
ResNet-50
VoxelNet
72.1
69.3
FusionFormer-S (Hu al., 2023a)
C+I.
1600xx40
VoV-99
VoxelNet
73.2
70.0
EA-LSS (Hu et al., 2023b)
C+L
704x256
Swin-T
VoxelNet
73.1
71.2
74.4
72.2
UniTR (Wang et al., 2023a)
C+L
704x256
73.3
70.5
74.5
70.9
Gaussian GaussianFusion(Ounn usion Ours)
C+L
704x256
Swin-T
VoxelNet
74.0
71.7
74.9
72.4
Additionally, we provide a comparison with other open-source state-of-the-art (SOTA) methods in inference latency and performance accuracy in Table 3. Benefiting from the unified architecture, it achieves excellent performance of 71.7 mAP while maintaining lower inference latency (132 ms) and memory consumption (4271 MB) compared to BEVFusion.
Table 3: Latency and performance on nuScenes val. set.
Method
Latency 4
Memory +
NDS f
mAP
BEVFusion
156
5140 M
71.4
68.5
GaussianFusion
132 ms
4271 M
74.0
71.7
Here, we design a simple temporal extension, termed GaussianFusion-T. historical Gaussian representations are warped to the current timestamp and then fused via Equations (5) and (6). In Equation (5), the mean Jmns scale sm. rotation Fm. and query yThh encompass not only the multi-modal (image and Li- DAR) Gaussians and queries, but also those from history frames. Experimen- tal results show that, compared BEV- Fusion4D (Liu et al., 2023b), our temporal variant GaussianFusion-T achieves significant improve- ments. Moreover, even without sophisticated temporal modeling, Graussinnuusion-T achieves com- petitive NDS against advanced temporal fusion methods such as SparseLIF-T (Zhang et al., 2024a).
Table 4: Comparison with temporal methods.
Method
NDS 1
mAP f
BEVFusion4D (Liu et al., 2023b)
73.5
72.0
FusionFormer (Hu et al., 2023a)
74.1
71.4
SparseLIF-T (Zhang et al., 2024a)
77.5
74.7
GaussianFusion-T
77.6
75.0
Table 5: Semantic scene completion results on nuScenes (Wei et al., 2023; Caesar et al., 2020) val set. i represents trained on nuScenes. For Camera-only and C+L, the top performance is indicated in bold black and bold blue, respectively.
Mudbiity
0O0SE voh
motonoyale
podesttaa
trahh cone
miau
Mehhod
1oU
600
REVFomer (Li et al, 20226)
30.30 16.75
14.22
6.38
23.46
28.28
8.66
10.77
6.64
4.05
11.20
17.08
37.28
18.00
22.88
22.17
13,80
22.21
TPVFomerr 2023)
FR-Oze(IO (1i et al., 2023)
51.55
20.17
20.31
12.20
26
31.07
10.78
15.95
13.31
11.14
13.24
22.13
39.56
22
25.14
23.59
13.92
21.64
SurrandOcee
2023)
FuusianFummer (1I al., 2024)
29:83
19.10
19.52
11.26
26
29.78
10:47
13.83
12.58
8.67
12.74
21.57
39.63
23
24.46
22.99
9.59
19.12
GaussianFusione
REVFusion) (1 et 2023b)
C-1.
30:11
24.65
23.78
12.29
30.67
34.95
14.62
17.23
30.76
15.75
19.83
26.3
40.01
23
25.47
36441
25.15
37.92
McCONe: (Wang 2023c)
14.70
20.00
26.00
37.10
CO Occ (Paa et 2024)
C+L
41.10
27.10
28.10
16.10
34.00
37.20
17.00
21.60
20.80
15.90
21.90
28.50
42.30
25
25.40
29.10
28.60
28.20
28.00
OccHusion (Mang al., 2024)
27.55
23.11
17.50
22.70
39.47
27.57
GaussianFuil usion
C+L
44.75
28.65
28.91
18.31
34.87
37.43
19.45
23.53
26.71
17.96
23.38
29.59
43.32
25.96
30.51
28.35
30.32
29.67
8

---

Published as a conference paper at ICLR 2026
Waymo Open Dataset Result. We further conduct experiments the Waymo Open Dataset (Sun et al., 2020) to evaluate the generalization capability our approach. GaussianFusion adopts the same heatmap-basee detection head BEVFusion for a fair comparison. Table 6 shows that the Gaussian representation outperforms the BEV representation, achieving higher detection accuracy, which demonstrates its greater potential for 3D perception.
Table 6: Waymo Open Dataset Result.
Model
mAP-L1
mAPH-LI
mAP-L2
mAPH-L2
BEVFusion (Liu et al., 2023b)
82.72 86.23
81.35
77.65
76.33
GaussianFusion(urrs)
84.65
81.47
80.75
4.4 3D SEMANTIC OCCUPANCY PREDICTION
Setting. We report the dntersection-ove-Ummeee (IoU) of occupied voxels the evaluation metric of the class-agnostic scene completion task and the mloU of all semantic classes for the Oce task following SurroundOce (Wei et al., 2023).
Results. As shown Table our GaussianFusion achieves SOTA perfor- mance at 28.65 mloU among all single- frame models. GaussianFusion outper- forms the multi-modal SOTA method OccFusion (Ming al., 2024), which based on multi-scale voxel fusion, by +1.11 mloU and significantly surpasses camera-only methods (H et al., 2023;
Table 7: Comprehensive comparison with Gaussian-
Former on nuScenes val set.
Method
Num. Gaussians mloU1 1 Latency
GaussianFormer
140,000
19.10
475
GaussianFusion-C
43,296
20.65
105 ms
Wei et al., 2023; Li et al., 2023). More importantly, benefiting from our proposed Gaussian ini- tialization strategy and iterative update mechanism. GaussianFusions achieves a 1.55 mIoU im- provement and nearly 4.5x computational efficiency compared GaussianFormer, while using only 30% of the Gaussians, as shown in Table 7. GaussianFormer randomly initializes a set of Gaussians 3D space and predicts new Gaussian parameters for these Gaussians an update. Extensive experiments demonstrate the effectiveness of our 3D Gaussian representation across multiple tasks, including both object-centric and dense semantic perception.
Table 8: Ablation of Gaussian initializa- tion strategy.
Table Ablation of the proposed Gaussian Encoder,
DA.G means Deformable Attention with Gaussian.
Share
Separate
DA.G
PE
Offset
NDS
mAP
Gaussian Initialization
NDS
mAP
74.0
71.7
Random Initialization
71.2
68.3
73.6
71.1
Backward Projection
72.4
70.0
73.4
71.0
Lidar Projection
73.6
71.1
V
73.6
71.2
Forward Projection
74.0
71.7
f
d
73.2
70.8
4.5 ABLATION STUDIES
Effect of Gaussian Initialization. Table provides detailed analysis of the performance im- pact of different Gaussian initialization strategies, including the classic random initialization, our proposed forward projection strategy, the backward projection BEVFormer-based strategy(Li al., 2023; 2022b), and the strategy of projecting Lidar points onto the image. The initialization details and corresponding Gaussian encoder for the latter two projection strategies are in Appendix. The forward projection and Lidar projection strategies show comparable performance (74.0 NDS 73.6 NDS), both outperforming the backward projection method (72.4 NDS). Notably, the forward projection strategy brings significant improvement of +2.8 NDS over random initialization.
Effect of Gaussian Encoder. In Table 9, we first compare the shared and separate Gaussian En- coders. We find that the shared Gaussian Encoder provides performance improvement of +0.7

---

Published as a conference paper at ICLR 2026
mAP. We attribute this to the unified Gaussian space, which helps the model learn uncertain cross- modal complementary features. And the sharing strategy makes the model leaner. We then con- duct ablation study the deformable attention module. Results show that deformable attention with Gaussian priors outperforms the vanilla variant by +0.4 NDS, demonstrating that the shape prior encoded by Gaussians facilitates model convergence and enhances detection accuracy. For the deformable iitention-basee query updating module, the results show that encoding the Gaussian properties PE into the query leads gain of +0.5 mAP. For the Gaussian updating module, predicting the offsets of the properties, rather than the properties themselves, improves multimodal fusion perception by +0.9 mAP. This validates the effectiveness the Gaussian encoder.
4.6 VISUALIZATIONS
As shown in Fig. 4, in BEV object detection, compared to previous BEV-based SOTA methods like UniTR (Wang et al., 2023a) and BEVFusion(Liu et al., 2023b), GaussianFusion achieve higher accuracy for distant small objects (white marks). Furthermore, UniTR exhibits significant object yaw errors (yellow marks). For Occ, GaussianFusion produces sharper object boundaries (red marks) and better class separation (yellow marks) compared to GaussianFormer. See Appendix for more visualizations.
Lidar Sean
Ground Truth
Gaussianl usion
BEV Pusson
Multi images
Ground
Gaussian
driveable surface
truck
terrain
vegetation
sidowalk
other flat
podestrian
bicycle
murmalle
motarcycle barrier
consinuction vehicle
trailer
traffic cone
Figure 4: Qualitative results on object detection and 3D semantic occupancy prediction.
4.7 LIMITATIONS
Several coproaches-coverinn both detection (Wang et al., 2023b) and Occ (Zhang et al., 2024b)-employ carefully designed temporal fusion modules to enhance performance. While our method naturally extends to multi-frame settings through simple temporal alignment and already achieves performance comparable to such multi-frame methods, this likely suboptimal. promis- ing direction for future work is to explore motion- Gaussian updates, for instance by predicting offsets, enabling more coherent 4D scene modeling over time.
CONCLUSION
We present GaussianFusion a novel multi-modal fusion perception framework grounded in unified 3D Gaussian representation that seamlessly integrates camera and LiDAR features in continuous spatial domain, effectively preserving fine-grained scene details. A shared Gaussian encoder is introduced to facilitate adaptive cross-modal interaction and alignment, with Gaussian properties it- eratively refined through optimization. To support aano-agi applications, we design an efficient Gaussian-to-voxe transformation module incorporating Gaussian pooling and aggregation mecha- nisms. Extensive experiments across multiple 3D perception tasks on the nuScenes dataset validate the effectiveness of GaussianFusion achieving state-of-the-art performance among task-agnostic baselines. Although it may slightly lag behind certain task musk-specialized approaches, our work rep- resents meaningful step toward generalizable and principled multi-modal fusion. We believe that the proposed Gaussian representation paradigm offers promising direction for future research in multimodal 3D perception.
10

---

Published as a conference paper at ICLR 2026
REFERENCES
Xuyang Bai, Zeyu Hu, Xinge Zhu, Qingqiu Huang, Yilun Chen, Hongbo Fu, and Chiew-Lan Tai. Transfusion: Robust lider-camera fusion for 3d object detection with transformers. Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 1090-1099, 2022.
Holger Caesar, Varun Bankiti, Alex H Lang, Sourabh Vora, Venice Erin Liong, Qiang Xu, Anush Krishnan, Yu Pan, Giancarlo Baldan, and Osear Beijbom. nuscenes: multimodal dataset for autonomous driving. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 11621-11631 2020.
Anh-Quan Cao, Angela Dai, and Raoul de Charette. Pasco: Urban 3d panoptic scene completion with uncertainty awareness. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 14554-14564, 2024.
Florian Chabot, Nicolas Granger, and Guillaume Lapouge. Gaussianbev: 3d gaussian representation meets perception models for bev segmentation. arXiv preprint arXiv:2407. 14108, 2024.
Yilun Chen, Zhiding Yu, Yukang Chen, Shiyi Lan, Anima Anandkumar, Jiaya Jia, and Jose M Alvarez. Focalformer3d: focusing on hard instance for 3d object detection. In Proceedings of the IEEE/CVF International Conference on Computer Vision. pp. 8394 8405, 2023.
Zehui Chen, Zhenyu Li, Shiquan Zhang. Liangji Fang, Qinhong Jiang. and Feng Zhao. Deformable feature aggregation for dynamic multi-modal 3d object detection. In European conference on computer vision, pp. 628-644. Springer, 2022.
Sara Krill, Alex Yu, Matthew Tancik, Qinhong Chen, Benjamin Recht, and Angjoo Kanazawa. Plenoxels: Radiance fields without neural networks. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 5501-5510, 2022.
Wanshui Gan, Fang Liu, Hongbin Xu, Ningkai Mo, and Naoto Yokoya. Gaussianoce: Fully self-supervised and efficient 3d occupancy estimation with gaussian splatting. arXiv preprint arXiv:2408. / 1447, 2024.
Chongjian Ge, Junsong Chen, Enze Xie, Zhongdao Wang, Lanqing Hong, Huchuan Lu, Zhenguo Li, and Ping Luo. Metabev: Solving sensor failures for 3d detection and map segmentation. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 8721-8731, 2023.
Yuanhui H. Wenzhao Zheng, Yunpeng Zhang, Jie Zhou, and Jiwen Lu. Tri-perspeetive view for vision-based 3d semantic occupancy prediction. In Proceedings of the IEEF/CVF conference on computer vision and pattern recognition, pp. 2233-9232, 2023.
Yuanhui H. Wenzhao Zheng, Yunpeng Zhang, Jie Zhou, and Jiwen Lu. Gaussianformer: Scene as gaussians for vision-based 3d semantic occupancy prediction. arXiv preprint arXiv:2405.17429, 2024.
Chunyong Hu, Hang Zheng, Kun Li, Jianyun Xu, Weibo Mao, Maochun Luo, Lingxuan Wang, Mingxia Chen, Kaixuan Liu, Yiru Zhao, al. Fusionformer: A multi-sensory fusion in bird's- eye-view and temporal consistent transformer for 3d objection. arXiv preprint arXiv:2309.0522 2023a.
Haotian Hu, Fanyi Wang, Jingwen Su, Yaonong Wang, Laifeng Hu, Weiye Fang, Jingwei Xu, and Zhiwang Zhang. Ea-lss: Edge-aware lift-splat-shot framework for 3d bev object detection. arXiv preprint arXiv:2303. 17895, 2023b.
Xu Hu, Yuxi Wang, Lue Fan, Junsong Fan, Junran Peng, Zhen Lei, Qing Li, and Zhaoxiang Zhang. Semantic anything in 3d gaussians. arXiv preprint arXiv:2401.17885 2024.
Junjie Huang and Guan Huang. Bevdet4d: Exploit temporal cues in multi-camera 3d object detec- tion. arXiv preprint arXiv:2203.1705: 2022a.
Junjie Huang and Guan Huang. Bevpoolv2: A cutting-edge implementation of bevdet toward de- ployment. arXiv preprint arXiv:2211.1711/ 2022b.

---

Published as a conference paper at ICLR 2026
Yang Jiao, Zequn Jie, Shaoxiang Chen, Jingjing Chen, Lin Ma, and Yu-Gang Jiang. Msmdfusion: Fusing lidar and camera at multiple scales with multi-depth seeds for 3d object detection. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 21643- 21652, 2023.
Bernhard Kerbl, Georgios Kopanas, Thomas Leimkühler, and George Drettakis. 3d gaussian splat- ting for real-time radiance field rendering. ACM Trans. Graph., 42(()) 2023.
Jinke Li, Xiao He, Chonghua Zhou, Xiaoqiang Cheng, Yang Wen, and Dan Zhang. Viewformer: Ex- ploring spatiotemporal modeling for multi-view 3d occupancy perception via view-guided trans- formers. In European Conference Computer Vision, pp. 90-106. Springer, 2025.
Yanwei Li, Yilun Chen, Xiaojuan Qi, Zeming Li, Jian Sun, and Jiaya Jia. Unifying voxel-based rep- resentation with transformer for 3d object detection. Advances in Neural Information Processing Systems, 35:1 8442-18455, 2022a.
Yingyan Li, Lue Fan, Yang Liu, Zehao Huang, Yuntao Chen, Naiyan Wang, and Zhaoxiang Zhang. Fully sparse fusion for 3d object detection. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2024.
Zhiqi Li, Wenhai Wang. Hongyang Li, Enze Xie, Chonghao Sima, Tong Lu, Yu Qiao, and Jifeng Dai. Bevformer: Learning hird's-eye-vier representation from multi-camera images via spatiotempo- ral transformers. European conference on computer vision, pp. 1-18. Springer, 2022b.
Zhiqi Li, Zhiding Yu, Wenhai Wang, Anima Anandkumar, Tong Lu, and Jose M Alvarez. Fb- bev: Bev representation from forward-backward view transformations. In Proceedings of the IEEE/CVF International Conference Computer Vision, pp.
2023.
Tingting Liang, Hongwei Xie, Kaicheng Yu, Zhongyu Xia, Zhiwei Lin, Yongtao Wang, Tao Tang, Bing Wang, and Zhi Tang. Bevfusion: A simple and robust lidar-camera fusion framework. Advances in Neural Information Processing Systems, 15:10111-10444, 2022.
Haisong Liu, Yang Chen, Haiguang Wang, Zetong Yang. Tianyu Li, Jia Zeng, Li Chen, Hongyang Li, and Limin Wang. Fully sparse 3d occupancy prediction. arXiv preprint a8Xiv22312.17118, 2024.
Shuai Liu, Quanmin Liang, Zefeng Li, Boyang Li, and Kai Huang. Gaussianfusion: Gaussian-basee multi-sensor fusion for end-to-end autonomous driving. arXiv preprint arXiv:2506.0003: 00034, 2025,
Yingfei Liu, Junjie Yan, Fan Jia, Shuailin Li, Aqi Gao, Tiancai Wang, and Xiangyu Zhang. Petrv2: A unified framework for 3d perception from multi multi-camera images. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 3262-32222222 -2722, 2023a.
Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. In Proceedings of the IEEE/CVF international conference computer vision, pp. 10012-10022 2021.
Zhijian Liu, Haotian Tang, Alexander Amini, Xinyu Yang, Huizi Mao, Daniela L Rus, and Song Han. Bevfusion: Multi-task multi-sensor fusion with unified bird's view representation. In 2023 IEEE international conference on robotics and automation (ICRA), pp. 2774-2781. IEEE, 2023b.
Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. 2017.
arXiv preprint
Yuhang Lu, Xinge Zhu, Tai Wang, and Yuexin Ma. Octreeoce: Efficient and multi caltiigranularity occupancy prediction using octree queries. arXiv preprint xxXv:2231..03774, 2023.
Junyi Ma, Xieyuanli Chen, Jiawei Huang. Jingyi Xu, Zhen Luo, Jintao Xu, Weihao Gu, Rui Ai, and Hesheng Wang. Cam4doce: Benchmark for camera-only 4d occupancy forecasting in tonomous driving applications. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 21486-21495 2024a.
12

---

Published as a conference paper at ICLR 2026
Qihang Ma, Xin Tan, Yanyun Qu, Lizhuang Ma, Zhizhong Zhang, and Yuan Xie. Cotr: Compact oc- cupancy transformer for vision-basee 3d occupancy prediction. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 19936-19945, 2024b.
Ben Mildenhall, Pratul P Srinivasan, Matthew Tancik, Jonathan T Barron, Ravi Ramamoorthi, and Ren Ng. Nerf: Representing scenes as neural radiance fields for view synthesis. Communications of the ACM, 65(1):99-106, 2021.
Zhenxing Ming. Julie Stephany Berrio, Mao Shan, and Stewart Worrall. Occfusion: A straightfor- ward and effective multi-sensor fusion framework for 3d occupancy prediction. arXiv e-prints, pp. arXiv-2403, 2024.
Thomas Müller, Alex Evans, Christoph Schied, and Alexander Keller. Instant neural graphics prim- itives with a multiresolution hash encoding. ACM transactions on graphics (TOG), (1): 2022.
Jingyi Pan, Zipeng Wang, and Lin Wang. Co-occ: Coupling explicit feature fusion with volume rendering regularization for multi-modal 3d semantic occupancy prediction. IEFE Robotics and Automation Letters, 2024.
Jonah Philion and Sanja Fidler. Lift, splat, shoot: Encoding images from arbitrary camera rigs by implicitly unprojecting to 3d. In Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK. August 23-28. 2020, Proceedings, Part XIV 16, pp. 194-210 Springer, 2020.
Leslie N Smith. Cyclical learning rates for training neural networks. In 2017 IEEE winter conference on applications of computer vision (WACV), pp. 4644472 IEEE, 2017.
Pei Sun, Henrik Kretzsehmar, Xerxes Dotiwalla, Aurelien Chouard, Vijaysai Patnaik, Paul Tsui, James Guo. Yin Zhou, Yuning Chai, Benjamin Caine, et al. Scalability in perception for au- tonomous driving: Waymo open dataset. In Proceedings of the IEEE/CVF conference on com- puter vision and pattern recognition, pp. 1446-2454 2020.
Xiaoyu Tian, Tao Jiang, Longfei Yun, Yucheng Mao, Huitong Yang, Yue Wang. Yilun Wang, and Hang Zhao. Occ3d: A large-scale 3d occupancy prediction benchmark for autonomous driving. Advances in Neural Information Processing Systems, 36, 2024.
Sourabh Vora, Alex H Lang, Bassam Helou, and Oscar Beijbom. Pointpainting: Sequential fusion for 3d object detection. In Proceedings of the IEEF/CVF conference computer vision and pattern recognition, pp. 4604-4612, 2020.
Haiyang Wang, Hao Tang, Shaoshuai Shi, Aoxue Li, Zhenguo Li, Bernt Schiele, and Liwei Wang- Unitr: A unified and efficient multi-modal transformer for bird's "edd's-eye-view representation. In Proceedings of the IEEF/CVE International Conference on Computer Vision, pp. 6792-6802, 2023a.
Shihao Wang, Yingfei Liu, Tiancai Wang, Ying Li, and Xiangyu Zhang. Exploring object-centric temporal modeling for efficient multi-view 3d object detection. Proceedings of the IEEE/CVF International Conference Computer Vision, pp. 166213633,, 2023b.
Xiaofeng Wang, Zheng Zhu, Wenbo Xu, Yunpeng Zhang. Yi Wei, Xu Chi, Yun Ye, Dalong Du, Ji- wen Lu, and Xingang Wang. Openoccupancy: A large scale benchmark for surrounding semantic occupancy perception. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 17850-17859, 2023c.
Yuqi Wang, Yuntao Chen, Xingyu Liao, Lue Fan, and Zhaoxiang Zhang. Panooce: Unified occupancy representation for camera-bases based 3d panoptic segmentation. In Proceedings of the IEEÉ/CVF conference on computer vision and pattern recognition, pp. 17158-17168, 2024a.
Zitian Wang, Zehao Huang, Yulu Gao, Naiyan Wang, and Si Liu. Mv2dfusion: Leveraging modality- specific object semantics for multi-modal 3d detection. arXiv preprint arXiv:2408. 05945, 2024b.
Yi Wei, Linging Zhao, Wenzhao Zheng, Zheng Zhu, Jie Zhou, and Jiwen Lu. Surroundoce: Multi- camera 3d occupancy prediction for autonomous driving. In Proceedings of the IEEE/CVF Inter- national Conference Computer Vision, pp. 21729-21748 2023.
13

---

Published as a conference paper at ICLR 2026
Junjie Yan, Yingfei Liu, Jianjian Sun, Fan Jia, Shuailin Li, Tiancai Wang, and Xiangyu Zhang. Cross modal transformer: Towards fast and robust 3d object detection. In Proceedings of the IEEE/CVF International Conference Computer Vision, pp. 18828-182788 2023.
Zeyu Yang, Jiaqi Chen, Zhenwei Miao, Wei Li, Xiatian Zhu, and Li Zhang. Deepinteraction: 3d object detection via modality interaction. Advances in Neural Information Processing Systems, 35:1992-2005, 2022.
Mingqiao Ye, Martin Danelljan, Fisher Yu, and Lei Ke. Gaussian grouping: Segment and edit anything in 3d scenes. In European Conference on Computer Vision, pp. 162-179. Springer, 2025.
Junbo Yin, Jianbing Shen, Runnan Chen, Wei Li, Ruigang Yang, Pascal Frossard, and Wenguan Wang. Is-fusion: Instance-scent collaborative fusion for multimodal 3d object detection. arXiv preprint arXiv:2403. 15241, 2024.
Hongcheng Zhang. Liu Liang, Pengxin Zeng. Xiao Song, and Zhe Wang. Sparselif: High- performance sparse lidar-camera fusion for 3d object detection. In European Conference on Computer Vision, pp. 109-128. 128. Springer, 2024a.
Shuo Zhang, Yupeng Zhai, Jilin Mei, and Yu Hu. Fusionoce: Multi-modal fusion for 3d occupancy prediction. In Proceedings of the 32nd ACM International Conference on Multimedia, pp. 787- 796, 2024b.
Xiao Zhao, Bo Chen, Mingyang Sun, Dingkang Yang, Youxing Wang, Xukun Zhang, Mingcheng Li, Dongliang Kou, Xiaoyi Wei, and Lihua Zhang. Hybridoce: Nerf enhanced transformer- multi-camera 3d occupancy prediction. IEEE Robotics and Automation Letters, 2024a.
Xiao Zhao, Xukun Zhang, Dingkang Yang. Mingyang Sun, Mingcheng Li, Shunli Wang, and Lihua Zhang. Maskbev: Towards unified framework for bev detection and map segmentation. arXiv preprint arXiv:2408.09122 2024b.
Yin Zhou and Oncel Tuzel. Voxelnet: End-to-end learning for point cloud based 3d object detection. In Proceedings of the IEEE conference computer vision and pattern recognition, pp. 4490- 4499, 2018
Zhizhuo Zhou and Shubham T. Sparsefusion: Distilling view-conditioned diffusion for 3d recon- struction. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recog- nition, pp. 15588-1297, 2023.
Benjin Zhu, Zhengkai Jiang, Xiangxin Zhou, Zeming Li, and Gang Yu. Class-balanced grouping and sampling for point cloud 3d object detection. arXiv preprint arXiv: 09408,0992,200 2019.
Xizhou Zhu, Weijie Su, Lewei Lu, Bin Li, Xiaogang Wang, and Jifeng Dai. Deformable detr: Deformable transformers for end-to-end object detection. arXiv preprint arXiv:2010.04159, 2020.
Sicheng Zuo, Wenzhao Zheng, Yuanhui Huang, Jie Zhou, and Jiwen Lu. Gaussianworld: Gaussian world model for streaming 3d occupancy prediction. arXiv preprint arXiv:24/2.10373, 2024.
14


Reviews:
[{"review_id": "3LmXienZLN", "reviewer": "Reviewer_RnZo", "summary": "The paper introduces GaussianFusion, a new framework for multimodal 3D perception that replaces the traditional bird’s-eye view (BEV) grid representation with a 3D Gaussian representation. This approach unifies multimodal features (e.g., from different sensors) in a shared, continuous 3D Gaussian space, effectively preserving fine details and edges. The framework includes a forward-projection Gaussian initialization module and a cross-modal Gaussian encoder that iteratively refines Gaussian properties using attention mechanisms. GaussianFusion is task-agnostic and demonstrates strong effectiveness.", "questions": "1. In Table 2, why is the modality listed as “C” when LiDAR (with VoxelNet backbone) is used?\n2. Can this method be applied to anchor-aware object detection models (e.g., FUTR3D [1], TransFusion [2])?\n3. Regarding Table 3, as far as I know, BEVFusion addresses long-latency issues for improved efficiency. Given that BEV-based methods quantize features early, I would expect them to have lower latency. However, GaussianFusion shows smaller latency than BEVFusion — where does this latency saving come from?\n4. Could this approach generalize to other modality combinations (e.g., Camera + Radar, LiDAR + Radar, Camera + LiDAR + Radar)?\n5. How effective is this approach on other datasets (e.g., Waymo) that have different sensor characteristics, such as varying LiDAR beam densities?\n\n**Reference**\n\n[1] FUTR3D: A Unified Sensor Fusion Framework for 3D Detection\n\n[2] TransFusion: Robust LiDAR-Camera Fusion for 3D Object Detection with Transformers", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. Clear and timely motivation. I initially thought BEV was the best in terms of representational capability and fusion suitability, but this work convincingly shows that 3D Gaussian representation is indeed a promising alternative.\n2. The paper is well-structured and flows logically.\n3. The proposed solution is intuitively convincing and demonstrates effectiveness in two visual representation use cases.", "weaknesses": "1. Missing ablation study on the impact of voxel size on task accuracy.\n2. For readability, the citation format should be revised: use ~\\citep or ~\\citet instead of ~\\cite.\n3. Lacks discussion of failure cases where the Gaussian representation performs worse than the BEV representation.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "i2QcOjNyI0", "reviewer": "Reviewer_eFff", "summary": "The authors presented GaussianFusion that replaces discrete BEV grids with a continuous 3D Gaussian space to fuse camera and LiDAR, aiming to preserve fine details and align modalities before any quantization. It introduces forward-projection camera Gaussian initialization, a shared Gaussian encoder with Gaussian-guided deformable attention and iterative property updates, then pools Gaussians to voxels for task heads; on nuScenes it reports +2.6 NDS over BEVFusion for detection and higher mIoU for occupancy with lower latency/memory", "questions": "Some questions are mentioned in the weakness section.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "Continuous fusion before BEV: avoids early discretization loss and enables richer cross-modal interaction in Gaussian space.\n\nForward-projection init for cameras; shared encoder with Gaussian-guided deformable attention and incremental updates to µ/s/r.\n\nSolid results + efficiency: NDS 74.0 / mAP 71.7 on val, latency 132 ms and memory 4271 MB vs. BEVFusion’s 156 ms/5140 MB; occupancy mIoU 28.65 SOTA among single-frame models.", "weaknesses": "Scope limited to nuScenes; no cross-dataset/domain validation shown, so generality beyond this setting is inferred rather than demonstrated.\n\nAlthough fusion happens in continuous Gaussian space, the method must voxelize Gaussians for task heads, which can bring back quantization artifacts the paper set out to avoid.\n\nAt the same 200×200 BEV size, GaussianFusion shows higher memory than BEVFusion (5418 MB vs. 5140 MB), even though latency is lower.\n\nTemporal modeling out of scope. Authors acknowledge that temporal fusion boosts SOTA in both detection and occupancy, but GaussianFusion leaves this to future work, so comparisons to temporal SOTA aren’t shown.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "lVFbm9ZRUC", "reviewer": "Reviewer_x14q", "summary": "This paper proposes *GaussianFusion*, a multi-modal fusion framework for 3D perception in autonomous driving. The core idea is to replace discrete Bird’s-Eye View (BEV) representations with a unified 3D Gaussian representation to preserve fine-grained geometric and semantic details. Key contributions include:  \n\n- A **forward-projection-based Gaussian initialization** strategy for camera and LiDAR data, avoiding random initialization used in prior work (e.g., GaussianFormer).  \n- A **shared Gaussian encoder** with deformable attention and iterative property updates to align cross-modal features.  \n- A **Gaussian-to-voxel fusion module** for task-agnostic 3D perception.  \n- State-of-the-art results on nuScenes for 3D object detection (74.0 NDS) and semantic occupancy prediction (28.65 mIoU), with improved efficiency over GaussianFormer.", "questions": "**Questions**:  \n\n1. How does the shared encoder resolve modality-specific conflicts (e.g., camera depth uncertainty vs. LiDAR precision)? Is there a mechanism to weight modalities dynamically?  \n2. The Gaussian updating module (§3.3) predicts offsets (Δµ, Δs, Δr). How stable is this optimization? Does it suffer from gradient vanishing/explosion with multiple encoder layers?  \n3. Could the framework support temporal fusion (e.g., multi-frame LiDAR/camera data)? The authors mention it as future work but provide no preliminary analysis.  \n\n**Suggestions**:  \n\n1. **Strengthen the fusion novelty**: Compare the deformable attention mechanism with prior Gaussian-guided attention (e.g., Zuo et al., 2024) and highlight differences.  \n2. **Expand experiments**:  \n   - Compare with recent sparse fusion methods (e.g., Wang et al., 2024b; Yin et al., 2024).  \n   - Report computational metrics (FLOPS, memory) for Gaussian-to-voxel conversion.  \n3. **Clarify technical details**:  \n   - Provide equations for LiDAR Gaussian initialization.  \n   - Discuss how Gaussian scale (**s**) and rotation (**r**) are regularized during training.  \n4. **Enhance visualization**: Include failure cases (e.g., occluded objects) and per-class IoU curves for occupancy prediction.  \n\n---\n\n**Rebuttal Potential**: The authors could improve the rating by:  \n\n- Demonstrating superior task-agnostic performance against sparse fusion baselines.  \n- Providing deeper analysis of computational trade-offs.  \n- Adding a ablation on the Gaussian mixture aggregation strategy.  \n\nOverall, the paper presents a promising direction but requires stronger differentiation from existing fusion paradigms and more rigorous evaluation to meet the acceptance threshold.", "limitations": "", "rating": 4, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "**Originality**:  \n\n- The use of 3D Gaussian representations for multi-modal fusion is novel. While 3D Gaussian Splatting (3DGS) has been explored for reconstruction (Kerbl et al., 2023) and single-modal perception (H et al., 2024), its application to multi-modal fusion is a creative extension.  \n- The forward-projection initialization addresses a critical limitation of random initialization in GaussianFormer, improving convergence and performance (Table 6).  \n\n**Quality**:  \n\n- Experiments are thorough, covering both object detection and occupancy prediction. The ablation studies (Tables 6–7) validate design choices, and comparisons with BEVFusion (Liu et al., 2023b) and GaussianFormer (H et al., 2024) are compelling.  \n- The shared encoder and Gaussian mixture model (Eq. 4–5) elegantly unify cross-modal features.  \n\n**Clarity**:  \n\n- The paper is well-structured, with clear figures (e.g., Fig. 2) illustrating the pipeline. Mathematical formulations (e.g., covariance matrix derivation in §3.2) are precise.  \n\n**Significance**:  \n\n- The framework advances multi-modal perception by addressing BEV’s quantization limitations. The 450% speedup over GaussianFormer (Table 5) demonstrates practical value.", "weaknesses": "**1. Limited Novelty in Fusion Mechanism**:  \n\n- While the Gaussian representation is novel, the fusion mechanism (shared encoder + deformable attention) closely follows prior work (e.g., BEVFusion’s CNN fusion, UniTR’s cross-modal attention). The paper does not sufficiently differentiate its fusion strategy from existing methods. For example, the deformable attention module (Fig. 2b) resembles Zhu et al. (2020) without novel adaptation for Gaussian properties.  \n\n**2. Incomplete Comparison with Sparse Detectors**:  \n\n- The authors acknowledge that sparse detectors (e.g., Zhang et al., 2024a; Li et al., 2024) outperform their method but dismiss them as “task-specific.” However, recent sparse fusion works (e.g., Wang et al., 2024b; Yin et al., 2024) achieve both high detection accuracy and occupancy prediction capability. A direct comparison with these methods is missing and weakens the claim of “task-agnostic” superiority.  \n\n**3. Shallow Technical Details**:  \n\n- Critical components lack implementation specifics:  \n  - The forward-projection initialization for LiDAR (§3.2) is described as “BEV-based” but lacks mathematical formulation (unlike the camera case).  \n  - The Gaussian mixture aggregation (Eq. 5) is presented as a summation without addressing computational complexity or optimization strategies for large-scale scenes.  \n\n**4. Under-Explored Limitations**:  \n\n- The computational overhead of Gaussian-to-voxel conversion (§3.4) is glossed over. For example, the “neighborhood radius” calculation for Gaussian-voxel pairing is mentioned but not analyzed (e.g., runtime vs. BEV resolution).  \n- The method’s reliance on dense LiDAR data (nuScenes) raises questions about its applicability to low-cost, camera-only systems.  \n\n**5. Weak Visualization**:  \n\n- Qualitative results (Fig. 3) are limited to two scenarios and lack comparisons with key baselines (e.g., OccFusion). The claimed “sharper boundaries” are not quantitatively validated.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "OnVW8Zj0oE", "reviewer": "Reviewer_mxcM", "summary": "GaussianFusion presents a unified 3D Gaussian-based framework for multimodal fusion, replacing discrete BEV with a continuous shared space. It aligns camera and LiDAR features via a shared encoder with deformable attention and iterative updates. A Gaussian-to-voxel module enables task-agnostic perception. On nuScenes, it achieves state-of-the-art results in detection and occupancy tasks with lower latency and memory.", "questions": "1. Have experiments been conducted on datasets such as nuPlan, NavSim, Waymo, or CARLA?\n\n2. Where is the real-time performance analysis that the autonomous-driving community cares about?\n\n3. What are the advantages of Gaussian-to-Voxel Pooling over BEV Pooling and Voxel R-CNN's Voxel ROI Pooling, and why weren't corresponding ablation experiments conducted?\n\n4. In Figure 2, the schematic diagrams in both the overall part (a) and part (b) do not clearly illustrate the correspondence with the inputs, leading to potential ambiguity and necessitating revision.\n\n5. What exactly is the \"&\" operation in Figure 2? It is not explicitly defined in the paper and needs to be clarified by the authors.\n\n6. The authors need to clarify whether the two tasks are trained separately or handled simultaneously by a single model; Figure 2 suggests joint inference, but the presentation is ambiguous.\n\n7. This work proposes to encode features by combining BEV and Gaussian representations, yet the detection head is not described in detail. Moreover, the paper does not explore whether alternative heads could be plugged in to verify the generality of the proposed encoding.", "limitations": "", "rating": 2, "confidence": 5, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "1. Unified continuous 3D Gaussian representation eliminates early discretization and preserves fine geometric/semantic detail.  \n\n2. Forward-projection initialization produces accurate, optimization-friendly Gaussians without random guesswork.  \n\n3. Shared Gaussian encoder with deformable attention enables natural cross-modal alignment and iterative refinement.  \n\n4. Gaussian-to-voxel pooling delivers task-agnostic features for both detection and dense occupancy, beating BEV baselines while using less memory and latency.", "weaknesses": "1. It lags behind sparse-detection specialists that optimize only for 3D object recall.  \n\n2. No temporal modeling is incorporated, so multi-frame cues remain unused.  \n\n3. Gaussian-to-voxel conversion still imposes a fixed voxel grid, re-introducing minor discretization error.  \n\n4. The shared encoder assumes synchronized full-modal inputs; robustness to sensor dropout is not evaluated.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)