How Do Images Align and Complement LiDAR? Towards a Harmonized Multi-modal 3D Panoptic Segmentation
Authors: Yining Pan
Qiongjie Cui
Xulei Yang
Na Zhao
Venue: ICML 2025 poster (2025)
Keywords: LiDAR and Image Perception
Multi-sensor
Panoptic Segmentation
Primary area: applications->computer_vision
TL;DR: We propose IAL, a new multi-modal framework that aligns and fuses image and LiDAR information at the data, feature, and mask levels to improve 3D panoptic segmentation, achieving state-of-the-art results.

Abstract: LiDAR-based 3D panoptic segmentation often struggles with the inherent sparsity of data from LiDAR sensors, which makes it challenging to accurately recognize distant or small objects. Recently, a few studies have sought to overcome this challenge by integrating LiDAR inputs with camera images, leveraging the rich and dense texture information provided by the latter. While these approaches have shown promising results, they still face challenges, such as misalignment during data augmentation and the reliance on post-processing steps. To address these issues, we propose **I**mage-**A**ssists-**L**iDAR (**IAL**), a novel multi-modal 3D panoptic segmentation framework.
In IAL, we first introduce a modality-synchronized data augmentation strategy, PieAug, to ensure alignment between LiDAR and image inputs from the start. Next, we adopt a transformer decoder to directly predict panoptic segmentation results. To effectively fuse LiDAR and image features into tokens for the decoder, we design a Geometric-guided Token Fusion (GTF) module. Additionally, we leverage the complementary strengths of each modality as priors for query initialization through a Prior-based Query Generation (PQG) module, enhancing the decoder’s ability to generate accurate instance masks. Our IAL framework achieves state-of-the-art performance compared to previous multi-modal 3D panoptic segmentation methods on two widely used benchmarks. Code and models are publicly available at https://github.com/IMPL-Lab/IAL.git.

LiDAR-based 3D panoptic segmentation often struggles with the inherent sparsity of data from LiDAR sensors, which makes it challenging to accurately recognize distant or small objects. Re- cently, few studies have sought to overcome this challenge by integrating LiDAR imputs with camera images, leveraging the rich and dense tex- ture information provided by the latter. While these approaches have shown promising results. they still face challenges, such as misalignment during data augmentation and the reliance on post- processing steps. To address these issues, we pro- pose Image-Assists- LiDAR (IAL), a novel multi- modal 3D panoptic segmentation framework. In IAL, we first introduce a modality-synchronized data augmentation strategy, PieAug. to ensure alignment between LiDAR and image inputs from the start. Next, we adopt a transformer decoder to directly predict panoptic segmentation results. To effectively fuse LiDAR and image features into tokens for the decoder, we design Geometric- guided Token Fusion (GTF) module. Addition- ally, we leverage the complementary strengths of each modality as priors for query initialization through a Prior-based Query Generation (PQG) module, enhancing the decoder's s ability to gen- erate accurate instance masks. Our IAL frame- work achieves state-of-the-ar performance com- pared to previous mentation methods on two widely used bench- marks. Code and models are publicly available at How Do Images Align and Complement LiDAR? Towards a Harmonized Multi-modal 3D Panoptic Segmentation
Yining Pan
Qiongjie Cui
Xulei Yang²
Na Zhao
Abstract
1. Introduction
3D Panoptic segmentation simultaneously assigns semantic labels and identifies distinct instances, effectively unifying 3D semantic (Zhao et al., 2021; Xu et al., 2023b) and in- stance (Li & Zhao, 2024) segmentation to provide a holistic understanding of the scene. This task is particularly crucial for real-world applications, such as dynamic object track- ing (Yang et al., 2023) and autonomous driving (Hong et al., 2021: Cao al., 2024). LiDAR is an indispensable sensor for perceiving the 3D world, with its LiDAR point cloud typically serving as the sole input for 3D panoptic segmenta- tion (Razani et al., 2021; Zhou et al., 2021; Li et al., 2022a). However, LiDAR inherently faces limitations in detecting small or distant objects due to its radial emission pattern, which results sparse returns along each laser ray (Li et al., 2022b). Consequently, entities that are small or located at a distance may not receive sufficient information. In contrast, camera images provide denser and more detailed represen- tations, effectively compensating for the sparsity in LiDAR data, particularly in these challenging scenarios.
This complementary nature has motivated the use of multi-modal information for enhanced panoptic segmen- tation. Recently, LCPS (Zhang al., 2023) and Panoptic- FusionNet (Song et al., 2024) have pioneered LiDAR-and- image fusion methods for multi-modal 3D panoptic segmen- tation. However, these methods only perform augmentation on the LiDAR side, leading to misalignment between the two modalities. This misalignment hinders effective inte- gration of information from both modalities, causing the models rely predominantly on LiDAR data, rather than fully utilizing both LiDAR and image data. Moreover, the prediction heads in LCPS and Canoptic-FusionNer do not directly predict 3D panoptic segmentation results. Instead, strategy that involves clustering
3D panoptic seg-
they use a postt-pooesseeeeee instances after semantic segmentation (Zhou et al., 2021). This strategy presents two issues: 1) The post-processing step is inefficient and limits the effectiveness of segmenta- tion by relying on preliminary results; 2) Their convolution- based prediction heads rely on local context, which may be suboptimal for panoptic segmentation as it requires global context for accurate predictions.
'Singapore University of Technology and Design (SUTD) "Institute for Infocomm Research (12R), A STAR, Singapore. Cor- respondence to: Na Zhao <nazzhoo@ sdud.edu.sg>.
Proceedings of the ndnd International Conference Machine Learning, Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).
To address the first limitation, we propose a modality-

---

Towards Harmonized Multi-modal 3D Panoptic Segmentation
synchronized augmentation strategy - PieAug. PieAug ensures that the augmentation of multi-view images is syn- chronized with the augmentation of their corresponding LiDAR pairs, enabling well-aligned and enriched LiDAR and image inputs. Notably, PieAug is a general multi-modal data augmentation strategy designed for outdoor segmen- tation tasks. Its LipAR-specific augmentation can be seen as a generalization of existing point cloud augmentation techniques, e.g., instance augmentation (Zhou al., 2021), PolarMix (Xiao et al., 2022), LaserMix (Kong et al., 2023b).
reliable instance queries, we introduce a set of learnable parameters as no-prior instance queries. Consequently, the three groups of instance queries, combined with a set of semantic queries, are input into the transformer decoder to predict instance masks and semantic labels.
Our contributions can be summarized as: 1) We present IAL, novel wransformerbbasee multi mottl framework for multi- modal 3D panoptic segmentation, eliminating the cumber- some post- pest-proesssing steps required by previous methods. 2) We propose PieAug, a multi-modal augmentation tech- nique that not only addresses the asynchronization issue but also serves as a generalized formulation of existing LiDAR augmentation methods. 3) We design the GTF and PQG modules that can effectively fuse image and LiDAR features as tokens and queries for the transformer decoder. 4) Our IAL achieves state-of-the-an art performance in outdoor panop- tic segmentation, surpassing previous methods by 2.5% and 4.1 % in PQ on the nuScenes and SemanticKITTI bench- marks, respectively.
To overcome the second post-processing limitation, we pro- pose adopting a transformer decoder for multi-modal 3D panoptic segmentation, inspired by its success in 3D panop- tic segmentation (Xiao et al., 2025) and multi-modal 3D object detection (Bai al., 2022; Yan al., 2023). By lever- aging global context and directly predicting class labels and mask outputs, the transformer decoder eliminates the inef- ficiencies and constraints associated with post-processing Despite its promise, adopting transformer decoder intro- duces new challenges, particularly in designing effective queries and tokens as inputs. To overcome these challenges, we introduce Geometric-guided Token Fusion (GTF) module and a Prior-based Query Generation (PQG) mod- ulc. Combined with the PieAug strategy, these components form our proposed solution: a novel Image-Assist-LiDA. transformer-base framework, named IAL, for multi-modal 3D panoptic segmentation in autonomous driving scenarios.
2. Related Work
3D Panoptic Segmentation. Most advanced approaches for LiDAR-based 3D panoptic segmentation can be categorized into three main groups: top-down, bottom-up, and single- path methods (Li & Chen, 2022). The top-down method typically follows a detection-first principle, where bounding boxes are predicted initially, followed by the generation of instance masks from points within those boxes (Sirohi et al., 2021; Xu al., 2023a; Ye et al., 2023). In contrast, hottom-up methods (Zhou al., 2021; Hong et al., 2021; Razani et al., 2021; Xu et al., 2022; Li et al., 2022a) begin with semantic segmentation predictions and then generate instance masks through operations such as grouping and clustering. Both top-down and bottom-up approaches are limited by the performance of their preliminary predictions (object detection or semantic segmentation, respectively). which hinders their ability to achieve holistic perception. On the other hand, single-path methods treat panoptic seg- mentation as a unified task, simultaneously addressing the segmentation of both "stuff" and "thing" classes. For ex- ample, RaskRange (Gu et al., 2022), MaskPLS (Marcuzzi et al., 2023), and P3Former (Xiao et al., 2025) utilize learn- able queries to predict masks and classes for both "thing" and "stuff" object types. Additionally, PUPS (Su al., 2023) and DQFormer (Yang et al., 2025) explore the use of auxiliary categorical and positional embeddings for query initialization. However, these methods mainly rely on Li- DAR. inputs and often struggle with recognizing small or distant objects, where geometric information degrades. This limitation inspires the use of camera images, which contain rich texture information, to enhance panoptic segmentation.
GTF module integrates the sparse, cylinder-shaped LiDAR features with the compact, grid-shaped image features to create input tokens. Specifically, we adopt Cylinder3DD (Zhu et al., 2020) to extract LiDAR features and use raw LiDAR points as geometric guidance to locate corresponding image patches for each cylindrical voxel. Additionally, we design a scale-aware positional embedding to encode the cylindrical voxels' locations and their receptive fields, facilitating the fusion of image patches and cylindrical voxels. This ap- proach enhances feature fusion while mitigating projection errors caused by variations in cylindrical voxel shapes.
Our PQG module leverages prior knowledge from Li- DAR and image inputs, which provide complementary strengths for object perception, to improve query initial- ization. Specifically, we generate two groups of instance queries - geometric-rrior and texture-prior instance queries derived from LiDAR and image modalities, respectively. Geometric-prior queries exploit LiDAR's geometric fea- tures, which are well-suited for detecting nearby or large objects rich in geometric information. In contrast, texture- prior queries leverage images by applying state-of-the-art detection and segmentation models, such as Grounding- DINO (Liu al., 2023a) and SAM (Kirillov et al., 2023), to better identify distant and small objects. To handle challeng- ing scenarios where both LiDAR and images fail to provide
Multi-modal 3D Scene Understanding. Multi-modal
2

---

Towards Harmonized Multi-modal 3D Panoptic Segmentation
Raw Inputs
Modality-symchronized Augmentation (PieAug)
Geometric-guided Token Fusion (GTF)
Prior-based Query Generation
(PQG)
params
tex.-prior
pair
EDD
Peh
qth
9eh
4eh
cylinder-img grids pairs
Told steulj LOAK
geo-prior
ne-prior
cylindriccl voxelization
point cloud
mixing
40
stale-sware PE
Transformer Decoder
class
O7
220
22
FFN
εDD
mask
inputs
multi-view images
modality tolkons
C/M /reg scan
feature encoder
feature
scale-aware PE
query
Figure 1. The architecture overview of our Image-Assistt-LiDAA (LAL) framework. We first voxelize the point cloud into cylindrical voxels. In PieAug, we synchronize augmentation by pairing cylindrical and image grids, mixing original and new scans in different modes. Using transformer-based structure, design token fusion (GTF) and query initialization (PQG) modules align and complement both modalities. In GTF, features from both modalities are bridged via LiDAR-to-image projection and scale-aware positional embeddings. We generate instance queries for "thing" classes using modality knowledge (specifically, geometric and texture priors from LiDAR and image features) and learnable parameters (for instances without advanced modality priors) in PQG. These instance queries, along with semantic queries, are iteratively updated through multiple transformer decoder layers to produce the final panoptic predictions.
learning with LiDAR and images has been extensively stud- ied for 3D semantic segmentation (Zhuang et al., 2021; Krispel et al., 2020; Yan al., 2022; Li et al., 2023b; Liu et al., 2023b; Man et al., 2023; Wu et al., 2024; An et al., 2025) and object detection (Liang et al., 2022; Yin et al., 2024; Bai et al., 2022; Li et al., 2024). A key challenge in these tasks is to effectively fuse the features of both modali- ties to leverage their complementary strengths. To address this, existing methods design fusion modules to align data from different sensors. Additionally, some object detection studies (Bai et al., 2022; Chen et al., 2023; Li et al., 2024; Zhang et al., 2024) have shown that image-driven queries improve the detection of challenging objects. While these approaches are not directly applicable to panoptic segmenta- tion, they inspire our design of the Geometric- goudedri--uuieed Token Fusion (GTF) and brior-based Query Generation (PQG). For multi-modal 3D panoptic segmentation, LCCPS (Zhang et al., 2023) and Panoptic-Fusion FusionNet (Song et al., 2024) are two pi- oneering works. LCPS designs a point-to-mask mapping for LiDAR to image fusion, while Panoptic-FusionNee directly applies point-to-pixel mapping by geometric information. However, these methods primarily apply augmentations only to LiDAR, leading to misalignment between modali- ties and an over-reliance LiDAR features. Additionally, their reliance on post-processing makes panoptic inference inefficient and limits the effectiveness of true multi-modal perception.
angles and selectively swap these intervals between scans. RangeFormer (Kong al., 2023a) represents the 3D scene range-view image and applies tailored augmentation strategies for range-view learning. Similarly, UniMix (Zhao et al., 2024) extends scene-wise mixing to different at- tributes, including intensity and semantic channels. In multi- modal 3D scene understanding, data augmentation for both modalities remains underexplored. LaserMix++ (Kong et al., 2024) extends LaserMix to multi-modal scenes but relies on single augmentation strategy. MSeg3D (Li et al., 2023b) applics asymmetric augmentation to each modality, using only simple local transformations - reducing the risk of mis- alignment but at the cost of data diversity. To address these limitations, we propose general multi-modal augmentation strategy that incorporates diverse instance- and acene-level mixing to enhance both cross- modal alignment and panoptic segmentation performance.
3. Methodology
In the multi- multi-modal 3D panoptic segmentation, are given 3D point cloud consisting of N discrete sampling points, denoted as P = {p, E I 11x41 contains its Cartesian coordinates in Euclidean space and its reflection intensity. The point cloud is associated with K view images, represented as I = [I& E TR Hx x31 K Hl and W denote the height and width of images. The goal of this task is to effectively utilize P and I to predict both semantic and instance labels for each point.
Jj=1- where each point p,
1k=1-
LiDAR-Based Data Augmentation. Studies on LiDAR- based data augmentation often rely on instance- or scene- level mixing. Instance mixing methods (Zhou et al., 2021: Xiao et al., 2022; Zhao et al., 2022) augment point clouds by copying instance points from one scan to another, while scene-wise mixing approaches, such LaserMix (Kong et al., 2023b) and PolarMix (Xiao et al., 2022), divide the scene into multiple intervals along inclination or azimuth
Framework Overview. In this paper, we introduce Image- Assist-LiDAR (IAL), a novel transformer- basedd framework for multi-modal 3D panoptic segmentation, as illustrated in Fig. To process the sparse and irregular LiDAR point cloud, we first apply cylindrical voxelization, converting points into gelindrical-shape voxels based on their polar coordinates. As result, each voxel v; contains a varying
3

---

Towards Harmonized Multi-modal 3D Panoptic Segmentation
PieAug
number of points, with voxel shapes differing along the
source of data
w/o image sync
w/ image sync
(a) lelinstonce
radial axis. The point cloud P can then be represented as == {v. where M is the number of valid cylindri-
cal voxels. We apply snddallt--ynccroniieee augmentation through our proposed Aug strategy (Sec. 3.1) ), ensuring consistency across LiDAR and image data by pairing each voxel with its corresponding image regions and employing a generalized augmentation operator for diverse effects.
swap (pltch)
(b) xcenn
swap (yaw)
The augmented 3D voxels and images are then processed by 3D encoder EaD and 2D encoder & extracting voxel- wise features F3D E IR M X and image features F2D E RxxxxWxD where D is the feature dimension. The 3D encoder uses Cylinder3D (Zhu et al., 2020), known for its strong generalization 3D panoptic segmentation (Xiao et al., 2025; Zhang et al., 2023), and 2D encoder is Swift- Net (Oršié& & Segvić, 2021) with a ResNet- 18 backbone.
Figure 2. Motivation and implementation variants of PieAug, Each column illustrates the motivation for LiDAR-image syn- chronized augmentation. Each row displays different pie-cut strategy. Point clouds are projected on camera images, with colors indicating semantic labels or data sources. Best viewed in color.
Here (-) is the operator that fits a bounding rectangle en- closing a set of pixels.
Next, we use F3D and F2D to create tokens and queries for a transformer decoder, enabling cross-modal interaction. To- ken features are formed by concatenating voxel features with their aligned image counterparts, guided by a unified, scale- aware positional embedding (Sec.3.2). Meanwhile, instance queries 9. are initialized using ors (Sec.3.3). These queries. along with semantic queries 9sm are fed into the transformer decoder to predicts the instance masks and semantic labels.
During augmentation, we determine the size and position of the pie-cut using 3D binary mask S E {0, 1 fxxxxx where R, θ, and Z represent the binning resolution along the radial, angular, and height axes of the cylindrical vox- elization, respectively. Each voxel is assigned a correspond- ing mask value S(r, 0, z). where S(r, z) = 1 indicates that the voxel is replaced with one from a new scan; other- wise, it remains unchanged from the original scene. Conse- quently, the augmented cylinder Vaug is obtained as follows:
pri-
3.1. Moddalty-SSnchhoniine Augmentation
(3) where & denotes an element- masking operation, multi- plying the voxel values by the corresponding mask. Since each voxel v, aligns with an image region gi, we apply image augmentation in parallel using the same mask S.
yaug = Vorg @ (1-S) S) + Vnew X S,
To mitigate modality misalignment and enhance diversity during data augmentation, we propose PieAug. The key idea is to extract flexible number of cylindrical voxels along the height, angle, or radius axes analogous to cutting variable-sized pie slice from a cake and swap it with a corresponding slice from another scene, as illustrated in Fig. 2. To maintain synchronization across modalities, each 3D "pie" is paired with its corresponding image patch, which is exchanged simultaneously.
Instance Pasting, We first illustrate how PieAug achieves instance-level augmentation (copy and paste) by selecting pie-cut voxels corresponding to s sampled instances from new scene. We apply transformations such as translation, rotation, and scaling to each instance. Next, we identify the indices of voxels that overlap with the s transformed instances as C. The mask S is then constructed as:
For simplicity, we illustrate the process of finding the corre- sponding image patch for single voxel from one camera view. This can be casily extended to a pie-shaped region with multiple image views. Given a voxel containing N; points, denoted as V;- = { first project the LiDAR point py from 3D coordinate system to its corresponding 2D coordinates in the image plane using the following transformation:
RxOx U
J.N., where ΣM N, = N, we
S =
1[ 0, C).
(4)
r=
Scene Swapping. We then illustrate how PieAug achieves scene-level augmentation, including scene swapping, which involves dividing the voxels evenly along a chosen axis (height or angle) and swapping them alternately. We achieve this by selecting all voxels along the radial axis, as well as one of the height or angle axes, and then freely choosing a number of slices from the remaining axis. For example, the mask for selecting b slices from the angle axis is defined as:
T T(p,) = K x T x IP. P3.2. P3.3, 1)T,
(1)
where K E 3x33 is the camera intrinsic matrix and T E 4x44 is the extrinsic transformation matrix. Next, we define gi as the bounding rectangle that encloses all projected points 7 (p:). This ensures that each voxel corresponds specific region in the image, denoted as g:):
1, if
((r,0,2) -1 0, otherwise
(5)
g: = B ((=(p))) Py E v:)).
(2)
4

---

Towards Harmonized Multi-modal 3D Panoptic Segmentation
Here, O denotes the set of indices corresponding the selected b angle slices. A similar masking strategy can be applied to height slices.
view image
no perceptive field
X: constraint field
Remarks. With the instance-level and scene-level aug- mentation capabilities described above, PieAug generalizes most existing LiDAR-based augmentation techniques, For example: 1) Panoptic-Polaan PolarNet: perform instance-level aug- mentation by sampling instances based on their semantic labels and applying transformations using translation and XY-plane rotation. 2) PolarMix (instance branch): perform instance-level augmentation by selecting instances based on their labels and applying transformations by rotating duplicated instances multiple times along the Z-axis. 3) Po- larMix (scene branch): perform scene-level augmentation by selecting half of the slices along the azimuth angle. 4) LaserMix: perform scene-level augmentation by selecting slices at different inclination angles. As result, PieAug offers greater flexibility in combining instance-level and scene-level augmentations by performing multiple augmen- tation rounds with different voxel indices. Furthermore, due to the synchronized transformations applied to both modalities, PieAug ensures well-aligned LiDAR and image augmentations.
www.mnae LIDAR coord.
V: sufficient field
Image coord.
(a) projection error
(b) positional embedding (PE)
Scale-aware Positional Embedding (SPE)
G virtual center physical point
center +
PE
cotreme point
SPE
projection
image patch
cylinder voxel
scale
MLP
LiDAR voxal
(c) module structure SPE
Figure 3. (a) and (b) illustrate two challenges in LiDAR-image
fusion introduced by cylindrical voxelization. In (a). relying on virtual voxel center can lead to projection errors, which become more pronounced for larger voxels and cause mismatches between LiDAR and image data. This motivates the use of physical points. In (b), canonical PE (row overlooks the varying sizes of voxel and image fcatures, while focusing solely on the region of physical points (row 2) limits the receptive field. Hence, we introduce scale-aware PE that uses extreme points, as shown in (c).
remains inadequate, as illustrated by the red region in the second row of Fig. 3(b). This limitation arises because phys- ical points provide only partial representation of objects in both the LiDAR and image domains, especially at greater distances where LiDAR points are sparsely distributed. To address this issue, we propose a unified Scale-aware Posi- tion Embedding (SPE) for both LiDAR and image tokens. SPE ensures consistent perception of perceptive fields by incorporating shared scale embeddings to both modalities, as shown in the third row of Fig. 3(c).
3.2. Geodetric-Guided Token Fusion
Since cylindrical voxelization results in voxels of varying sizes (larger in regions farther from the central origin), this poses two key challenges for generating multi-modal tokens: 1) how to align image features with LiDAR features, and 2) how to effectively fuse. To address these issues, we propose Geometric-guided Token Fusion (GTF), as illustrated in Fig. 3, which leverages the rich geometric information from LiDAR to guide alignment and enable effective fusion.
Specifically, as illustrated in Fig. 3(c), we introduce an extreme point set, denoted as V; = {p,) ) consisting of the eight corners of each cylindrical voxel to represent its scale. In the LiDAR space, these extreme points define the spatial partitioning of the scene. In the image space, they outline potential perceptive field regions, particularly in areas with sparse LiDAR points, allowing for more adaptive perception range. In SPE, each cylindrical voxel is embedded with both its scale and centroid position:
Specifically, we align features at the voxel level by project- ing all physical points within a voxel v, onto the image plane and averaging their corresponding image features to create aggregated representation:
(6) We refer to the paired voxel-wise LiDAR feature F&D and image feature F?D as the contents of the i-th multi-modal token. Notably, aggregating image features by projecting all physical points within a voxel preserves feature validity. As shown in Fig. 3(a), using only the voxel centroid can lead to misalignment, as the projected location may not correspond accurately to the relevant image region.
FPD =
N Σ, 2 (T(P)))
N:
(7) where Avg(-) denote the averaging method and and || is L2 norm operation in Cartesian space. 1á represents mixed- parameterized positional encoding function (Xiao et al., 2025), which embeds the centroid position in both Cartesian and polar spaces. ø is a multi-layer perceptron (MLP) that projects the scale feature into the same dimension as the centroid embedding. With SPE, the final fused multi-modal token Ffuse for i-th cylindrical voxel is obtained by:
S; = (Avg(vv)) + <(I|V, - Avg(v+)ll2),
Position embedding (PE) has proven effective in aligning features from different modalities (Yan et al., 2023). How- ever, due to the varying sizes of cylindrical voxels, encoding only the voxel centroid is suboptimal, as single point may not sufficiently represent the voxel's I's perceptive field, Even when using physical points for PE, capturing the full per- ceptive field of a voxel or its corresponding image region
F Ffuse
Cat + s.), (F? @ ss)..
(8)
3.3. Prior-Basee Query Generation
Previous works, such as P3Former (Xiao et al., 2025), ini- tialize queries by a set of learnable parameters. However,
5

---

Towards Harmonized Multi-modal 3D Panoptic Segmentation
Table 1. Preliminary study of positional embedding for objects of thing classes. We conduct the experiment our LiDAR branch. "GT" denotes using the ground truth center position, while "Noise" denotes adding Gaussian noise with kernel size of to the GT center position. "th" and "st" is the thing and stuff classes.
the unsupervised DBSCAN (Ester et al., 1996) algorithm. Finally, the centroids of these clusters serve location hints for texture-prior instances.
Given location hints from both modalities, we apply Far- thest Point Sampling (FPS) to obtain a fixed number 4pr of location hints. It is worth noting that for large objects easily recognizable by both LiDAR and images, global sampling provides a holistic view and reduces redundant candidate proposals. Similar to the positional embedding used for multi-modal tokens, we apply SPE to embed both the query location and scale features. We then extract the query con- tent by indexing into the corresponding voxel features Ffuse, and finally add the SPE to this content to form the final query representation.
Modality
GT
Noise
PQ
mIoU
PQ
PQ"
LiDAR LiDAR LiDAR
77.0
75.9
77.8
75.7
/
83.2
82.3
88.5
74,4
81.8
79.8
86.8
73.6
such queries tend to prioritize easier samples while neglect- ing more challenging ones. Additionally, preliminary results have validated that accurate positional embedding signifi- cantly enhances the model's ability to locate objects. As demonstrated in Table 1, applying the ground truth center position for thing classes resulted in a 6.2% improvement in overall PQ and a 10.7% improvement in PQ for thing classes specifically. Even adding noise on the position, the improvement still significant.
No-Prior Query. We hypothesize that instances without advanced priors exhibit specific feature representation paradigm, allowing them to be recognized through a set of learnable parameters. We set the number of no-prior queries as 4n- This implicit paradigm learning enables the model to search within a smaller candidate pool and effectively identify potential instances.
Inspired by this observation, we propose the Prior-based Query Generation (PQG) module to explicitly leverage texture features from the image domain, and geometric infor- mation from LiDAR domain as prior knowledge to generate well-informed initializations for instance queries. Specifi- cally, we design three groups of queries: geometric-prior, texture-prior, and no-prior queries.
All geometric-, texture- and no-prior queries are concate- nated and fed into the transformer decoder for "thing" class prediction, i.e., 3D instance segmentation. Semantic queries are initialized following the approach in P3Former, with auxiliary semantic supervision applied. These semantic queries are used to predict segmentation results. We fol- low the process in P3Former to combine the instance and semantic predictions into the final panoptic segmentation.
Geometric-Prior Query. Potential geometric-pric -rrior in- stances are those for which LiDAR features exhibit min- imal degradation, allowing geometric characteristics to pro- vide sufficient information for accurate positional hints. Compared to the rich texture features captured in images, LiDAR features offer more precise location predictions. Therefore, we generate location hints for geometric-prion queries by predicting center heatmap and performing Non- Maximum Suppression (NMS) sampling according to con- fidence scores and range radius threshold. Specifically, we predict the class-agnostic heatmap in the polar --rde View (BEV) space using a structure similar to (Yin et al., 2021; Bai et al., 2022). For each selected instance proposal (identified by its coordinates in the center heatmap), we lift it into 3D space by averaging all valid voxels across the height dimension.
4. Experimental Results
4.1. Experimental Setting
Datasts. nuScenes (Caesar et al., 2020; Fong et al., 2022) is large-scale, multi-modal dataset designed for autonomous driving, containing data from a 32-beam LiDAR, radars, and 6 RGB cameras. It includes 40,157 frames of outdoor scenes, with 34,149 frames labeled for training and valida- tion, and the remaining reserved for testing. The panoptic annotations cover 10 "thing" classes, 6 "stuff" classes, and class for noisy labels. SemanticKITTI (Behley et al., 2019; 2021) is an outdoor dataset derived from KITTI Vi- sion Benchmark (Geiger et al., 2012). It includes data from beam LiDAR sensor and two front-view cameras, in- cluding 8 "thing" classes and 11 "stuff" classes, comprising 19,130 frames for training, 4,071 frames for validation, and 20,351 frames for testing.
Texture-Prior Query. For objects that are small or located far from the sensor, geometric information often becomes unreliable, making accurate location prediction difficult. To address this issue, we use texture information from images to discover potential texture- prior instances. First, we ex- tract mask proposals using the pre- trained image segmenta- tion models Groonningg-INOO and SAM (Liu et al., 2023a; Kirillov et al., 2023). We then lift each 2D mask into a 3D frustum and collect all the 3D points that fall within it. To mitigate noise caused by overlapping along the depth dimension, we cluster these points into several groups using
Evaluation Metries. Consistent with the standard works (Kirillov et al., 2018; Zhang et al., 2023; Xiao et al., 2025), panoptic quality (PQ) is selected as the primary met- ric. PQ is defined as the product of segmentation quality
6

---

Towards Harmonized Multi-modal 3D Panoptic Segmentation
Table Comparison of panoptic segmentation performance the nuScenes validation set. Top results are shown in bold. "M."
indicates which modality (or modalities) each method uses. "P.- PE-PCSCNet is the LiDAR branch of "P.- FusionNet" (Song et al., 2024).
Method
M.
PQ
PQ'
RQ
SQ
PQ"
RQ
SQ"
PQ
RQ*
SQ"
mloU
DS-Net (Hong al., 2021)
L
42.5
51.0
50.3
83.6
32.5
38.3
83.1
59.2
70.3
84.4
70.7
EfficientLPS (Sirohi et al., 2021)
62.0
65.6
73.9
83.4
56.8
68.0
83.2
70.6
83.6
83.8
65.6
P.PPlarNet (Zhou et al., 2021)
L
67.7
71.0
78.1
86.0
65.2
74.0
87.2
71.9
84.9
83.9
69.3
P.-PHNet (Li et al., 2022a)
L
74.7
77.7
84.2
88.2
74.0
82.5
89.0
75.9
86.9
86.8
79.7
CFNet (Li ct al., 2023c)
L
75.1
78.0
84.6
88.8
74.8
82.9
89.8
76.6
87.3
87.1
79.3
CenterL.PS (Mei al., 2023)
76.4
79.2
88.0
86.2
77.5
88.4
87.1
74.6
87.3
84.9
77.1
LCPS (Zhang et al., 2023)
72.9
77.6
82.0
88.4
72.8
80.5
90.1
73.0
84.5
85.5
75.1
P.PCSCNNet (Song et al., 2024)
L
72.7
75.4
84.8
86.4
71.2
82.9
86.6
75.1
84.2
84.2
69.8
P3Former (Xiao et al., 2025)
L
75.9
78.9
84.7
89.7
76.9
83.3
92.0
75,4
87.1
86.0
76.8
IAL (our LiDAR branch)
L
77.0
79.6
85.1
90.2
77.8
83.8
92.6
75.7
87.3
86.2
75.9
LCPS (Zhang et al., 2023)
L+C
79.8
84.0
88.5
89.8
82.3
89.6
91.7
75.6
86.5
86.7
80.5
P.-FusionNet (Song et al., 2024)
L+C
77.2
79.3
87.2
87.8
77.5
87.7
88.2
76.2
85.9
86.0
73.4
IAL (ours)
L+C
82.3
84.7
89.7
91.5
85.3
90.6
94.1
77.3
88.2
87.2
80.6
Table Comparison on the nuScenes test set. Top and runner-up results are marked in bold and underline, respectively. nert indicates the use of additional temporal frames and detection annotations. Our method is evaluated without test time augmentation or ensembling.
Method
M.
PQ
PQ
RQ
SQ
PQ"
RQ"h
SQ
PQ
RQ
SQ*
mloU
EfficientLPS (Sirohi et al., 2021)
L
62.4
66.0
74.1
83.7
57.2
68.2
83.6
71.1
84.0
83.8
66.7
P.PPolarNet (Zhou al., 2021)
L
63.6
67.1
75.1
84.3
59.0
69.8
84.3
71.3
83.9
84.2
67.0
P.-PHNet (Li et al., 2022a)
L
80.1
82.8
87.6
91.1
82.1
88.1
93.0
76.6
86.6
87.9
80.2
CPSeg (Li et al., 2023a)
L
73.2
76.3
82.7
88.1
72.9
81.3
89.2
74.0
85.0
86.3
73.7
MaskPLS (Marcuzzi et al., 2023)
L
61.1
64.3
68.5
86.8
54.3
58.8
87.8
72.4
84.5
85.1
74.8
LCPS (Zhang al., 2023)
L
72.8
76.3
81.7
88.6
72.4
80.0
90.2
73.5
84.6
86.1
74.8
LidarMultiNet (Ye et al., 2023)
L.
81.4
84.0
88.9
91.3
83.9
89.9
93.1
77.3
87.1
88.2
82.2
IAL (our LiDAR branch)
L
75.1
77.7
83.0
90.1
75.0
80.9
92.4
75.2
86.5
86.4
73.3
4DFormer (Athar et al., 2023)
L+C
78.0
81.4
86.6
89.7
80.0
87.8
90.9
74.6
84.5
87.6
80.4
TCPS (Zhang et al., 2023)
L+C
79.5
82.3
87.7
90.3
81.7
88.6
92.2
75..9
86.3
87.3
78.9
IAL (ours)
L+C
82.0
84.3
89.3
91.6
84.8
90.2
93.8
77.5
87.8
87.8
79.9
(SQ) and recognition quality (RQ):
strategies to 0.4:0.05:0.05, respectively. Additionally, we perform basic transformations including random rotation, flipping, and scaling. We set the number of prior-based and no-prior instance queries to 4pr = l1 128. We use Adam (Kingma & Ba, 2014) as the optimizer, with a default weight decay of 0.01. The entire model is trained from scratch with a batch size of 2, using 4 NVIDIA A40 GPUs. The training spans 80 epochs for nuScenes and 36 epochs for SemanticKITTI The initial learning rate is to 0.0008 and decays by half at epochs [60,75] for nuScenes and [30,32] for SemanticKITTI respectively. All model results listed in the following sections are NOT employed with any test-time augmentation (TTA) method.
Σ
IOU
TP
PQ
(9)
TP
TP
+
FP|
FN
SQ
RQ
where IoU denotes the Intersection over Union, TP denotes True Positives and so as for others. Theses metrics can be further extended to "thing" and "stuff" classes, denoted as thth, PQ", RQth, RQ" SQth and SQ We also report PQ (Porzi et al., 2019), which replaces PQ with mIoU for stuff classes.
Implementation Details. We follow standard practice (Zhou et al., 2021; Li et al., 2022a; Xiao et al., 2025) to rep- resent point clouds by discretizing the 3D space into cylin- drical voxels of size [480 x 360 x 32). The LiDAR branch built upon the architecture of P3former. For nuScenes, the polar coordinate range is defined as [-50m, 50m] x [0, 2m] x [-5m, 3m], while for SemanticKIrTI, the height range is adjusted to [-4m, 2m]. All images resized to 640 x 360. For augmentation, we employ the following strategies: instance pasting and scene-swapping (split the scene along the height and angle axes, with the number of splits randomly chosen from [3, 4,5] 5] each time). We
4.2. Benchmark Results
nuScenes. We present comprehensive comparison results for LiDAR panoptic segmentation performance on the nuScenes validation and test sets, as shown in Table and Table 3. Due to the limited number of multi-modal methods, currently only LCPS (Zhang et al., 2023) and Panoptic-FussionNet (Song al., 2024), we also include LiDaR-only methods for comparison. Notably, our method IAL achieves the best performance across all metrics on the validation set and ranks first or second on most metrics
set the ratio of application for these three augmentation
7

---

Towards Harmonized Multi-modal 3D Panoptic Segmentation
Table 4. Comparison of panoptic segmentation performance on the SemanticKITTI validation set. Top results are shown in bold.
Table 5. Ablation study of the proposed modules in our framework.
"PIE" denotes the PieAug module.
Method M.
PQ
PQ
RQ
SQ
mloU
PIE
GTF
PQG
PQ
PQ
RQ
SQ
mloU
P.PPolaNNet
L
59.1
64.1
70.2
78.3
64.5
75.7
78.1
84,4
88.3
73.8
DS-Net
L
57.7
63.4
68.0
77.6
63.5
78.4
81.0
86.9
90.0
78.2
EfficientLPS
L
59.2
65.1
69.8
75.0
64.9
81.1
83.5
89.0
90.9
80.2
P.PHNNet
L
61.7
65.7
82.3
84.7
89.7
91.5
80.6
CenterLPS
L
62.1
67.0
72.0
80.7
LCPS
L
55.7
65.2
65.8
74.0
61.1
Table 6. Ablation study of PQG module. "Geo.", "Tex.". and "NP" represent geometric prior, texture prior, and no-prior queries, re- spectively. We set the total number of queries to 256 for fair comparison. In configurations combining prior (geometric or tex- ture) and no-prior queries, 128 queries are allocated to each set.
P3Former
L
62.6
66.2
72.4
76.2
IAL (LiDAR)
L
62.0
65.1
71.9
76.0
64.9
LCPS
L+C
59.0
68.8
68.9
79.8
63.2
IAL (ours)
L+C
63.1
66.3
72.9
81.4
66.0
Geo.
Tex.
NP.
PQ
PQ
PQ
mloU
in the test set. Specifically, IAL outperforms LCPS and Panoptic-Fusionte FusionNet by significant margin of 2.5% and 5.1% in PQ on the validation set, as shown in Table 2. This improvement is attributed to superior performance in both recogmition (surpassing the two previous works by 1.2% and 2.5% in RQ, respectively) and segmentation (surpassing them by 1.7% and 3.7% in SQ). Furthermore, our model demonstrates superior performance on both "thing" and "stuff" classes, achieving a 7.8% and 1.1% improvement in metrics compared to the latest work, Aanoptic-FusionNee Compared to the LiDAR-oni, baseline (using the same aug- mentation strategies as P3Former adopts), IAL achieves a 5.3% improvement, primarily due to a 7.5% increase from thing classes, demonstrating the effectiveness of image as- sistance in detecting and recognizing objects. In Table 3, IAL also demonstrates superior performance, achieving the highest scores across most metrics the nuScenes leader- board. These outstanding results highlight the effectiveness of our modules for modality alignment and compensation.
81.2
83.8
76.8
79.8
/
81.3
83.9
77.0
80.0
/
81.1
83.4
77.2
80.0
80.7
83.0
77.0
80.0
82.3
85.3
77.3
80.6
Table 7. Comparison of augmentation strategies. "Img" indicates whether snnge-synnhronizzed augmentation applied.
Method
Img
PQ
PQ
RQ
SQ
mloU
PolarMix
80.3
82.8
87.9
91.0
78.6
LaserMix
80.6
83.0
88.5
90.8
79.3
PieAug (ours)
81.4
83.7
89.1
91.0
80.1
PieAug (ours)
82.3
84.7
89.7
91.5
80.6
unified scale embedding and accurate projection enhance multi-modal representations. Finally, incorporating the PQG module brings an additional 1.2% gain in PQ, validating our hypothesis that initializing queries with modality priors leads to more precise object predictions than using purely learnable parameters.
SeTTaticKITTI presents a significant challenge due to its use of only two front-viey view cameras, limiting the availability of image features to support LiDAR. As shown in Table 4, despite these constraints, our IAL achieves 4.1% improve- ment in PQ over the state-of-the-art multi-modal baseline LCPS, demonstrating the robustness of our method even under limited image supervision.
The effectiveness of the PQG module is validated in Table 6. Rows 1-3 show comparable performance, suggesting that purely learnable queries tend to overfit easy or redundant samples, even when uni-modal priors are available. Row 4 shows slight drop, likely due to an excessive number of prior-based queries exceeding the number of ground-truth instances, resulting in more false positives. In contrast, our design (row 5) assigns strong geometric and texture priors to confident regions, while reserving lcarnable queries for harder, low-prior cases. This balanced allocation improves the model's 's ability to handle both easy and difficult samples, leading superior overall performance.
4.3. Ablation Studies
To validate the effectiveness of our proposed components, we conduct comprehensive ablation studies on the overall proposal framework in Table 5 and provide detailed analyses for each individual module in Table All experiments are conducted on the nuScenes validation set using the same Syper-parameters for fair comparison.
4.4. Augmentation Methods Comparison
As shown in Table 7, we compare PieAug with LiDAR- only augmentation methods, including PolarMix (instance pasting and scene mixing) and LaserMix (inclination an- gle splitting) by 2.0% and 1.7% in PQ. Even with LiDAR- only augmentation, PieAug achieves superior performance, demonstrating its effectiveness as a generalized framework.
As shown in Table 5, compared to the baseline that uses only basic point cloud transformations (row 1). PicAug im- proves PQ by 2.7%, benefiting from better input alignment and enriched scene context. Building on this, GTF further boosts PQ by 2.7% and RQ by 2.15 %, demonstrating that
8

---

Towards Harmonized Multi-modal 3D Panoptic Segmentation
Ground Truth
LCPS [23'ICCV]
IAL (LiDAR Branch)
IAL
barrier
bus
pedestrian
truck
drivable surface
sidewalk
terrain
man-made
vegetation
Figure Qualitative comparison of our method with preliminary multi-modal panoptic segmentation baseline, LCPS. To highlight the differences, mark false positive and false negative predictions, which affect recognition quality, well well-matched and mismatch points in true positive predictions, which impact segmentation quality. GT colorized by semantic label. Best viewed in color.
image
4.5. Qualitative Results and Discussion
We present qualitative evaluations nuScenes validation set. As illustrated in the error maps in Fig. 4, our method notably reduces false positives (red points) and false nega- tives (green points) compared to LCPS. Furthermore, IAL outperforms its LiDAR branch detecting remote objects (highlighted in the black boxes) and recognizing ambiguous classes (in yellow boxes), leveraging the assistance of image data. In Fig. 5, we compare instance prediction with GT, Li- DAR branch, and IAL alongside the corresponding images. IAL showeases significant performance improvements in: (1) distinguishing multiple objects when they are clustered together (rows and 2); (2) detecting distant objects (row 3); (3) recognizing false positive objects (rows and 5).
FO
5. Conclusion
This paper proposes IAL, a multi-modal 3D panoptic seg- mentation framework that harmonizes LiDAR and im- ages through PieAug (synchronized augmentation), GTF (geometry-guidee fusion), and PQG (prior-based queries). IAL directly predicts panoptic results via a transformer decoder, eliminating post- processing and achieving state- of-the-art performance on nuScenes (82.3% PQ) and Se- manticKITTI (63.1% PQ). Texture-prior queries enhance small/distant object recognition, while geometric-pen prior queries improve large/nearby instance localization.
Figure 5. Visualization of instance predictions. Red circles high- light instances where the LiDAR branch fails segment correctly, but our multi-mont modal method succeeds. Best viewed in color.

---

Towards Harmonized Multi-modal 3D Panoptic Segmentation
Impact Statement
databases with noise. In Proceedings of the Second Inter- national Conference on Knowledge Discovery and Data Mining, KD,'96, pp. 226-231. AAAI Press, 1996.
This paper aims to enhance multi-modal 3D panoptic seg- mentation. There are minor potential societal consequences of our work, none of which we feel must be specifically highlighted here.
Fong, W. K., Mohan, R., Hurtado, J. V., Zhou, L., Caesar, H., Beijbom, O., and Valada, A. Panoptic nuscenes: large-scale benchmark for lidar panoptic segmentation and tracking. In ICRA, 2022.
References
An, Z., Sun, G., Liu, Y., Li, R., Wu, M., Cheng, M.-M., Konukogiu, E., and Belongie, S. Multimodality helps few-shot 3d point cloud semantic segmentation, 2025.
Geiger, A., Lenz, P., and Urtasun, R. Are we ready for Autonomous Driving? The KITTI Vision Benchmark Suite. In Proc. of the IEEE Conf. on Computer Vision and Pattern Recognition (CVPR), pp. 2354-3361, 2012.
URL https: arxiv. http://..//212...... 22489.
Athar, A., Li, E.,, Casas, S., and Urtasun, R. 4d-form::: Mul- timodal 4d panoptic segmentation. In Tan, J., Toussaint, M., and Darvish, K. (eds.), Proceedings of The 7th Con- ference on Robot Learning, volume 229 of Proceedings of Machine Learning Research, pp. 2151-2164. PMLR, 06- 09 Nov 2023. URL wwwpthttp//ccm http///rrceeee...c.oom mml.. press/ /v229/ athar23a. html.
Gu, Y., Huang, Y., Xu, C., and Kong, H. Maskrange: A mask-classfficttion model for range-view based lidar segmentation. ArXiv, abs/2206.12073. 2022. URL attp::/api..smaatiicchhllaa CorpusID:250000 .emanticcccoolar....... org/
Hong, F., Zhou, H., Zhu, X., Li, H., and Liu, Z. LiDAR- based Panoptic Segmentation via Dynamic Shifting Net- work. In 2021 IEEF/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 13085- 13094, Nashville, TN, USA, June 2021. IEEE. ISBN 978-1-66544-509-2. doi: 10.1109/VVP44447.2221. 01289. URL https:/tttsstttt http:///eeeepppee wreexploee... www.:plooor.eee.rgg......... document/ 9577777//
Bai, X., Hu, Z., Zhu, X., Huang, Q., Chen, Y., Fu, H., and Tai, C.-L. Transfusion: Robust lidar-camera fusion for 3d object detection with transformers. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 1090-1099, 2022.
Behley, J., Garbade, M., Milioto, A., Quenzel, J., Behnke, S., Stachniss, C., and Gall, J. SemantieKITTI A Dataset for Semantic Scene Understanding of LiDAR Sequences. In Proc. of the IEEE/CVF International Conf. on Computer Vision (ICCV), 2019.
Kingma, D. P. and Ba, J. Adam: A method for stochastic optimization. CoRR, abs/1412.6980, 2014. URL https: //ppi. semanticscholar. seg/CorpuuID:662810
Behley, J., Garbade, M., Milioto, A., Quenzel, J., Behnke, S., Gall, J., and Stachniss, C. Towards 3D LiDAR-basee se- mantic scene understanding of 3D point cloud sequences: The SemanticKITTI Dataset. The International Jour- nal on Robotics Research, 40(8-9) (000889:95999990 9679999967277679 2021. doi: 10.1 00.11770027334491110000 1006735.
Kirillov, A., He. K., Girshick, R. B., Rother, C., and Dollár, P. Panoptic segmentation. 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition 2018. URL https: /api. semanticscholar: org/ http:/oo:p:DDD3333330
(CVPR), pp.
Kirillov, A., Mintun, E., Ravi, N., Mao, H., Rolland, C., Gustafson, L., Xiao, T., Whitehead, S., Berg, A. C., Lo, W.-Y., Dollár, P., and Girshick, R. Segment any- thing. In 2023 IEEE/CVF International Conference on Computer Vision (ICCV), pp. 10. 1019/CCC510000222..0000
Caesar, H., Bankiti, V., Lang, A. H., Vora, S., Liong, V. E., Xu, Q., Krishnan, A., Pan, Y., Baldan, G., and Beijbom, O. nuscenes: A multimodal dataset for autonomous driving. In CVPR, 2020.
2023. doi:
Cao. A.- Dai, A., and de Charette, R. Pasco: Urban 3d panoptic scene completion with uncertainty awareness. In CVPR, 2024.
Kong, L., Liu, Y.. Chen, R., Ma. Y., Zhu, X., Li, Y., Hou, Y., Qiao, Y., and Liu, Z. Rethinking range view rep- resentation for lidar segmentation. International Conference on Computer Vision (ICCV), pp. 228-240, 2023a. doi: 10. 10.111/CCV5510002222...000
Chen, X., Zhang, T., Wang, Y., Wang, Y., and Zhao, H. Futr3d: A unified sensor fusion framework for 3d de- tection. In proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 172-181, 2023.
2023 IEEE/CVF
Kong, L., Ren. J., Pan, L., and Liu, Z. Lasermix for semi- supervised lidar semantic segmentation. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 21705-21715, 2023b.
Ester, M., Kriegel, H.-P., Sander, J., and Xu, X. A density- based algorithm for discovering clusters in large spatial
10

---

Towards Harmonized Multi-modal 3D Panoptic Segmentation
Kong, L.,, Xu, X., Ren, J., Zhang, W., Pan, L., Chen, K., Ooi, W. T., and Liu, Z. Multi-modal data-efficient 3d scene understanding for autonomous driving, 2024. URL battps://axxx org/ abs/ 2405. 05258.
Liang, T., Xie, H., Yu, K., Xia, Z.,, Lin, Z., Wang, Y., Tang, T,, Wang, B., and Tang, Z. Bevfusion: A simple and robust lidar-camera fusion framework. volume 35, pp. 10421-10434 2022.
Krispel, G., Opitz, M., Waltner, G., Possegger. H., and Bischof, H. Fuseseg: Lidar point cloud segmentation fusing multi-moda data. In Proceedings the IEEE/CVF winter conference applications of computer vision, pp. 1874-1883, 2020.
Liu, S., Zeng. Z., Ren, T., Li, Zhang, H., Yang, J., Li, i C., Yang, J., Su, H., Zhu, J., et al. Grounding dino: Marry- ing dino with grounded pre-training for open-set object detection. arXiv preprint M.Xiv:2333305499 2023a.
Liu, Y., Runnan, C.,, Li, X., Kong, L., Yang. Y., Xia, Z.,, Bai, Y., Zhu, X., Ma, B., Li, Y., Qiao, Y., and Hou, Y. Uniseg: A unified multi-modal lidar segmentation network and the openpeseg codebase. pp. 21605-21616, 10 2023b.
Li, E., Razani, R., Xu, Y., and Liu, Cpseg: Cluster-free panoptic segmentation of 3d lidar point clouds. pp. 8239-8245, 2023a. URL https: //api. www.eteeincnolllaii.....i wwt.eoorrooooouooeee 140419848,
doi: 10.1109/CCV510000222..0099
Man, Y., Gui, L.-Y., and Wang, Y-X. BEV-Guided Multi- Modality Fusion for Driving Perception. In CVPR, 2023,
Li, J., He, X., Wen, Y., Gao, Y., Cheng, X., and Zhang, D. Panoptic-phned Towards real-time and high-precision lidar panoptic segmentation via clustering pseudo heatmap. 2022 IEEF/CVF Conference Computer Vi- sion and Pattern Recognition (CVPR), pp. 11799-11808, 2022a. URL https: api. serd.seeaan semant insnantccccolar. org/Cor TD: 248811222
Marcuzzi, R., Nunes, L., Wiesmann, L., Behley, J., and Stachniss, C. Mask-Based Panoptic LiDAR Segmentation for Autonomous Driving. 8(2):1141- 1148, 2023. doi: 10.10099LRA.2223336666. URL wp pprconteent/ppperitte-daaat marcuzz1202rall.pa
https: htl:://www.ee ipb wni.bonn.nnnnnn.. de/
Li, J., Dai, H., Han, H., and Ding, Mseg3d: Multi-modal 3d semantic segmentation for autonomous driving. In CVPR, pp. 21694-21704, 2023b.
Mei, J., Yang. Y., Wang, M., Li, Z.,, Hou, X., Ra, Li, L., and Liu, Y. CenterLPS: Segment Instances by Centers for LiDAR Panoptic Segmentation. In Proceedings of the st ACM International Conference on Multimedia, MM
Li, L. and Zhao, N. End-to-end semi-supervised 3d in- stance segmentation with peteacher. In 2024 IEEE Inter- national Conference on Robotics and Automation (ICRA), pp. 5352-5358. IEEE, 2024.
'23. pp. 1884-1894, New York, NY, USA, 2023. Associa- tion for Computing Machinery, ISBN 9798400701085. URL https::/dl. 361008
doi: 10.
Li, X. and Chen, D. A survey on deep learning- based panoptic segmentation. Digital Signal Processing, 120:103283, 2022. ISSN 1051-2004. doi:
acm htt.or///oo/11..1111
Oršić, M. and Šegvić, S. Efficient semantic seg- mentation with pyramidal fusion. Pattern Recog- nition, 110:107611, 2021. ISSN 0031-3203. doi: http://0iiogyl...0d///pptooo0000
URL https::/www.tt sttp:///wwwwcieeceedreeetttt com/ science/artiicleppii/S/st
URL http:://ww.eeleeccddeeet... coctenee/articeee//////311222222
Li, X., Zhang, G., Wang, B., Hu, Y., and Yin, B. Center fo- cusing network for real-time lidar panoptic segmentation. pp. 13425-13434, 06 2023c. doi: 10. 19.1099/CP 2023.01290.
Porzi, L., Buld, R., Colovic, A., and Kontschieder, P. Seamless scene segmentation. 2019 IEEE/CVF Conference Computer Vision and Pattern Recog- nition (CVPR), pp. 8269-8278, 2019. URL https: //api semant wwsccholar.org/oorppoi 145047913.
Li, Y., Yu, A. W., Meng, T., Caine, B., Ngiam, J., Peng, D., Shen, J.. Lu, Y., Zhou, D.. Lc, Q V., et al. Decpfu- sion: Lidar-camera deep fusion for multi-modal 3d object detection. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 17182- 17191, 2022b.
Razani. R., Cheng, R., Li, E., Taghavi, E. M., Ren, Y.. and Liu, B. Gp-s3net: Graph-based panop- tic sparse semantic segmentation network. 2021 IEEE/CVF International Conference on Computer Vision (ICCV), pp. 16056-16065, 2021. URL https: //api. hempp.ssemanticscccoeooo wrtp:orpo///ooo.rr.o 237213370.
Li, Y., Fan, L., Liu, Y., Huang, Z., Chen, Y., Wang, N., and Zhang, Z. Fully sparse fusion for 3d object detec- tion. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2024.

---

Towards Harmonized Multi-modal 3D Panoptic Segmentation
Sirohi, K., Mohan, R., Buscher, D., Burgard, W., and Val- ada, A. Efficientips: Efficient lidar panoptic segmenta- tion. IEEE Transactions on Robotics, 38: 1894944-1 2021. URL https: /ttp://app.. wtwppp.smmttticcchoooooooo httpp::/pp....mmmnniccccceeee org/CorpusTDD 221::33113357 933783.
Yan, X., Gao, J., Zheng. C.,, Zheng, C., Zhang, R., Cui, S., and Li, Z. 2dpass: 2d priors assisted semantic segmen- tation on lidar point clouds. In European Conference on Computer Vision, pp. 677-695. Springer, 2022.
Yang, J., Cen, J., Peng, W., Liu, Shuai amd Hong, F., Li, X., Zhou, K., Chen, Q. and Liu, Z. 4d panoptic scene graph generation. In NeurlPS, 2023.
Song. H., Cho, J., Ha, J., Park, J., and Jo, K. Panoptic- fusionnet: Camera-lida -idar fusion-based point cloud panop- tic segmentation for autonomous driving. Expert Syst. Appl., 25I(C), October 2024. ISSN 0957-4174, doi: URL actps://dck. 2024.1
Yang, Y., Mei, J., Du, S., Xiao, Y., Wu, H., Xu, X., and Liu, Y. Dqformer: Toward unified lidar panoptic segmentation with decoupled queries for large-scale outdoor scenes. IEEE Transactions on Geoscience and Remote Sensing, 63:1-15 2025. doi: N.L.100TGRR.2222555555
10.1016/j. (.000offeesaa222222333555 org/10. // eswa 2024. 123950.
Su, S., Xu, J., Wang, H., Miao, Z., Zhan, X., Hao, D., and Li, X. PUPS: Point Cloud Unified Panoptic Segmen- tation, February 2023. URL http: / arxiv.org/ abs/2302 06185. arXiv:2302.06185 [cs].
Ye, D., Zhou, Z.,, Chen, W. Xie, Y., Wang, Y., Wang, P., and Foroosh, H. Lidarmultinet: towards unified multi-task network for lidar perception. In Proceed- ings of the Thirty-Seventh AAAI Conference on Artifi- cial Intelligence and Thirty-Fifth Conference on Inno- vative Applications of Artificial Intelligence and Thir- teenth Symposium on Educational Advances in Artificial Intelligence, AAAI"23MAANN233EAANA 2023. ISBN 978-1-57735 1-------77-----0-0-000 doi: 10. 10.1609aa v37i3. 25429. URL /t/ wtt.:////......../..../.. /aaai. v3713.25429.
Wu, Y., Liu, J., Gong, M., Miao, Q.. Ma, W., and Xu, C. Joint semantic segmentation using representations of lidar point clouds and camera images. Information Fusion, 108: 102370, 2024.
AAAI Press,
Xiao, A., Huang, J., Guan, D., Cui, K., Lu, S., and Shao, L. Polarmix: general data augmentation technique for lidar point clouds. 2022.
Yin, J., Shen, J., Chen, R., Li, W., Yang, R., Frossard, P., and Wang, W. Is-fusion: Instance-scene collaborative fusion for multimodal 3d object detection. In CVPR, 2024.
Xiao, Z., Zhang, W., Wang, T., Loy, C. C., Lin, D., and Pang. J. Eesition-Guider Point Cloud Panoptic Segmentation Transformer. International Journal of Computer Vision, 133(())275-290, January 2025. ISSN 1573-1405. doi: 111007//s1123--2440000
Yin, T., Zhou, X., and Krähenbühl, P. Center-based 3d object detection and tracking. CVPR, 2021.
z. URL https:///di. /doi. 0rg/10.10072 000000//11133331 024- 020-222222666 021
Zhang, H., Liang, L., Zeng, P., Song, X., and Wang. SparseLIF: High-Performance Sparse LiDAR-Camera Fusion for 3D Object Detection, pp. 109-128. 09 2024. ISBN 978-3-031-72760-3, doi: 10.1007/ 978- 778-3-031-72761-07 7.
Xu, S., Wan, R., Ye, M. Zou, X., and Cao, T. Sparse cross-scale attention network for efficient lidar panoptic segmentation. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 36, pp. 2920-2928, 2022.
Xu, Y., Fazlali, H., Ren, Y., and Liu, B. Aop-net: All-in- one perception network for lidar-based joint 3d object detection and panoptic segmentation. In 2023 IEEE In- telligent Vehicles Symposium (IV), pp. 1-7, 2023a. doi: 10. 10.1109//5551122222333.. 10186777
Zhang, Z., Zhang, Z., Yu, Q. Q., Yi, R., Xie, Y., and Ma, L. Lidar-camera panoptic segmentation via geometry- consistent and semantic-awar alignment. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pp. 3662-3671, October 2023.
Xu, Y., Hu, C., Zhao, N., and Lee, G. H. Generalized few- shot point cloud segmentation via geometric words. In Proceedings of the IEFF/CVF International Conference on Computer Vision, pp. 21506-21515, 2023b.
Zhao, H., Zhang. J., Chen, Z., Zhao. S., and Tao, D. Unimix: Towards domain adaptive and generalizable lidar seman- tic segmentation in adverse weather. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 14781-1479) June 2024.
Yan, J., Liu, Y., Sun, J., Jia, F. Li, S., Wang. T., and Zhang, X. Cross modal transformer: Towards fast and robust 3d object detection. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 18268- 18278, 2023.
Zhao, N., Chua, T.-S., and Lee, G. H. Few-shot 3d point cloud semantic segmentation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 8873-8882, 2021.
12

---

Towards Harmonized Multi-modal 3D Panoptic Segmentation
Zhao, Y., Zhao, N., and Lee, G. H. Synthetic-to-real domain generalized semantic segmentation for 3d indoor point clouds. arXiv preprint arXiv:2212.0444 0446668, 2022.
Zhou, Z., Zhang. and Foroosh, H. Panoptic-Polaree PolarNet: Proposal-free LiDAR Point Cloud Panoptic Segmentation
In 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 13189-13198, Los Alamitos, CA, USA, June 2021. IEEE Computer Society. doi: 10111090CCVPP4444772221...111110 20129999222210019 URL bttpss:/doi... 10.1109 10..119//vVE www.11///VVV24447..2 201..
http://l....ccccopppprrrroeeeeea h /
Zhu, X., Zhou, H., Wang. T., Hong, F., Ma, Y., Li, W., Li, H., and Lin, D. Cylindrical and asymmetrical 3d convo- lution networks for lidar segmentation. arXiv preprint arXiv:2011. 10033, 2020.
Zhuang, Z., Li, R., Jia, K., Wang. Q. Li, Y., and Tan, M. Perception-aware multi-sensor fusion for 3d lidar semantic segmentation. In Proceedings of the /EEE/CVF International Conference on Computer Vision, pp. 16280- 16290, 2021.
13

---

Towards Harmonized Multi-modal 3D Panoptic Segmentation
Supplementary Materials
The supplementary materials are organized as follows: Sec. A extends the ablation study with token fusion stage; Sec. provides additional qualitative results to visualize the improvements brought by each module; Sec. C anaalyzes the efficiency of IAL; Sec. D investigates how image imputs enhance LiDAR under perturbations from lighting and weather conditions; Sec. E discusses the potential broader impact of IAL; and Sec. F outlines current limitations and future work.
A. Ablation Study of Token Fusion
To analyze the effectiveness of the crometric-guideer Token Fusion (GTF) module, we divide GTF into two components: Token Selection (Sel) and Token Positional Embedding (PE). The full version of GTF uses all physical points within cylindrical voxel (denoted as "sef") for token selection and embeds the scale between all extreme points ("scl") to indicate perception regions. Alternative designs degrade token selection virtual center ("ctr") and positional embedding the center extreme points ("ext") of the voxel. All results evaluated using the same experimental setting as the ablation studies in the main manuscript.
Table 8 reveals that using point set rather than the virtual center for every token to construct the image feature contributes up to 0.9% increase in PQ performance, and an up to 0.7% improvement RQ. This verifies that precise LiDAR image projection helps LiDAR voxels find corresponding image patches, and image features assist in recognition, For positional embedding. using scale-awarg embedding indicates the potential perception regions of both LiDAR and image tokens, achieving the highest performance. This advanced improvement diminishes when scaling is degraded to using extreme points solely the center of the voxel.
Table Ablation study of the GTF module. "Scl" and "PE" denote the designs for token selection and positional embedding, respectively. We evaluate different configurations for component ablation: "ctr" represents the voxel virtual center, "set" refers physical points within a voxel, and "L", "ctr", "ext", , and "scl" indicate not implementing PE, the use of center, extreme, or scale embeddings.
Sel
PE
PQ
PQ
RQ
SQ
mIoU
ctr
78.4
81.0
86.9
90.0
78.2
ctr
79.7
82.2
87.8
90.5
78.3
ext
79.7
82.3
87.7
90.6
78.5
ctr
scl
80.4
82.7
88.3
90.7
78.4
set
79.3
81.7
87.5
90.3
77.7
set
ctr
80.6
83.0
88.4
90.8
80.1
set
ext
80.0
82.4
88.0
90.6
77.9
set scl
81.1
83.5
89.0
90.9
80.2
B. Qualitative Results for Modular Performance
We present qualitative evaluations of the ablation studies for the GTF and PQG modules on the nuScenes validation set, as illustrated in the еттor maps in Fig. 6. With the assistance of the GTF module (comparing column 3 with column 2), our model demonstrates improved performance in distinguishing ambiguous objects, such as recognizing barriers and pedestrians from the background (highlighted in yellow boxes in rows 2 and 5), as well as differentiating buses and trucks from cars (rows 3-5). These classes often share similar geometric appearances, especially when point clouds are sparse. However, images provide rich texture features that help distinguish each class, even in limited regions. The GTF module enhances LiDAR voxel data by embedding it with more accurate image features, allowing for better receptive field estimation for each cylindrical voxel and facilitating a strong alignment of LiDAR and image features. The PQG module further enhances object perception, as shown by the comparison between column 4 and column 3, especially for small-scale and remote objects highlighted in black boxes. Even for small objects like pedestrians, bicycles, cars, and trailers, which have few points, the PQG module succeeds in accurate detection, demonstrating its superior performance by utilizing both geometric and texture priors, as well as the learnable capability of no-prior queries.
14

---

Towards Harmonized Multi-modal 3D Panoptic Segmentation
Ground Truth
XGTF XPQG
/GTF XPQG
/GTF PQG
barrier
bicycle
pedestrian
trailer
truck
drivable
sidewalk
terrain
man-made
vege
Figure 6. Qualitative comparison of the ablation study for GTF and PQG modules. To emphasize the differences, mark false positive and false negative predictions, which affect recognition quality, well as well-matched and mismatch points in true positive predictions, which impact segmentation quality, GT is colorized by semantic label. Best viewed in color.
C. Time and Memory Cost
We compare inference speed, model size, and performance between method and the main baseline, LCPS. We also report lightweight variant (denoted by *), which excludes the 2D mask pre-processing step (Grounding-DINO and SAM) to highlight the efficiency of our framework's ' major components. All latency measurements are conducted on a single NVIDIA A40 GPU with batch size 1. For a fair comparison, we measure LCPS latency using its official codebase on our hardware. As shown in Table 9, our method achieves over 2x faster inference and a +2.5% gain in PQ compared to LCPS. Even when including the mask generation time, our approach remains comparable in speed.
15

---

Towards Harmonized Multi-modal 3D Panoptic Segmentation
Table Comparison of models in of inference speed (FPS), model size (#Params), and Panoptic Quality (PQ). * denotes the result of core components of our model. All latency measurements are conducted on the same device.
Model
FPS
#Params (M)
PQ
LCPS
1.7
77.7
79.8
4.0"
81.8*
82.3
IAL
0.9
859.9
82.3
Table 10. Performance the full nuScenes validation and its nighttime/rain subsets. Best results are highlighted in bold.
Model
Full Val Set
Night Split
Rain Split
# of scan
6,019
602
1,088
LCPS
79.8
64.3
76.8
IAL (LiDAR branch)
77.0
63.2
73.1
IAL (full model)
82.3
70.5
81.2
D. Image Assists LiDAR Under Adverse Conditions
We evaluate the performance of our IAL on the nighttime and rain splits of the nuScenes val set. In the nighttime scenario, image quality is significantly degraded; in the rain scenario, both LiDAR and image encounter perturbations. As shown in Table 10, IAL (row 4) outperforms LCPS (row 2) not only on the full set but also under each adverse condition. We further ablate cross-modal interaction by comparing the full IAL to its LiDAR-only branch (row 3). Even under the degraded nighttime and rain splits, the full model gains +7.3% and +8.1 % on PQ, respectively, confirming the image's effective assistive role for LiDAR. This improvement augmentation (PicAug), which exposes the model to more diverse samples, including nighttime and rain scenarios, by mixing synchronized LiDAR and image data. This allows the model to generalize better to rare conditions like nighttime scenes. 2. The combination of three types of queries in our PQG module, where no-prior queries complement the texture- prior and geometric-prion queries, helping the model to effectively identify potential instances. Additionally, pre-trained Grounding-DIie DINO and SAM further stabilise 2D mask generation under distribution shifts thanks to their large-scale training.
be attributed two main factors: 1. Modality-synchronize
E. Potential Broader Impact of This Work
Based on the details in the paper, the broader impacts of this work can be highlighted as follows:
The proposed multi-modal 3D panoptic segmentation framework (IAL) advances the field of autonomous driving by improving object detection and segmentation through the integration of LiDAR and image data. This technology has direct implications for the safety, accuracy, and efficiency autonomous vehicles, particularly in complex, real-world environments. By addressing challenges such as the sparsity LiDAR data and the difficulty of recognizing small or distant objects, this work enhances perception systems, enabling more reliable decision-maken in autonomous driving.
Furthermore, the advancements in (GTF) represent significant contributions to the broader field of sensor fusion in robotics and autonomous systems. These innovations could be adapted for other applications requiring high-precision environmental understanding, such as robotics, agriculture, and urban planning.
augmentation (PieAug) and geometric-guided token fusion
As with all AI technologies, ethical considerations should be taken into account, particularly concerning privacy, security, and the potential for job displacement industries such as transportation and logistics. However, the broader societal impact is generally positive, particularly in terms of enhancing public safety and reducing the risks associated with human error in driving.
16

---

Towards Harmonized Multi-modal 3D Panoptic Segmentation
F. Limitations
While our work achieves strong results across key metrics, important to acknowledge certain limitations. Specifically, the sampling method in our Query Initialization (PQG) module relies on a relatively simple and generic approach. While our work demonstrates strong performance across key metrics, queries relies generic, large-scale pre-traine pre-trained models rather than methods specifically designed for this task benchmark. Although this approach ensures broad applicabilityy, it may not fully leverage task-specific characteristics that could further enhance performance. Nevertheless, through our carefully designed PQG module, we still achieve competitive results. In future work, we plan to explore more specialized sampling methods tailored to the task, which could further improve the quality texture- texture-prior queries and overall performance.
important note that the extraction of texture-prior
17


Reviews:
[{"review_id": "TwrqAOBh1K", "reviewer": "Reviewer_eBKA", "summary": "This paper proposes a new multi-modal framework for multi-modal 3D panoptic segmentation. First, the authors adopt a PieAug strategy to ensure consistency across LiDAR and image data during data augmentation. Then, they use a Geometric-Guided Token Fusion mechanism to fuse LiDAR and image tokens. Finally, several types of queries—geometric-prior queries, texture-prior queries, and no-prior queries—are used to obtain the final segmentation results. The authors conduct experiments on the nuScenes and Semantic KITTI datasets and demonstrate that the proposed method achieves better performance than state-of-the-art (SOTA) methods.\n\n## update after rebuttal\nThe rebuttal address most of my concerns. I increase my rating to weak accept.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your valuable feedback on the voxel representation, the model efficiency, and the design of query initiation. Regarding your questions, we provide the following detailed explanation.\n\n### Q1: Justification of the adoption of cylindrical representation in the LiDAR branch.\nWhile Cartesian voxelization ensures uniform scales, it inherently struggles with the density imbalance in LiDAR point clouds. Due to **radial sparsity**—where points are denser near the sensor and sparser at a distance—Cartesian voxels fail to accommodate this variation, potentially causing uneven feature distribution and information loss, particularly in distant regions.\n\nThe advantage of cylindrical over Cartesian voxel representations is well-documented in prior work, e.g., Cylinder3D (in CVPR’21, Sec. 3.2) and PolarNet (in CVPR’20, Sec 3.3), showing superior performance through adaptive partitioning of sparse distant points.\nAdditionally, adopting cylindrical representation aligns with high-performing baselines like P3Former (IJCV’25) and LCPS (ICCV’23), ensuring consistency and fair comparison.\n\n### C2: Computation cost comparison\nWe compare inference speed (FPS), model size (Params), and performance (PQ, the primary evaluation metric) between our method and LCPS, the main baseline. Additionally, we report a lightweight variant of our model, denoted by *, which excludes the 2D preprocessing step (Grounding-DINO and SAM) to highlight the efficiency of our framework’s major components. Since the 2D preprocessing can be replaced with any off-the-shelf model, this variant demonstrates the trade-off between speed and performance. All latency measurements are conducted on a single NVIDIA A40 GPU with batch size 1. For fair comparison, we measure LCPS latency using its official codebase on our hardware.\n\n|        | Mask Proposal Method       | FPS   | Params (M)   | PQ  |\n|--|--|--|--|--|\n| LCPS        | -                  | 1.7   | 77.7    | 79.8|\n| IAL*   | Grounding DINO+SAM         | 4.0 | 81.8   | 82.3|\n| IAL   | Grounding DINO+SAM         | 0.9   | 859.9   | 82.3|\n| IAL   | Mask R-CNN                   | 2.7   | 123.8  | 81.7|\n\nAs shown in the table above, with Grounding-DINO and SAM, IAL achieves slightly lower FPS than LCPS (0.9 vs. 1.7) but **significantly improves PQ** (82.3 vs. 79.8), making it ideal for accuracy-critical applications. Removing the 2D preprocessing step (row 2) makes our model 2.4× faster than LCPS. The choice of 2D preprocessing is flexible, balancing accuracy and efficiency. For instance, replacing Grounding-DINO and SAM with **Mask R-CNN** (ResNet50 backbone, IAL-MaskRCNN, row 4) yields 2.7 FPS (1.6× faster than LCPS) while maintaining a strong PQ of 81.7 (+1.9\\% over LCPS).\n\n### C3.1: Novelty of prior-based query generation\nWe respectfully disagree with the comment. Our prior-based query generation (PQG) is both novel and well-motivated, as acknowledged by reviewer #DcuJ. PQG fundamentally differs from TransFusion and FSF in both design and query generation mechanism.\n\nFrom a design perspective, unlike TransFusion and FSF, which solely rely on prior-based queries, our no-prior query solution specifically targets objects missed by geometric (3D) and texture (2D) priors, a critical gap for cross-modal fusion that has been **unaddressed in prior works**.\n\nIn terms of query generation, PQG introduces a **more advanced** approach that enhances prior utilization. While TransFusion fuses image features with BEV LiDAR features for heatmap prediction (which doesn't fully leverage the rich texture features and may harm each modality during fusion), our PQG module generates 2D and 3D prior-based queries independently. \nThis allows us to fully leverage the rich texture information without information loss and perform late fusion, better complementing the two modalities. As for FSF, they do not explicitly use queries; instead, they extract potential instances from both 2D and 3D branches and merge them via self-attention for final prediction. \nOur 2D query generation differs in that we use clustering to group points within a frustum and then apply Farthest Point Sampling (FPS) to select potential candidates, effectively mitigating outliers (e.g., background points). In contrast, FSF assumes all points within a mask frustum belong to the same semantic object, which can cause semantic ambiguity.\n\n### C3.2: Performance explanation when adding \"no-prior queries\"\nWe would like to clarify that we **keep the total number of queries the same** (as stated in L616-617) across all settings in Table 8. This ensures that the observed differences in performance are not influenced by the total number of queries, but instead reflect the contribution of different query combinations. Thus, the performance improvement when adding no-prior queries cannot be attributed to an increased number of queries, but rather to the complementary benefits of no-prior queries in enhancing the model’s ability to handle missed objects.", "claims_and_evidence": "Yes.", "methods_and_evaluation_criteria": "Yes. nuScenes and SemanticKITTI are popular used dataset in LiDAR panoptic segmentation tasks.", "theoretical_claims": "There are not theoretical claims in this paper.", "experimental_designs_or_analyses": "Yes. I check the main experiments and the ablation studies.", "relation_to_broader_scientific_literature": "1. Prior methods also adopt prior queries\\proposals from LiDAR or Image modality to enhance the performances in LiDAR detection tasks, such as TransFusion and FSF. This paper adopts prior queries in the task of panoptic segmentation.\n2. Prior methods adopt image painting, cross-attention, deformable attention to fuse multi-modal information. This paper adopt directly add fusion with scale aware positional embedding.", "essential_references_not_discussed": "No.", "other_strengths_and_weaknesses": "Strengths\n1. The paper is well-written, with clear presentation and illustrations.\n2. The authors provide detailed ablation studies to demonstrate the effectiveness of each component.\n3. The authors conduct experiments on nuScenes and SemanticKITTI and achieve state-of-the-art (SOTA) performance.\n\nWeaknesses\n\n1. The adoption of cylindrical representation in the LiDAR branch lacks justification. There is no explanation as to why cylindrical representation is preferred over Cartesian representation. The scale ambiguity problem only exists in cylindrical representation. If Cartesian representation were used, the scale-aware position embedding might not be necessary, as the scale of each voxel would be consistent.\n2. The overall pipeline is computationally expensive. The framework incorporates additional image models, such as Grounding-DINO and SAN, to extract 2D masks and use them as texture-prior queries. Comparing the proposed method with other methods without considering FLOPs or latency is unfair.\n3. The idea of adopting proposals from heatmaps (e.g., TransFusion) or image modalities (e.g., Fully Sparse Fusion ) as priors is not novel. As shown in the supplementary materials, combining texture priors with geometric priors does not achieve better performance. The authors hypothesize that this is due to an excessive number of prior-based queries. However, adding no-prior queries increases the total number of queries but results in better performance. The authors should provide a more convincing explanation for this observation.", "other_comments_or_suggestions": "No.", "questions_for_authors": "Please refer to the Weaknesses in Other Strengths And Weaknesses.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "GovQkOBdqw", "reviewer": "Reviewer_CFKM", "summary": "This work proposed a novel framework for multi-model 3D panoptic segmentation. By leveraging the proposed modality-synchronized data augmentation (PieAug) with geometric-guided token fusion (GTF) and prior-based query generation (PQG), this work achieves new state-of-the-art (SOTA) performance on two challenging benchmarks, i.e. nuScenes and semantic-KITTI.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank the reviewer for their constructive feedback and positive comments on our contributions. \n\n### C1: Fig. 3 is too far from the Sec. 3.1\nThank you for the suggestion. We will update the placement of Fig. 3 in the final version to ensure that PieAug is introduced and illustrated in a more cohesive manner. \n\n### C2: The impact statement on page 9 should be in the supplementary material\nThank you for the suggestion. However, we followed the official guidelines, which recommend placing the impact statement in a separate section at the end of the paper, co-located with the Acknowledgements, before the References.\n\n### C3: Add qualitative comparison among ablation studies\nGreat suggestion! Unfortunately, due to the constraints of the rebuttal policy, we are unable to include images or provide links to them on the OpenReview platform. However, we will add these qualitative comparisons in the final version of the paper.\n\n### C4: Moving Tab. 8 to the Main Paper\nThank you for your suggestion. We agree that Table 8 is crucial for showing how different prior queries impact the results. We will move it to the main paper in the final version.\n\n### C5: Will the mask prior strongly impact texture queries' effectiveness? E.g., replacing Grounding SAM with Mask DINO or Mask R-CNN?\nGood catch! We have replaced the mask generation module (Grounding-DINO and SAM) with alternatives, including Mask R-CNN (as you suggested) and its follow-up, the HTC model [1], and report the results in the table below. As shown, the performance of our model remains robust to the choice of mask priors from different mask proposal methods, with only a minimal performance drop when using Mask R-CNN or HTC. This indicates that our texture-prior query generation design is flexible, accommodating different 2D mask proposal models, and can be integrated with advanced off-the-shelf methods (in the future) for further performance improvements.\n\n| **2D Mask Proposal**           | **PQ** |\n|---------------------------------|--------|\n| Mask R-CNN (ResNet50)           | 81.7   |\n| HTC (ResNeXt101)                | 81.9   |\n| Grounding-DINO + SAM            | 82.3   |\n\n\n[1] Chen Kai, et.al. Hybrid task cascade for instance segmentation. CVPR'19.", "claims_and_evidence": "The authors claim the proposed PieAug can align multi-camera augmentation with corresponding LiDAR cylindrical voxels. This is proved by eq. 1 - 5 (theoretically), Fig. 3 (qualitatively), and Tab. 5 & 6 (quantitatively).\n\nAuthors also claim that GTF can better align LiDAR and multi-camera image features by eliminating projection errors and obtaining sufficient fields to fully utilize dense image features. This is supported by eq. 6 - 8 plus Fig. 2 (theoretically) and Tab. 5 (quantitatively).\n\nThe final claim is that PQG can provide sufficient proposals for panoptic segmentation using all kinds of priors. This is proved by Tab. 1 (theoretically) and Tab. 5 (quantitatively).", "methods_and_evaluation_criteria": "The methods are novel and interesting. Extensive experimental results support the claims and demonstrate its value for multi-modal 3D panoptic segmentation tasks. The evaluation criteria are standard.", "theoretical_claims": "As stated in the claims and evidence, the theoretical claims are correct and supported by quantitative results.", "experimental_designs_or_analyses": "The experimental designs are correct and solid. The analyses are sufficient and insightful both in the main paper and supplementary material.", "relation_to_broader_scientific_literature": "I believe the key contributions of the paper are related to the broader scientific literature, especially the proposed GTF. This interesting multi-modal features fusion will be quite valuable for other 3D perception tasks that are using multiple sensors.", "essential_references_not_discussed": "N/A", "other_strengths_and_weaknesses": "Strength:\n- The proposed PieAug is interesting and seems quite effective. It solves the misalignment between modalities which is quite important and overlooked in the previous works.\n- The proposed GTF is novel and meaningful. It provides a better way to align multi-model features and can be used in the popular transformer architectures.\n- PQG is also novel and interesting, which derives from the observation of using ground truth (GT) center position. This finding is quite valuable to 3D panoptic segmentation.\n\nWeakness\n- Fig. 3 is too far from the Sec. 3.1, where the PieAug is introduced. It will be much better for the reader to get the full pictures of PieAug without page jumps and affect the reading.\n- The impact statement seems to be on Page 9. I believe it should be in the supplementary material, or the overall contents management needs to be more careful.", "other_comments_or_suggestions": "- It will be nice to have a qualitative comparison among the ablation studies so that readers can understand how much the PieAug, GTF, and PQG affect the prediction.\n- Tab. 8 is important for the readers to know how different prior queries affect the final results. It might be better to move it in the main paper.", "questions_for_authors": "- Will the effectiveness of texture queries change a lot when changing the mask prior? For example, replacing Grounding SAM with Mask DINO or Mask R-CNN?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "BnRD5abIlJ", "reviewer": "Reviewer_qZ7o", "summary": "Aiming at 3D panoptic segmentation, the paper proposes using a transformer decoder to directly predict class labels and mask outputs. The authors further introduce a Geometric-guided Token Fusion (GTF) module and a Prior-based Query Generation (PQG) module to obtain effective queries and fuse tokens as input. In addition, a multi-modal data augmentation strategy is designed to augment both 3D and 2D inputs. The experiments show the effectiveness of the proposed method.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank the reviewer for the positive feedback and constructive suggestions. \n\n### C1: Suggestion for adding additional references.\nThank you for pointing out these two papers. We will add them in the final version and provide the following discussions:\n\n1. **UniSeg (ICCV’23)** proposes a unified multi-modal fusion strategy for LiDAR segmentation, focusing on fusing voxel-view and range-view LiDAR features with image features. However, UniSeg primarily targets semantic segmentation and does not explore how to utilize multimodal information for instance segmentation, which focuses on thing classes. In contrast, our design of *prior-based query generation* is specifically tailored for 3D instance segmentation. Furthermore, while UniSeg employs Cartesian voxelization, our method uses *cylindrical voxelization*, which has been proven more effective for panoptic segmentation in prior works like PolarNet (CVPR'20) and Cylinder3D (CVPR'21). We also incorporate *Geometric-guided Token Fusion* (including scale-aware positional encoding) to better accommodate this cylindrical representation.\n\n2. **MM-FSS (ICLR’25)** focuses on few-shot 3D semantic segmentation in indoor scenarios by leveraging both explicit textual modalities and implicit 2D modalities for cross-dataset adaptation. This differs significantly from our work, which focuses on fully-supervised multi-modal 3D panoptic segmentation in outdoor scenarios. Additionally, the fused modalities in MM-FSS (text, image, and point cloud) differ from our approach, which uses multi-view images and LiDAR point clouds.\n\n\n###  C2: Could the authors provide the efficiency analysis such as comparing the computational cost and the parameter count with previous methods?\n\nThank you for your suggestion! We provide a comparison of inference speed (FPS), parameter count, and performance (PQ, the primary evaluation metric) between our method and LCPS, the main baseline, as shown in the table below.\n\n| Model          | Mask Proposal Method    | FPS  | Params   | PQ   |\n|--              |--                       |--    |--        |--   | \n| LCPS           | -                       | 1.7  | 77.7M    | 79.8 |\n| IAL (ours)     | Grounding DINO + SAM    | 0.9  | 859.9M   | 82.3 |\n| IAL (ours)*    | Grounding DINO + SAM    | 4.0* | 81.8M*   | 82.3 |\n| IAL (ours)     | Mask R-CNN              | 2.7  | 123.8M   | 81.7 |\n\n\nTo better understand the computational cost and model size of our framework’s major components, we report a lightweight variant of our model (denoted by \\* in the 3rd row), which excludes the 2D preprocessing step (Grounding-DINO and SAM). This analysis reveals that the main computational cost and parameter count stem from the sophisticated 2D preprocessing step. However, our core components - including GTF, geometric-prior query generation, 2D and 3D encoder, and transformer decoder - are highly efficient, achieving 4.0 FPS with only 81.8M of parameters.\n\nTo further evaluate efficiency, we replace Grounding-DINO and SAM with Mask R-CNN (ResNet50 backbone), a lightweight 2D mask generation alternative, as reported in the 4th row. This variant achieves a well-balanced trade-off between speed and performance, with the full pipeline running at 2.7 FPS (1.6× faster than LCPS) while maintaining 81.7 PQ (+1.9% over LCPS). These results demonstrate that while external proposal models contribute to computational cost, our core method remains highly efficient and adaptable to different 2D preprocessing choices.", "claims_and_evidence": "The claims are reasonable and supported. PieAug augments both modalities while maintaining synchronization across them. The GTF module addresses mismatches between modalities and scale issues by mapping the eight corner points during fusion, and the PQG module leverages priors from both modalities to initialize queries for better localization and performance.", "methods_and_evaluation_criteria": "The proposed designs are logical, and the chosen evaluation benchmarks are suitable for the task.", "theoretical_claims": "There are no theoretical claims.", "experimental_designs_or_analyses": "The experiments are conducted on two standard outdoor datasets, demonstrating the effectiveness of the proposed method. However, the ablation study is somewhat limited. For example, GTF requires projecting all the corner points for computing positional encoding and token fusion, and PQG uses three sets of queries, with two query groups needing external proposal models for initialization. It would be beneficial to know the computational efficiency of the method and the parameter count compared to previous methods.", "relation_to_broader_scientific_literature": "The proposed designs complement the broader literature and offer new insights.", "essential_references_not_discussed": "The paper should also reference existing fusion modules proposed in multi-modal 3D segmentation literature, such as \n+ ICCV 2023, UniSeg: A Unified Multi-modal LiDAR Segmentation Network and the OpenPCSeg codebase\n+ ICLR 2025, Multimodality Helps Few-shot 3D Point Cloud Semantic Segmentation", "other_strengths_and_weaknesses": "The paper is well-written and easy to follow. The motivations for the design choices are clear and reasonable.", "other_comments_or_suggestions": "Please see the Questions.", "questions_for_authors": "GTF requires projecting all the corner points for computing positional encoding and token fusion, and PQG uses three sets of queries, with two query groups needing external proposal models for initialization. Could the authors provide the efficiency analysis such as comparing the computational cost and the parameter count with previous methods?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "vf5Ul9ceuB", "reviewer": "Reviewer_DcuJ", "summary": "This paper presents Image-Assists-LiDAR (IAL), a new panoptic 3D multi-modal segmentation method combining camera images with LiDAR point clouds. The key contributions are PieAug, multi-modal data augmentation with aligned data, Geometric-guided Token Fusion (GTF) for effective cross-modal fusion of features, and Prior-based Query Generation (PQG) with geometric (LiDAR) priors in a transformer decoder, with texture (image) priors. IAL achieves state-of-the-art results for nuScenes (PQ=82.0%) and SemanticKITTI (PQ=63.1%)", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely appreciate the reviewer’s thoughtful and valuable feedback. \n\n### C1: Contribution of performance improvement\n\nAs in Table 8 of our supplementary material, using only texture-prior and no-prior queries (row 3) performs worse than the combination of all three query types (last row). This indicates that the performance improvements primarily stem from our novel fusion strategy—the combination of 3 types of queries in the PQG module—rather than solely relying on pre-trained strong image models, which only influence texture-prior query generation.\n\n\n### C2: Discussions with other top-performing LiDAR-only methods\n\nThank you for suggesting these two references. We will include them in the final version and provide the following discussion:\n\n**LidarMultiNet**: IAL outperforms LidarMultiNet on **all key panoptic segmentation metrics** despite using only a single frame and standard annotations, whereas LidarMultiNet leverages **extra** temporal frames, detection annotations, and multi-task learning. While LidarMultiNet slightly surpasses IAL in mIoU, this metric primarily reflects 3D semantic segmentation and is not the main evaluation metric for 3D panoptic segmentation. Moreover, some of LidarMultiNet’s effective components, such as GCP and two-stage, **do not conflict with** our multimodal method and could potentially be integrated into IAL to further enhance multi-modal 3D panoptic segmentation performance.\n\n**PUPS**: PUPS exibits inconsistent performance across datasets. Specifically, it significantly outperforms the baseline method Panoptic-PHNet on SemanticKITTI (Table 1 & 2 of PUPS) but achieves only comparable performance to Panoptic-PHNet on nuScenes validation set (Table 3 of PUPS). In contrast, IAL consistently achieves superior result over Panoptic-PHNet on both datasets. Since PUPS has not released its code for reproducibility, we hypothesize that its inconsistent performance trend may stem from overfitting to the simpler SemanticKITTI. Compared to SemanticKITTI, nuScenes presents greater challenges due to its larger scale, more diverse domains.\nIAL attains the highest performance on nuScenes, further demonstrating its robustness and practical effectiveness.\n\n\n\n### C3: Inference speed and model size comparison\n\nGood catch! We provide a comparison between across all methods in the table below.\n\n| Model               | Mask Proposal Method    | FPS  | Params (M)    | PQ   |\n|--|--|--|--|--|\n| LCPS                | -                       | 1.7  | 77.7     | 79.8 |\n| IAL         | Grounding DINO + SAM    | 0.9  | 859.9    | 82.3 |\n| IAL*         | Grounding DINO + SAM    | 4.0* | 81.8*    | 82.3 |\n| IAL          | Mask R-CNN              | 2.7  | 123.8M   | 81.7 |\n\nTo isolate the impact of external image models (SAM, GroundingDINO), we exclude 2D preprocessing step and report the efficiency of the remaining pipeline in 3rd row (denoted by *). Notably, the inference speed of our core pipeline is faster than LCPS, showing the efficiency of our design.\n\nWe replace the *heavy* 2D preprocessing step (GroundingDINO and SAM) with a *lightweight alternative*, Mask R-CNN (ResNet50 backbone), as reported in 4th row. This variant strikes a balance between speed and performance, running at 2.7 FPS (1.6× faster than LCPS) while maintaining 81.7 PQ. These results indicate that while external 2D mask generation models introduce additional computational costs, our **query generation design remains flexible**, accommodating different 2D mask proposal models to **balance high-quality performance and fast inference speed**. More importantly, our **core framework remains highly efficient and robust** to different 2D preprocessing choices. Also, further optimizations such as model pruning or quantization could further accelerate inference.\n \n\n### C4: Robustness when a modality is degraded\n\nGreat suggestion! We evaluate the performance of IAL by comparing the LiDAR-only branch with the full model on the **nighttime** split (602 scans) of the nuScenes val set (6,019 scans), where image quality is significantly degraded. The results, (63.2 vs 70.5 PQ), show that despite the low image quality, our IAL (full model) still achieves a **+7.3\\% PQ gain** over the LiDAR-only model. This improvement can be attributed to two factors: 1. **Modality-synchronized augmentation (PieAug)**, which exposes the model to more diverse samples, including nighttime scenarios, by mixing synchronized LiDAR and image data. It allows the model to generalize better to rare conditions like night scenes. 2. The combination of **3 types of queries** in PQG, where no-prior queries complement the texture-prior and geometric-prior queries, helping the model to effectively identify potential instances. Additionally, pre-trained Grounding-DINO and SAM models contribute to enhancing the robustness of 2D mask generation in night conditions, as they have been trained on vast amounts of data, improving generalization to such challenging scenarios.", "claims_and_evidence": "They are generally backed by extensive experiments and ablations. The significant improvements in PQ in typical benchmarks supports the performance of each of the proposed modules.", "methods_and_evaluation_criteria": "The chosen methods and datasets (nuScenes and SemanticKITTI) are apt and standard for the evaluation of panoptic segmentations in the 3D domain. The proposed modules (PieAug, GTF, PQG) are well-motivated, addressing apparent limitations in the prior work.", "theoretical_claims": "There were no proofs given or required.", "experimental_designs_or_analyses": "Experimental design is solid and rigorous. Ablation experiments nicely demonstrate the contribution of each component. Authors should nevertheless indicate whether performance improvements are primarily the result of new fusion strategies or the application of pre-trained strong image models (SAM, GroundingDINO).", "relation_to_broader_scientific_literature": "The paper correctly positions itself in the context of the most up-to-date multi-modal panoptic segmentation techniques (Panoptic-FusionNet, LCPS). It correctly identifies the limitations of the state-of-the-art approaches (post-processing-intensive, misaligned augmentation) and correctly positions its contributions (aligned augmentation, fusion using transformers) in the context of the state-of-the-art.", "essential_references_not_discussed": "The work does not mention comparisons or discussions with other top-performing LiDAR-only methods including LidarMultiNet [AAAI'23](PQ~81.4%, mIoU~82.2% on nuScenes test set) and PUPS [AAAI'23] (PQ~64.4% on SemanticKITTI validation set). A mention of those will give a better context to the performance gains and clearly highlight the strength of their multi-modal approach.", "other_strengths_and_weaknesses": "Strength\nTransparent methodological developments (PieAug, GTF, PQG)\nStrong experimental validation with comprehensive ablations.\nClear text with good figures presented.\nOpen Source Code Release Commitment\n\nWeakness\nExclusion of high-performance LiDAR-only methods (for example, PUPS, LidarMultiNet) from comparison\nThere is no mention of the efficiency of inferences or real-world runtimes of heavy external image models (SAM, GroundingDINO).", "other_comments_or_suggestions": "Minor improvements could include explicitly comparing inference speed and model size against baseline models.", "questions_for_authors": "1. How quickly is IAL’s inference compared to its complexity and reliance on external models like SAM? Could it be run in real-time?\n\n2. How significant is the contribution of the use of strong pre-trained image models compared to the new fusion strategy? Would less complex models be able to produce comparable results?\n\n3. How robust is IAL when a modality is highly degraded (i.e., low image quality at night)? Does it degrade gracefully to the performance of LiDAR alone?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper presents a framework for multi-modal 3D panoptic segmentation by introducing PieAug, a modality-synchronized data augmentation strategy designed to ensure alignment between LiDAR and image inputs. Additionally, it proposes the GTF and PQG modules, which effectively combine image and LiDAR features as tokens and queries for the transformer decoder.

Reviewers raised concerns regarding the novelty of the work and sought clarification on aspects such as cylindrical representation, the prior-based query generation module, and computational costs. 
The final ratings are mixed, weak accept, 2 accept, and weak reject. 

After a thorough review, including consideration of both the authors' responses and the reviewers' comments, AC has decided to accept the paper. The authors must adhere to their commitments and rigorously revise the paper per their promises.
Decision: Accept (poster)