4D-Former: Multimodal 4D Panoptic Segmentation
Authors: Ali Athar
Enxu Li
Sergio Casas
Raquel Urtasun
Venue: CoRL 2023 Poster (2023)
Keywords: Panoptic Segmentation
Sensor Fusion
Temporal Reasoning
Autonomous Driving

Abstract: 4D panoptic segmentation is a challenging but practically useful task that requires every point in a LiDAR point-cloud sequence to be assigned a semantic class label, and individual objects to be segmented and tracked over time. Existing approaches utilize only LiDAR inputs which convey limited information in regions with point sparsity. This problem can, however, be mitigated by utilizing RGB camera images which offer appearance-based information that can reinforce the geometry-based LiDAR features. Motivated by this, we propose 4D-Former: a novel method for 4D panoptic segmentation which leverages both LiDAR and image modalities, and predicts semantic masks as well as temporally consistent object masks for the input point-cloud sequence. We encode semantic classes and objects using a set of concise queries which absorb feature information from both data modalities. Additionally, we propose a learned mechanism to associate object tracks over time which reasons over both appearance and spatial location. We apply 4D-Former to the nuScenes and SemanticKITTI datasets where it achieves state-of-the-art results.

4D-Former: Multimodal 4D Panoptic Segmentation
Ali ,t Enxu Li Sergio Casas Raquel Urtasum 1.2
Waabi University of Toronto 3WWTH Aachen University
faathar, tli, sergio, urtasun (raauii ai
Abstract: 4D panoptic segmentation is challenging but practically useful task that requires every point in LiDAR point-cloud sequence to be assigned a se- mantic class label, and individual objects to be segmented and tracked over time. Existing approaches utilize only LiDAR inputs which convey limited information in regions with point sparsity. This problem can, however, be mitigated by uti- lizing RGB camera images which offer appearance- based information that reinforce the geometry-based LiDAR features. Motivated by this, we propose 4D-Former: a novel method for 4D panoptic segmentation which leverages both LiDAR and image modalities, and predicts semantic masks as well as temporally consistent object masks for the input point-cloud sequence. We encode semantic classes and objects using set of concise queries which absorb feature informa- tion from both data modalities. Additionally, we propose learned mechanism associate object tracks over time which reasons over both appearance and spa- tial location. We apply 4D-Former to the nuScenes and SemanticKITTI datasets where it achieves state-of-the-arr results. For more information, visit the project website: https://waabs: ai/4dformer.
Keywords: Panoptic Segmentation, Sensor Fusion, Temporal Reasoning, Au- tonomous Driving
Introduction
Perception systems employed in self-driving vehicles (SDVs) aim to understand the scene both spa- tially and temporally. Recently, 4D panoptic segmentation has emerged as an important task which involves assigning a semantic label to each observation, as well as an instance ID representing each unique object consistently over time, thus combining semantic segmentation, instance segmentation and object tracking into single, comprehensive task. Potential applications of this task include building semantic maps, auto-labelling object trajectories, and onboard perception. The task is, however, challenging due to the sparsity of the point-cloud observations, and the computational complexity of 4D spatio-tempora reasoning.
Traditionally, researchers have tackled the constituent tasks in isolation, iee, segmenting classes [1, 2. 3, 4], identifying individual objects [5, 6], and tracking them over time [7, 8). However, combining multiple networks into single perception system makes error-prone, potentially slow, and cumbersome train. Recently, end-to-end approaches [9, 10, 11] for 4D panoptic segmentation have emerged, but they utilize only LiDAR data which provides accurate 3D geometry, but is sparse at range and lacks visual appearance information that might be important to disambiguate certain classes (e.g., a pedestrian might look like pole at range). Nonetheless, combining LiDAR and camera data effectively and efficiently is non-trivial as the observations are very different in nature.
In this paper, we propose 4D-Former, a novel approach for 4D panoptic segmentation that effec- tively fuses information from LiDAR and camera data to output high quality semantic segmentation labels well as temporally consistent object masks for the input point cloud sequence. To the best of our knowledge, this is the first work that explores menti-sensor fusion for 4D panoptic point cloud segmentation. Towards this goal, we propose novel transformer-based architecture that
Indicates equal contribution. "ork done while an intern at Waabi.
7th Conference Robot Learning (CoRL. 2023), Atlanta, USA.

---

fuses features from both modalities by efficiently encoding object instances and semantic classes as concise queries. Moreover, we propose a learned tracking framework that maintains a history of previously observed object tracks, allowing us to overcome occlusions without hand-crafted heuris- ties. This gives us an elegant way to reason in space and time about all the tasks that constitute 4D panoptic segmentation. We demonstrate the effectiveness of 4D-Former on both nuScenes [12] and SemanticKIITI [13] benchmarks and show that significantly outperform the state-of- afteeo..
2 Related Work
This task combines semantic and instance segmentation, but does not require temporally consistent object tracks. Current approaches often utilize a multi-branch architecture to independently predict semantic and instance labels. A backbone network is used to extract features from the LiDAR point cloud with various representations e.g. points [14], vox- els [1, 3]. 2D range views [15, 16], or birds-eye views [17]. Subsequently, the network branches into two paths to generate semantic and instance segmentation predictions. Typically, instance pre- dictions are obtained through deterministic [18, 19, 20] or learnable clustering [6, 21]. proposal generation [22], or graph-based methods [23, 24). These methods are not optimized end-to-end. Several recent work [25, 26, 27] extends the image-level approach from Cheng et al. [28] to per- form panoptic segmentation the LiDAR domain an end-to-end fashion. We adopt similar approach to predict semantic and instance masks from learned queries, however, queries attend to multi-modal features whereas the former utilizes only LiDAR inputs.
3D Panoptic Segmentation:
LiDAR Tracking: This task involves predicting temporally consistent bounding-boxed for the objects in the input LiDAR sequence. We classify existing approaches into two main groups: and end-to-end methods. The tracking- -bedeettn paradigm [7, 8, 29] has been widely researched, and generally consists of a detection framework followed by a tracking mechanism. Since LiDAR point clouds typically lack appearance information but offer more spatial and geometric cues, existing approaches usually rely on motion cues for tracking (e.g. Kalman Fil- ters [30], Hungarian matching [31] or Greedy Algorithm [8] for association). Recently, end-to-end frameworks [32] have also emerged where a single network performs per-frame detection and tem- poral association. In contrast to these, 4D-Former utilizes both LiDAR and image modalities, and performs point-level instance tracking and semantic segmentation with a single unified framework.
4D Panoptic Segmentation: This is the task we tackle in our work, and it involves extending 3D panoptic segmentation to include temporally consistent instance segmentation throughout the input sequence. Most existing methods [9, 11, 33] employ a sliding-windor approach which tracks instances within short clip of upto frames. 4D-PL. [9] models object tracklets Gaussian dis- tributions and segments them by clustering per-point spatio-temporal embeddings over the 4D input volume. 4D-StOP [11] proposes a center-based voting technique to generate track proposals which are then aggregated using learned geometric features. These methods associate instances across clips using mask loU in overlapping frames. CA-Net [10), on the other hand, learns contrastive em- beddings for objects to associate per-frame predictions over time. Recently, concurrent work from Zhu et al. [34] develops -auuivn-qquivariant networks which provide more robust feature learning for 4D panoptic segmentation. Different to these, 4D-Former utilizes multimodal inputs, and adopts transformer- architecture which models semantic classes and objects concise queries.
Multimodal approaches have recently become popular for object detection and semantic segmentation. Existing methods can be grouped into two categories: (1) point-level fusion methods, which typically involve appending camera features to each LiDAR point [35, 36, 37] or fusing the two modalities at the feature level [38, 39, 40]. (2) Proposal- fusion, where object detection approaches [41, 42] employ transformer based architectures which represent object as queries and then fuse them with camera features. Similarly, Li al. [43] perform seman- tic segmentation by modeling semantic classes as queries which attend scene features from both modalities. 4D-Former, on the other hand, tackles 4D panoptic segmentation whereas the aforemen- tioned methods perform single- frame semantic segmentation or object detection.
LiDAR and Camera Fusion:

---

Imagee
U4, 4
Trooks from iteration
Imange MOOOSAA
Trackied Meaion
Trackist Module
RosNet
FPM
Fuston
Pale-lewl
imsaee
UDAR
Samantic
Track Masks
[Vi Vs,
Encodar
Decader
2
Header
Panoptoo
4D Panoptic Predictions for ittrrationn
Beration
Figure I: 4D-Former inference iteration i. Note that tracking history from i is used in the Tracklet Association Module.
Multimodal 4D Panoptic Segmentation
In this paper we propose 4D-Former to tackle 4D panoptic segmentation. The task consists of la- belling each 4D LiDAR point with a semantic class and track that specifies a consistent instance time. Camera images provide rich additional context to help make accurate predictions, particularly in regions where LiDAR is sparse. To this end, propose novel transformer-bas -based architecture that effectively combines sparse geometric features from LiDAR with dense contextual features from cameras. In particular, it models object instances and semantic classes using concise, learnable queries, followed by iterative refinement by self-attention and cross-attention to LiDAR and camera image features. Using these queries, our method able to attend only to regions of the sensor data that are relevant, making the multimodal fusion of multiple cameras and LiDAR tractable. In order handle sequences of arbitrary length as well continuous streams of data (e.g., in the onboard setting), 4D-Former operates in sliding window fashion, illustrated in Fig. 1. At each iteration, 4D-Former takes input the current LiDAR scan at time 4, the past scan at - 1, and the camera images at time t. It then generates semantic and tracklet predictions for these two LiDAR scans. To make the tracklet predictions consistent over the entire input sequence, we propose a novel Tracklet Association Module (TAM) which maintains a history previously observed object tracks, and associates them based on a learning- matching objective.
3.1 Multimodal Encoder
Our input encoder extracts image features from the camera images, and point-level and voxel-level features by fusing information from the LiDAR point clouds and camera features. These features are then utilized in our waasformer-basee panoptic decoder presented in Sec. 3.2.
Image feature extraction: Assume the driving scene is captured by set of images of size H X W captured from multiple cameras mounted on the ego-vehicle. We employ a ResNet-50 [44] backbone, followed by Feature Pyramid Network (FPN) [45], to produce a set of multi-scale, D dimimensional feature maps (Tx 4, 8 for each of the images, where T, E TRH/sxW/axD
Point/voxel feature extraction: The
network architecture is inspired by [46]
Z
and consists of point-branch and
voxel-branch. The point-branch learns
LIDA
MLP
MUP
point-level embeddings, thus preserving fine details, whereas the voxel-branch performs contextual reasoning using 3D sparse convolutional blocks [47] and provides multi-scale feature maps. Each of the N points in the input LiDAR point-cloud is represented as 8-D feature which include the zyz coordinates, relative timestamp, intensity, and 3D relative offsets to the nearest voxel center. An MLP is applied to obtain initial point
Figure Overview point and voxel feature extraction. p2v: point-to-voxel. v2p: voxel-to-point.

---

embeddings which are then averaged over a voxel to obtain voxel features. These voxel features are processed through four residual blocks with 3D sparse convolutions, each of which downsamples the feature map by 2x. Four additional residual blocks are then used to upsample the sparse feature maps back to the original resolution, thus yielding a set of D-dimensional voxel features at various = 2, 4, ). where N, denotes the number of non-empty voxels at the i-th stride. At various stages in this network, point-level features are updated with image features via point-level fusion (as explained in the next paragraph). Moreover, we exploit point- to-voxel and voxel-to-point operations to fuse information between the point and voxel branches at different scales, as illustrated in Fig. 2. We denote the final point-level features as Z E NND.
strides V {Vi E N
Point-level fusion: We enrich the geometry-based LiDAR features with appearance- image features by performing a fine-grained, point-level feature fusion. This is done by taking the point features Zudar E IR NxD at intermediate stages inside the LiDAR backbone, and projecting their corresponding (2r, y, 2) coordinates to the highest resolution image feature map 44. Note that this can be done since the image and LiDAR sensors are calibrated, which is typically the case in modern self-driving vehicles. This yields set of image features Zimg RMxD, where M I N since generally not all LiDAR points have valid image projections. We use I RMMx to denote the subset of features in Zudar which have valid image projections, and "'daar E (N) for the rest. We then perform point-level fusion between image and LiDAR features as follows:
(1) where both MLPs contain 3 layers, and [ denotes channel-wise concatenation. Intuitively, MLP fusion performs pairwise fusion for corresponding image and LiDAR features. On the other hand, MLP pscudo updates the non-projectable LiDAR point features to resemble fused embeddings.
Z MLPfusion (1 Zimgl) Tddar
lidar < MLP pscudo (Z
3.2 Transformer-based Panoptic Decoder
We propose a novel decoder which predicts per-point semantic and object track masks with a unified architecture. This stands in contrast with existing methods [9, 11, 48, 6] which generally have sepa- rate heads for each output. Our architecture is inspired by image-level object detection seriettion/semmeaatiionnn methods [49, 28], but the key difference is that our decoder performs multimodal fusion.
We initialize set of queries E Tx randomly at the start of training where the number of queries (I) is as- sumed to be an upper-bound on the number of objects in a given scene. The idea to use these queries to segment a
LIDAR to Image Projection
Is
V,
F,
varying number of objects as well as the non-instantiable 'stuff" classes in the scene. The queries are input to se- ries of 'fusion blocks'. Each block composed of multi- ple layers where the queries Q are updated by: (1) cross-
14
Figure 3: LiDAR image projection.
attending the voxel features V; E N.C at given stride, (2) cross-attending to the set of image
features F, E IR MC which are obtained by projecting the (x, 2) coordinates for the voxel fea- tures V/2 into each of the multi-scale image feature maps' 1 {I1, Is } (see Fig. 3 for an illustration), and (3) self-attending to each other twice intermittently, and also passing through 2x Feedforward Networks (FFN). The architecture of these fusion blocks is illustrated in Fig. 4.
These queries distill information about the objects and semantic classes present in the scene. To this end, self- attention enables the queries to exchange information be- tween one another, and cross-attention allows them to learn global context by attending to the features from both modalities across the entire scene. This mitigates the need for dense feature interaction between the two modal- ities which, if done naively, would be computationally in- tractable since N; and M, are on the order of 10a. Our
Salf-atttor cattonatiiiioon
Self-attionioon aoti-aatttnioon
Q+1
atllentiinnn
FFM
alttertion
FFN
VG
F,
Figure 4: Fusion block architecture.
M. < 2N, since each voxel feature is projected into 2 image feature maps (14, 1s), but not all LIDAR voxel features have valid image projections

---

fusion block avoids this by leveraging set of concise queries which attend to the scene features from both modalities sequential fashion where the computational footprint of each operation is manageable since T < N₂ and < M,
Our transformer-based panoptic decoder is composed of four such fusion blocks, each involv- ing cross-attention to voxel features different strides, and their corresponding image features. We proceed in coarse-to-fine manner where the inputs the fusion blocks are ordered Fs), Fa), (V2, F2), (V1, Fi). Note that this query-level fusion compliments the fine- grained, point-level fusion in the LiDAR backbone explained in Sec. 3.1. The updated queries output by the decoder, denoted by Q' E RT'DD are used to obtain logits for the object tracklet and semantic masks, where each logit represents the log probability of a Bernoulli distribution capturing whether the query represents a specific instance or class. Per-point object tracklet masks Mp are calculated the dot- dot-product of the updated queries Q' with the point-level features Z E RNx:;
Mp + Z Q" E R NXT
(2)
Semantic (per-class) confidence scores are obtained by passing Q' through a linear layer. This layer has fan-out of 1 C predict classification score for each of the C' semantic classes, and an additional 'no-object' score which is used during inference to detect inactive queries that represent neither an object nor 'stuff" class. We use the semantic prediction to decide whether the query mask belongs to an object track, or to one of the 'stuff" classes.
Soft-masked Cross-attention: Inspired by [50, 28), we employ soft-masked cross-attention to improve convergence. Given a set of queries Q, the output Qx-entn of cross-attention is computed as: [Q(K + E)TT + a.M
Qx attn f -softmaa
(3)
VDD
Here, K E (N,.M.) and V E (N) denote the keys and values (derived as linear projec- tions from V- or Fi), respectively, E' E R (N,,M.)xD x denotes positional encodings (explained in the next paragraph), a is a scalar weighting factor, and M is the voxel-level query mask computed by applying Eq. to Q, followed by voxelization and downsampling to the required stride. Intuitively, the term "aM,T" amplifies the correspondence between queries and voxel/image features based on the mask prediction from the previous layer. This makes the queries foeus their respective ob- ject/class targets.
Positional Encodings: We impart the cross-attention operation with 3D coordinate information of the features in V; by using positional encodings (E' in Eq. 3). These contain two components: (I) Fourier encodings [51] of the (sr v,) coordinates, and (2) a depth component which is obtained by applying sine and cosine activations at various frequencies to the Euclidean distance of each voxel feature from the LiDAR sensor. Although the depth can theoretically be inferred from the zyz coordinates, we find it beneficial to explicitly encode it. Intuitively, a multi-modal setup the depth provides a useful cue for how much the model should rely on features from different modalities, e.g., for far-away points the image features are more informative as the LiDAR very sparse. Both components have 2 dimensions and are concatenated to obtain the final positional encoding E M.N.. For the image features F,, we use the encoding of the corresponding voxel.
3.3 Tracklet Association Module (TAM)
The 4D panoptic task requires object track IDs be consistent time. Since 4D-Former pro- cesses overlapping clips, one way to achieve temporal consistency is associate tracklet masks across clips based on their respective mask IoUs, as done by existing works [9, 11, 33]. However, this approach cannot resolve even brief occlusions which frequently arise due to inaccurate mask predictions and/or objects moving out of view. To mitigate this shortcoming, we propose a learnable Tracklet Association Module (TAM) which can associate tracklets across longer frame gaps and reasons over the objects' appearances and spatial locations.
The TAM is implemented The input to our TAM is constructed by concatenating the following attributes of the input tracklet
MLP which predicts an association score for given pair of tracklets.
5

---

LiDAR
Front Camera
LiDAR
Front Camera
T=1
T=2
Semantic
Tracks
Figure 5: Qualitative results on nuScenes sequence 0798 with both LiDAR and image views.
pair along the feature dimension: (I) their (I, y, =) mask centroid coordinates, (2) their respective tracklet queries, (3) the frame gap between them, and (4) their mask IoU. We refer the readers to the supplementary material for illustration. Intuitively, the tracklet queries encode object appear- ances, whereas the frame gap, mask centroid and mask IoU provide strong spatial cues. Our TAM contains 4 fully connected layers and produces a scalar association score the final output. The mask IoU is set to zero for tracklet pairs with no overlapping frames. Furthermore, the mask cen- troid coordinates and frame gap are expanded to 64-D each by applying sine and cosine activations at various frequencies, similar to the depth encodings discussed in Sec. 3.2.
During inference, we maintain a memory bank containing object tracks. Each track is represented by the tracklet query, mask centroid and frame number for its most recent occurrence. This memory bank is maintained for the past hist frames. At frame f, we compute association scores for all pair- wise combinations of the tracks in the memory bank with the predicted tracklets in the current clip. Subsequently, based on the association score, each tracklet in the current clip is either associated with a previous track ID, or is used to start a new track ID.
3.4 Learning
We employ a two-stage training approach for the proposed architecture. During the first stage, we exclude the TAM and optimize the network for a single input clip. The predicted masks are matched to ground-truth objects and stuff class segments with bi-partite matching based on their mask loUs and classification scores. Note that during training, each stuff class present in the scene is treated object track. Subsequently, the masks are supervised with combination of binary cross entropy and DICE losses, denoted by Loe and Ldice respectively, and the classification output is supervised with a cross-entropy loss Les-- These losses are computed for the output of each of the B fusion blocks in the Panoptic Decoder, followed by summation. Lastly, the point-level pseudo-fusion output discussed in Sec. 3.1 is supervised by the following L2 regression loss Epr:
Ept + MLP MLPiusion (1 Zime)) MLP pscudo
(Zidter) 11
(4)
The final loss Loval is computed by taking the sum of the individual losses with empirically chosen weights. The superscript L denotes the loss for the output of the b-th fusion block in the decoder.
B
Lioaal + Lpt + c5 2 + C%.)
(5)
b=1
The second stage involves optimizing the TAM with the remaining network frozen. We generate tracklet predictions for multiple clips separated by different frame gaps, and then optimize the TAM using all pairwise combinations of tracklets in the given clip set. The predicted association scores are supervised with a binary cross-entropy loss.
4 Experiments
Implementation Details: We process clips containing frames each, with voxel size of 0.1 m, and the feature dimensionality D 128. The images are resized in aspect-ratio preserving

---

Method
Validation
Test
PAT
LSTQ
PTQ
PQ
TQ
Sanoe
PAT
LSTQ
PTQ
PQ
TQ
Sper
PanopiccTracNNed [53]
44.0
43.4
50.9
$1.6
38.5
32.3
45.7
44,8
$1.6
51.7
40,9
36.7
4D.PL
59.2
56.1
51.4
60.5
64.9
Cylinder3D++ OGR3MOT 154]
62.7
61.7
61.3
61.6
63.8
59,4
(AFy -33Net [3] OGR3MOT (54)
62.9
62.4
60.9
61.3
64.5
59.9
[22] [F]
64.6
67.6
58.6
67.1
71.2
60.2
EfficionLPP [33]
70.4
66.0
67.0
67.9
71.2
4D.Former
78.3
76.4
75.2
77.3
79.4
73,9
79.4
78.2
75.5
78.0
75.5
76.1
Table
1: Benchmark results for nuScenes validation and test set.
manner such that the lower dimension is 480px. For training time data augmentation, randomly subsample the LiDAR pointcloud to 10t points, and also apply random rotation and point jitter. The images undergo SSD-based color augmentation [52], and are randomly cropped to 70% of their original size. In the first stage, we train for 80 epochs with A optimizer with batch size across 8x Nvidia T4 GPUs (14GB of usable VRAM), The learning rate is set to 3 x 10 3 for the LiDAR feature extractor, and 10-4 for the rest of the network. The rate is decayed in steps of 0.1 after 30 and 60 epochs. For the second stage, we train the TAM for 2 epochs on single GPU with learning rate 10 During inference, associate tracklets over temporal history Thist = 4.
Datasets: To verify the efficacy of approach, apply it
popular benchmarks: nuScenes [12] and SemanticKITTI [13]. nuScenes [12] contains 1000 sequences, each 20s long and annotated at 2Hz. The scenes are captured with 32-beam LiDAR sensor and 6 cameras mounted at different angles around the ego vehicle. The training set contains 600 sequences, whereas validation and test each contain 150. The primary evaluation metric is Panoptic Tracking (PAT). Compared to nuScenes, SemanticKITTI [9] contains fewer but longer sequences, and uses LiDAR Segmentation and Tracking Quality (LSTQ) as the primary evaluation metric. One caveat is that image input is only available from a single, forward-facing camera. As a result, only small fraction (~ 15%) of LiDAR points visible in the camera image. For this reason, following existing multimodal methods [43], we evaluate only those points which have valid camera image projections.
Comparison to state-of-the-art: Results nuScenes are shown in Tab. 1 and visualized in Fig. 5. We see that 4D-Former outperforms existing methods across all metrics. In terms PAT, 4D-Former achieves 78.3 and 79.4 on the val and test sets, respectively. This is signifi- cantly better than the 70.4 (+9.0) achieved by EfficientLPT [33] the test set and the 64.6 (+13.7) achieved by EfficientL.PP [2 on val. We attribute this to 4DD-Fommer' 's ability to reason over multimodal inputs and segment both semantic classes and object tracks in an end-to-end learned fashion. The results on SemanticKITTI validation set are reported in Tab. 2. For fair compari- son, we also evaluated existing top-performing methods on the same sub-set camera- points. We see that 4D-Former achieves 73.9 LSTQ which is higher than the 71.0 (+2.8) achieved by 4D-StOP and also the 65.4 (+8.4) achieved by 4D-PLS [9]. Aside from S 4D-Former is also better for other metrics.
Method
LSTQ
Sanse
Sak
loU
loU
4D-PLS [9]
65.4 72.3
59.1
62.6
61.8
4D-SIOP [1I]
71.0
82.5
61.0
63.0
66.0
Ours
73,9
80.9 67.6
64.9 71.3
Table 2: SemanticKITTI validation results.
Effect of Tracklet Association Module: The effec- tiveness of the TAM is evident from Tab. 3 where compare it to baseline which uses only use mask IoU in the overlapping frame for association. This results in the PAT dropping from 78.3 to 76.3. This highlights the importance of using a learned temporal association mechanism with both spatial and appearance cues.
Setting
PAT
LSTQ
PTQ
PQ
Mask IoU TAM
76.3
74.6
73.9
77.3
78.3
76.4
75.2
77.3
Table 3: Ablation results for temporal as- sociation on nuScenes validation set.
Next, we ablate other aspects our method in Tab. 4.
For these experiments, subsample the training set by using only every fourth frame to save time and resources. The final model is also re-trained with this setting for a fair comparison (row 6).
7

---

Effeet of Image Fusion: Row I is LiDAR-
CAF
only model which does not utilize image input
#
PF
DE
PC
PAT
LSTQ
PTQ
PQ
in any way. This achieves 59.7 PAT which is significantly worse than the final model's 66. I. This shows that using image information yields significant performance improvements. Row 2 utilizes point-level fusion (Sec. 3.1), but does not apply cross-attention to image features in the decoder (Sec. 3.2). This setting achieves 61.8 PAT which is better than the LiDaR-only setting (59.7), but still much worse than the nal model (66.1) 1). Row 3 tests the opposite con- figuration: the decoder includes cross-attention to image features, but point-level fusion is applied. This yields 63.2 PAT which is slightly higher than row 2 (61.8) but worse than the final setting (66.1) 1). We conclude that while both types of fusion are beneficial in standalone setting, combining them yields a larger improvement.
59.7
64.3
60.8
63.6
61.8
65.2
64.3
67.6
3.
v
63.2
66.1
63.1
66.3
4.
64.1
66.4
65.7
69.1
5.
64.6
66.7
66.0
69.4
6.
66.1
67.4
66.2
69.9
Table 4: Ablation results on nuScenes val set. PF: Point Fusion, CAF: Cross-attention Fusion, DE: Depth encodings, PC: Pseudo-camera feature loss.
Effect of Depth Encodings: As discussed in Sec. 3.2, our positional encodings contain depth component which is calculated by applying sine/cosine activations with multiple frequencies to the depth value each voxel feature. Row 4 omits this component and instead only uses Fourier encodings based on the zyz coordinates. This setting yields 64.1 PAT which is lower than the full model (66.1 1), thus showing that explicitly encoding depth beneficial.
Effect of Pseudo-camera Feature Loss: Recall from Sec. 3.4 that we supervise pseudo-camera features for point fusion with an L2 regression loss. Row shows that without this loss the PAT reduces from 66.1 I to 64.6. Other metrics also reduce, though to lesser extent.
Limitations
Our method performs less effectively on SemanticKITTI compared to nuScenes, particularly in crowded scenes with several objects. In addition to lower camera image coverage, this is due the limited number of moving actors in the SemanticKITTI training set which, on average, contains only 0.63 pedestrians and 0.18 riders per frame. Existing LiDAR-only methods [2, 20, 55] overcome this by using instance cutmix augmentation which involves randomly inserting LiDAR scan cutouts of actors into training scenes. Doing the same in multimodal setting is, however, non-trivial since it would require the camera images to also be augmented accordingly. Consequently, a promising future direction is to develop more effective augmentation techniques for multimodal training.
Our tracking quality is generally good for vehicles, but is comparatively worse for smaller object classes e.g. bicycle, pedestrian (see class-wise results the supplementary material), and although the TAM is more effective than mask IoU, the improvement plateaus This 4 (i.e. 2s into the past). Another area for future work thus involves improving the tracking mechanism to handle longer time horizons and challenging object classes.
6 Conclusion
We proposed a novel, online approach for 4D panoptic segmentation which leverages both LiDAR scans and RGB images. We employ transformer-basee Panoptic Decoder which segments semantic classes and object tracklets by attending to scenes features from both modalities. Furthermore, our Tracklet Association Module (TAM) accurately associates tracklets over time in a learned fashion by reasoning over spatial and appearance cues. 4D-Former achieves state-of thee-art results on the nuScenes and SemanticKITTI benchmarks, thus demonstrating its efficacy on large-scale, real-world data. We hope our work will spur advancement SDV perception systems, and encourage other researchers to develop multi-sensor methods for further improvement.

---

Acknowledgments
We thank the anonymous reviewers for the insightful comments and suggestions. We would also like thank the Waabi team for their valuable support.
References
[1] X. Zhu, H. Zhou, T. Wang, F. Hong, Y. Ma, W. Li, Li, and D. Lin. Cylindrical and asym- metrical 3d convolution networks for lidar segmentation. In CVPR, 2021.
[2] J. Xu, R. Zhang, J. Dou, Y. Zhu, J. Sun, and S. Pu. Rpvnet: A deep and efficient range-point- voxel fusion network for lidar point cloud segmentation. ICCV, 2021.
[3] R. Cheng, R. Razani, Taghavi, E. Li, and B. Liu. af2-s3net:
Attentive feature fusion with adaptive feature selection for sparse semantic segmentation network. In CVPR, 2021.
[4] E. Li, S. Casas, and R. Urtasun. Memoryseg: Online lidar semantic segmentation with latent memory. In ICCV, 2023.
[5] Y. Zhao, X. Zhang, and X. Huang. A divide-ande and-merge point cloud clustering algorithm for lidar panoptic segmentation. ICRA, 2022.
[6] E. Li, R. Razani, Y. Xu, and B. Liu. Smac-seg: Lidar panoptic segmentation via sparse multi- directional attention clustering. In ICRA, 2022.
[7] X. Weng, J. Wang, D. Held, and K. Kitani. 3d multi-object tracking: A baseline and new evaluation metrics. In IROS, 2020.
[8] H.-k Chiu, Li, R. Ambruş, and J. Bohg. Probabilistic 3d molti-modal, multi-object tracking for autonomous driving. In ICRA, 2021.
[9] M. Aygun, A. Osep. M. Weber, M. Maximov, C. Stachniss, J. Behley, and Leal-Taixé. 4d panoptic lidar segmentation. In CVPR, 2021.
[10] R. Marcuzzi, L. Nunes, L. Wiesmann, I. Vizzo, J. Behley, and C. Stachniss. Contrastive instance association for 4d panoptic segmentation using sequences of 3d lidar scans. IEEE Robotics and Automation Letters, 1(2):1550-1557 2022.
[11] L. Kreuzberg, E. Zulfikar, Mahadevan, F. Engelmann, and Leibe. 4d-stop: Panoptic segmentation of 4d lidar using spatio-temporal object proposal generation and aggregation. In ECCV Workshops. 2023.
[12] H. Caesar, V. Bankiti, A. H. Lang, S. Vora, V. E. Liong, Q Xu, A. Krishnan, Y. Pan, G. Baldan, and O. Beijbom. nuscenes: A multimodal dataset for autonomous driving. In CVPR, 2020.
[13] Behley, M. Garbade, A. Milioto, Quenzel, S. Behnke, C. Stachniss, and J. Gall. Se- mantickitti: dataset for semantic scene understanding of lidar sequences. ICCV, 2019.
[14] H. Thomas, C. R. Qi, J.-B. Deschaud, B. Marcotegui, F. Goulette, and L. J. Guibas. Kpconv: Flexible and deformable convolution for point clouds. In ICCV, 2019.
[15] A. Milioto, I. Vizzo. J. Behley, and C. Stachniss. Rangenet++: Fast and accurate lidar semantic segmentation. In IROS, 2019.
[16] C. Xu, B. Wu, Z. Wang. W. Zhan, Vajda, Keutzer, and M. Tomizuka. Squeezesegv3: Spatially-adaptic convolution for efficient point-cloud segmentation. In ECCV, 2020.
[17] Y. Zhang, Z. Zhou, P. David, X. Yue, Z. Xi, B. Gong, and H. Foroosh. Polamet: An improved grid representation for online lidar point clouds semantic segmentation. In CVPR, 2020.

---

[18] A. Milioto, J. Behley, C. McCool, and C. Stachniss. Lidar panoptic segmentation for au- tonomous driving. IROS, 2020.
[19] Li, X. He, Y. Wen, Y. Gao, Cheng, and D. Zhang. Panoptic-phnet, Towards real-time and high-precision lidar panoptic segmentation via clustering pseudo heatmap. In CVPR, 2022.
[20] Z. Zhou, Y. Zhang, and H. Foroosh. Panoptic-polarret: Proposal-free lidar point cloud panoptic segmentation. In CVPR, 2021.
[21] S. Gasperini, M.-A. N. Mahani, A. Marcos-Ramirg N. Navab, and Tombari. Panoster: End- to-end panoptic segmentation of lidar point clouds. IEEE Robotics and Automation Letters, (2):3216-322) 2021.
[22] K. Sirohi, R. Mohan, D. Büscher, W. Burgard, and A. Valada. Efficientlps: Efficient lidar panoptic segmentation. IEEE Transactions on Robotics, 2021.
[23] R. Razani, R. Cheng, E. Li, E. Taghavi, Y. Ren, and L. Bingbing. Gp-s3net: Graph-based panoptic sparse semantic segmentation network. In ICCV, 2021.
[24] E. Li, Razani, Y. Xu, and Liu. Cpseg: Cluster-free panoptic segmentation of 3d lidar point clouds. In ICRA, 2023.
[25] R. Mareuzzi, L. Nunes, L. Wiesmann, J. Behley, and C. Stachniss. Mask-based panoptic lidar segmentation for autonomous driving. IEEE Robotics and Automation Letters, 8(2): 141- 1148, 2023.
[26] S. Su, J. Xu, H. Wang, Z. Miao, X. Zhan, D. Hao, and X. Li. Pups: Point cloud unified panoptic segmentation. In AAAI, 2023.
[27] Z. Xiao, W. Zhang, T. Wang, C. C. Loy, D. Lin, and Pang. Dosition-guides point cloud panoptic segmentation transformer. In arXiv, 2023.
[28) B. Cheng, I. Misra, A. G. Schwing, A. Kirillov, and R. Girdhar. Masked-attention mask trans- former for universal image segmentation. In CVPR, 2022.
[29] C. Zheng, Yan, Gao, Zhao, W. Zhang, Li, and S. Cui. Box-aware feature enhance- ment for single object tracking on point clouds. In ICCV, 2021.
[30] R. E. Kalman. A new approach to linear filtering and prediction problems. 1960.
[31] H. W. Kuhn. The hungarian method for the assignment problem. Naval research logistics quarterly, 2(1-2):83-99 97, 1955.
[32] C. Luo, X. Yang, and A. Yuille. Exploring simple 3d multi-object tracking for autonomous driving. In ICCV, 2021.
[33] R. Mohan and A. Valada. 7th driving olympics: 1st place report for panoptic tracking, arXiv,, 2021.
[34] M. Zhu, S. Han, H. Cai, S. Borse, M. G. Jadidi, and F. Porikli. 4d panoptic segmentation as invariant and equivariant field prediction. In arXiv, 2023.
[35] Yin, X. Zhou, and P. Krähenbühl. Multimodal virtual point 3d detection. NewrIPS, 34: 14494-165011 2021.
[36] S. Vora, A. H. Lang, B. Helou, and O. Beijbom. Pointpainting: Sequential fusion for 3d object detection. CVPR, 2020.
[37] C. Wang, C. Ma, M. Zhu, and X. Yang. Pointaugmenting: Cross-modal modal augmentation for 3d object detection. In CVPR, pages 11794-11808 1803, 2021.
10

---

[38] M. Liang B. Yang, S. Wang, and R. Urtasun, Deep continuous fusion for multi multi-sensor 3d object detection. In ECCV, 2018.
[39] Z.. Zhuang, R. Li, K. Jia, Q. Wang. Y. Li, and M. Tan. Perception-aware multi-sensor fusion for 3d lidar semantic segmentation. In ICCV, 2021.
[40] Y. Li, A. W. Yu, T. Meng, B. Caine, Ngiam, D. Peng, Shen, Lu, D. Zhou, Q. V. Le, al. Deepfusion: Lidar-camera deep fusion for multi-modal 3d object detection. In CVPR, 2022.
[41] X. Bai, Z. Hu, X. Zhu, Q. Huang, Y. Chen, H. Fu, and C.-L. Tai. Transfusion: Robust lidar- camera fusion for 3d object detection with transformers. In CVPR, 2022.
[42] X. Chen, T. Zhang, Y. Wang, Y. Wang, and H. Zhao. Futr3d: A unified sensor fusion framework for 3d detection. In arXiv, 2022.
[43] Li, H. Dai, Han, and Y. Ding. autonomous driving. In arXiv, 2023.
Mseg3d: Multi-modal 3d semantic segmentation for
[44] K. He, X. Zhang, S. Ren, and J. Sun. Deep residual learning for image recognition. In CVPR, 2016.
[45] T.Y. Lin, P. Dollár, R. Girshick, K. He, B. Hariharan, and S. Belongie. Feature pyramid networks for object detection. In CVPR, 2017.
[46] H. Tang, Z. Liu, S. Zhao, Y. Lin, J. Lin, H. Wang, and S. Han. Searching efficient 3d architec- tures with sparse point-voxel convolution. In ECCV, 2020.
[47] H. Tang, Z. Liu, X. Li, Y. Lin, and S. Han. Torchsparse: Efficient point cloud inference engine. In MLSys, 2022.
[48] F. Hong, H. Zhou, X. Zhu, H. Li, and Z. Liu. Lidar-based panoptic segmentation via dynamic shifting network. In CVPR, 2021.
[49] N. Carion, F. Massa, G. Synnaeve, N. Usumier, A. Kirillov, and S. Zagoruyko. End-to-end object detection with transformers. In ECCV, 2020.
[50] A. Athar, Luiten, A. Hermans, Ramanan, and B. Leibe. Hodor: High-level object de- scriptors for object re-segmentation in video learned from static images. In CVPR, 2022.
[51] M. Tancik, P. Srinivasan, B. Mildenhall, S. Fridovich-Keil, N. Raghavan, U. Singhal, R. Ra- mamoorthi, J. Barron, and R. Ng. Fourier features let networks learn high frequency functions in low dimensional domains. In NeurIPS, 2020.
[52] W. Liu, D. Anguelov, D. Erhan, C. Szegedy, S. Reed, C.-Y. Fu, and A. C. Berg. Ssd: Single shot multibox detector. In ECCV, 2016.
[53] V. Hurtado, R. Mohan, W. Burgard, and A. Valada. Mopt: Multi-object panoptic tracking. In CVPR-W W. 2020.
[54] J.-N. Zaech, Liniger, Dai, M. Danelljan, and L. Van Gool. Learnable online graph representations for 3d multi-object tracking. IEFE Robotics and Automation Letters, 2022.
[55] X. Yan, J. Gao, C. Zheng, C. Zheng, R. Zhang, S. Cui, and Z. Li. 2dpass: 2d priors assisted semantic segmentation on lidar point clouds. ECCV, 2022.

---

Supplementary Materials
A Tracklet Association Module
We provide an illustration of the proposed Tracklet Association Module (TAM) in Fig.6. The input to our TAM is constructed by concatenating the following attributes of the input tracklet pair along the feature dimension: (I) their y. mask centroid coordinates, (2) their respective tracklet queries, (3) the frame gap between them, and (4) their mask IoU. The frame gap and mask centroid coordinates are expanded to 64-D each by applying sine/cosine activations with various frequencies. The concatenated set of features input to 4-layer MLP which produces scalar association score for the input tracklet pair.
Mask Centroids
Queries
Frame Gap
Mask IoU
concatenate
MLP
Association Score
Figure 6: Illustration of the Tracklet Association Module (TAM).
B Detailed Quantitative Results
In this section, we first present the 3D panoptic metrics on the two benchmarks for reference and then provide the detailed class-wise metrics on the two datasets.
Specifically, we present the 3D panoptic metries on nuScenes validation set in Tab. 5. Please note that we did not include any other methods in the table since it's unfair to directly compare with other single-scan based methods the 3D benchmark. For completeness, ManticKITTI results using 3D panoptic metrics and report the results in Tab. 6 for both cases; eval- uating only those points which are projectable into the camera (Camera FoV) and also the Full Scan which includes all LiDAR scan points. Unsurprisingly, because of the missing camera image input, our performance on the full scan (60.7 PQ) is lower than that on the camera FoV only (64.3 PQ). Lastly, we present the detailed per-class results for: nuScenes val set (Tab. 7), nuScenes test set (Tab. 8), and SeTT val set (Tab.9).
evaluate our Se-
PQ
PQ
PQ
PQ
RQ
RQ
RQ
SQQ
SQ
SQ
4D-Forer [Ours]
77.3
80.9
73.5
79.6
86.5
84.1
87.8
89.0
86.7 90.4
Table 5: Results on nuScenes 3D panoptic segmentation validation benchmark
PO1
PO
Pg
RQ
RQ
RQ.h
SQ
SQ
mloU
Fall Scan
60.7
65.4
66.4
70.3
68.8
72.4
76.0
72.9
80.1
66.3
Camera FoV only
64.3
66.7
60.6
69.5
73.6
72.1
75.6
80.6
80.6
80.5
67.6
Table 6: Results on SemanticKITTI 3D panoptic segmentation validation benchmark
12

---

Driabic
OWher FlAt
Nagetation
Bayal
Traffcc
Tniler
Trick
Metric
PTQ
ML.33
79.05
90.89
69.64
8187
R3,04
38.07
26.92
95.61
31.92
68.92
39.61
82.46
87.59
#TO
65.25
75.33
79.39
91.16
64.94
82.43
88.47
83115
39.14
77.25
95.01
51.92
68.92
54.61
$2.46
87.59
loU
78.56
82.74
$2.69
90.41
94.31
54.95
88.90
82.66
68.58
65.41
82.57
96.32
71.33
73.36
75.54
91.80
89.
7734
68.59
79.51
80.98
93.51
67.63
86.77
91.71
87.74
61.00
78.91
95.61
51.92
68.92
51.61
82.46
87.59
98.95
84.50
89.64
Table Class-wise results on nuScenes val set. Metrics are provided [%]]
Consruccioon
Motorcyole
Pidestrain
Manierr
Bleyele
Drivable
Sidewalk
Bus
Metric
PTQ
75.47
63.20
73.20
75.21
90.14
62.44
81.01
89.11
84,95
65.46
75.15
97.30
46.13
71.44
58.00
85.16
89.35
sPTQ
75.90
64.63
73.98
75.42
90.45
63.73
81.92
89.57
75.43
46.13
71.44
58.00
85.16
89.85
77.99
68.63
30
77.48
93.01
69.07
86.6
92.64
89.13
68.17
77.05
97.10
16.13
71.M
38.00
SQ
89.66
81.69
89.1
94.77
95 80
87.12
92.62
93.94
91.63
88.30
94.29
97.36
85.46
8185
78.04
91.08
91.51
86.59
84.01
87.85
81.78
97.00
79.28
93.60
98.62
97.28
77.19
81.71
99.33
53.98
87.28
74.31
93.50
98.19
Table 8: Class-wise results on nuScenes set. Metrics provided in [%]
C Qualitative Comparison (LiDAR-only vs. Fusion)
Figures 7 and 8 provide a qualitative comparison of our proposed method with the LiDAR-only baseline (Tab, 4, row the main text). We provide the segmentation results the LiDAR domain for both LiDAR-onfe -only and fusion models in the first two columns, respectively, and the corresponding camera view in the third column. The region of interest in each case is highlighted in red.
In the first example (Fig. 7), the baseline wrongly segments the building at range vegetation due to the limited information obtained from the LiDAR input. By contrast, the final model with fusion effectively leverages the rich contextual information from the camera (highlighted by the red box) and segments the correct class.
the second example (Fig. 8), the baseline fails to track pedestrians when they are close to each other (the two pedestrians on the left merged together as single instance). By contrast, the camera view provides distinct appearance cues for each pedestrian, enabling model to accurately segment and track them.
13

---

Other Vehicle
Ground
mesin
Bleycls
Monoeyele
TTuck
Bieyclit
Monorsyelaa
Road
Parking
Sidewalk
Building
Fence
Vegetation
Thonk
Torain
Pble
Tralikc Sign
Metric
67.6
97.0
61.0
78.0
$4.0
73.0
83.0
95.0
0.0
96.0
44.0
80.0
88.0
56.0
89.0
71.0
76.0
66.0
45.0
Table 9: Class-wise results on SemanticKITTI val set.
Metrics are provided in [%)].
Note that
association metrics are not available for 'stuff" classes.
LiDAR-only
With Fusion
Camera View
T=2
Figure 7: Qualitative comparison of semantic segmentation for LiDAR-only fusion model on sequence 0105 from nuScenes.
LiDAR-only
With Fusion
Camera View
T=2
Figure 8: Qualitative comparison of instance segmentation and tracking for LiDAR-only vs. fusion model on sequence 0003 from nuScenes.
14


Reviews:
[{"review_id": "rBkjDeo2Unj", "reviewer": "Reviewer_56js", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "This paper proposes a multi-modality transformer-based framework for 4D LiDAR-based point clouds panoptic segmentation. Specifically, it takes multi-view images and LiDAR-based point clouds as input, uses a multimodal encoder to fuse image and point features, and subsequently, designs a query-based unified decoder for panoptic segmentation prediction. Furthermore, a tracking module named \"Tracklet Association Module (TAM)\" is devised to associate tracklets across longer frame gaps in the multi-frame panoptic segmentation problem. The proposed framework achieves new state-of-the-art on two popular benchmarks, nuScenes and SemanticKITTI, and detailed ablation studies show the efficacy of each component.", "summary_of_recommendation": "This paper proposes an effective transformer-based framework for multi-modality 4D LiDAR-based panoptic segmentation. Although there are minor issues regarding missing related works and potentially weakened contributions regarding the transformer-based framework, the overall paper has strong performance and detailed ablation studies, making it potentially beneficial to the community. Therefore, I vote for weak accept.", "technical_quality": "Good: Generally convincing, but some additional experiments or analyses needed.", "clarity_of_presentation": "Good: Minor revision needed to improve organization or clarity.", "potential_impact": "", "robotics_focus": "Highly relevant to robotics but no hardware experiments", "quality_of_limitations": "Additional details required", "questions_for_rebuttal": "See the weaknesses.", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Strengths:\n\n- This paper proposes the first multi-modality transformer-based framework for 4D LiDAR panoptic segmentation, and it is also effective and practical, as shown in the experiments.\n- The basic idea is easy to follow, and the overall methodology is simple yet effective.\n- Detailed ablation studies show the importance of each component.\n- It achieves new state-of-the-art and surpasses previous methods by a large margin, validating its effectiveness.\n\nWeaknesses:\n\n- There are some missing related works regarding transformer-based methods for LiDAR-based panoptic segmentation, such as PUPS[1] and P3Former[2]. From this perspective, the contribution of a unified query-based decoder for LiDAR-based panoptic segmentation should be weakened. Technically, there are few new things in the multimodal encoder and transformer-based panoptic decoder except for incorporating multimodal data fusion during the procedure.\n- (Minor) It would be better to have more ablation studies for alternative model designs. For example, how about sharing several MLP layers in Eq. (1), i.e., using the same MLP layers to encoder point clouds and using additional layers to fuse part of them with image features?\n- (Minor) The overall training is two-stage and there are no experiments comparing two-stage and one-stage training, although the author gives the reason and claims end-to-end training may yield better results.\n\n[1] PUPS: Point Cloud Unified Panoptic Segmentation\n\n[2] Position-Guided Point Cloud Panoptic Segmentation Transformer", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed."}, "extra_text": {}}, {"review_id": "ghDZ5k4ENB", "reviewer": "Reviewer_g5r2", "summary": "", "questions": "", "limitations": "", "rating": 2, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "This paper proposes a system that incorporates 3D panoptic segmentation, 4D tacking, and multi-modal fusion to fulfill multimodal 4D Panoptic Segmentation. It fuses image information in the 3D feature extraction stage and decode stage. It also designs a Tracklet Association Module to associate object tracks.", "summary_of_recommendation": "Considering the limitations as stated in the weaknesses, my rating is Weak Reject. ", "technical_quality": "Good: Generally convincing, but some additional experiments or analyses needed.", "clarity_of_presentation": "Good: Minor revision needed to improve organization or clarity.", "potential_impact": "", "robotics_focus": "Highly relevant to robotics but no hardware experiments", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "The authors shall address the weaknesses in the Review section. ", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Strengths:\n1. Comprehensive Approach: The paper integrates different modules to address the challenges of multimodal 4D panoptic segmentation, making the system complete and sensible.\n2. Clear Methodology: The methodology is presented in a straightforward manner, allowing readers to easily follow the proposed approach. Figure 1 provides a clear overview of the system's structure, while subsequent subfigures and sections delve into the details.\n3. Analysis of Limitations: The paper acknowledges the limitations of their methods, providing an insightful analysis of the deficiencies and inspiring future work in the field.\n\nWeaknesses:\n1. Fragmented Components: The proposed components sometimes lack sufficient insight into the problems they aim to solve. This issue is also evident in the related works section. Instead of highlighting the novelty of specific fields, the paper introduces components from other fields to showcase superiority without thoroughly addressing the underlying problem.\n2. Incomplete Experiments: This paper misses some important comparisons and metrics in the experiment part. While the focus is on 4D panoptic segmentation, it is crucial to include comparisons with 3D panoptic segmentation methods, as it serves as a fundamental task for 4D panoptic segmentation. And the metric Panoptic Quality (PQ) is both important in 3D and 4D panoptic segmentation. Besides, 3D panoptic segmentation is more mature. This method, incorporating multi-modal and temporal information, is expected to surpass previous 3D panoptic segmentation to validate its effectiveness. The comparison on the Semantickitti dataset is also missing in terms of PQ.\n3. Lack of Related Works: The paper should discuss more related works, particularly in the field of 3D unified panoptic segmentation, which forms an important foundation for this work. Including works such as PUPS (Zhao et al., 2023) and P3Former (Xiao et al., 2023) would enhance the comparison and provide a broader context for the proposed system.\n", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed."}, "extra_text": {}}, {"review_id": "Xi41qcNhKP", "reviewer": "Reviewer_WZbn", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "This paper addresses the challenging task 4D panoptic segmentation by utilizing both LiDAR and RGB inputs. To this end, the authors propose a novel method called 4D-Former that leverages both modalities to predict semantic masks along with temporally consistent object masks for the given input sequence. This paper employs concise queries to encode semantic classes and objects from both modalities. Further, this work proposes a learnable association mechanism to associate objects over time that considers both appearance and spatial location. \nLastly, 4D former achieves SOTA results for nuScenes and SemanticKITTI dataset\n", "summary_of_recommendation": "To the best of my knowledge, this study represents the first effort in tackling the multi-modal 4D panoptic segmentation task. The results achieved by the 4D Former model are convincing. It is my anticipation that this work will lay the groundwork for subsequent advancements in the realm of multi-modal research, serving as a good starting point.", "technical_quality": "Very Good:  Convincing.", "clarity_of_presentation": "Very Good:  Well organized and clear.", "potential_impact": "", "robotics_focus": "Relevant but unlikely to deploy to hardware in near future", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "The supplementary addresses the question I would have such as qualitative results and details of TAM. Hence, I have no questions. ", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "- The paper is easy to understand and logically organized into sections.\n- The figures complement the text and help in understanding the tasks easily.\n-This work is one of the first to address the task of 4D panoptic segmentation in a multi-modal setting.\n-4D-Former achieves SOTA results on both Semantic KITTI and Panoptic-nuScenes.\n-The ablation experiments exhibit the usefulness of each architecture design choice quantitatively. \n", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed."}, "extra_text": {}}, {"review_id": "KvLdY8IRLZ", "reviewer": "Reviewer_oeTB", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "The task of this paper is to conduct semantic segmentation, instance segmentation and tracking of Lidar point clouds across time, and the paper proposed to incorporate multi-view camera images as input to enhance the features and improve the performance.", "summary_of_recommendation": "This paper proposed to solve the 4D panoptic segmentation task in a new way by incorporating extra camera input. The method has good incorporation of existing solutions in building the overall framework, and also proposed novel submodules to improve the performance. The experimental results are good overall, although not as strong on the SemanticKITTI dataset. I would like to see the soft-masked cross attention clarified, and it is a good paper overall. \n\nAfter rebuttal:\n\nI'd like to keep my score at Weak Accept.", "technical_quality": "Very Good:  Convincing.", "clarity_of_presentation": "Good: Minor revision needed to improve organization or clarity.", "potential_impact": "", "robotics_focus": "Highly relevant to robotics but no hardware experiments", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "1. Please clarify on the soft-masked cross-attention module. \n\n2. Add the missing references.", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Originality:\n\nThis paper is the first to use monocular images for point cloud 4D panoptic segmentation. From the technical perspective, this paper combined recent successful solutions for 3D point cloud segmentation and multi-view monocular 3D object detection together for this new task, and the authors also proposed a novel method for temporal association of objects. The discussion of related work is proper. A missing reference is the current SOTA of 4D panoptic segmentation on semantic-kitti: EQ-4DSTOP (https://arxiv.org/pdf/2303.15651.pdf). Another work Panoster (https://arxiv.org/abs/2010.15157) presented a post-processing-free instance prediction module, which is also relevant to this paper's contribution. Overall, this paper has decent novelty. \n\nQuality:\n\nThe paper presents a well-assembled framework to process multi-modal inputs to conduct the segmentation task. Besides, I found three designs particularly interesting. First, the point-level fusion has a pseudo fusion module for points without RGB correspondence so that the features are better aligned with those with RGB information, with a corresponding loss function. Second, it employed a query-based approach for cross-modal fusion and segmentation. In terms of segmentation, it naturally solved the problem of consistency between instance and semantic labels. Third, the tracklet association module goes beyond IoU-based association, making use of both the semantic and geometric features. The experiments validate the effectiveness of the methodology. Overall, this work is complete in the standard of a conference paper. \n\nClarity:\n\nThe paper is overall well written, but I found the section of soft-masked cross-attention confusing. The K and V do not match the dimension of M_v. Since K and V are of shape TxD, representing the query features (Q also query features), while M_v is the attention matrix between queries and voxels, it is hence unclear whether this module is self-attention or cross-attention, and where this module is placed in the overall network. \n\nSignificance:\n\nThe results are important. It pushed forward the state-of-the-art 4D panoptic segmentation in nuScenes, and more importantly showed a viable way of handling this task with additional image input. \n\nRelevance:\n\nThe panoptic segmentation task with temporal knowledge is a very relevant problem to the robotic community. This paper brings new modality to this task and obtained good results, thus I believe can attract interest of CoRL audience. \n\nLimitations:\n\nA major weakness of this paper is the performance on the SemanticKITTI dataset. The performance is only marginally better than the baselines while having extra monocular image input, which could make people question the effectiveness of the multimodal fusion. However, the authors did provide a paragraph of discussion on the possible reasons, which I found acceptable. \n\nStrengths:\n\n1. First to propose to solve point cloud 4D panoptic segmentation with camera modality. \n\n2. Reasonably designed framework, fusion, prediction, and association modules. Incorporated recent progress in different subfields. \n\n3. Good results on the nuScenes dataset. \n\nWeakness:\n\n1. Confusing introduction of the soft-masked cross attention module. \n\n2. Mediocre SemanticKITTI performance.", "extra_scores": {"impact": "4: The work contains interesting new ideas that will potentially have major impact in robotics or machine learning."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposes an approach for 4D panoptic segmentation from camera images and LiDAR scans. The reviewers find the paper to be clearly written and addressing an important problem. Some concerns are convincingly addressed in the rebuttal whereas some regarding the incremental nature of the contributions still remain. Nevertheless, all the reviewers agree that the work is technically correct and well executed.
Decision: Accept (Poster)