Towards Learning to Complete Anything in Lidar
Authors: Ayça Takmaz
Cristiano Saltori
Neehar Peri
Tim Meinhardt
Riccardo de Lutio
Laura Leal-Taixé
Aljosa Osep
Venue: ICML 2025 poster (2025)
Keywords: Zero-shot segmentation; Lidar scene completion; Lidar perception
Primary area: applications->robotics
TL;DR: Zero-Shot Lidar Panoptic Scene Completion

Abstract: We propose CAL (Complete Anything in Lidar) for Lidar-based shape-completion in-the-wild. This is closely related to Lidar-based semantic/panoptic scene completion. However, contemporary methods can only complete and recognize objects from a closed vocabulary labeled in existing Lidar datasets. Different to that, our zero-shot approach leverages the temporal context from multi-modal sensor sequences to mine object shapes and semantic features of observed objects. These are then distilled into a Lidar-only instance-level completion and recognition model. Although we only mine partial shape completions, we find that our distilled model learns to infer full object shapes from multiple such partial observations across the dataset. We show that our model can be prompted on standard benchmarks for Semantic and Panoptic Scene Completion, localize objects as (amodal) 3D bounding boxes, and recognize objects beyond fixed class vocabularies.

Towards Learning to Complete Anything in Lidar
Ayça Takmaz 12 Cristiano Saltori'
Neehar Peri 13 Tim Meinhardt Laura Leal-Taixé
Riccardo de Lutio
Aljosa Osep
Complete Anything in Lidar (CAL)
Application D: Panoptic & Semantic Scene Completion
Application ② Amodal 3D Object Detection
Input Sparse
Output Completed Objects Semantic CLIP Features
Test. Time Vocabulary: "car, road, vegotation,
Query:
Application ④ Open-Vocabulle Vocabulary 3D Panoptic Completion
Query: "camper van"
Query: "oaa"
Query: "portable toilet"
Query: "laçade"
Query similarity:
low
high
Figure 1. Learning to Complete Anything in Lidar. Given a sparse I Lidar point cloud, CAL (Complete Anything in Lidar) localizes, reconstructs, and, optionally, recognizes objects a zero-shot fashion. By providing a semantic class vocabulary of specific object classes at test time, CAL can be prompted to perform Semantic Scene Completion (SSC), Panoptic Scene Completion (PSC), (amodal) 3D Object Detection. Note that CAL only takes single Lidar scan as input; RGB images are shown for visualization purposes only.
Abstract
object shapes from multiple such partial observa- tions across the dataset. We show that our model be prompted on standard benchmarks for Se- mantic and Panoptic Scene Completion, localize objects as (amodal) 3D bounding boxes. and rec- ognize objects beyond fixed class vocabularies.
We propose CAL (Complete Anything in Lidar) for Lidar-based shape-completion in-the-wild. This is closely related to Lidar-based seman- tic/panoptic scene completion. However, con- temporary methods can only complete and rec- ognize objects from a closed vocabulary labeled in existing Lidar datasets. Different to that, our zero-shot approach leverages the temporal con- text from multi-modal sensor sequences to mine object shapes and semantic features of observed objects. These are then distilled into Lidar-only instance-level completion and recognition model. Although we only mine partial shape completions, we find that our distilled model learns infer full
1. Introduction
Understanding the complete spatial layout and semantics of objects and scene geometry from raw sensor data is crucial for embodied 3D perception and safe navigation.
Contemporary methods for Lidar semantic (Behley et al., 2019) and instance (Behley et al., 2021) segmentation only group and classify points directly observable from the Lidar sensor. In contrast, methods for (amodal) 3D object detec- tion (Zhou & Tuzel, 2018). Semantic Scene Completion (SSC) (Behley et al., 2019; Li et al., 2024) and Panoptic Scene Completion (PSC) (Cao et al., 2024) learn to com- plete objects and scenes directly from labeled sensor data to predict occluded regions not directly observable Lidar. However, prior work can only localize and complete around
Work done during research intersship NVIDIA. 'NVIDIA ETH Zurich 'Carnegic Mellon University (CMU), Correspon- dence to: Ayça Takmaz wwyca.takmaze......a tat..aaaamma@nn.ftt.h...h...
Proceedings of the nd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

Towards Learning to Complete Anything in Lidar
20 classes labeled in existing Lidar datasets. This is far be- low the label diversity and scale compared to state-of-the-art image-based datasets (Kirillov et al., 2023).
Contributions. We propose the first method for Zero-Shot Lidar Panoptic Scene Completion. Our approach is enabled by our pseudo-labeling engine, which mines 3D shape priors from unlabeled Lidar sequences using 2D vision foundation models. We show that such pseudo-labels can be used to train a model for scene-scale object-level completion from noisy and partial pseudo-labels
Mining shape priors from unlabeled data. Unlike prior work, we extend beyond typical fixed taxonomies by learn- ing object shape priors from temporal context in Lidar se- quences. However, this requires (i) segmenting objects and regions in space and time, followed by (ii) temporal ag- gregation to obtain 3D shape estimates. Precisely tracking objects is crucial for 3D reconstruction: tracking errors like identity switches can lead to errnneos incomplete geom- etry. Even when objects are correctly localized in space and time, object-level temporal registration is challenging (Groß et al., 2019; Seidenschwarz et al., 2024; Huang et al., 2022).
2. Related Work
In the following section, we discuss prior work with a focus on Lidar-based scene understanding, scene completion, and generative modeling.
Lidar-based segmentation methods for semantic (Behley et al., 2019) and panoptic (Behley et al., 2021) segmenta- tion classify directly observed Lidar points into pre-defined semantic classes and identify individual instances. While most existing methods rely manually labeled datasets (Behley et al., 2019; 2021; Fong et al., 2021), several works (Unal et al., 2022; Li et al., 2023a) address weakly super- vised segmentation to reduce labeling efforts. More recent works embrace foundation models for open-vocabulay auto- labeling. Liu et al. (2023) use contrastive pre-training to distill rision-foundation model features for label-efficient segmentation. Osep et al. (2024) distill vision foundation models into zero-shot Lidar panoptic segmentation model. Peng et al. (2023); Xiao et al. (2024) similarly distill 2D foundational knowledge into 3D but rely on Lidar and cam- era inputs to classify Lidar/RGB-D points at test-time. Such modal scene recognition only localizes the visible portion of objects and is therefore sensitive to signal sparsity and (self) occlusions. In contrast, our method addresses zero-shot completion of shapes in Lidar, which comes with distinct challenges beyond the scope of segmention.
To address these challenges, we leverage image (Kirillov et al., 2023) and video (Ravi et al., 2024) segmentation foundation models localize and track objects in video. Such foundation models are already trained on diverse data and are capable of accurately segmenting any object in a video. After segmenting and tracking all objects. we lift each masklet (i.e. spatio-temporal object masks) to Lidar space and integrate them over time using a calibrated multi- modal sensor setup with known ego-vehicle poses (Behley et al., 2019). To enable vero-shot recognition, we addition- ally obtain CLIP (Radford et al., 2021) features for each masklet averaged per-timestep, effectively connecting the completed shapes with semantic information. Objects may be only partially completed, and not all objects are static however, in practice, we learn to fully complete static and dynamic objects from such partial observations.
Learning to complete anything. We utilize these mined pseudo-label pairs to train an instance-level completion net- work. Following prior methods for SSC (Behley et al., 2019), we learn scene-level occupancy in a fixed-size voxel grid using sparse generative encoder-decoder net- work (Cao et al., 2024). To segment individual object in- stances, we train a decoder (Cao et al., 2024), which predicts instance masks in the (occupied) voxel space in a class- agnostic fashion. Finally, we regress CLIP token (Najibi et al., 2023) for each predicted instance to capture object se- mantics in fixed-dimensional embedding vector, enabling zero-shot prompting at test time.
Lidar-based object detection methods localize objects as oriented 3D bounding boxes (Petrovskaya & Thrun, 2009; Yin et al., 2021; Liu et al., 2021; Ma et al., 2023; Peri et al., 2023a;b), including regions not directly observed by the Lidar sensor. As these methods require manually labeled 3D boxes, recent work (Najibi et al., 2022; 2023; Zhang et al., 2023; Seidenschwarz al., 2024; Khurana al., 2024) utilize foundational priors and temporal context to automatically obtain amodal 3D boxes for moving objects. In contrast, our work is not limited to the subset of thing classes observed in a state of motion we learn to segment and complete objects for any category.
We empirically demonstrate that our method is versatile and can be used for semantic (Behley et al., 2019; Li et al., 2024), and panoptic scene completion (Cao et al., 2024) (Fig. 1, @) via test-time prompts. Moreover, we qualitatively show that our approach can localize objects as 3D bounding boxes (Fig. ② and demonstrate that our method can recognize and complete arbitrary objects not captured in canonical semantic vocabularies (Fig. 1, ().
Object-level shape completion from partially observed scans (e.g., a single viewpoint) is commonly addressed us- ing data-driven methods that rely on object shape priors (Yuan et al., 2018; Dai et al., 201 7b; Mescheder et al., 2019; Park al., 2019; Mittal et al., 2022). Prior art utilizes gener-
2

---

Towards Learning to Complete Anything in Lidar
Vidvo-object localization
2D-1o-3D lift and refinement
Accumulated labeled points
Reffned pscsado-lsbel lubels
Mined pseudo-label pairs
Sparse Lidar
Completed shages + CLIP features
CRF
refinemmntt
deforeod Gaxe
Completed shapes CLIP features
Hinary occupancy
iddr sequence
2D-tn-3D licting
"truck"
'CEE"
Toad
Pseudo-labeling engine
low similarity
high similarity
Figure2 2. Pscudo-labeling engine. Given a calibrated RGB camera and Lidar sensor, ! we use video-object segmentation models (Ravi al., 2024) to localize object instances in video, @ pseudo-label the Lidar point clouds over time, and ( generate completed voxclized object representations, cach enriched with per-instance CLIP fcature extracted from RGB images. In @ wc accumulate 360 Lidar scans to obtain full-scene binary occupancy, used for refining the aggregated pseudo-labels A via CRE-guided label refinement process ① As output @. our method pairs each sparse and incomplete Lidar scan with pseudo-labels labels for object-level scene completion (top-right) and CLIP features, which are temporally aggregated by averaging per-instance features across the sequence. These CL.IP features enable zero-shot recognition via text queries (hottom-right). Mined pseudo-label pairs are then used to train the CAL model.
ative models such adversarial networks (Wu et al., 2016; Achlioptas et al., 2018), auto- entoeenccders (Wu et al., 2015; Dai et al., 201 7b; Yuan et al., 2018), 8), diffusion models (Vahdat et al., 2022; Nam et al., 2022; Ntavelis et al.. 2023; Jun & Nichol, 2023), and vector-quantized auto-encoders (Van Den Oord et al., 2017; Mittal et al., 2022). Such models commonly trained on datasets that provide partial obser- vations and corresponding 3D meshes (Chang et al., 2015; Deitke et al., 2023), or via data augmentations (Dai et al., 2017b; Mittal et al., 2021). In contrast, we address object and scene-level completion via auto- labeling.
occupied voxels to estimate set of segmentation masks for each (learned) query vector. Although our model is inspired by Cao et al. (2024), our work goes beyond fixed class taxonomies, and does not require 3D datasets with manual labels. Our work addresses these limitations by extracting geometry from temporal cues and open- vocabullyy seman- tics from foundational priors. Finally, open-vocabulary SSC from multi-view images has been explored in Vobecky et al. (2023); Zheng et al. (2024). To the best of our knowledge, no prior work combines open-vocabulary, instance-level, and LiDAR- bassed scene completion in a single framework.
Scene-level completion methods typically represent la- beled, pre-registered scans of static environments using dense voxel grids, which hinders scalability to large scenes (Dai et al., 7a; 2018). To address this, Dai al. (2020) utilize sparse encoders in conjunction with coarse-to-fine generation in feed-forward manner, while Ren et al. (2024) perform diffusion in the latent space. This differs from streaming robot perception, where range sensors provide sparse measurements (Song et al., 2017; Behley et al., 2019; Li et al., 2024) and methods must infer dense scene geom- etry in the presence of dynamic objects. Li et al. (2023b) explore implicit scene completion of sparse Lidar data, whereas Nunes et al. (2024) explore diffusion-based com- pletion of Lidar point clouds. Prior work addressing se- mantic scene completion (Xia et al., 2023; Liang et al., 2024; Roldao et al., 2020; Rist et al., 2021; Mei et al., 2023) requires labeled datasets (which are expensive to curate) (Behley al., 2019; Li al., 2024; Tian et al., 2024) with amodal completion data.
3. Method
We describe our task formulation and method CAL (Com- plete Anything in Lidar) for zero-shot completion of objects in a sparse Lidar scan. CAL has two key components: (i) a pseudo-labeling engine (Fig. 2) that mines pairs of par- tially observed point clouds with completed 3D shapes and CLIP features (Radford et al., 2021), and (ii) a model for zero-shot, class-agnostic object completion (Fig. 3).
Preliminary: Panoptic Scene Completion. Semantic Scene Completion (SSC) (Behley et al., 2019) assumes input in the form of single Lidar point cloud P (pm W. Pa E RM, consisting of spatial positions and inten- sity channel. Given a set of semantic classes (known at train time and accompanied by labeled instances). the task is to estimate per-class scene occupancy. In addition, Panoptic Scene Completion (PSC) (Cao et al., 2024) requires assign- ing K instance identities to semantic classes designated as thing classes. Object shapes are localized and parametrized via a per-object occupancy function O₂ IR + N3, k < K, defined in regular voxel grid G c N in fixed-size bound- ing volume in front of the Lidar sensor.
Cao et al. (2024) jointly tackle SSC and instance segmen- tation (L.., PSC) using a generative encoder-decoder in conjunction with a transformer decoder that interacts with
3

---

Semantic feature aggregation. We compute per-instance CLIP features following Ding et al. (2023), and aggregate (Takmaz et al., 2023) them in the temporal domain using mask information to obtain multi-view vasion-language fea- tures. These features are then normalized and averaged to Towards Learning to Complete Anything in Lidar
Complete Anything. Our problem setting follows the gen- eral setting for PSC. However, particular to our setting, semantic information is not provided during training. Our method takes a semantic vocabulary consisting of free-form semantic class descriptions only at test time. We address this challenge by performing class-agnostic segmentation and reconstruction of objects, optionally followed by zero-shot classification, that assigns each instance a semantic class in the specified test-time vocabulary.
obtain S& E Rf for each accumulated shape & in the refer- ence frame t, where F is the CLIP embedding dimension.
Lift and refine @ For each frame in the temporal window, we backproject each 2D mask m2 to the Lidar coordinate frame using the provided camera-to-Ie Lidar transformations and obtain 3D masks 3D E {0, 1]xXXXX2 X 0 < k < K for each instance k. These masks may suffer from projec- tion artifacts due to imperfect calibration and rolling shutter. We follow the single-frame mask refinement procedure pro- posed by Osep et al. (2024) for precise localization (details in Appx. A.3).
3.1. Mining 3D Shape Priors From Unlabeled Data
Overview. Given a Lidar sean with partial scene obser- vations, our shape-mining pseudo-labeler creates an occu- pancy grid O where each voxel can be empty, unlabeled, or assigned to an observed instance. Each instance is repre- sented by a 3D occupancy grid O₁,, accumulating per- per-point instance observations over time. Additionally, each O₂ is accompanied by a semantic CLIP feature fu, that connects instance geometry with vision- lsionnlanguage features (Radford et al., 2021). We depict our approach for mining shape priors in Fig. 2.
Temporal aggregation D Once masks migh lo- calized in the Lidar sequence, we project them into the reference coordinate frame using known ego-poses and ag- gregate them over time to obtain densified masks m 3D We obtain per-instance occupancy O₂ by voxelizing each m and assign instance indices via majority voting. Relying solely on O₂ provides limited guidance. This due (i) image-based object localization providing only partial cov- erage of Lidar scans where unseen regions (e.g., the back of car) lack completion signals and (ii) errors in mask localization and tracking. We complement O₂ with a binary occupancy signal obtained by directly accumulating 360* Lidar points (Fig. 2 ) to improve label coverage (Tab. 5).
Key challenge. Our approach stems from the observa- tion that object shapes can be inferred from temporal con- text while driving down the street. However, obtaining per-object shape priors requires precise object localization, tracking and accurate object-level registration - a challeng- ing problem sensitive to registration errors and drift (Huang et al., 2022; Seidenschwarz et al., 2024; Groß et al., 2019).
CRF refinement 5 To further improve label coverage, we refine instance masks i using Conditional Random Fields (CRF) (Krähenbühl & Koltun, 2011), constructed over binary occupancy + We analyze the benefits of this CRF refinement (whose final output is illustrated in Fig. 2 ② in Tab. 3 and report further details in the Appx. A.
Video-object localization O. To this end, we utilize seg- mentation foundation models (Kirillov et al., 2023; Ravi et al., 2024; Radford et al., 2021), trained on a large amount of diverse visual data and proven capable of segmenting ar- bitrary objects. Given a video sequence v consisting of RGB images I E RR x333, we first segment images with SAM (Kirillov et al., 2023) using grid-based prompting and obtain set of 2D mask proposals m E (0,1) 0 k < K, that represent the set of < K objects in the reference image, observed at time t. We propagate the masks from the reference frame at time to the frames within the given temporal window [t- - Thw, t + Ty using video-object seg- mentation model SAM (Ravi et al., 2024). Specifically, we perform backward propagation for Thw frames and forward propagation for T frames, with a stride of w. This allows us integrate observations from various viewpoints. The output is set of < K class-agnostic masklets, providing temporal instance association in the video v.
3.2. Learning To Complete Objects
Overview. CAL takes single input Lidar scan P. provid- ing sparse and incomplete observations of scene geometry (Fig. 4, 1st col.), and outputs set of completed object in- stances (Fig. 4, 2nd col.), represented via voxel occupancy and instance IDs. Each instance ID is accompanied by a predicted semantic feature that allows us to match instance semantic features with the class vocabulary provided at test time (Fig. 4, 3rd col.). Different from prior art (Cao et al., 2024), we do not differentiate between stuff and thing classes, i.e., cars, road, and (individual) bushes are treated as individual instances. CAL overview is provided in Fig. 3.
Model architecture. We follow Cao et al. (2024) and employ a genes--generativeeeee 3D U-Net (Dai et al., 2018) architecture that estimates scene-level occupancy, and a Transformer instance decoder (Cheng et al., 2022) operating directly on occupied voxels. The backbone consists of a sparse feature encoder (0) (Choy et al., 2019) followed by a dense 3D convolutional block The multi-scale genera-
4

---

Towards Learning to Complete Anything in Lidar
CLIP Festure
cupancy completion, (ii) class-agnostic instance mask pre- diction, (iti) CLIP feature distillation, and (iv) per-voxel prototype assignment. During each training iteration, the generative decoder produces coarse-to- fine- voxel grids for each scale I,, supervised with a binary occupancy loss (Coce: binary-cross entropy wrt. aggregated binary oc- cupancy labels) and prototype classification loss (Epros: cross-entropy and Lovasz wrt. pseudo-category labels from CLIP prototype assignments). The transformer decoder produces instance masks and CLIP features, supervised by the mask-loss (fmask: binary-cross entropy and Dice loss) and the feature distillation loss (CCLIP: cosine sim- ilarity loss). For Luask. we perform Hungarian matching to pair predicted and pseudo-labeled instance masks. We ignore unlabeled voxels for Hungarian matching, and also for computing Lmask. and Eprot to ensure the network is not penalized for predictions outside the pseudo- labeled area. The final training objective is the weighted sum of {Loces Eprot, LCLIP, } with weighting parameters specified as {Aoce: Aprot ACIIP, Amask }- Further imple- mentation details are provided in Appx. B.3.
Loss
Loss
Procatypes
Rinary
cassilication
occupancyy
Pseado-Labbls
Occupancy Insaacce Labels
Genernaivve
CLIP Fext
(pr-maok)
Fondar
Desse
Decocer
Block D
Smak
Objectass
Tepur: Lādae Point Cloud
Transformer Decnder
pediciiom
Mask
-Loss Monk
CLIP
Lony
Output: Complated Ohjeets
& CLIP Features
Figure 3. CAL model architecture and training pipeline. The backbone consists of a sparse encoder and dense 3D convolu- tional block. We estimate scene-level occupancy using a multi- scale sparse generative decoder that consists of decoder blocks D, two occupancy heads B., and B., and a pscudo-semantic head (S) each scale L.. The Transformer decoder then predicts segmenta- tion masks over the completed scene and regresses CLIPP features.
tive decoder uses three decoding blocks D estimating occupancy at three different resolution levels L E {1, 4). The input to the Transformer decoder is set of learn- able queries that interact with the multi-resolltion features learned by the generative decoder in voxel space. To ac- commodate zero-shot recognition, we estimate an object- ness score, a segmentation mask (parametrized in the voxel space), and CLIP features for each query.
Inference. Given a Lidar point cloud as input, CAL pro- duces set of object instance masks over the voxel grid and CLIP feature for each predicted instance. We suppress small overlapping masks with the overlap threshold Towr and filter low confidence masks with the objectness thresh- old Tobj (Appx. B.2). We use the predicted CLIP features to classify each query in a zero-shot manner. Using the CLIP text encoder, we encode the semantic vocabulary, i.e., set of text prompts (additional details in Appx. C.1). We then obtain a posterior over the specified vocabulary for each query by computing the cosine similarity between a predicted CLIP feature and encoded text descriptions.
Generative decoder, completion heads & CLIP feature prototypes. The key component for accurately complet- ing the scene is the sparse generative decoder (Dai al., 2018; Cao et al., 2024), comprising three decoder blocks interleaved with pruning layers to maintain sparsity. Each decoder block D II provides features at scale L, used by a binary occupancy head B, to predict scene occupancy at that scale. Training the current model with pseudo-labels presents two key challenges: partially completed pseudo- labels bias training toward well-covered regions, and the lack of a semantic grouping of instances hinders the learn- ing of shape priors. We address the first issue by guiding B. with binary occupancy (Fig. 2 @). We address the sec- ond challenge by quantizing instance CLIP features into C² ceeudo-protoypess with clustering and introduce an addi- tional (pseudo) semantic head S to predict prototype class for each occupied voxel. We use prototype predictions only during training (additional details in Appx. B.2) during test-time, we only use the predicted CLIP features for zero- shot recognition. Additionally, we add a second occupancy head B.. which processes per-class logits from and learns to convert them into a binary occupancy prediction. We find that B, further regularizes our training.
4. Experiments
We evaluate CAL's ability to localize and complete the full 3D extent of instances from a Lidar point cloud given a class vocabulary prompt at test time. Key details on the experi- mental setup (Sec. 4.1), benchmark comparisons (Sec. 4.2), and ablations on design choices (Sec. 4.3- 4.3) are discussed below, with further implementation details in the Appendix.
4.1. Experimental Setup
Task. We quantitatively assess CAL's zero-shot completion and recognition performance on Semantic Scene Comple- tion (SSC) (Behley et al., 2019) and Panoptic Scene Com- pletion (PSC) (Cao et al., 2024) benchmarks. SSC assumes single Lidar point cloud as input and requires vox-v occupancy estimation and semantic classification of the occupied voxels. PSC additionally requires assigning iden- tities to instances of thing classes, Given a class vocabulary
Training. We train our network jointly for (i) binary oc-
5

---

Towards Learning to Complete Anything in Lidar
Input Scan
Completion + Masks
Completion + Prompting
Ground Truth
Figure 4. Qualitative results on SemanticKITTI. Given a single Lidar scan (1 col.). CAL completes object-level observations as a set of masks over the voxel grid (2nd col.) and predicts CLIP feature for each mask. We can prompt with any semantic class vocabulary and obtain panoptic and semantic scene completion (3rd col.) results. Our model predicts shape priors for both thing (e.g., 'car'. 'cyclist") and stuff classes (e.g. 'vegetation', 'road') y and can correctly predict the intersection geometry in 4th row, despite limited direct evidence.
Table Panoptic Scene Completion. We compare CAL against LMSCNet (Roldao et al., 2020) + MaskPLS (Mareuzzi et al., 2023). JS3CNet (Yan et al., 2021)+ + MaskPLS, SCPNet (Xia et al., 2023) + MaskPLS, and PaSCo (Cao et al., 2024) (M=1 and Ensemble).
Semantic KITTI
(Behley al., 2019) (vail set)
SSCBench KITTI360 (Li et al., 2024) (test
-SCBench-KITT1300
All
Thing
Stuff
All
Thing
Stuff
Method
PO
PQT
SQ
RQ
PQ
SQ
RQ
PQ
SQ
RQ
mloUt
P
PO1
SQ
RQ
PQ
SQ
RQ
PQ
SQ
RQ
mloUt
Filly supervised
LMSCNet MaskPLS
13.81
4.17
36.13
6.82
1.62
29.87
2.68
6.02
40
60.69
9.82
1740
12.76
4.14
26.52
6,45
088
20.41
1.58
5.78
29:58
8,88
15.10
JS3CNer MaskPLS
18.41
6.85
41.90
11.34
4.18
43.10
7.22
8.79
41
03
14.34
22.70
16.42
6.79
51.16
10.31
3.36
48.41
5.83
8.51
52.54
13.15
21.31
SCPNet MasPLLS
19.39
49.49
46.41
7.70
11:30
51.73
18.04
7.09
52.55
11.70
PaSCo (M-I)
26.49
15.36
54.15
23.65
12.33
47.4
18.78
17.55
59.05
27.19
28.22
19.53
9.91
58.81
15.40
3.46
57.72
6.10
13.14
59.35
20.05
21.17
PaSCo (Ensemble)
31.42
16.51
54,25
25.13
13.71
48.07
20.68
18.54
58.74
28.38
30.11
26.29
10:92
56.
10
17.09
4.88
57.53
8.48
13.94
55.39
21.39
22.39
Zero-shor
CAL (SO)
17.12
6.27
43.40
10.06
3:48
44.39
5.65
8.30
42.67
13.27
20.71
12.56
1.71
33.18
3.10
2.05
45.57
3.76
1.54
26.99
2.76
13.34
CAL (ZS)
13.12
5,26
27.45
8.44
2/42
22.79
3.89
7.33
30.84
11.76
13,09
8:57
1.46
21.01
2,63
1.39
27.62
2.54
1.49
17.81
2.68
8.49
for evaluation, we prompt our method at inference time to perform these tasks in zero-shot manner.
our model is trained to localize and classify larger set of objects (i.e., objects that appear in the Lidar data and are localized by vision-based segmentation foundation models, Kirillov et al. (2023)), we utilize labeled instances as a proxy for assessing zero-shot (ZS) shape completion and recog- nition on classes that are labeled. In contrast to baselines trained ground-truth (GT) data, we use GT labels solely for evaluation and ablations.
Datasets and benchmarks. We follow prior work (Cao et al., 2024) and evaluate CAL on two datasets that provide semantic and instance-level labels for PSC: SSCBench- KITTI360 (Li et al., 2024; Liao et al., 2021) and Se- manticKITTI (Behley et al., 2019; Geiger et al., 2012; 2013) whose instance-level labels are provided by Cao et al. (2024). The hyperparameters used by our pseudo-labeling engine for each dataset are given in Appx. A, and additional details are provided in Appx. C.I.
Metries. We assess SSC performance using mean Intersection-over-Unin (mloU). For PSC, we follow Cao et al. (2024) and use the Panoptic Quality (PQ = SQ x RQ), Segmentation Quality (SQ), and Recognition Quality (RQ) metries (Kirillov et al., 2019), evaluated on the full voxel grid. Similar to Cao et al. (2024), we focus on the modified variant of PQ, i.e. PQ' (Behley et al., 2021), and remove the minimum 0.5 IoU overlap requirement for stuff classes, as this can be too restrictive for regions that do not have well-defined boundaries. As PQ mixes seg-
We evaluate our model's segmen- tation, completion, and recognition capabilities by specify- ing target classes (defined in each respective dataset) via prompts at test time (additional details Appx. C.I) and assign labels based on cosine similarity between encoded text prompts and the predicted CLIP features fx. While
Zero-shot evaluation.
6

---

Towards Learning to Complete Anything in Lidar
Table 2. Panoptic scene completion results with zero-shot base- lines. We compare CAL against the zero-shot baselines we con- struct: LODE (Li et al., 2023b) SAL (Osep et al., 2024) and LiDiff (Nunes et al., 2024) SAL (Osep et al., 2024). Results reported on the SemanticKITT dataset.
All
Thing
Stuff
SSC
Method
PQ'+
PQr
RQ
PQ
SQ
SQ
RQ
mloUt
LODE SAL
7.74
1.96
11.12
3.54
1.000
6.36
0.00
3.39
14,59
6,11
8.12
LIDiI's SAI.
7.35
0.36
23,95
066
0.22
34.81
16.06
0,83
7.38
CAL (ZS)
13.12
5,26
27.45
8:44
244
22.79
3.89
7.33
30.84
11.76
13.09
in the supervised setting'. Remarkably, we reach ~ 50%
of PaSCo on SemanticKITTI, and ~ 40% of PaSCo on SSCBench-K/TT1366 in the zero-shot (ZS) setting, while even achieving comparable results to fully-supervised base- line LMSCNet + MaskPLS. Specifically, we achieve 13.12 PQ (49.51 % of PaSCo) and 13.09 mIoU (46.37 % of PaSCo) in the 7S setting SemanticKITTI and further im- prove to 17.12 PQ' (64.63 % of PaSCo) with the semantic oracle (SO). Similarly. CAL achieves 8.57 PQ (43.87 % of PaSCo) and 8.49 mIoU (40.11 % of PaSCo), narrowing the gap to 17.12 PQ' (64.63 %) with SO on SSCBench- KITTI-360. We find that the gap between CAL and the supervised baselines is largely due to zero-shot recogni- tion performance, which is limited by the underlying vision foundation model. In our per-class analysis (further details in Appx. C.3), we also see that the gap between CAL and the supervised baselines is affected by rare classes (c.g.. pedestrian and cyclist).
Figure 5. Completion and amodal detection on KITTI-360. Given input Lidar scan (left), CAL outputs set of completed object shapes (middle). We visualize recognized objects (right) for queries 'vehicle' (top), 'car' (middle) and 'tree' (bottom), and fit 3D bounding boxes to the identified object instances, demonstrat- ing the zero-shot amodal 3D object detection ability of CAL,
mentation and recognition accuracy, we additionally report semantic oracle (SO) results (similar to Osep et al. (2024)) by assigning each mask to the GT semantic label based on a per-voxel majority vote. This allows to further decouple completion performance and semantic understanding.
4.2. Experimental results
Comparison to zero-shot baselines. As there are no prior works tackling Lidar PSC in zero-shot setting, we construct two baselines adhering to the following criteria for a fair zero-shot comparison: (1) input should be single Lidar scan, (2) scene completion model should be trained without semantic labels, and (3) instance prediction and semantic inference should rely on zero-shot recognition. Accordingly, we combined recent Lidar completion methods that were trained without semantic labels- LODE (Li et al., 2023b) and LiDiff (Nunes et al., 2024) - with SAL (Osep et al., 2024), a zero-shot panoptic segmentation model.
First, we compare CAL with four fully supervised baselines in Tab. 1. To the best of our knowledge, method is the first method to address zero-shot Lidar panoptic scene com- pletion; therefore. we also construct two zero-shot baselines, whose results are reported in Tab. 2. Additionally, we report qualitative results on SemanticKITTI in Fig. 4 and Fig. 6, and on
in Fig. 5 and Appx. Fig. 7.
Comparison to supervised baselines. Tab. reports CAL results for SSC and PSC on the SemanticKITTI and SSCBench-KITT1366 datasets. We also compare with four fully supervised baselines. We report two versions of the PaSCo baseline, the current ofte-of-the-an method for fully supervised SSC/PSC: single network (M- = 1), whose set- tings are consistent with ours, and the ensembled version (Ensemble). reported only for completeness. Fully super- vised baselines have clear advantage over CAL as they train on closed-set, noise-free annotations with full scene coverage. Nonetheless, even early supervised SSC base- lines (Behley et al., 2019) struggle, often achieving less than 15 m outlining the difficulty of the task even
LODE performs implicit scene completion from sparse Li- dar, trained with ground-truth completion data. We employ the LODE variant that does not use any semantic labels. We extract a surface mesh from its output, convert it to an occupancy grid, and propagate SAL's zero-shot per-point panoptic labels to occupied voxels (LODE+SAL, Tab. 2). LiDiff is a diffusion- based completion method that learns to complete Lidar point clouds from GT completion data without semantic labels. Similarly, we convert its output to an occupancy grid and propagate SAL's zero-shot panoptic labels to voxels (LiDift+SAL, Tab. 2).
The SSC baselines reported in (Behley et al., 2019) achieve 9.53 mloU (SSCNet (Song al., 2017)). 9.54 mIoU (TS3D (Gar- bade et al., 2019)), and 17.70 mloU (w/ SATNet (Liu et al., 2018))
As can be seen in Fig. 6 and Tab. 2, our method outperforms zero-shot baselines across nearly all metrics. Notably, while
7

---

Towards Learning to Complete Anything in Lidar
Input Scan
LiDiff + SAL
LODE + SAL
Ours (CAL)
Ground Truth
Figure 6. Comparison to zero-shot baselines on SemanticKITTI. Given single Lidar scan (1" col.), we compare CAL (4th col.) to zero-shot baselines (2nd and 3rd cols.) combining LiDiff (Nunes al., 2024) and LODE (Li et al., 2023b) with SAL (Osep et al., 2024).
Table CRF refinement ablation. We evaluate pseudo-label quality with and without CRF refinement SemanticKITTI and SSCBench- KITTI360. Results show that CRF refinement significantly improves pseudo-label quality in both datasets and settings.
Semantic KITTI (Bebley al., 2019) (val
SSCBench- KITTII-0 (li et al,, 2024) (test set)
All
Thing
Stuff
All
Thing SQ
Stuff
Method
pQ't
PQt
SQ
RQ
PQ
5Q
RQ
PQ
SQ
RQ
mlooU
PQ 1
PQt
SQ
RQ
PQ
RQ
PQ
SQ
RQ
mloUU
Psnudo-Label
CAL (SO)
12.78
5.59
46.22
9.39
5.00
49.79
8.56
6.02
43.63
10.00
17.77
11.48
4.44
35.06
7.14
3:24
47.17
5.81
5.04
29.00
7.81
13.22
CAL (ZS)
9.34
3.45
35.07
5.83
1.75
48.14
3.08
4.69
25.56
7.83
12.79
9.37
3.66
28.80
5.82
1.67
47.23
3.00
4.66
19.58
7.22
9.72
Pseudo label CRF
CAL (SO)
25.90
17.71
64.06
26.55
16.93
67.91
23.75|
18.28
61.25
28.58
13.10
14.75
4.57
40,79
7.98
7.08
48.18
12.35
3.32
37.10
5.79
17.14
CAL (ZS)
16.98
10.67
54.02
16.19
7.30
67.19
10.21
13.12
44.43
20.54
20.62
10.98
3.18
33.95
5.58
3.91
48.30
6.86
2.82
26.78
4.95
11.04
Table 4. Pseudo-labeling engine ablations, semantic oracle (SO). Pseudo-labels benefit from forward and backward propagation, with notable improvements up to I w
Table 5. Coverage analysis. Coverage of mask pseudo-labels (w/o CRF, Label) and binary occupancy (w/o 360 aggr., Occ.).
32 and T Tim. 8 frames.
SemanticKITTHeall
SSCBench. KITTI36 (val)
Parameters
All
Thing SQ
Stuff
SSC 1607
CRF
Label
360
Occ.
CRF
Label
360 o
Occ.
Tjo
PQ
PQT
SQ
RQ
RQ
PQ
mle01
Coverage %
Ager.
Coverage %
Coverage %
Agar
Coverage %
X
28.04
x
37.36
27.38
44.15
7.79
0.75
36,62
1.32
1.41
46.97
2.42
0.42
31.44
0.77
13,48
8.82
70.13
/
99.96
50.48
/
67.63
16
9.67
2.91
34.13
5.07
2.46
79
4.32
3.13
27,80
5.44
18.98
10.97
10.48
4.11
34.56
6.76
2.73
47.76
4.74
4.80
25.41
7.77
22,02
12.05
10.49
4.15
34.86
6.00
2.74
47.85
4.75
4.85
28.37
7.83
22.09
12,10
and only marginal improvements using Tyu = 48 (10.49
10.48
34.86
6.76
47.76
4.74
25.41
7.77
22,02
12.05
PQ*). Increasing Tow also improves performance. We find that the the best combination is T Tow 8 and Tyw = 32
11,65
4:80
41.08
7.77
3:51
56.61
6.2%
5:41
33.24
24.53
13.32
12:21
5.08
40.84
8:20
3.96
47.4
6.97
5.63
37.55
8.81
25.64
13.93
10.97
4.13
37.28
7:00
2.98
55.00
5.35
4,71
28.39
1.88
22.10
12.55
(12.21 PQt). Interestingly, backward propagation improves completion of partially visible instances near the reference camera. We observe no significant improvements between t = [1, and a degradation in performance when in- creasing to w = 4 (10.97 PQ') due to the large temporal gap the video propagation model (Ravi et al., 2024) must bridge. While the best results are achieved with T = 64, Tww 16, = 1 (13.10 PQt), we use the combination T fw 32 The 8, = 2 in our experiments due to the improved runtime of pseudo-labeling.
47.4
6.17
57.55
25.64
13,93
27.96
the baselines leverage completion models trained fully completed GT, our approach excels despite using pseudo- labels with only partial coverage. This highlights that zero- shot panoptic Lidar scene completion is challenging and not trivially addressed by prior art.
Completion and amodal 3D detection. Our method com- pletes the full amodal extent of objects, highlighting its potential for zero-shot amodal perception tasks. We qualita- tively assess CAL for amodal 3D object detection in Fig. 5 by fitting 3D bounding boxes to the recognized objects.
Refinement with CRF. We study the pseudo-label quality with & w/o CRF refinement (Fig. 2 )) and report results in Tab. 3. Due to the limited visibility within a camera frustum, our pscudo labels cover approximately 28% of the binary occupancy obtained with full 360 Lidar scans (Tab. 5). CRF refinement greatly improves pseudo-label quality on SemanticKITTI and SSCBench-KITTi3d datasets (Tabb. 3). For instance, we notice an improvement from 12.78 PQ' to 25.90 PQt in the SO setting on SemanticKITTI.
4.3. Pseudo-labeling engine analysis
Temporal mask aggregation. In Tab. 4, we evaluate pseudo-label quality w.r.t GT labels, using PSC metrics in the SO setting (ZS results are provided in Appx. C.2). We ablate the number of frames used for forward Ty and backward The propagation and stride w. We perform this analysis before CRF refinement, directly using the output from Fig. 2 ④ We notice that propagation is beneficial in both forward and backward directions. We observe an increasing improvement up to Tjw = 32 (10.48 PQt)
Pseudo-label coverage. In Tab. 5, we study the pseudo- label coverage (Label Coverage %) with and w/o CRF re- finement and binary occupancy coverage (Occ. Coverage %) with and w/o full LiDAR scan aggregation. Pseudo- label coverage is lower without CRF due to limited visi-
8

---

Towards Learning to Complete Anything in Lidar
Table CAL model ablations. We analyze the contribution of CAL's 's key design choices and components: training B. with par- tial coverage (Bg") or with full coverage (B!*), introducing S, and adding B,, Introducing S provides significant improvement, likely due to its implicit semantic regularization. Training with full coverage (B.") and B, further improve performance.
Table 7. Number of CLIP prototypes. We evaluate SSC/PSC performance on SemanticKITTI when varying the number of CLIP prototypes C. We observe similar performance with C' {6, 18, 50, 100). indicating general robustness to C. Extreme cases (C = and C 500) result in performance degradation.
All
Thing
Stuff
SSC
Training Components
AlI
Thing
Staff
SSC
C
PQ
+
PQT
SQ
RQ
SQ
RQ
PQ
SQ
RQ
mloUt
D/-
B.
p
PQT
SQ
BQ
PQ
SQ
RQ]
PQ
SQ
RQ
mloU?
3.73
0.01
5.68
0.01
0.00
0.02
Semnanic ommlee
x
X
4.81
0.01
R44
0.01
0.0
7.10
800
0.02
9.38
0.04
4.84
6
12:22
4.74
24.36
7.61
2.16
22.92
3.30
6.61
26.41
10.75
11.07
X
X
16.08
5.25
41.81
8,67
3:46
51.62
5.00
6.54
54.72
10,50
20.80
18
13.12
5.26
45
8.4
22.79
7.33
30.84
11.76
x
17.01
6.00
$4.09
9.71
5:14
51.08
5.1
8.12
40:05
13.05
21.08
50
12.62
4.69
27
98
7.00
1.83
23.93
2.82
6.77
30.93
11.07
12:57
17.32
6.27
43:40
10.06
3.48
44.39
5.65
5.30
42,67
13:27
20.71
100
13.16
4.81
$2.82
7.90
1.82
29.07
2.83
6.99
35.55
11.59
12.55
CLIP Sonuntics
500
4.76
1.59
5.65
2.82
0.00
0.00
0.00
2.75
9.75
4.88
4.45
0000
0.0
4.66
0.00
3.15
11.98
23.73
262
19.90
13.12
not constrain the types of objects that can be completed and recognized. For example, if car and van are in the same pseudo-class, our instance decoder can distinguish them as individual instances, while their estimated CLIP token allows us to recognize them as instances of different classes. Finally, we note that our approach under-segments rare semantic classes due to k-means-based CLIIP feature space quantization.
bility with the camera frustums (Sec. 3.1). CRF improves this coverage by 1.9x on WSCBench-KITTT366 and x.5x on SeTTnticKITTI. Similarly, binary occupancy coverage benefits from full Lidar scan aggregation, improving cover- age from 37 36% to 99.96% on SemanticKITTI and from 44.15% to 67.63% on JSCBBnch--ITTT1660
4.4. CAL model analysis
Limitations. While our method demonstrates strong per- formance under zero-shot conditions, several limitations remain. Pseudo-label accuracy depends on the robustness of video foundation models, which can suffer from tracking errors such as ID switches and recognition failures, espe- cially under significant view changes or occlusions. These errors can lead to incomplete or noisy label coverage, par- ticularly in poorly visible regions. Another limitation is the performance gap between our zero-shot approach and fully- sullyvsuee methods, driven by challenges in zero-shot recognition and lower pseudo-label coverage due to camera- LiDAR visibility constraints. To address these limitations, future work could explore improved temporal association us- ing geometric cues, integration of improved foundation mod- els, and strategies for scaling to larger unlabeled datasets to further close the gap with supervised basclines.
Model ablations. In Tab. 6, we analyze the effects of the key CAL design choices and components: training B, with partial coverage (B training B, with full coverage (B!9), introducing the (pseudo) semantic head S, and the additional binary head B,, Tab. 6 reports the SO and ZS results for CAL trained on the SemanticKITTI dataset; we Training
observe similar results for SSCBench- with partial coverage pseudo-labels (Bge) achieves the worst performance of 4.81 PQ due to the model overfitting on common and fully visible instances. Introducing S improves the results to 16.08 PQ' and 11.98 PQ' in the SO and ZS settings, respectively. This highlights that CLIIP feature prototypes are beneficial for learning alass-agnostic shape priors. Introducing full coverage guidance (B("") and B, improves performance most settings.
5. Conclusion
CLIP prototypes. We study CAL performance when vary- ing the number of CLIP prototypes C (Sec. 3.2) and report results on SemanticKITTI (ZS) in Tab. 7. We observe that the performance is not particularly sensitive to the num- ber of clusters C E {6, 18, 50, 100}. Specifying C' = 18 clusters (close to the number of annotated semantic groups in common datasets) yields the highest overall PQ. The extreme case of C = shows significant performance degra- dation, demonstrating the benefits of CLIP prototypes for learning semantic occupancy priors. Unsurprisingly, perfor- mance is negatively affected when trained with too many clusters (C = 500). In practice, the number of clusters does
We propose the first method for Zero-Shot Lidar Panoptic Scene Completion, capable of completing objects in a sparse and incomplete Lidar scan. The key components of CAL are a pseudo-labeling engine that mines completed shape priors with queryable semantic features, and a zero-shot model trained on pseudo-labels. Our work takes a step towards learning shape priors from temporal context, laying the foun- dation for amodal perception. However, our labeling engine remains computationally expensive, and despite CRF refinement, label coverage is limited. A potential solution is leveraging self-supervised LiDAR forecasting to improve label coverage in fully unobserved regions. We believe these are promising directions for future work.
"The discrepancy likely originated from undocumented point accumulation strategy, which we could not clarify with authors.

---

Towards Learning to Complete Anything in Lidar
Acknowledgements
Dai, A., Chang, A. X., Savva, M., Halber, M., Funkhouser, T., and Nießner, M. Scannet: Richly-annotated 3d recon- structions of indoor scenes. In IEEE Conf. Comput. Vis. Pattern Recog., 2017a.
We thank Zan Gojeic and Dávid Rozenberszki for their valuable feedback and comments!
Impact Statement
Dai, A., Ruizhongtai Qi, C., and Nießner, M. Shape comple- tion using 3d-encoder predictor preeccccr-reedictoo cmns and shape synthesis. IEEE Conf. Comput. Vis. Pattern Recog. 2017b.
We present method for performing zero-shot object com- pletion in Lidar scans, with possible applications in robotics and autonomous driving. Our method benefits from existing large vision-language models and vision foundation mod- els, and relies on their generalization ability connect text queries to our model's zero-shot object-completion output. Our work is intended as initial step towards achieving such abilities, and we acknowledge that in its current form our method shares the limitations of the foundation models we rely on, restricting its operational robustness in safety critical enviromments. As these are not specifically our method's limitations but instead limitations of the founda- tion models, such consequences are already discussed in detail in the respective technical reports of the foundation models we employ. We believe there are no particular soci- etal consequences we must specifically highlight here.
Dai, A., Ritchie, D., Bokeloh, M., Reed, S., Sturm, J., and Nießner, M. Scancomplete: Large-scale scene completion and semantic segmentation for 3d scans. In IEEE Conf. Comput. Vis. Pattern Recog., 2018.
Dai, A., Diller, C., and Nießner, M. Sg-nn: Sparse gener- ative neural networks for self-supervised scene comple- tion of rgb-d scans. IEEE Conf. Comput. Vis. Pattern Recog., 2020.
Deitke, M., Schwenk, D., Salvador, J., Weihs, L., Michel, O., VanderBilt, E., Schmidt, I.., Ehsani, K., Kembhavi, A., and Farhadi, A. Objaverse: A universe of annotated 3d objects. In IEEE Conf. Comput. Vis. Pattern Recog.. 2023.
References
Ding, Z., Wang, J., and Tu, Z. Open-vocabulary universal image segmentation with maskclip. In Int. Conf. Mach. Learn., 2023.
Achlioptas, P., Diamanti, O., Mitliagkas, I., and Guibas, L. Learning representations and generative models for 3d point clouds. In Int. Conf. Learn. Represent., 2018.
Fong, W. K., Mohan, R., Hurtado, J. V., Zhou, L., Caesar, H., Beijbom, O., and Valada, A. Panoptic nuscenes; A large-scale benchmark for lidar panoptic segmentation and tracking. IEEE Rob. Automat. Letters, 2021.
Behley, J., Garbade, M., Milioto, A., Quenzel, J., Behnke, S., Stachniss, C., and Gall, SemanticKITTI: Dataset for Semantic Scene Understanding of LiDAR Sequences. In Int. Conf. Comput. Vis., 2019.
Garbade, M., Chen, Y.-T., Sawatzky, J., and Gall, J. Two stream semantic scene completion. In CVPR Work- shops, 2019.
Behley, J., Milioto, A., and Stachniss, C. A Benchmark for LiDAR-based Panoptic Segmentation based on KITTI. In Int. Conf. Rob, Automat., 2021.
Geiger, A., Lenz, P., and Urtasun, R. Are we ready for autonomous driving? The KITTI vision benchmark suite. In CVPR, 2012.
Cao, A.Q Dai, A., and de Charette, R. Pasco: Urban 3d panoptic scene completion with uncertainty awareness. In IEEE Conf. Comput. Vis. Pattern Recog., 2024.
Geiger, A., Lenz, P., Stiller, C., and Urtasun, R. Vision meets robotics: The KITTI dataset. Int. J. Rob. Research, 2013.
Chang, A. X., Funkhouser, T., Guibas, L., Hanrahan, P., Huang. Q.. Li, Z. Savarese, S., Savva, M., Song, S., Su, H., et al. Shapenet: An information-rich 3d model repository. arXiv preprint 1x11 2015.
GroB, J., OSep, A., and Leibe, B. Alignnet-3d: Fast point cloud registration of partially observed objects. In Int. Conf. 3D Vis., 2019.
Cheng, B., Misra, I,, Schwing, A. G., Kirillov, A., and Gird- har, R. Masked-attention mask transformer for universal image segmentation. In IEEE Conf. Comput. Vis. Pattern Recog., 2022.
Huang, S., Gojeic. Z., Huang, J., Wieser, A., and Schindler, K. Dynamic 3d scene analysis by point cloud accumula- tion. In Eur. Conf. Comput. Vis., 2022.
Choy, C., Gwak, J., and Savarese, S. 4D seatio-temporal convnets: Minkowski convolutional neural networks. In IEEE Conf. Comput. Vis. Pattern Recog., 2019.
Jun, H. and Nichol, A. Shap-e: Generating conditional
3d implicit functions. arXiv preprint 23X.:23005.02..5 2023.
10

---

Towards Learning to Complete Anything in Lidar
Khurana, M., Peri, N., Ramanan, D., and Hays, J. Shelf- supervised multi-modal pre-training for 3d object detec- tion. arXiv preprint arXiv:2406.1011 2024.
Marcuzzi, R., Nunes, L.,, Wiesmann, L.., Behley, J., and Stachniss, C. Mask-based panoptic lidar segmentation for autonomous driving. IEEE Rob. Automat. Letters, 8 (2):1141-1148, 2023.
Kirillov, A., He, K., Girshick, R., Rother, C., and Dollár, P. Panoptic segmentation. In IEEE Conf. Compus. Vis. Pattern Recog., 2019.
Mei, J., Yang, Y., Wang, M., Huang, T., Yang, X., and Liu, Y. Ssc-rs: Elevate lidar semantic scene completion with representation separation and bev fusion. In Int. Conf. Intel. Rob. Sys., 2023.
Kirillov, A., Mintun, E., Ravi, N., Mao, H., Rolland, C., Gustafson, L.,, Xiao, T., Whitehead, S., Berg, A. C.,, Lo, W.-Y., et al. Segment anything. In Int. Conf. Comput. Vis., 2023.
Mescheder, L., Oechsle, M., Niemeyer, M., Nowozin, S., and Geiger, A. Occupaney networks: Learning 3d recon- struction in function space. In IEEE Conf. Comput. Vis. Pattern Recog., 2019.
Krähenbühl, P. and Koltun, V. Efficient inference in fully connected crfs with gaussian edge potentials. In Adv. Neural Inform. Process. Syst., volume 24, 2011.
Mittal, H., Okorn, B., Jangid, A., and Held, D. Self- supervised point cloud completion via inpainting. Brit. Mach. Vis. Conf., 2021.
Li, L., Shum, H. P. H., and Breckon, T. Less More: Re- ducing Task and Model Complexity for 3D Point Cloud Semantic Segmentation. In IEEE Conf. Comput. Vis. Pat- tern Recog., 2023a.
Mittal, P., Cheng, Y.-C., Singh, M., and Tulsiani, S. Au- tosdf: Shape priors for 3d completion, reconstruction and generation. In JEEE Conf. Comput. Vis. Pattern Recog.. 2022.
Li, P., Zhao, R., Shi, Y., Zhao, H., Yuan, J., Zhou, G., and Zhang, Y.QQ. LODE: Locally Conditioned Eikonal Implicit Scene Completion from Sparse LiDAR. In Int. Conf. Rob. Automat,, 2023b.
Najibi, M., J., Zhou, Y., Qi, C. R., Yan, X., Ettinger, S., and Anguelov, D. Motion inspired unsupervised percep- tion and prediction autonomous driving. In Eur. Conf. Comput. Vis., 2022.
Li, Y., Li, S., Liu, X., Gong, M., Li, K., Chen, N., Wang, Z., Li, Z., Jiang, T., Yu, F., Wang, Y., Zhao, H., Yu, Z.,, and Feng, Sscbench: A large-scale 3d semantic scene completion benchmark for autonomous driving. In Int. Conf. Intel. Rob. Sys., 2024.
Najibi, M., Ji, J., Zhou, Y. Qi, C. R., Yan, X., Ettinger, S., and Anguelov, D. Unsupervised 3d perception with 2d vision- lasion- distillation for autonomous driving. In Int. Conf. Comput. Vis., 2023.
Liang, L., Akhtar, N., Vice, J., and Mian, A. Voxel-and bird's-eye--e s-ee----e-----eeeeee semantic scene completion for li- dar point clouds. Remote Sensing, 16(13):2266, 2024.
Nam, G., Khlifi, M., Rodriguez, A., Tono, A., Zhou, L., and Guerrero, 3d-ldm: Neural implicit 3d shape generation with latent diffusion models. arXiv preprint arXiv:22 2022.
Liao, Y., Xie, J., and Geiger, A. KITTI-360: A Novel Dataset and Benchmarks for Urban Scene Understanding in 2D and 3D. In IEEE Trans. Pattern Anal. Mach. Intell., 2021.
Ntavelis, E., Siarohin, A., Olszewski, K., Wang, C., Gool, L. V., and Tulyakov, S. Autodecoding latent 3d diffusion models. Advances in Neural Information Processing Systems, 36:67021-67047, 2023.
Liu, S., HU, Y., Zeng, Y., Tang, Q. Jin, B., Han, Y., and Li, X. See and think: Disentangling semantic scene completion. Adv. Neural Inform. Process. Syst., 2018.
Nunes, L., Marcuzzi, R., Mersch, B., Behley, J., and Stach- niss, C. Scaling Diffusion Models to Real-World 3D LiDAR Scene Completion. In IEEE Conf. Comput. Vis. Pattern Recog.. 2024.
Liu, Y., Kong, L.. Cen, J., Chen, R., Zhang, W., Pan, L., Chen, K., and Liu, Z. Segment any point cloud sequences by distilling vision foundation models. In Advances in Neural Information Processing Systems, 2023.
Osep, A., Meinhardt, T., Ferroni, F., Peri, N., Ramanan, D., and Leal-Taixe, L. Better call sal: Towards learning to segment anything in lidar. In Fur Conf. Comput. Vis., 2024.
Liu, 7.,, Zhang, 7.,, Cao, Y., Hu, H., and Tong, X. Group- free 3d object detection via transformers. In Int. Conf. Comput. Vis., 2021.
Park, J. J., Florence, P., Straub, J., Newcombe, R., and Lovegrove, Deepsdf: Learning continuous signed distance functions for shape representation. In JEEE Conf. Comput. Vis. Pattern Recog., 2019.
Ma, Y., Peri, N., Wei, S., Hua, W., Ramanan, D., Li, Y., and Kong, S. Long-tailed 3d detection via 2d late fusion. arXiv preprint arXiv:23/2. 10986, 2023.

---

Towards Learning to Complete Anything in Lidar
Peng, S., Genova, K., Jiang, C., Tagliasacchi, A., Pollefeys, M., and Funkhouser, T. Openscene: 3d scene understand- ing with open vocabularies. In IEEE Conf. Comput. Vis, Pattern Recog., 2023.
Tian, X., Jiang, T., Yun, L.,, Mao, Y., Yang. H., Wang. Y., Wang, Y., and Zhao, H. Occ3d: A large-scale 3d occupancy prediction benchmark for autonomous driving. Adv. Neural Inform. Process. Syst. 2024.
Peri, N., Dave, A., Ramanan, D., and Kong, S. Towards long-tailed 3d detection. In Conf. Rob. Learn.; 2023a.
Unal, O., Dai, D., and Van Gool, L. Scribble-supervised lidar semantic segmentation. In IEEE Conf. Compul. Vis. Pattern Recog., 2022.
Peri, N.. Li, M., Wilson, B., Wang. Y.-X., Hays, J., and Ramanan, D. An empirical analysis of range for 3d object detection. Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV) Workshops, pp. 4074-4083, October 2023b.
Vahdat, A., Williams, F., Gojeic, Z., Litany, O., Fidler, S., Kreis, K., et al. Lion: Latent point diffusion models for 3d shape generation. Adv. Neural Inform. Process. Syst., 2022.
Petrovskaya, A. and Thrun, S. Model based vehicle de- tection and tracking for autonomous urban driving. Aut. Rob., 26:123-1339 2009.
Van Den Oord, A., Vinyals, O., et al. Neural discrete rep- resentation learning, Adv. Neural Inform. Process. Syst., 2017.
Radford, A., Kim, J. W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., et al. Learning transferable visual models from natural language supervision. In Int. Conf. Mach. Learn., 2021.
Vobecky, A., Siméoni, O., Hurych, D., Gidaris, S., Bursuc, A., Pérez, P., and Sivie, J. Pop-3d: Open-vocabulary 3d occupancy prediction from images. Adv. Neural Inform. Process. Syst., 2023.
Ravi, N., Gabeur, V., HU, Y-T.,, Hu, R., Ryali, C., Ma, T., Khedr, H., Rädle, R., Rolland, C., Gustafson, L., Mintun, E.,, Pan, J., Alwala, K. V., Carion, N., Wu, C.-Y., Girshick, R., Dollár, P., and Feichtenhofer, C. SAM 2: Segment Anything in Images and Videos. arXiv preprint arXiv:2408. 000114, 00714, 2024. URL https:// 00714.
Wu, J., Zhang, C., Xue, T., Freeman, B., and Tenenbaum, J. Learning a probabilistic latent space of object shapes via 3d generative-adversai modeling. Adv. Neural Inform. Process. Syst., 2016.
Wu, Z., Song, S., Khosla, A., Yu, F., Zhang, I.., Tang, X., and Xiao, J. 3d shapenets: deep representation for volumetric shapes. In IEEE Conf. Comput. Vis. Pattern Recog., 2015.
arxiv. org/ abs 00714
Ren, X., Huang, J., Zeng, X., Museth, K., Fidler, S., and Williams, F. Xcube: Large-scale 3d generative modeling using sparse voxel hierarchies. In IEEE Conf. Comput. Vis. Patlern Recog,, 2024.
Xia, Z., Liu, Y.. Li, X., Zhu, X., Ma, Y., Li, Y., Hou, Y., and Qiao, Y. SCPNet: Semantic Scene Completion on Point Cloud. In CVPR, 2023.
Rist, C. B., Emmerichs, D., Enzweiler, M., and Gavrila, D. M. Semantic scene completion using local deep im- plicit functions on lidar data. IEEE Trans. Pattern Anal. Mach. Intell., 44(10))2005-7211 2021.
Xiao, Z., Jing, L., Wu, S., Zhu, A. Z., Ji, J., Jiang, C. M., Hung. W.-C., Funkhouser, T., Kuo, W., Angelova, A., et al. 3d open-vocabulary panoptic segmentation with 2d-3d vision vision-language distillation. In Eur. Conf. Comput. Vis., 2024.
Roldao, L., de Charette, R., and Verroust-Blonder A. Lmsc- net: Lightweight multiscale 3d semantic completion. In Int. Conf. 3D Vis., 2020.
Yan, X., Gao, J., Li, J., Zhang, R., Li, Z., Huang, R., and Cui, S. Sparse Single Sweep LiDAR Point Cloud Seg- mentation via Learning Contextual Shape Priors from Scene Completion. In AAAI, 2021.
Seidenschwarz, J., OSep, A., Ferroni, F., Lucey, S., and Leal-Taixé, L. Semoli: What moves together belongs together. IEEE Conf. Comput. Vis. Pattern Recog., 2024.
Yin, T., Zhou, X., and Krähenbühh, P. Center-based 3d object detection and tracking. In IEEE Conf. Comput. Vis. Pattern Recog., 2021.
Song. S., Yu, F., Zeng, A., Chang, A. X., Savva, M., and Funkhouser, T. Semantic scene completion from a single depth image. IEEE Conf. Comput. Vis. Pattern Recog., 2017.
Yuan, W., Khot, T., Held, D., Mertz, C., and Hebert, M. Pen: Point completion network. In Int. Conf. 3D Vis., 2018.
Takmaz, A., Fedele, E., Sumner, R. W., Pollefeys, M., Tombari, F., and Engelmann, F. OpenMask3D: Open- Vocabulary 3D Instance Segmentation. In Ady. Neural Inform. Process. Syst., 2023.
Zhang, L., Yang, A. J., Xiong, Y., Casas, S., Yang, B., Ren, M., and Urtasun, R. Towards unsupervised object detection from lidar point clouds. In IEEE Conf. Comput. Vis. Pattern Recog., 2023.
12

---

Towards Learning to Complete Anything in Lidar
Zheng, J., Tang, P., Wang, Z., Wang, G., Ren, X., Feng.
B., and Ma, C. Veon: Vocabulary-enhanced occupancy prediction. In Eur. Conf. Comput. Vis., 2024.
Zhou, C., Loy, C., and Dai, B. Extract free dense labels from clip. In Eur. Conf. Comput. Vis., 2022.
Zhou, Y. and Tuzel, O. Voxelnet: End-to-end learning for point cloud based 3d object detection. In IEEE Conf. Comput. Vis. Puttern Recog. 2018.
7hu, X., Zhou, H., Wang, T., Hong, F., Ma, Y., Li, W., Li, H., and Lin, D. Cylindrical and asymmetrical 3d convolution networks for lidar segmentation. In CVPR, 2021.
13

---

Towards Learning to Complete Anything in Lidar
A. CAL Pseudo-Labeling Engine Details
this section, we provide additional implementation details of our pseudo-labeling engine, and provide further insights into the pseudo-labeled data we generate for training purposes. An overview of the pseudo-label generation configuration can be found in Tab. 8 for both SemanticKITTI (Behley et al., 2019) and KITTI360 (Liao al., 2021) datasets.
Table Pseudo-laheling engine configuration with datasct-specifie parameters SemanticKITTI and KITT1360.
Parameter
SemanticKITTI
KITT1360
Temporal Window for Video-Ohject Localization
Number of forward frames (Trw)
32
32
Number of backward frames (Thw)
8
Stride (w)
2
2
Temporal Window for Binary Occupancy Aggregation
Number of forward frames
72
Number of backward frames
36
Stride
CRF Settings
Number of iterations
5
Other Settings
Dynamic object removal
X
SAM Automatic Mask Generator Configuration
Model
sam
vit
4b8939
sam
vit 4b8939
Points per side
32
32
Points per batch
64
64
Pred. IoU threshold
0.84
0.88
Stability score threshold
0.90
Stability score offset
1.0
1.0
Crop N layers
0
Box NMS threshold
0.7
0.8
Crop N layers downscale factor
2
1
Min. mask region area
100
100
Crop NMS threshold
0.7
0.8
SAM2 Video-Object Localization Configuration
Model
sam2 sam2_hicra_lan hicra_large
Himn_hicra_larg
Points per side
32
32
Points per batch
64
64
Pred. IoU threshold
0.5
0.8
Stability score threshold
0.9
Stability score offset
1.0
1.0
Crop N layers
1
Box NMS threshold
0.7
0.7
Crop N layers downscale factor
2
Min. mask region area
100
100
Crop NMS threshold
0.7
0.7
Voxelization and Alignment Settings
Voxel size
0.2
0.2
Scene (m)
(51.2, 51.2, 6.4)
(51.2, 51.2, 6.4)
Voxel grid origin
(0, -25.6, 22)
(0, -25.6, -2)
Grid size
(256, 256, 32)
(256, 256, 32)
Camera-to-Lidar alignment shift
(0.0, 0.0, 0.0)
(0.79, 0.3, -0.25)
14

---

Towards Learning to Complete Anything in Lidar
A.1. CRF-based refineeent module
In our pseudo-labeling engine (as depicted in Fig. 2 of the main paper), we first accumulate two types of information: binary occupancy, and completed instance masks. As depicted in Tab. 10, the binary occupancy (voxels that are occupied) information we obtain has much higher coverage compared to our pscudo-labeled mask coverage. As the mask-level information relies on whether we were able to successfully detect and track mask across the RGB sequence, there are areas in the full scene for which we only have occupancy information from the Lidar scan, but no intance-level labels. For example, if the ego-vehicle drives by a parked car, the RGB camera only observes the visible parts of the car, hence only tracking those areas across the video. This often results in incomplete mask coverage of objects that are only observed from one side, such as parked cars. However, our occupancy signal is more complete as it is obtained from 360 Lidar scans, To benefit from this additional information and improve our partial pseudo-label coverage, we employ Dense Conditional Random Fields (DenseCRF) propagate our mask-level pscudo-labels to unlabeled but occupied areas, in order to improve and refine our masks. For this purpose, we rely on the DenseCRF Python wrapper provided in pydensecrf, based on the work of Krähenbühl et al. (Krähenbühl & Koltun, 2011).
We perform CRF-guided refinement only in the occupied area by operating on the sparse voxel representation. This implies that CRF does not change the occupancy coverage, but only increases mask-level label coverage. We use a multi-class setting where we define each separate instance mask ID as a class label (e.g. mask I has class ID and mask has class ID etc.) in a class-agnostic setting. The unary energy is obtained from class- probabbbbaiisss which represent the initial confidence in each voxel belonging to a particular class. For voxels which already belong to a mask, we initialize unary-potentials based on the original instance mask IDs, meaning that we set probability of 1.0 for the original class. For the occupied but unlabeled voxels, we initialize the probabilities as a uniform distribution over all classes (i.c., instance mask IDs), Pairwise potentials for DenseCRF are defined based on the voxel coordinates, using features formulated as the normalized voxel coordinates. This way, the features represent the spatial proximity of the voxels. These pairwise potentials encourage neighboring voxels to have similar labels, thereby refining the initial pseudo labels, and improving the level of completion. We then run DenseCRF inference for 5 steps to refine the labels iteratively. Finally, we ensure that none of the original voxel mask labels change after the CRF-guided refinement by ensuring that the voxels with initial labels preserve their original mask labels. Furthermore, the only update that the CRF-guidance performs is regarding the class-agnostic masks over the voxel grid, meaning that the CLIP features are not affected by this operation.
A.2. Handling dynamic objects
Our video-object localization process segments points that correspond to both static and dynamic parts of the scene. However, since we need to correctly accumulate a dense point cloud representation for the objects in the scene, handling dynamic objects would require registering their partial observations and correctly mapping them to the canonical pose for accurate reconstruction. Existing methods that perform dynamic object synchronization and rectification (Li et al., 2024; Xia et al., 2023) rely on panoptic segmentation labels for identifying and registering such objects. However, (1) we assume no access to semantic labels, and (2) performing such registration is error-prone in our setting.
In the original SemanticKITTI completion benchmark (Behley et al., 2019), while performing point level aggregation, spatio-temporal tubes for dynamic objects are preserved, and such objects are neither registered to the canonical pose and synchronized per frame, nor removed. For consistency with the supervised baselines, we perform the same, where we preserve the moving- object tubes. However, for the KITTI-360 (Liao et al., 2021) dataset where the 3D object tracks are available as an asset in the dataset, we perform dynamic object removal operation and only keep the static parts of the scene during Lidar point cloud accumulation. We would like to highlight that while such object tracks are commonly available in most autonomous driving datasets, it also possible to instead use automatic 3D object detection approaches for this purpose. For this dynamic object removal operation, after backprojecting the predicted 2D instance masks onto the Lidar point cloud, we first fit 3D z-axis aligned bounding boxes to each 3D object point cloud. If the overlap between the 3D bounding boxes of dynamic objects (from the 3D object tracks available as an asset) and the predicted (fitted) object boxes exceeds a certain limit, we classify this predicted object as dynamic. Any object mask ID associated with a dynamic object in any frame within the temporal window [t - Thw, t + T yw] is discarded, and not used for final aggregation of pseudo-labeled instance masks. To note once again, we perform this process only for KITTI360, and we keep the moving object tubes for our experiments in SemanticKITTI in alignment with the original benchmark from (Behley et al., 2019). Our results show that our method CAL learns to complete both static and dynamic objects despite the fact that only the static parts get correctly aligned (registered) for completion in our pseudo-labeled training data for both datasets.
15

---

Towards Learning to Complete Anything in Lidar
A.3. Lifting and refinement
Following Osep al. (2024), we perform refinement of the 3D masks mi,k based on 3D segments obtained via DBSCAN clustering the points from the Lidar sean Lt which are inherently less noisy in the 3D space. More specifically, for each Lidar scan, we perform DBSCAN at varying density thresholds E {11.2488, 0.8136, 0.6952, 0.594, 0.4353, 0.3221) to compensate for varying point density in Lidar scans. This operation is performed after ground plane fitting following Osep et al. (2024). After obtaining clusters (segments) with each of these thresholds, we match each predicted 3D mask mi,k with the best-matching segment (matching in terms of IoU). If the overlap between the predicted mask and the matched segment is above a certain threshold (0.5), we replace each predicted mask mi,k with the matched segment. Differently from (Osep et al., 2024), we perform this operation for each scan in our temporal window [t Tow t + Tyw].
A.4. Voxelization details
Following the voxelization process from SemanticKITTI (Behley et al., 2019), which was also followed SSCBench- KITTI360 (Li et al., 2024), we crop the volume in front of the camera using a minimum extent of [0, 22.66--550 2 and maximum extent of [51.2, 25.6, 4.4] in meters. Each aggregated point cloud is first cropped within these bounds, and then centralized using the predefined voxel-origin, by subtracting (0, 25.66 22 from the aggregated point coordinates. Next, we use a voxel size of 0.2 following prior work to voxelize the point coordinates obtained in the previous step. For SSCBench-KITTT1600, we observe that the camera placement procedure is different than the procedure in SemantieKITTI, as the ground truth voxel grids had undergone a Lidar-to-Camera alignment transformation according to the discussions available in the official repository of CCCBench-KITTISSS process after reaching out, so we empirically determined a transformation vector of [0.79, 0.3, -.255] with our best efforts to obtain an alignment vector apply our pseudo-labels before voxalization, in order provide correct GT alignment for evaluation, and to have fair comparison between our method and the supervised baselines.
However, we were unable to confirm the exact transformation
A.5. CLIP features
To obtain CLIP features associated with each object instance, we employ MaskCLIP (Zhou al., 2022) on each masklet across the video sequence, and average-pool normalized CLIP features obtained at each timestamp, per object. The output from this operation is separate CLIP feature vector of dimension 768, for each object in the scene.
B. CAL Model Details
B.1. Architecture Details
Overview. Our model architecture is built upon sparse geauss-generttiee 3D U-Net, largely following the implementation in (Cao et al., 2024), which consists of a feature backbone, an encoder followed by a dense 3D convolutional network, and a multi-scale generative decoder consisting of 3 decoder blocks for each of the 3 scales of resolution. The features from the generative decoder are passed to a transformer decoder which aims to produce a set of binary instance masks over the predicted occupied voxels, along with associated CLIP feature for each mask.
The input to our model is Lidar point cloud that is cropped within the bounding volume of the voxel-grid, which is represented unstructured set of coordinates. This cropping operation follows the problem formulation introduced for SSC and PSC benchmarks (Cao et al., 2024). This input point cloud is given as input to MLP and voxelization layers based on Cylinder3D (Zhu et al., 2021) as in PaSCo (Cao et al., 2024). The voxelized features are then passed through a sparse convolutional encoder that is composed of 4 encoder blocks which gradually upsample the features, resulting in fl:1 fl.2 fl:4 fl:8 First encoder block consists of 3 residual blocks, and each of the remaining 3 encoder blocks consists of fene.
Encoder.
a sparse convolutional layer (intended for feature upsampling) and 3 residual blocks. The final output from the encoder is 1:8 resolution features fla8 Since the original input is sparse, and we would like to perform scene-scale completion in a dense voxel grid, we employ dense 3D convolutional network following PaSCO. This dense 3D convolutional network takes f1:8 from the encoder, and outputs the features fasD.
Multi-scale generative decoder. The decoder in our sparse-generativ 3D U-Net consists of 3 decoder blocks D for each scale L E [4, 2, 1J. Each decoder block DI:L is structured generative block designed for sparse tensor operations. It begins with an upsampling operation using transposed convolution to increase spatial resolution, followed by batch
16

---

Towards Learning to Complete Anything in Lidar
normalization and a LeakyReL.U activation layer. The resizing step adjusts feature dimensions via point-wise convolution. The output features from the decoder block D1LL is denoted as fdee-
Completion heads. The output features from each decoder block at each scale is passed through set of completion heads: binary occupancy prediction heads (B, and B. well as S!:T.) for prototype prediction formulated as a classification task. Please note that in our setting, we only use CLIP prototypes as proxy for semantic classes, but we never have access to the ground truth semantic classes at any capacity. To provide further detail, we have two types of binary occupancy prediction heads: B I. and B. The binary occupancy head B, is a sparse convolutional layer which directly takes decoder features fl:I, and predicts per pox-v binary occupancy. B. on the other hand, takes the semantic voxel logits from the semantic completion head S i.e. 51. (fdec) as input. The reason why we have the additional B.- head is the following: since our binary occupancy labels are more complete, whereas the pseudo-labeled area with instance masks and CLIP features have smaller amount of coverage compared to the full occupied area, we are supervising binary occupancy and semantic head with two separate completion signals with differing volume coverage. Therefore, by adding an additional binary occupancy head to the output of the prototype- caaippppcaasiiiittiinn head, we aim to ensure that the classification head output will not diverge from the binary occupancy head output, which could potentially result in mismatched regularization. This way, we intend to encourage the network to complete the scene even in areas for which we do not have pseudo-labell (masks and associated CLIP features).
Pruning. Similar to Cao et al. (2024), we also perform pruning of voxels based on the logits predicted by the prototype- classification head to preserve sparsity, and continue refining per-voxel predictions only for voxels that were predicted be occupied in the coarser resolution. Further details regarding the pruning could be found in (Cao et al., 2024). The decoder block output from coarser scale is processed through the pruning layer, and the output is then passed to the next decoder block as input. This process is repeated until the final decoder block with the highest resolution.
Transformer predictor. We adopt the mask-centric transformer architecture from (Cao et al., 2024), which is based on the multi-scale decoder layer from Mask2Former (Cheng et al., 2022). This transformer model aims to perform panoptic scene completion based on the multi-scale features (l:L extracted in the generative decoder backbone. We perform the operations only on the voxels that are predicted to be occupied in the generative decoder. The transformer decoder consists of 3 layers. In addition to the original mask prediction heads from (Cao et al., 2024) our transformer decoder also includes a CLIP distillation head which aims to predict a CLIP token for each query. This MLP block consists of layers with dimensions (384,512 512, 1024, 768, 768). where the final dimension, 768, is the dimensionality of the CLIP embedding space. In essence, for each query, our method regresses CLIP feature vector. Importantly, for mask prediction, our method outputs a soft mask over the voxels: e.g., per-voxel probability over the list of sparse voxels (which are the voxels predicted to be occupied) for a voxel to be included in the instance mask.
B.2. Implementation Details
Processing the predictions. A voxel probability threshold parameter (Teos is applied to obtain the final binary instance mask. The objectness of the predicted instance mask is computed as the average vox- probability of the predicted instance mask. We only keep the instance masks with an objectness score that is at least Tokj. Lastly, we suppress overlapping masks, if the pairwise mask overlap is above a specified mask overlap threshold, Tour. The mask overlap threshold, objectness threshold and voxel probability threshold parameters are used both during training and test to refine the output. Since with our pseudo-labeled dataset with occasional incomplete observations, we expect to have in general lower confidence in the predicted masks and completed voxels. We empirically find that with the SemanticKrTTI dataset which consists of a smaller number of training samples, we benefit from setting lower confidence thresholds during inference. Therefore, while performing inference for SemanticKITTI, we use overlap threshold of Toor 0.1, an objectness threshold of Tobj 0.1, and voxel probability threshold of Toor = 0.1. For the KITTI-360 dataset which has a higher number of samples, we use an overlap threshold of Toer = 0.4, and objectness threshold of Tabj = 0.5 and voxel probability threshold of Toox = 0.3.
CLIP prototypes. To obtain the fixed prototype centers as described also in the main paper, we perform a clustering of the CLIP features associated with each instance mask we have in our pseudo-labeled dataset. For SemanticKITTI, we have 1 18333 individual CLIP feature instances in our pseudo-labeled dataset, and for for KITTI-360 we have 301332 CLIP feature instances. For clustering these features, we apply k-means, defining the number of groups we expect to create. We experiment with different numbers of clusters. The setting k=/ refers to treating all masks with the same prototype class, which in essence reduces to a trivial classification problem, reducing overall task of the completion heads to binary
17

---

Towards Learning to Complete Anything in Lidar
occupancy prediction. We also use the setting k=6 as a proxy for supergroups of expected number of classes, following Osep al. (2024), and k= which represents the general number of commonly annotated number of classes for urban scene datasets for autonomous driving such as Semantic and KITTI-360. We also experiment with higher numbers of clusters to assess our method's sensitivity to the number of clusters, by setting l E 50, 100, 500. For all settings, the prototypes are fixed at the beginning, and are not updated during training. At inference time, we completely discard our model's protolype class predictions, and only use the predicted instance masks over the voxel grid as well as the predicted CLIP features.
B.3. CAL Training Details
Overall Loss Definition. Following the notation in the main paper, our total training objective combines four main terms: (i) binary occupancy completion loss Loce, (ii) class-agnostic instance mask prediction loss Lmask. (iii) CLIP feature distillation loss CCLIP, and (iv) per-voxel prototype classification loss pprot-
Total loss formulated as
Lvotal Aooe Loce + Aprot Eprot + Amask Lmask + ACLIP LCLIP + Laux:
(I)
where each À is scalar weight, Acompi 1.0, Amask 40.0, ACIIP 1.0, Aprot 1.0., Laux is an auxiliary loss term following Cao et al. (2024). Below, we provide detailed overview of the loss terms.
Completion Losses, Loce and Sprut. As described in the main paper, supervise both geometric occupancy and coarse semantic structure (via CLIP-based prototypes) multiple scales. Specifically, we define binary occupancy loss (Coce) as the term which supervises whether each voxel is occupied or empty via binary cross entropy. Next, we define Sprot, which is the per-voxel prototype classification loss which classifies each voxel into one of C' prototype categories derived from CLIP features. This combines cross-entropy and Lovász losses. While computing this loss, we explicitly ignore voxels that do not have a corresponding pseudo-label associated with them, as the pseudo-labeling system often has a limited coverage of the full voxel grid volume. In practice, Loce and Fprot terms are computed and averaged across scales (from coarse to fine, L E {4, 2, 1}) based on the outputs from the generative decoder. These multi-scale loss terms are averaged across scales, i.e.
Spoot
Σ
prot and Loce 21,
Σ
CL
[11,2,4] 4H
(E(11,24)
(E(1,2,4)
More specifically,
c.d =
Cl: I.LL De,, +
Loee,B. B.
with individual binary occupancy loss terms for B, and Bg, where Chdk Bo,, B, and Coc,, are defined as a weighted binary cross entropy (wBCE) loss terms with respect to the pseudo-label binary occupancy at level L,, Oli and predicted occupancy outputs from B, and By, defined as O and Ole respectively:
Cli Lc wBBCE iabbel O.L.) and
LIL o w wBCC Labell O
In our implementation, we use a class weight of 20 for the occupied voxels in the weighted binary cross-entropy computation.
Instance Mask-Matching Loss Lmask. Our transformer decoder predicts class- instance masks over the completed 3D volume. To compute Lmask, we perform a one-to-one Hungarian matching between predicted masks and pseudo-labeled instance masks. Unlabeled voxels are ignored both during Hungarian matching as well as mask loss computation to avoid penalizing predictions outside the limited pseudo-labeled regions. Following previous work on 3D instance-segmentation. we define:
Lmass
>cE LCE
\Dic L
where Lce and Lpice are per-voxel binary cross-entropy and Dice terms, respectively, and ACE- Abice are scaling factors. set ACE = 2.0, ADice 1.0 in our experiments and apply an overall factor Amask when adding to (1).
CLIP Distillation Loss LCLIP. Simultaneously, our transformer regresses CLIP features per predicted instance to align with the pseudo-labeled embeddings, as also described in the main text. We use a cosine embedding loss between predicted
18

---

Towards Learning to Complete Anything in Lidar
CLIP embeddings, and the pseudo-label GT embedding associated with the matched instance mask:
LCLPP = CosineEmbedddnglLos/// Loss(pedd__bbeels gt_embeds).
This ensures that predicted instance-level embeddings preserve the semantic structure learned by CLIP. We weight CCLP by ACLIP in (I).
Further Implementation Details and Model Overview. Our model consists of 118M trainable parameters in total, with 8.2M parameters allocated to the transformer predictor and 59.3K parameters for the CylinderFeat backbone. We train the model for 50 epochs on 8 NVIDIA A100 GPUs, using a batch size of 8 with I item per GPU, and a learning rate of 0.0001. As also described earlier in A.4, we follow (Behley et al., 2019) and define a voxel grid volume extending 51.2 m forward, 25.6 m the sides, and 6.4 m in height, with voxel size of 0.2 m. We adopt this setting for both model training and inference. During training, we treat all pseudo-labeled instances equally, making no distinction between stuff and things-our our approach is entirely zero-shot and class-agnostic. All unlabeled voxels are ignored during loss computation for mask and Cprot. ensuring no penalty for predictions outside the spatial coverage of pseudo-labeled instances.
C. Additional Evaluation Details and Analysis
C.1. Dataset and Implementation Details
Datasets and benchmarks. We follow prior work (Cao et al., 2024) and evaluate CAL on two datasets that provide semantic and instance-level labels for PSC: SSCBench-KITTT360 (Li al., 2024; Liao et al., 2021) and SemanticKITTI (Behley al., 2019; Geiger et al., 2012: 2013) whose instance-level labels are provided in (Cao et al., 2024). These datasets provide per-voxel semantic labels (ScmanticKITTI: 20 classes, 8 are thing; SSCBench-KITTI366 19 classes, 6 are thing) that we only use during evaluation. Data was recorded using a 64-beam Velodyne Lidar sensor at 10H accompanied by front RGB stereo camera (camera placement differs across the two datasets, that were recorded in different periods). We only utilize the left front RGB camera for pseudo-labeling, but we only use the Lidar scans at inference time, not requiring any camera data. During temporal accumulation of pseudo-labels (Fig. 2, )), we identify and remove dynamic objects in KITTI-360 using 3D object tracks. For SemanticKITTI, we follow the original benchmark (Behley et al., 2019) and keep spatio-temporal moving object tubes for consistency with supervised baseline methods.
Data statisties. To have a fair comparison between our method and the supervised panoptic completion methods reported by Cao et al. (2024), we generate pseudo-labels for the same Lidar scans that were used for training these methods. More specifically, for SemTn we use one scan in every 5 scans, resulting in a total of 4649 pseudo-label samples. For KITTI-360, we use the Lidar scan IDs used in SSCBench-KITTI KITTT166,,, resulting in 8487 training scans, 1780 validation scans, and 2165 test scans as pseudo-labels. While we limited our training data scale for fair comparison, our pseudo-labeling engine can label an arbitrary number of samples if desired, enabling easy scaling of data size.
Text prompt tuning. For our PSC and SSC experiments, we follow the prompt-tuning strategy from previous work (Osep et al., 2024), and we define per-class text prompts for SemanticKITTI and KITTI-360 by associating each semantic class with multiple potential queries describing the class, resulting in more informative set of prompts. For instance, the query "car" can be addressed by "car. jeep, SUV, van" or the query "bicycle" can be addressed by "bicycle, bike". Full list of such queries can be found in "Table 1. Dataset vocabulary text prompts" of SAL (Osep al., 2024). We also employ the additional text prompt augmentation strategy from Osep et al. (2024) by including different variants of sentence structures describing the semantic class in an upen-vocabulary setting. Overall similarity between predicted CLIP feature and a semantic class is obtained by averaging the per-prompt similarity obtained for all prompts for this class.
C.2. Additional Ablations for the Pseudo-Labeling Engine
In this section, we provide additional ablation studies and further analysis on our pseudo-labeling engine.
Parameter analysis. First, in Tab. 9, we present an ablation on the key hyperparameters governing the pseudo-label aggregation process. In the main paper, we provided the SO setting for this experiment to decouple tracking and completion performance from the CLIP feature quality. To complement this, here we share the ZS counterpart of this experiment. Particularly, we ablate the affect of the number of frames for mask propagation T Tw (forward) and Tow (backward), as well as the stride, w. This table presents the performance metries for various combinations of these parameters (wr Tow, w)
19

---

Towards Learning to Complete Anything in Lidar
evaluated on the KITTI-360 validation set using PSC and SSC parameters PQ', PQ, SQ, RQ, IoU, mloU. We additionally report the relative number of time units required to obtain pseudo-labels for each setting, well as the average pseudo-label coverage achieved with respect to the GT labels. To ensure consistency across different parameter settings, we perform these measurements before performing CRF.
Table 9. Pseudo-labeling engine ablations using CLIP semantics. This table presents an analysis on the key parameters of the pseudo-label aggregation process: the number of frames for tracking The and Thw well the stride, w.
Parumeters
Thing
Stuff SQ
SSC
Time
Label Cov:1
Tra
Thue
PQ'1 1
PQ2
SQ
RQ
PQ
SQ
RQ
PQ
RQ
1oUt
mloU1
To
6.30
21.94
0.57
1.01
0.24
13.96
13.47
5.54
14.50
7.91
2.17
24.99
3.78
1.26
2.26
2.63
14.12
18.97
7.28
2
20.32
8.61
25. 70
5:33
1:34
47.97
3.37
4.21
6.81
22:00
8:23
23.31
48
0
8.70
3.34
25:75
5.45
1.34
48,06
2.37
4.34
14.59
6.99
22:07
8.34
23.32 6
32
0
8.61
3.25
25.70
5.33
1.34
47.97
2.37
4.21
14.56
6.81
22.00
8.23
4
23.31
9.32
28.85
56.40
3.25
4.66
15.08
24.51
26.13
10.04
3.96
25:30
0.32
2.25
47.15
4.01
4.82
15:12
7.48
25.
9.32
5
27.38
16
4
8.73
3.37
27.68
5.37
1:45
54.73
2.67
4.03
14.15
6.72
22.09
8.37
2,5
23.47
12
10.04
3.96
25.88 80
6.32
2.25
47.15
4.01
4.82
15.12
7.48
25.60
9.32
5
27.38
16
10.63
28.98
6.93
3.01
57,07
5.48
5.05
14.93
7.65
27.96
9.87
10
29.98
How well is our pseudo-label quality in the high-coverage area? Our pseudo-labeled dataset covers only limited portion of the full grid compared to the ground truth (70.14% for SemanticKITTI and 50.48% for KITTI-360) due to occasional failures in SAM2 mask detection or tracking across the video sequence, leading to missing label areas in the voxel grid. The panoptic scene completion evaluation using pseudo-labels penalizes these missing areas when performed over the full voxel grid (full-grid eval). To better assess the CLIP feature and instance-level completion performance of our pseudo-labels in areas for which we do have labels, we also evaluate performance restricted to areas with pseudo-labels by masking out excluded voxels (masked-voxel eval) in Tab. 10. The difference between full-grid and masked voxel evaluations highlights that where completion signals exist, CLIP features and instance masks are generally informative and well-separated. However, the observed gap underscores the limited label coverage of pseudo-labels, which could be improved through enhancements in pseudo-label construction or training augmentations in future work.
Table 10. Pseudo-label evaluation restricted to the areas in the voxel grid for which we have pseudo-labels. Analysis of the accuracy full- full-grid eval setting refers to evaluating our pscudo-labet labels using the usual PSC evaluation with respect to the GT. The masked-voxel eval setting refers to excluding the voxels for which we don't 't have any pseudo-labels during PSC evaluation.
of pscudo-lahels on the SemanticKITTI (Behley et al., 2019) validation set. The
Semuntic KITTlcBehley et al., 2019) (val set)
Thing
Stuff
PQ't
PQT
5Q
RQ
PQ
SQ
RQ
PQ
SQ
RQ
mIoUt
Parudo-labels (full-grid eval)
CAL (Sem. oracle)
25.90
17.71
26.55
16.93
67.91
23.75
18.28
61.25
33.10
CAL (CLIP Semanties)
16.98
10.67
54.02
16.19
7.30
67.19
10.21
13.12
44.43
20.54
20.62
Psesdo--lale label (masked-vvorel eval)
CAL (Sem. oracle)
33,20
26.09
68.05
36.62
20.57
69.39
28.21
30.10
67.07
42,74
42.37
CAL (CLIP som.) )
20.89
15:65
62.36
22.42
8.34
68.53
11.48
20.97
57.87
30.37
25.47
Discussion on the possibility of pseudo-labeling efficiency. Video-object localization over long video sequences is costly, particularly due to the costs associated with SAM2 (Ravi et al., 2024) mask propagation, which often results in a single pseudo-label pair to be generated in the order of a few minutes. Since this process can be costly, we also investigated whether it is possible to make the overall process more efficient by reducing the number of frames that one needs to process and aggregate for a single pseudo-labeled training sample. To do so, we experimented with reducing the number of required frames in conjunction with CRF-refinement, as in our earlier experiments we observed that CRF plays significant role in improving the completion of scene objects. Therefore, designed an experiment where, instead of using a long tracking horizon (T₁ 32, Thm = 8), we used short tracking horizon but also performed CRF refinement. Tab. 11 presents our findings from this experiment on SemanticKITTI, using the SO evaluation setting. In the first block of Tab. 11, we propagate masks from reference frame for w = 8 frames (FW), and compare to the second block with our original setting(T/ = 32, Tou = 8), after bas-d mask propagation. Here we see that while using a long- oongttracking horizon improves completion performance, it is also possible to instead use a shorter horizon in conjunction with CRF to reduce
20

---

Towards Learning to Complete Anything in Lidar
video-object localization prcessing time. We believe that this observation might be relevant for future scaling efforts by saving processing time, and enabling the more sparse labeling of sequences for future data-scaling endeavors.
Table 11. Impact of the number of frames completion quality with CRF on Semantic KITTI (Behley et al.. 2019) (val). This study demonstrates that CRF significantly enhances completion quality, allowing the use of fewer frames the pseudo-labeling pipeline which is computationally expensive due to the costly mask propagation step. The first block evaluates mask tracking over 8 frames, while the second block uses our original setting with 32 frames forward and frames backward tracking.
Semantic KITTicBehley et al., 2019) (val set)
All
Thing
Stuff SQ
Label Cov't
PQ
PQT
SQ
RQ
PQ
SQ
RQ
PQ
RQ
mloU
Tyu
Thu
with CRF
CAL (SO)
22.06
11.41
62.01
17.79
14.16
67.90
20.25
9.40
57.73
16.00
27.00
$2.36
CAL
13.53
5.65
50.30
8.92
5.53
62.06
7.68
5.74
41.72
15:46
52.36
Tyu
Th.
8, with CRF
CAL (5O)
25.90
17.71
64.06
26.55
16.93
67.91
23.75
18.28
61.25
28.58
33.10
70.13
CAL (CS)
16.98
10.67
54,02
16.19
7.30
67.19
10.21
13.12
44.43
20.54
20.62
70.13
C.3. Additional Ablations and Analysis for the Model Performance
Training data ablations for model training. In the main paper, we discussed the improvements that CRI-refinement brings to pseudo-label quality, demonstrated in Tab. 3. In this section, we also discuss the effect of training with pseudo-labels with and without CRF-refinement on the model performance for completeness. The findings from this experiment are presented in Tab. 12 and Tab. 13. We observe that training with pseudo-labels that use CRF-refinement (resulting in higher label coverage) clearly improves PQ as well as PQthing, SQthing and RQthing results. However, we see that Stuff categories are negatively impacted by this variant for the SSCBench- SSCBench-KITTTI66 dataset, which we suspect is due to CRF being relatively less effective in successfully propagating labels in KITTI360 dataset, where our binary occupancy coverage is only around 70% compared the setting SemanticKITTI where we reach almost 100% binary occupancy coverage. This is important, because CRF can only propagate labels towards unlabeled but occupied areas. Overall, it is evident that CRF refinement brings significant benefits to zero-shot PSC performance.
Table 12. Model ablations for data on SemanticKITTL. We train the CAL model using two different sets of data: pseudo-labels w/o CRF refinement, and pseudo-laber with CRF refinement. report PSC metrics for both variants individually. We observe that training CAL with data that has higher label coverage, (pseudo labels w. CRF) clearly improves overall performance.
Training data
Thing
Stuff
ssc
Sctting
PQ't
PQt
SQ
RQ
PQ
SQ
RQ
PQ
SQ
RQ
mloU?
Preudo Labels wño CRP
Semantic oracle CLIP semanties
12.25
3.60
43.86
5.98
0.77
41.06
1.40
5:65
45.89
9.31
15:37
9.65
3.27
26.27
5.40
0.32
20.47
0.57
5.43
30.48
8,90
9.40
Porudo Labels w. CRF
Semantic oracle
17.12
6.27
43.40
10.06
3.48
44.39
5.65
8.30
42.67
13:27
20.71
semantics
13.12
27.45
8.44
2.42
22.79
3.89
30.84
11.76
13:09
Pseudo-labels vs. model predictions. In this section, we analyze the gap between our pseudo-labels and model predictions (both evaluated on the SemanticKITTI validation set; semantic classes are assigned based on CLIP- featuree prompting). We note that our model is only given a single Lidar point cloud input and predicts occupancy, segmentation masks, and corresponding CLIP features used for prompting. On the other hand, pseudo-labels are obtained by (i) segmenting objects in images/video, (ii) averaging CLIP features, extracting from images, and (ii) accumulating Lidar measurements across a temporal window. Pseudo-labels, therefore, derive occupancy estimates from geometry observed from multiple views, and semantic features from visual information, while the model must rely on learned occupancy priors and semantic features distilled from the image- to the Lidar domain. In the semantic oracle setting, our model obtains 17.12 PQt (66.10 % of pseudo-labels), and 13.12 (77.27 % of pscudo-labels), suggesting that currently, the bottleneck is instance-level completion. Finally, we note that this gap is not surprising and largely stems from the inherent task difficulty- state-of-the-an PSC model (Cao et al., 2024) obtains 19.53 PQt (26.29 when ensembling models, see Tab. 14) when trained on perfect GT data, and classes are known a-priori
21

---

Towards Learning to Complete Anything in Lidar
Table 13. Model ablations for data on SSCBench-KITTI336 (Li et al., 2024). We train CAL model using two different sets data: pseudo-labels w/o CRF refinement, and with CRF refinement. We report PSC metrics for both variants individually. We observe that training CAL with data that has higher label coverage, (pseudo labels w. CRF) clearly improves overall performance except for stuff categories.
Training data for CAL
Thing
Stuff
SSC
Sctting
PQ'T
PQT
SQ
RQ
PQ
SQ
RQ
PQ
SQ
RQ
mloUt
Preudo Labels w/o CRF
Semantic oracle CLIP semantics
2.96
4.77
0.06
0.11
4.42
7.11
10.15
7.01
2.89
15.56
4.56
0.04
18.49
0.07
4.31
14.10
6.95
7.34
Preudo Labels CRF
Semantic oraele
12.56
1.71
33.18
3.10
2.05
45.57
3,76
1.54
26.99
2.76
13.34
CLJIP semantics
8.57
1,46
21.01
2,63
1.39
27.62
2.54
1.49
17.81
2.68
8.49
Table 14. Panoptic Scene Completion. On Semantic KITTI (Behley et al., 2019) (val) and SSCBench KITT1360 (Li et al., 2024) (test). Compared against LMSCNet (Roldao et al., 2020) +MaskPLS (Marcuzzi et al., 2023), JS3CNet (Yan et al., 2021) +MaskPLS (Marcuzzi al., 2023), SCPNet (Xia al., 2023) +MaskPLS (Marcuzzi al., 2023) and PaSCo (Cao et al., 2024) (M=1 and ensemble).
Semantic KITTl(Behley al., 2019) (val set)
SSCBouch-K/TT13000 (Li 2024) (test set)
All
Thing
Stuff
All
Thing
Staff SQ
Method
PQ" 1 PO1
SQ
RQ
PQ
SQ
RQ
PQ
SQ
RQ
mloU1
PO*
PO1
SQ
RQ
PQ
SQ
RQ
PQ
RQ
mloOU
Fudly supervised
LMSCNet -MasPPLS
13.81
4.17
36.13
6.82
1.62
29.87
2.68
6.02
40.69
9.82
17.02
12.76
4.14
26.52
6.45
0.88
20.41
1.58
5.78
29.58
8.58
15.10
353CNet +MarkPLS
11334
22.70
3,36
5.83
8.51
52.54
13.15
SCPNet -MasPPLS
19.39
8.59
49.49
13.69
488
46.41
7.30
11:30
51.73
18.04
22 44
16.54
6.14
51.18
10.15
4.23
48.46
7.05
7.09
52.55
11.70
21.47
PaSCo (M-1)
26.49
15.36
54.15
23.65
12.33
47.42
18.78
17.55
59.05
27.19
28.22
19.53
58.81
15.40
3,46
57.72
6.10
13.14
59.35
20.05
21.17
PaSCo (Ensemble)
31.42
16.51
54.25
25.13
13.71
48.07
20.68
18.54
58.74
28.38
30.11
10.92
56.10
17.09
4.88
57.53
55.39
21.39
22.39
Psoudo
CAL (Semantic oracle)
25.90
17.71
64.06
26.55
16.93
67.91
23.75
18.28
61.25
28.58
33.10
14.75
4.57
40.79
7.98
7.08
48.18
12.35
3.32
37.10
5.79
17.14
(CLIP semantics)
16.98
10.67
54.02
16.19
7.30
67.19
10.21
13:12
44.43
20.54
20.62
10.98
3.18
33,95
5.58
3.91
48.30
6.86
2.82
26.78
4.95
11.04
Zero- Shor
CAL (Semantic ocacle)
17.12
6.27
43.40
10.06
3:48
44.39
5.65
8.30
42.67
13.27
20.71
12.56
1.71
33.18
3.10
2.05
45.57
3.76
1.54
26.99
2.76
13.34
CAL (CLIP semantics)
13.12
5,26
27.45
8.44
2.42
22.79
3.89
7.33
30.84
11.76
13.09
8.57
1.46
21.01
2.63
1.39)
27.62
2.54
1.49
17.81
2.68
8.49
Per-class results. In Tab. 15, we provide a detailed per-class break-down of the pscudo-label quality and model performance for the SemanticKITTI dataset. We notice that most of the gap between CAL and the baselines is due to rare classes (e.g.. pedestri an, cycl ),,, which suggests that fully supervised baselines can exploit class frequency information during training-allovnng them to re-balance and weight examples in a way that benefits rare categories. In contrast, because our approach is zero-shot, it does not have access to the ground-truth frequency distributions of classes, and thus it struggles more with less frequent or smaller structures. Additionally, since we rely on K-means for clustering, our method does not incorporate prior knowledge of the data distribution class frequencies, making it less effective in representing long-tail categories. While this limitation is challenging to address under the zero-shot paradigm, future work could explore more distribution aware clustering techniques or other strategies that better capture underrepresented object shape priors without relying on labeled data.
C.4. Additional Qualitative Results
In Fig. 7, we share qualitative results from our model's predictions on the KITTI360 dataset. In Fig. 8, we compare our method CAL with zero-shot baselines for the panoptic scene completion task. We also provide a supplementary video visualizing zero-shot panoptic scene completion results for a sequence of Lidar scans.
22

---

Towards Learning to Complete Anything in Lidar
Table 15. Per-class performance analysis for Panoptic Scene Completion, evaluated on SemanticKITTI (Behley et al., 2019) dataset.
Per-class scores baselines and fresuecreqeescs are taken from (Cao et al., 2024).
mustonc cenneyyliss... (0.00)) (0.05))
bicycle (0025))
mnooorcycle (0.035)
mruck (n 1083
veh. (0.20))
person (0.0%)
bicyelist (0.075)
road (15 303)
parking EL (112))
sidewalk (1.133)
other gmd (0.509))
building (M (1.108)
fence (008))
vegetation (9.300))
trunk (0.5%%)
terrain (9, 1963
traff taff-sign (0.05)
car (39229))
pole
mesn
Metbod
LMSCNet MaskPLS
9.43
0.76
2.32
0.47
0.00
1.82
0.26
0.27
0.00
4.17
JS3CNet + MaskPLS
9.57
1.07
4.19
17.54
0.91
0.12
0.00
0.00
58.45
5.32
15.89
0.00
1.02
1.33
0.00
0.76
13.63
0.28
0.00
6.85
SCPNet + MaskPLS
18.44
4,84
6.72
4.42
2.79
1.81
0.00
63.89
7.92
19.92
0.00
3.11
3.28
0.13
2.29
21.55
1.99
0.17
8.59
PaSCo (Ensemble)
24.55
18.09
44.89
0.00
76.22
4.85
4.22
32.61
3,26
CAL
14.11
0.00
0.00
4.34
0,88
0.00
0.00
0,00
$4.08
0.00
3.09
0.00
0.12
0.00
1.00
0.00
22.20
0.14
0.00
5.26
7.29
0.00
LMSCNet+ + MasPLS
62.65
0.00
53.44
53.87
0.00
69.00
0.00
0.00
63.30
57.83
52.70
0.00
53.93
52.58
0.00
59.76
54.12
53.37
0.00
36.13
JS3CNet MaskPLS
59.88
$3.79
55.17
57.73
55.70
62.50
0.00
65.98
55,70
54,53
0.00
$2.62
53.41
0,00
01
56.38
57.68
0.00
41.90
SCPNer + MaskPI.S
66.69
57.78
65. 30
55.30
65.15
61.01
0.00
0.00
56
58.72
55.81
0.00
54.94
54.45
$1.04
59.86
52.97
57.14
49.49
PaSCo (Ensemble)
70.10
67.33
62.15
77.52 52
62.6
55.87
51.29
63.88
54,78
54.25
CAL
65.84
0.00
0.00
52,80
63.71
0.00
0.00
0.00
63.05
0.00
$4.81
0.00
50.02
0.00
51.47
0.00
62,58
57.32
0.00
27.45
CAL Pscudo Labels
74.38
54.58
67.01
74.56
74.12
64.88
71.65
56.35
58.48
58.24
0.00
56.52
57.95
0.00
68.15
60.64
59.67
54.02
IMSCNet + Ma<PI
15.05
0.00
1.42
4.30
0.00
0.67
0.00
0.00
84.56
3.15
10.69
0.00
0.48
0.37
0.00
0.45
6.50
1.87
0.00
6.82
JS3CNet MaskPLS
15.98
7.59
30.38
0.20
0.00
89
9.55
29.14
0.00
1.93
2.49
1.39
0.48
0,00
SCPNet + MaskPLS
27.65
8,38
10.29
8.00
4,28
2.96
0.00
0.00
98
13.50
35.69
0.00
5.66
6.03
0.25
4.12
36.00
3.76
0.30
13.69
(Ensemble)
44.91
50.73
7.33
17.70
5.91
CAL
21.52
0.00
0.00
8.22
1.39
0.00
0.00
0.00
85.77
0.00
5.64
0.00
0.24
0.00
1.94
0.00
35.47
0.24
0.00
8.44
Pscudo
27.21
10.88
11.91
1.64
18.18
90.54
3.31
0.00
22.08
0.00
33.75
1.72
2.04
23

---

Towards Learning to Complete Anything in Lidar
Input Scan
Completion + Masks
ZS Prompting
"vehicle" "
"car"
"tree"
"road"
"façade"
"sidewalk"
Figure Qualitative results on KITTI-360 (Liao al., 2021). Given single Lidar scan as input (1*t column), CAL completes object-level observations as set of masks over the voxel grid nd column) with semantic CLIP feature for each predicted mask. We can prompt with any semantic class vocabulary and perform panoptic and semantic scene completion (3rd column).
24

---

Towards Learning to Complete Anything in Lidar
Input Scan
LiDiff + SAL
LODE SAL
Ours (CAL)
Ground Truth
Figure Qualitative comparison to zero-shot baselines SemanticKITTL Given single Lidar (1#t col.), compare method (CAL, 4th col.) to zero-shot baselines (2nd and 3rd cols.) combining LiDiff (Nunes et al., 2024) and LODE (Li et al., 2023b) with SAL (Osep al., 2024). While baselines struggle with coherent structure and semantic accuracy, CAL produces clcaner and complete outputs that align closely with the ground truth.
25


Reviews:
[{"review_id": "WmQGbwp4kB", "reviewer": "Reviewer_KxNA", "summary": "This paper introduces a novel zero-shot approach for completing data from a single LiDAR scan, including both object and instance completion. The method is potentially scalable as it leverages a pre-trained foundational video segmentation model, eliminating the need for labeled video data. CLIP features are extracted and fused across multiple views to supervise the completion model. The extensive experimental results, along with the implementations and discussions provided in the supplementary materials, offer valuable insights into the system’s design. While the results do not outperform fully supervised methods, the proposed approach holds significant value for the research community due to its scalability.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We are thrilled that the reviewer finds our task of zero-shot Lidar-based panoptic scene completion challenging and novel. We are particularly happy that the reviewer recognizes our method's scalability potential (wrt. data) and appreciates the extensive experimental results and sound methodology. Below, we address questions raised by the reviewer. \n\n\n**Q1. The reviewer asks about the effect of the video-foundation model on the final performance and asks how errors it introduces can be mitigated.**\n\nGreat point; indeed, predictions from the video foundation model may not always be perfectly accurate, as noted by the reviewer. In practice, we observed two types of errors: (i) noisy masks, especially inaccurate around borders, that cause artifacts during image-to-Lidar lifting. To address these, we perform several post-processing steps described in the paper, most importantly, DBSCAN-based segment refinement and CRF-based refinement of aggregated labels. (ii) video-segmentation models may also produce tracking errors, such as ID switches. To address this, we empirically determined the window size and stride in which state-of-the-art models are reliable (which we also ablated in *Table 2* of the paper), ensuring that such errors are infrequent. In practice, we find that as long as we have a sufficiently high signal-to-noise ratio, our model learns to ignore such artifacts. To mitigate these errors in future work, one could additionally utilize 3D/4D geometric cues from the Lidar sequence to improve temporal association.\n\n**Q2. Performance gap between fully-supervised and zero-shot methods.**\n\nGreat question! We would like to elaborate further on the main reasons behind this gap (beyond the limitations due to rare classes). As shown by our findings in *Table 1* (semantic oracle vs. zero-shot results), this gap is largely due to zero-shot semantic recognition— which is a challenging and active research area. Several opportunities exist to improve performance, such as enhancing the underlying Vision Language Model (VLM) for zero-shot recognition, or incorporating manually labeled data for supervised fine-tuning. \n\nAnother potential cause for this gap is that our pseudo-labels have lower coverage (approximately 50% on KITTI-360 and 70% on SemanticKITTI, please refer to *Table 3*) compared to ground truth labels, due to their construction requiring camera-Lidar co-visibility. In contrast, fully supervised methods benefit from training on a complete ground-truth signal with full-grid coverage (see *Figure 4*, 4$^{th}$ column), effectively using more labeled data. While this lower coverage is a limitation when training on fixed-size datasets like SemanticKITTI, as the *Reviewer KxNA* notes, it also underscores our approach’s potential: scaling with more data could help close the gap with fully supervised, closed vocabulary baselines. Our method can already be applied to the application scenarios from *Fig. 1* (as also shown by quantitative ZS-panoptic completion results and qualitative results). However, scaling with more data could enable our method to be more effective and robust for these tasks in the future.\n\nOn a related note, following *Reviewer C64v*’s suggestion, we added zero-shot panoptic scene completion baselines (please refer to our response to *Reviewer C64v*, *Q2*), which further confirm that zero-shot Lidar panoptic scene completion is a problem with non-trivial challenges.\n\nWe appreciate *Reviewer KxNA*'s interpretation that our method *“holds significant value for the research community even though it does not necessarily outperform the fully supervised methods”*. We will ensure to include this extended discussion in the paper to further elaborate on the gap between our zero-shot method and fully supervised baselines, as well as potential solutions to address this gap.\n\n**Q3. The reviewer highlights that Figure 2 does not clearly illustrate how the CLIP features are fused.**\n\nThanks for pointing this out! We compute CLIP features per-instance in every frame and *average* these (normalized) CLIP features over time across the frames in which this instance was observed and tracked. We will improve the clarity of this figure to highlight this temporal aggregation step, and also expand our textual description in L198.\n\n**Q4. Reviewer notes that our title may be misleading as the pseudo-labeling engine in our approach still relies on camera images.**\n\nThank you for raising this point! Our trained (distilled) model indeed takes only a single Lidar scan as input at test time, producing a completed scene representation without using any camera data during inference. However, as the reviewer points out, our pseudo-labeling engine—used solely during training—leverages camera images and Lidar sequences to generate training labels. To avoid any confusion, we’re happy to revise the title (for example, a possible alternative could be \"Towards Learning to Complete Anything in Lidar Using Vision Foundation Models\").", "claims_and_evidence": "The paper addresses a novel and underexplored problem: completing parts from sparse lidar inputs. The motivation for the study is well-articulated, and the research problem itself holds significant value. \n\nAddressing the completion problem in an open-vocabulary setting is highly valuable, as it presents intriguing possibilities for real-world applications.\n\nThe authors claim to present the first method for Zero-Shot Panoptic Scene Completion using LiDAR data, which adds a unique contribution to the field.", "methods_and_evaluation_criteria": "Using a video foundation model to extract masks for associating temporal information, and then extracting and aggregating CLIP features in 3D space, is a sound and reasonable approach. However, reliance on the video foundation model, which may not always be perfectly accurate, could introduce errors and potentially limit the overall performance of the system.\n\n\nThe title suggests that the method completes objects solely from LiDAR data, while the approach described in the paper still relies on camera images. This discrepancy could be misleading.\n\nThe figures do not clearly illustrate how the CLIP features are fused together.", "theoretical_claims": "The methods presented follow a typical learning-based formulation, and therefore, extensive theoretical proofs are not required. The foundational theories related to occupancy networks, LiDAR/camera geometry, and the reliance on the video foundation model are sound and reasonable.", "experimental_designs_or_analyses": "The experiments are quite extensive and provide comparisons with various fully supervised baselines. However, Table 1 shows a noticeable gap in performance compared to these methods. Could you provide insights into the main reasons behind this gap and suggest ways to improve the performance further? Although the authors claim that the main gap arises from rare classes, the ratio of data for these rare classes does not fully explain the significant performance gap.", "relation_to_broader_scientific_literature": "N/A", "essential_references_not_discussed": "N/A", "other_strengths_and_weaknesses": "Completing data from sparse views is a highly valuable problem, as it provides fundamental information for downstream tasks such as object detection, as demonstrated by the authors in the paper.", "other_comments_or_suggestions": "N/A", "questions_for_authors": "How does the reliance on the foundational video model potentially affect the final performance, and what steps can be taken to mitigate any errors introduced by it?\n\nWhat are the main reasons behind the performance gap observed in Table 1 when comparing your method to fully supervised baselines?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "8lckM7Clzs", "reviewer": "Reviewer_C64v", "summary": "The paper introduces CAL (Complete Anything in Lidar), a zero-shot panoptic scene completion framework that infers dense 3D object and scene geometry from sparse Lidar scans without relying on predefined class vocabularies. To achieve this, the authors propose a pseudo-labeling engine that mines 3D shape priors from unlabeled Lidar sequences by leveraging vision foundation models for object segmentation and tracking in videos. These mined pseudo-labels, which combine shape completion and semantic features, are then used to train CAL, a sparse generative encoder-decoder network with a transformer-based instance decoder that performs class-agnostic segmentation and completion. Unlike prior methods, CAL enables zero-shot semantic and panoptic scene completion, amodal 3D object detection, and recognition of novel object classes at test time via text-based prompting. The experiments on SemanticKITTI and SSCBench-KITTI360 show that CAL does not match the performance of fully supervised baselines.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We’re delighted that the reviewer found our paper well-structured with clear motivation and methodology. We appreciate the detailed feedback, and we are excited to address the concerns raised by the reviewer.\n\n**Q1. Novelty of distilling vision foundation models (VFMs) to Lidar**\n\nWe agree with the reviewer that distilling VFMs to Lidar is not entirely novel in the broader literature; for instance, [1,2] explore this for *segmentation*. Our work, however, introduces zero-shot Lidar *panoptic completion*, which extends beyond segmentation. While we leverage prior insights on distilling CLIP features for zero-shot (ZS) prompting—as in [1]—we find this alone insufficient for *completing* 3D shapes from sparse LiDAR scans. To this end, we combine ZS prompting with temporal aggregation of objects, providing essential cues for shape completion. Generating supervision for ZS panoptic completion presents non-trivial challenges: such as associating objects across time, partial coverage and occlusions, and training models to reconstruct full shapes from incomplete labels. These unique complexities in *completion* distinguish our method from prior work on ZS Lidar *segmentation* such as [1]. We believe that, as *Reviewer KxNA* also noted, our work tackles a *“novel and underexplored problem”* with *“significant value for the research community”*.\n\n**Q2. Comparison to zero-shot (ZS) baselines**\n\nWe thank the reviewer for the suggestion, and agree that ZS baseline comparisons would strengthen our analysis. As the reviewer noted, our method is the *“first for ZS panoptic scene completion in LiDAR”*— making it *“difficult to find a second existing ZS method for direct comparison”*. \n\nTo this end, we constructed two baselines adhering to the following criteria for a fair ZS comparison: (1) input is a single Lidar scan, (2) scene completion model is trained *without semantic labels*, and (3) instance prediction and semantic inference rely on *zero-shot* recognition. Accordingly, we combined recent Lidar completion methods w/o semantic labels—LODE [5] and LiDiff [6]—with SAL [1], a ZS panoptic segmentation method. As SAL’s codebase is not public, we obtained its ZS predictions on SemanticKITTI directly from its authors. Our baselines are:\n\n- LODE + SAL: LODE [5] performs implicit scene completion from sparse LiDAR, trained with GT completion but no sem. labels. We extract a surface mesh from its output, convert it to an occupancy grid, and propagate SAL’s ZS panoptic labels to voxels. \n- LiDiff + SAL: LiDiff [6], a diffusion-based completion method using GT completion data (no sem. labels), densifies LiDAR point clouds. We convert its output to an occupancy grid and similarly propagate SAL’s ZS panoptic labels to occupied voxels.\n\n|                            | All PQ† | All PQ | All SQ | All RQ | Thing PQ | Thing SQ | Thing RQ | Stuff PQ | Stuff SQ | Stuff RQ | mIoU  |\n|-|-|-|-|-|-|-|-|-|-|-|-|\n| LODE + SAL  | 7.74    | 1.96   | 11.12  | 3.54   | 0.00     | 6.36     | 0.00     | 3.39     | 14.59    | 6.11     | 8.12  |\n| LiDiff + SAL  | 7.35    | 0.36   | 23.95  | 0.65   | 0.22     | **34.81**    | 0.40     | 0.46     | 16.06    | 0.83     | 7.38  |\n| **Ours**              | **13.12** | **5.26** | **27.45** | **8.44** | **2.42** | 22.79    | **3.89** | **7.33** | **30.84** | **11.76** | **13.09** |\n\nResults show our method outperforms the baselines across nearly all metrics. Notably, while the baselines leverage completion models trained on fully completed GT, our approach excels despite using pseudo-labels with only partial coverage. This highlights that ZS panoptic Lidar scene completion is a challenging task, not trivially solved by existing methods. \n\n**Q3. Performance gap between fully-supervised and zero-shot methods**\n\nDue to space limits, please see our response to *Reviewer KxNA (Q2)*.\n\n**Q4. VFM biases and limitations**\n\nDue to space limits, please see our responses to *Reviewers KxNA (Q1) and ptaj (Q1-2)*.\n\n**Q5. Additional references**\n\nThanks for the valuable references! We'll add them to the Lidar-based *segmentation* section. [3] and [4] explore weakly supervised segmentation, while [2] uses contrastive pre-training to distill VFMs for segmentation. As the reviewer noted, [2–4] reduce manual labeling efforts. In contrast, our method addresses ZS Lidar panoptic *completion* with distinct challenges beyond the segmentation scope [2–4].\n\n[1] Osep et al., Better Call SAL: Towards Learning to Segment Anything in Lidar, ECCV '24\n\n[2] Liu et al., Segment Any Point Cloud Sequences by Distilling Vision Foundation Models, NeurIPS '23\n\n[3] Li et al., Less is More: Reducing Task and Model Complexity for 3D Point Cloud Semantic Segmentation, CVPR '23\n\n[4] Unal et al., Scribble-Supervised LiDAR Semantic Segmentation, CVPR '22\n\n[5] Li et al., LODE: Locally Conditioned Eikonal Implicit Scene Completion from Sparse LiDAR, ICRA '23\n\n[6] Nunes et al., Scaling Diffusion Models to Real-World 3D LiDAR Scene Completion, CVPR '24", "claims_and_evidence": "Yes.", "methods_and_evaluation_criteria": "Checked. See Other Strengths And Weaknesses", "theoretical_claims": "Checked.", "experimental_designs_or_analyses": "Checked. See Other Strengths And Weaknesses", "relation_to_broader_scientific_literature": "The idea of distilling vision foundation models (VFMs) into LiDAR-specific models is no longer novel in the broader literature; for example, prior works have successfully applied VFMs to LiDAR panoptic segmentation [1] and semantic segmentation [2] tasks. This paper extends a similar method to a different yet related task—specifically, instance-level completion and recognition using LiDAR data alone. Thus, the key contribution here builds upon existing insights, following established approaches from other LiDAR VFM-based tasks to instance-level scene completion and recognition.\n\n[1] Osep, A., Meinhardt, T., Ferroni, F., Peri, N., Ramanan, D.,and Leal-Taixe, L. Better call sal: Towards learning to segment anything in lidar. In Eur. Conf. Comput. Vis., 2024.\n[2] Liu, Youquan, et al. \"Segment any point cloud sequences by distilling vision foundation models.\" Advances in Neural Information Processing Systems 36 (2023): 37193-37229.", "essential_references_not_discussed": "In LiDAR-based segmentation, numerous semi-supervised or weakly supervised methods [1, 2], which do not fully rely on (fully-) manually labeled datasets, have not been discussed. Additionally, related works [3] utilizing vision foundation models have also not been discussed. The authors should explicitly compare their method against these relevant methods.\n\n\n[1] Li Li, Hubert P. H. Shum, Toby P. Breckon; Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2023, pp. 9361-9371\n\n[2] Ozan Unal, Dengxin Dai, Luc Van Gool; Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2022, pp. 2697-2707\n\n[3] Liu, Youquan, et al. \"Segment any point cloud sequences by distilling vision foundation models.\" Advances in Neural Information Processing Systems 36 (2023): 37193-37229.", "other_strengths_and_weaknesses": "Strengths:\n- The paper introduces CAL (Complete Anything in Lidar), a novel zero-shot panoptic scene completion approach. It extends beyond traditional fixed taxonomies by learning object shape priors from unlabeled temporal Lidar sequences.\n- The paper is well-structured, with clear motivation and methodology.\n\nWeakness:\n- The method is compared to fully supervised baselines (Tab. 1), but an ablation against other zero-shot methods would strengthen the effectiveness. Although I acknowledge that the authors claim this is the first method for zero-shot panoptic scene completion in LiDAR, and therefore it might be difficult to find a second existing zero-shot method for direct comparison, perhaps a practical alternative could be modifying current supervised methods (e.g., by adding specific modules or other adjustments) to adapt them for zero-shot evaluation. \n- It is important to point out that there remains a substantial performance gap (Tab. 1) between existing zero-shot methods and supervised approaches. Consequently, I am not convinced that the zero-shot method proposed in this paper could be effectively applied to the application scenarios mentioned in Fig. 1.\n- The reliance on pre-trained 2D models (CLIP, SAM) may inherit their biases and limitations.", "other_comments_or_suggestions": "My concerns mainly focus on the performance (see Other Strengths And Weaknesses) and novelty (see Relation To Broader Scientific Literature) of the proposed method. If the authors can adequately address these concerns, I am willing to increase my rating.", "questions_for_authors": "N/A", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "3dliAAtVVk", "reviewer": "Reviewer_ptaj", "summary": "The paper proposes a zero-short learning method CAL (Complete Anything in Lidar) to use the temporal context from multi-modal sensor sequences to mine object shapes and semantic features that are then distilled into a Lidar-only instance-level completion and recognition model. The experiments on real-world lidar benchmarks demonstrate that the approach can do zero-shot shape completion with promising results.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their feedback. We are happy to hear that the reviewer found that our experiments demonstrate promising results for zero-shot shape completion. We are also glad that the reviewer has found our ablation studies thorough, and our method well-documented. Below, we address the comments and questions posed by the reviewer.\n\n**Q1. How good is pseudo-label accuracy in practice and when does it fail?**\n\nThis is a great question, and we acknowledge its importance. In the main paper, we carefully investigated pseudo-label accuracy through extensive experiments and discussions. As reported in *Section 4.3* and *Appendix C.2*, our experiments indicate strong overall pseudo-labeling accuracy. The accuracy of pseudo-labels primarily depends on recognizing and completing objects by tracking them across video frames, integrating multi-view observations, and leveraging zero-shot recognition via vision-foundation model features. In the following, we report the main findings from our experiments:\n\n- *Table 2* investigates pseudo-label accuracy when varying the number of tracked frames ($T_{fw}$ and $T_{bw}$), and the tracking stride $w$. Our experiments suggest that a sufficient number of frames is essential for completing objects based on observations across multiple views. Our model is generally robust to view changes, and CRF refinement allows to reduce the number of tracked frames (*Table 10*). We notice that failure cases primarily stem from the foundation model incorrectly switching tracking IDs (in case of strong view changes or occlusions), or failing to recognize objects, highlighting areas for further improvement.\n- *Table 3* evaluates pseudo-label accuracy by measuring coverage completeness against GT annotations. While pseudo-labels may initially provide partial scene completions due to tracking limitations, our CRF refinement significantly enhances label coverage. However, challenges persist when objects are completely undetected or are located in poorly visible regions.\n- *Table 4* evaluates pseudo-label quality on two datasets for both semantic-oracle and zero-shot settings (see last two rows). We notice that our pseudo-labels demonstrate strong performance even under zero-shot conditions, while revealing room for improvement in terms of the quality of vision-foundation model features for zero-shot recognition.\n\n**Q2. How sensitive is the approach to inaccuracies or biases from the 2D foundation models?**\n\nOur method is generally robust to minor errors from video foundation models like SAM2, thanks to both our pseudo-labeler and our training strategy. The pseudo-labeler enhances robustness by aggregating labels across frames, refining 3D masks per scan, and applying CRF refinement to improve label coverage. On the training side, we employ losses and training task formulations (see *Table 5*) that are specifically designed to help the model learn effectively from potentially-noisy pseudo-labels, ensuring it remains resilient even when pseudo-label coverage is imperfect. However, inaccuracies can still arise, particularly from 2D mask tracking failures, such as ID switches during significant view changes or heavy occlusions. To minimize these issues, our pipeline selectively generates pseudo-labels only for objects with high-confidence completions, filtering out lower-confidence outputs. Additionally, we also fine-tuned video-based tracking parameters (as shown in *Table 2*) to further reduce errors.  Importantly, our pseudo-labeling engine is modular, meaning that one can easily integrate improved 2D foundation models as they become available, which may translate to direct enhancements in pseudo-labeling performance. \n\n**Q3. The proposed pseudo-labeling approach is computationally heavy, which may limit applications to real-time scenarios.**\n\nWhile we acknowledge that our pseudo-labeling engine is computationally intensive, we optimize this by performing pseudo-labeling offline to generate a training dataset, which is then distilled into a single, efficient model during training. Once this model is trained, inference (i.e., completing a sparse LiDAR scan) only requires a single forward pass through the model, making it more suitable for real-time scenarios. Although real-time performance is not the main focus of this work, future efforts could improve the efficiency of our pseudo-labeler to further improve scalability.\n\nIn this direction, we note that the primary computational cost arises from the video foundation model used for object tracking across an RGB sequence. To improve efficiency, one may consider reducing the number of frames used during 2D mask propagation. Our preliminary results reported in *Table 10* suggest that we can (significantly) reduce the tracking horizon and minimize computational costs while achieving comparable performance.", "claims_and_evidence": "Yes, the claim “CAL performs zero-shot PSC in lidar ” is shown by quantitative experiments on two established datasets with comparions on zero-shot performance to baselines. Furthermore, the paper also shows qualitative results on the recognition of unlabeled classes.", "methods_and_evaluation_criteria": "Yes, the panoptic scene completion metrics, such as PQ, SQ, RQ, and mIoU are standard and appropriate for existing benchmarks \n including SemanticKITTI and KITTI-360.", "theoretical_claims": "The paper does not contain complex proofs or new theoretical analysis, and is more on experimental results. The pipeline looks standard and well-documented.", "experimental_designs_or_analyses": "The ablation studies are thorough, showing the impact of different design and pseudo-labeling choices. However, the pseudo-labeling approach is complicated, which may introduce errors, which makes the whole pipeline difficult to train. Hence, it would be good to explain this more to make the paper more solid and convincing.", "relation_to_broader_scientific_literature": "The paper is directly related to unsupervised learning and semi-supervised learning. Moreover, the paper is also related to large vision language models, which is helpful for multi-modal learning.", "essential_references_not_discussed": "None", "other_strengths_and_weaknesses": "Weakness\n1. The method heavily relies on the quality of 2D foundation models and multi-frame projection. Even though CRF-based refinements can compensate for partial or noisy coverage, the paper acknowledges that coverage remains imperfect.\n2. The whole pipeline looks computationally heavy: pseudo-labels are built using multi-frame and multi-sensor pipelines, which might be acceptable for offline training but limits applications on real-time scenarios.", "other_comments_or_suggestions": "None", "questions_for_authors": "1. How good is pseudo-label accuracy in practice? It would be good to provide some analysis of where the pseudo-label pipeline fails.\n2. How sensitive is the approach to inaccuracies or biases from the 2D foundation models?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: The paper received three "weak accept" ratings after the rebuttal. Two of the reviewers maintained their initially positive ratings.
The reviewer who initially gave a "weak reject" rating later changed it to a "weak accept," as the authors satisfactorily addressed the concerns regarding performance and novelty.
The area chair agrees with the consensus of the reviewers and therefore recommends a "weak accept" if there is room in the program.
Decision: Accept (poster)