TALoS: Enhancing Semantic Scene Completion via Test-time Adaptation on the Line of Sight
Authors: Hyun-Kurl Jang
Jihun Kim
Hyeokjun Kweon
Kuk-Jin Yoon
Venue: NeurIPS 2024 poster (2024)
Keywords: semantic scene completion
test-time adaptation
point cloud
Primary area: machine_vision
TL;DR: We propose TALoS, a novel test-time adaptation approach for SSC that leverages situational information available in driving environments.

Abstract: Semantic Scene Completion (SSC) aims to perform geometric completion and semantic segmentation simultaneously. Despite the promising results achieved by existing studies, the inherently ill-posed nature of the task presents significant challenges in diverse driving scenarios. This paper introduces TALoS, a novel test-time adaptation approach for SSC that excavates the information available in driving environments. Specifically, we focus on that observations made at a certain moment can serve as Ground Truth (GT) for scene completion at another moment. Given the characteristics of the LiDAR sensor, an observation of an object at a certain location confirms both 1) the occupation of that location and 2) the absence of obstacles along the line of sight from the LiDAR to that point. TALoS utilizes these observations to obtain self-supervision about occupancy and emptiness, guiding the model to adapt to the scene in test time. In a similar manner, we aggregate reliable SSC predictions among multiple moments and leverage them as semantic pseudo-GT for adaptation. Further, to leverage future observations that are not accessible at the current time, we present a dual optimization scheme using the model in which the update is delayed until the future observation is available. Evaluations on the SemanticKITTI validation and test sets demonstrate that TALoS significantly improves the performance of the pre-trained SSC model.

TALoS: Enhancing Semantic Scene Completion via Test-time Adaptation on the Line of Sight
Hyun-Kurl Jang" Visual Intelligence Lab.
Jihun Kim* Visual Intelligence Lab. KAIST
Hyeokjun Kweon" Visual Intelligence Lab. KAIST
KAIST jhg00018kaist. kc
j jihun1990kaiist.ard krr
0327june@kaist.cc ac.
Kuk-Jin Yoon Visual Intelligence Lab. KAIST kjyoon@kaist.ac.kk ac.
Abstract
Semantic Scene Completion (SSC) aims to perform geometric completion and semantic segmentation simultaneously. Despite the promising results achieved by existing studies, the inherently ill-posed nature of the task presents significant challenges in diverse driving scenarios. This paper introduces TALoS, novel test-time adaptation approach for SSC that excavates the information available in driving environments. Specifically, certain moment can serve as Ground Truth (GT) for scene completion at another moment. Given the characteristics of the LiDAR sensor, an observation of object at a certain location confirms both 1) the occupation of that location and 2) the absence of obstacles along the line sight from the LiDAR to that point. TALoS utilizes these observations to obtain self-supervision about occupancy and emptiness, guiding the model to adapt to the scene test time, In similar manner, we aggregate reliable SSC predictions among multiple moments and leverage them as semantic pseudo-Go for adaptation. Further, leverage future observations that are not accessible at the current time, we present a dual optimization scheme using the model in which the update is delayed until the future observation is available. Evaluations on the SemanticKITTI validation and test sets demonstrate that TALoS significantly improves the performance of the pre-trained SSC model. Our code available at httpt::////g thub. com/ htte/blue-550 Saaee-533TAALSS
focus that observations made at
Introduction
LiDAR is predominant 3D sensor in autonomous vehicles, effectively capturing the 3D geometry of the surroundings as point cloud. However, LiDAR inherently records the surface of objects, crucial for safety and driving
leaving the areas behind initial contact points empty. Therefore, planning to predict the state of these hidden regions using only the limited information available. Addressing these challenges, Semantic Scene Completion (SSC) has emerged as pivotal research topic, enabling simultaneous geometric completion and semantic segmentation of the surroundings.
mainly from architectural perspective. By amalgamating the models specialized for each task, these approaches have shown promising results the last few years. Nevertheless, the nature of the completion
Existing SSC studies have focused tackling both tasks 2 4
denotes equal contribution.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

task task-filling filling in the unseen parts from the given observation hervvill relies on the prior structural distribution learned from the training dataset. Therefore, our view, the classical SSC paradigm is inevitably vulnerable to handling the diverse scene structures encountered in driving scenarios.
As a remedy, this paper pioneers a novel SSC approach based Test-time Adaptation (TTA), which adjusts a pre-trained model to adapt to each test environment. Due to the absence of ground truths (GTs) during test time, the existing TTA studies for various fields have endeavored to design optimization goals, like meta-learning or auxiliary tasks [0] 12 14. Instead, we focus on the driving scenarios assumed by SSC, excavating the information helpful for adapting the model the scene in test time.
Our main idea is simple yet effective: observation made at moment could serve as supervision for the SSC prediction at another moment. While traveling through an environment, an autonomous vehicle can continuously observe the overall scene structures, including objects that were previously occluded (or will be occluded later), which are concrete guidances for the adaptation of scene completion. Given the characteristics of the LiDAR sensor, an observation of a point at specific spatial location at specific moment confirms not only the occupation at that location itself but also the absence of obstacles along the line of sight from the sensor to that location.
The proposed method, named Test-time Adaptation via Line of Sight (TALoS), is designed to explicitly leverage these characteristics, obtaining sulerviinn for geometric completion. Additionally, we extend the TALoS framework for semantic recognition, another key goal of SSC, by collecting the reliable regions only among the semantic segmentation results predicted each moment. Further, to leverage valuable future information that is not accessible at the time of the current update, devise novel dual optimization scheme involving the model gradually updating across the temporal dimension. This enables the model to continuously adapt to the surroundings at test time without any manual guidance, ultimately achieving better SSC performance.
We verify the superiority of TALOS on the Semantic KITTI 15] benchmark. The results strongly confirm that TALoS enhances not only geometric completion but also semantic segmentation per- formance by large margins. With extensive experiments, including ablation studies, we analyze the working logic of TALoS in detail and present its potential as viable solution for practical SSC.
Related Works
2.1
Semantic scene completion
Starting from SSCNet I1 semantic scene completion task has been extensively studied [] 6.7
2 3 4 5
22 23 In SSC using LiDAR, as both tasks of geometric completion and semantic understanding should be achieved simultaneously, the existing studies have mainly presented architectural approaches. For example, LMSC 4 and UtD (8) utilize UNet- based structures with multi-scale connections. JS3C-Net [5] and SSA-SC [6] propose architectures consisting of semantic segmentation and completion networks to utilize them complementarily. Although these approaches show promise, handling the diversities inherent in outdoor scenes remains challenging problem. In this light, we would like to introduce test-time adaptation to the field of semantic scene completion.
S 19 20
Notably, SCPNet 3 proposes use distillation during the training phase, transferring the knowledge from the model using multiple scans to the model using a single scan. Although this approach also aims to use information from various moments, the proposed TALoS is distinct as it leverages such information online, adapting the model to the diverse driving sequence. Also, the recently published OceFiner [24] is noteworthy, as it aims to enhance the already existing SSC model. However, OceFiner is a post- processioossig method that refines the results of the pre-traince model, performing in an offline manner, unlike online TTA-based approach.
2.2 Test-time adaptation
Test-time Adaptation (TTA) aims to adapt a pre-trained model to target data in test time, without access to the source domain data used for training. One widely used method involves attaching additional self-supervised learning branches to the model 10] 12]. point cloud TTAs, using actively studied. However, these
auxiliary tasks such as point cloud reconstruction [3]

---

approaches require the model to be trained with the additional branches, primarily in the training stage on the source dataset.
To relieve the requirements on the training stage, various online optimization goals have been explored,
such information maximization [25] 26 27 and pseudo laheling 226
28
29
30 schemes. Similar
approaches have also been proposed for the point cloud, as in
31 32 using pseudo labeling.
Unfortunately, despite the natural fit between these TTA approaches and the goal of SSC, which involves completing diverse driving environments, the use of TTA has been scarcely explored in the SSC field. Against this background, we pioneer the TTA-based SSC method, especially focusing on excavating the information from the point clouds consecutively observed at various moments.
3 Method
3.1
Problem definition
This section begins by defining the formulation of our approach and the notations used throughout the paper. The Semantic Scene Completion (SSC) task aims to learn mapping function from an input point cloud to the completed voxel representation. We formally denote the input point cloud X E N2 as set of points, where each point represents its XYZ coordinate. Following the conventional SSC studies, the completion result is denoted as Y E C Here, L, W, H are the dimensions of the voxel grid, and = {0, I..... C} is set of class indices indicating whether voxel is empty (0) or belongs to a specific class (I,...,C) C).
As our approach is based TTA, assume the existence of pre-trained SSC model F follows:
p= F(X),
(1)
where p [0, 1|C++)xXxx>H is the probability of each voxel belonging each class. The final class prediction Y can be obtained by applying an argmax function on p. In this context, the goal of TALoS is to adjust the pre-trained model to adapt to an arbitrary test sample X by optimizing the parameters, making them more suitable. Here, note that the proposed approach does not have explicit requirements on F, such as the architectures or pre-training policies.
3.2 Test-time Adaptation via Line of Sight (TALoS)
The proposed TALoS targets a realistic application of the pre-trained SSC model, assuming an autonomous vehicle drives through arbitrary environments in test time. In this scenario, we suppose the point clouds captured by LiDAR are continuously provided as time proceeds, and accordingly. the model should perform SSC for each given point cloud instantly. We denote the input sequence of point clouds {X where {I....] indicates the moment when the point cloud captured.
The main idea behind TALoS is that for guiding the model prediction of a certain moment (let i), the observation made another moment (let j) can serve supervision. However, the ego- vehicle moves as time proceeds, the input point clouds captured at the two different moments, i.e., X, and X,, on different LiDAR coordinates. To handle this, we use transformation matrix Ty- between two coordinates to transform the jth point cloud X5 with respect to the ith coordinate system, by
X = TT
where Xy-s is the transformed point cloud.
Subsequently, we exploit X, to obtain a binary self-supervision for geometric completion, indi- cating whether voxel is empty or occupied. We implement this process as shown in Fig. [ For voxelization, we first define a voxel grid a pre-defined size (L x W x H), initialized with an ignore index (255). Then, using X we set the value of voxels containing at least one point to I (green color in Fig. D. while keeping the other voxels as 255. Here, note that we discard the points of the non-static classes (e.g., car), as such object can change their location between ith and jth observation. For this rejection, we use Pj. the model prediction at jth moment. The resulting binary mask indicates which voxels are occupied the jth moment with respect to the ith coordinate.
Additionally, we use the Line of Sight (LoS), an idea utilized in various fields K3 identify which voxels should not be occupied. Considering LiDAR's characteristics, the space
35) to further
3

---

Transformed point cloud
Marking occupied voxels
Marking empty voxels
Example of obtained binary completion map
point de: LiDAR
ignored
occupied
LoS
empty
Figure 1: Left: Visualization of constructing a binary map 3-+; from the transformed point cloud
ji- Although we represent our process using a 2D grid for intuitive visualization, note that the real process is performed on a 3D voxel. Right: The real example of the binary map projected on 2D.
between the LiDAR and the voxels filled with should be empty. To check which voxels are crossed by the LoS, we employ Bresenham's 's algorithm [36]. For this process, we use the LiDAR position at the jth moment converted to the ith coordinate system, not the ith LiDAR position. Finally, we set the value of the identified voxels to 0 (red color in Fig. D], indicating that the voxels should be empty.
The obtained مーر {0, 1, 255] 2 Lxxx then serves as supervision for Pt. the ith prediction. Here, p, is obtained by the pre-tra pre-trained SSC model F as follows:
p; F(X.).
Since P; is the prediction for all the classes, including the empty class, convert it into the somp prediction for completion, denoted as p, comp E [0, xx xH Here, the first element of p. simply p and the second is max pf, the maximum value among the scores of the non-empty classes. Here, p represents the predicted probability of the voxels belonging to eth class ith moment.
Finally, the loss function using the binary completion map is as
Coomp (p:) = Lee (p. comp g + Liovaes (p. comp jー )
where Lee and Llovass are cross-entropy loss and lovasz-softmax loss 37 respectively.
3.3 Extension for semantic perception
In the previous section, we described a method that leverages an observation made in one moment (j) to obtain the binary occupancy map of another moment (i). As the method mainly focuses enhancing the model's 's capability of understanding the test-time scene structure, i.e., scene completion, this section further extends approach to address the semantic perception of surroundings, another key goal of SSC. Specifically, we carefully identify the reliable regions from the prediction of the pre-trained model each moment, and then build consensus among these predictions across various moments.
To achieve this, we first define metric similar to follows:
25 26 27
based Shannon entropy [38 H, as
R H(p) = - p log
--EP- c=0
(5)
where R E [0, 1] is the measured reliability. As reported in conventional works based on confidence-basee self-supervision, we also confirmed high positive correlation between the reliability R and the actual accuracy of voxel-wise classification, as in Fig. 2
To identify the confident regions from the model predictions, simply threshold the measured reliability by pre-defined value - (0.75 in ours) as follows:
aargmax, p p0 2.2) if R(
A v.
(6)
255
otherwise,

---

Accuracy (%)
90
85
Reliabilit
0.15
0.45
0.75
Prediction
CuT
Aggregated PGT
GT
Figure 2: Left: Verification of the reliability metric. The voxels having higher reliability show higher semantic completion accuracy. Right: Examples of pseudo-GT (pGT) construction. The blue box depicts the successful rejection of misprediction using reliability, while the red boxes show the benefit of using the prediction of another moment, providing more completed pGT.
where (x, y, =) denotes the voxel coordinate. Here, A E (CU (255)) LXW* the reliable self- supervision, which can function pseudo-GT for semantic segmentation.
Following the above process, we first acquire Aj using jth model prediction. Subsequently, we project its coordinate to ith coordinate, similar to Equ. (2). After obtaining the projected pseudo-GT, denoted as A, we aggregate with A, the pseudo-GT obtained at the current moment i. In detail, we replace only the unconfident voxels in A; (indexed as 255) with the corresponding voxels of A j-vi- Meanwhile, if the classes predicted by A, and A,- differ on certain voxels and both are confidently predicted, we conservatively drop those voxels as 255. As depicted in the colored boxes in Fig. right, this aggregation helps our framework build consensus among the semantic perceptions performed at various moments, significantly enhancing the quality of pseudo-GT from the perspective of SSC.
We denote the result of aggregation as Veen E (C U (255)) ( LxW*H, Since View contains semantic S information about all the classes, we can utilize as direct pseudo-GT for guiding P,. Accordingly, the loss function is defined as:
(p:) Lee (P:, V + Ltovasz (Pi, Veem). 3-+2)
where the notations are similar to those of Equ.
3.4 Dual optimization scheme for gradual adaptation
The previous sections introduced how TALoS guides the prediction of a pre-trained SSC model at certain moment (i), leveraging observations made at another moment (j). From a methodological perspective, the remaining step to consider how effectively adapt the model by selecting the appropriate moments.
Essentially, cannot observe the future. Therefore, assuming TTA in real driving scenarios, can only use past observations when updating the model at the current moment, which implies i > j. However, from the perspective of the SSC task, the main region of interest is intuitively the forward driving direction of the autonomous vehicle. This implies that guidance from future observations can be more important and valuable than guidance from past observations.
So, how can we leverage future information without actually observing it at the current moment? Our key idea is to hold the model and its prediction at the current moment without updating and delay the update until future observations become available.
We develop this idea as in Fig. In detail, our approach involves two models of FM and FG The goal of FM is an instant adaptation on the sample of the current moment. Therefore, we initialize FM every moment with the pre-trained model and discard it at the end of the moment. Here, FM can be instantly updated at the moment i, using the past information already observed at jth moment. The loss function for J.M is defined as:
C C (p.M) + (p
(8)
where MM is the output of FM ith moment.
5

---

TpM can be immediately updated at ith moment.
i updated when arrive at kh moment, where X is available.
k
X
X
FM
FM
FG
FM
Pj
L MM
Pi
D
P
P
P
talos
The update of F is delayed until kth moment.
Figure 3: Conceptual visualization of the dual optimization scheme. JFM is instantly updated at
moment i, using the past information provided from jth moment. On the other hand, the update of JG using ith prediction is delayed until kth moment, when the future information becomes available. We unify the predictions of the models, p.a and p, to get the final prediction p. paalos The red dashed line denotes the back- back-propagation.
However, as aforementioned, want to also leverage future information at kth moment, which is not available yet. For this, we define F which aims to gradually learn the overall scene distribution. Therefore, F is initialized only at the first step and continuously used for prediction. As in Fig. 3 the inference of JG for X, is instantly done and is used for the final output of the ith moment. On the other hand, the update of FG should stand by, until X is available. In other words, once the model JG arrives at kth moment (which was future at the time of prediction), the update performed. The loss function for JG is defined as:
(p + feem (p,"),
where p is the output of JGG at ith moment.
This update cannot directly affect the prediction of FG made ith moment, as is already over in the past from the perspective of kth moment when the update occurred. However, this continuous accumulation future information gradually enhances the model, allowing it to better learn the overall scene structure time progresses. We provide a detailed illustration in Algorithm
In summary, the proposed TALoS framework involves two models, moment-wisely adapted JMM and gradually adapted FG To obtain the final prediction palos of ith moment, we individually run JM and F using X; as an input. Both results p." and p are unified into a single voxel prediction. Here, we use D. as base, while trusting P only for the voxels predicted as static categories (such as roads or buildings) by p. The rationale behind this strategy is that continual adaptation makes FG gradually adapt to the overall sequence, leading to better understanding of the distribution of static objects. We empirically found that the continuous adaptation is more facilitated for the static pattern than the movable objects having diverse distribution.
4 Experiments
4.1 Settings
Datasets Metrics. We primarily experiment on SemanticKITTI [15], the standard benchmark for SSC, comprising 22 LiDAR sequences. Sequences 00 to 10 are used for pre-training SSC models, except for 08, which is employed as validation set. For testing, we use sequences 11 to 21. Additionally, we verify TALoS on cross-dataset evaluation from SemantickITTI to SemanticPOSS 39.. From the sequences of SemanticPOSS, we utilize only the validation sequence (02). For more details, please refer to the supplementary material. For evaluation, we employ intersection over union (IoU), a standard metric for semantic segmentation. We report both the completion IoU (cIoU) for binary occupancy prediction and the mean IoU (mloU) for all classes.

---

Algorithm Dual optimization scheme (single iteration)
1: Input: Moment-model F. Gradual-model FC, Pre-trained SSC model parameters lo temporal
distance and Buffer B
2: Output: Adapted gradual-model parameters 8G
3: Initialize gradual-mode parameters: @ <- 0g
4: for each timestep
to unknown T do
5: Initialize moment- model parameters: BM <- Bu
6: i.j t,
7: Receive current LiDAR observation X;
8:
Perform prediction with the moment-model: p 4- FM (X;; 0M)
(X;
9:
Perform prediction with the gradual-model: p F (X,; (.()) Save P. in the buffer B.
10:
11:
Save p and the corresponding forward propagation graphs of FG to the buffer B.
12:
j >== 1 then
13:
Load P from the buffer B.
Compute - using P, and P Update moment-model parameters: 8M +- BM - V GM L.M
15:
16:
Load p and the corresponding forward propagation graphs of JG from the buffer B. Compute Chs using p and D
18:
Update moment-mode parameters: 8 B v C.
19:
Perform prediction with the updated noment-modell p
M f- FMX;;; (M;
20:
Perform prediction with the updated gradual-model: p t- JG (g
end if
22:
pialos + Agg(p,, .p.')
23:
Return p. as the final SSC result of the current timestep i
24:
end for
25: Return adapted gradual-model parameters g
Implementation. We employ the officially provided SCPNet [3] which is pre-trained on the SemanticKITTI [15] train set, as baseline. To apply TALoS in test time, only update the last few layers of SCPNet. Additionally, to prevent SCPNet's architecture from automatically making the voxels far from existing points empty, tensor computation. This ensures that distant voxels are properly involved in the test-time adaptation. For optimization, we use Adam optimizers [40), where the learning rates are 3e-4 and 3e-5 for JM and FG, respectively. For more details, refer to Section
3D convolution layer to expand the region of sparse
4.2 Ablation studies
We conducted ablation studies evaluate the effectiveness of each component of TALoS, The configurations and results of the ablation studies are demonstrated in Table First, we verify the effectiveness of the loss functions we devised. The result of Exp A confirms that minimizing Coomp indeed increases both cloU and mloU performance over the baseline, helping the model adapt to each test sample. addition, Exp B shows that using _ also effective, especially for semantic perception, resulting in better mloU performance. Further, Exp C using both losses achieves even higher performance, demonstrating
effectiveness of the proposed TALoS.
Additionally, we check the validity of the dual optimization scheme, ablating either J or FG, The results of Exp C and Exp show that the use moment-wist adaptation and gradual adaptation are both effective. Further, Exp E confirms that our dual scheme effectively unifies both gains into our TALoS framework, significantly outperforming the baseline on both cloU and mloU.
4.3 Results on SemanticKITTI
Table 2 provides the performance of existing SSC methods on the SemanticKITTI test set. Compared with SCPNet, which serves as our baseline, the proposed TALoS achieves significantly higher performance on both cloU and mloU. Considering that SCPNet also involves knowledge distillation

---

Table 1: The results of ablation studies for the proposed TALoS framework, conducted on Se- MaTTicKITTI val set. COMP and SEM denote the use of loss function defined in Equ. 44) and Equ. 77 respectively. Meanwhile, MOMENT and GRADUAL represent the use of F and J for the optimization scheme in Sec. 3.4 respectively. All metrics are in %. Best results are in bold.
Loss functions
Dual optimization scheme
Metries
CОмР
SEM
MOMENT
GRADUAL
mloU
cloU
Baseline
37.56
50.24
A
37.97
52.81
B
38.35
52.47
38.38
52.95
38.81
55.94
(Ours)
39,29
56.09
Baseline
Ours
Ground Truth
Car
A CASSI
(a)
Thuck
Road
Parking
Sidkwalk
Building
Fence
Vegetation
(4)
Thunk
Temmin
Pole
Thafffic sign
Qualitative comparisons between baseline (SCPNet) and ours TALoS on SemanticKITTI val set. The highlighted regions depict the improvements achieved by TALoS, better completing the scene while also recovering the mispredictions.
Figure 4:
using future frames during training, this performance gain confirms that our TTA bTsedd method is more effective for leveraging future information. Figure 4 provides a qualitative comparison between baseline and ours, demonstrating the advantages TALoS.
Additionally, it is noteworthy to mention the difference between ours and OccFiner [24]. OccFiner designed refine the results of existing SSC methods an offline manner. It first generates predictions for LiDAR sequence using an SSC method and then fuses these predictions post-driving to refine the results. contrast, TALoS aims to perform test-time adaptation instantly in an online manner. We assume the sequential sensing of LiDAR data during driving, and TALoS gradually enhances the model the test-time adaptation progresses, As both methods have advantages their respective practical settings, we simply mention it here, rather than comparing them in Table 2
4.4 Comparisons with the existing TTA methods
To demonstrate the benefit of our approach from the perspective of TTA, we integrated existing TTA studies into our framework and tested them. The results can be found in Table First, performed optimization via entropy minimization, as TENT [25], instead minimizing the proposed loss functions. For this experiment, we exclusively use AM for clear comparison. This setting achieved 37.92% and 49.865 99.86% in mloU and cloU, respectively. Note that cloU of this setting is slightly lower than that of the baseline. Further, the setting of Exp C in Table which also uses FM only, still outperforms TENT in both metrics. These highlight the effectiveness of our losses, which utilize observations made various moments.
To further clarify the superiority our dual optimization scheme, we also implemented CoTTA 28 continuous TTA approach based the widely used student-teacher scheme. We update both FM

---

Table 2: Quantitative comparison between the existing SSC methods with our TALoS on Se-
ManticKITTI [1] test set. We use an online benchmark server for evaluation.
other vehervehiele
motorgyclist
other ground
motoreyclo
traffic sign
mloD
clol'
bicydle
truck
poison
bicyclist
200d
parking
Sidewalk
buildinng
Tenoc
Vegestation
Strmin
gole
Methods
SSA-SC
23.5
58.8
36.5
13,9
4.6
577
74
4.4
2.6
0.7
72.2
37.4
43.7
10,9
43.6
30.7
43.5
25.6
41.8
14.5
6.9
JS3C-Net 5
23.8
56.6
33.3
14.4
8.8
72
12.7
8.0
5.1
0.4
64.7
3499
39.9
14.1
39.4
30.4
43.1
19.6
40.5
18.9
15.9
S3CNet
45.6
41.5
79
39.5
31.0
24,3
SCPNet
36.7
56.1
46.4
33,2
34,9
13.8
28.2
24.7
1.8
68.5
51.3
49.8
30.7
38.8
44.7
46.4
40.1
48.7
40.4
25.1
TAL
37.9
60.2
46.4
34.4
36.9
14,0
30.0
30.5
27.3
2.2
73.0
51.3
53.6
28.4
40.8
45.1
50.6
38.8
51.0
40.7
24.4
Table 3: Comparisons between the proposed TALoS and the existing TTA methods
25
28
We use
SCPNet baseline and conduct evaluation on SemanticKITTI val set.
Methods Bascline
TENT [25]
Ours (Exp. C) CoTTA 228
Ours
mloU 37.56
37.92
38.38
36.55
39.29
cloU
50.24
49.86
52.95
50.61
56.09
and F are optimized using entropy minimization, where the update of JGG is assisted by CoTTA scheme. We verify that this setring achieves 36.55% of mloU and 50.61% % of cloU, where the mloU decreases from the baseline. The results strongly confirm the superiority of the proposed optimization goals and schemes, which effectively leverage the information from driving scenarios for SSC.
4.5 Experiments under the severe domain gap
We check the potential of TALoS under the test scenarios of a target domain considerably different from the source domain. Specifically, we tested SCPNet pre-trained SemanticKITTI, the driving sequences of SemanticPOSS. Unlike SemanticKITTI, which uses a 64-beam LiDAR, SemanticPOSS captured with 40-beam and targets campus rather than on typical roads, resulting in significantly different class distribution. Note that although TTA-based approaches could enhance performance by adapting the pre-trained model to the scene, the capability of the initial model itself still essential.
Table compares our performance with that of the baseline, which uses the pre-trained SCPNet without any adjustments. Unfortunately, due to the severe drastic gap between SemanticKITTI and SemanticPOSS, the mloU performance actually low for both methods, as we expected. Therefore, in this section, we would like focus on the significant improvement achieved by TALoS over the baseline. In particular, TALoS shows its potential by achieving an improvement of over 10 in cloU, which is less affected by changes in class distribution. We believe that combining TALoS with the prior studies targeting domain gaps could be an interesting direction for future research.
4.6 Additional experiments
Playback experiment. We further clarify the advantages of our continual approach with an intuitive experiment named "playback". Specifically, we first run TALoS the SemanticKITTI validation sequence, from start to the end. During this first round, we save the weights of the gradual model FGG certain moment. Subsequently, again on the same sequence from the start. Here, during this playback round, we do not update the continual model at all. If the FG indeed learned the distribution of the scene while not being biased to certain moment during the first round, the playback performance would be better than that of the first round. Table 5 verifies this expectation, where the playback performs better compared to not only the baseline but also the first round of TALOS. We also provide a qualitative comparison between their results in Fig. where the prediction of playback is clearly enhanced in terms of SSC.
initialize the JG with the saved weights, and run TALoS once
Impact of number of iterations. Table shows the impact of the number of iterations for updating Frl on the performance of TALOS. Notably, TALoS achieves significant gains in both mloU and cloU, even with a single iteration. Performance is enhanced as the number of iterations increases: however, we observe saturation after five iterations per sample. The results imply that the proposed method efficiently excavates the information we targeted, fully leveraging it with only few iterations.

---

Table Results of the cross-dataset evaluation, pre-training on SemanticKITTI [5]. and evaluating
on SemanticPOSS [39]. We compare the performance of TALoS with the baseline (SCPNet).
mloU
ctoU
person
rider
trunk
plants
traffie-sip
pole
building
fence
bike
ground
car
Methods
Bascline (SCPNet (1
7.6
25.2
0.8 0.3
16.8
0.6
9.5
28.7
10.0
3.8 7.2
Ours
9.6
36.2
1.0
0.6
3.5
3.5
27.7
0.9
8.2
31.6
6.1
9.4
13.2
Table 5: Results of the playback exp.
Table 6: Impact of the number of iterations.
Methods
Baseline
Ours
(Ours-Playback
# of iterations
O (baseline)
1
3 (ours)
cloU 50.24
56.09
56.91
cloU mloU
50.24
55.99
56.05
56.09
56.07
mloU 37.56
39.29
39.38
37.56
39.09
39.22
39.29
39.31
Baseline
Ours
Ours- Playback
Ground Truth
Figure 5: The results of the playback experiment. The red boxes depict the sequential improvements, implying that the gradual model indeed adapts to the scene as TTA proceeds.
Limitations
One of our approach's 's main limitations is that it addresses point clouds only. As there exist number of SSC approaches using image data, exploring TTA for SSC in this setting could be an interesting research direction. We believe that the main philosophy of this paper, using the observation of one moment to guide the prediction of another moment, can be seamlessly extended to the image- approaches, enhancing the practicality of SSC.
Further, we are aware that the current state of SSC performance is still in its infancy, and thereby the gain achieved by TTA can be seem to less meaningful. However, we strongly believe that the proposed TALoS can even more promising approach for autonomous driving, as the field of SSC grows in the future.
6 Conclusion
This paper pioneers a novel Semantic Scene Completion (SSC) approach based on Test-time Adapta- tion (TTA). The proposed method, named TALoS, focuses that observation made one moment can serve as Ground Truth (GT) for scene completion at another moment. For this, we present several approaches acquiring self-supervision that can be helpful in adapting the model to the scene in test time, from both the perspectives of geometric completion and semantic segmentation. To further capitalize on future information that inaccessible at the current time, introduced dual optimization scheme that delays the model update until future observations become available. Evaluations on the Semantic- KITTI validation and test confirmed that TALoS significantly enhances the performance of the pre-trained SSC model, compared to the existing TTA approaches.
Acknowledgement
This work was supported by the National Research Foundation Korea (NRF) grant funded by the Korea government (MSIT) (NRF2022R1 (NRF2022RA22BB330000000 A2B5B03002636).
10

---

References
[1] Shuran Song, Fisher Yu, Andy Zeng, Angel X Chang, Manolis Savva, and Thomas Funkhouser. Semantic scene completion from a single depth image. In Proceedings of the IEEE conference computer vision and pattern recognition, pages 1746-1754, 2017.
[2] Christoph B Rist, David Emmerichs, Markus Enzweiler, and Dariu M Gavrila. Semantic scene completion using local deep implicit functions on lidar data, IEEE transactions on pattern analysis and machine intelligence, 44:1007705-- 2021.
[3] Zhaoyang Xia, Youquan Liu, Xin Li, Xinge Zhu, Yuexin Ma, Yikang Li, Yuenan Hou, and Yu Qiao. Sepnet: Semantic seene completion on point cloud. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 17642 17651, 2023.
[4] Luis Roldao, Raoul de Charette, and Anne Verroust-Blondet. Lmsenet: Lightweight multiscale 3d semantic completion. In 2020 International Conference on 3D Vision (3DV), pages 111-119. IEEE, 2020.
[5] Xu Yan, Jiantao Gao, Li, Ruimao Zhang, 7hen Li, Rui Huang, and Shuguang Cui. Sparse single sweep lidar point cloud segmentation via learning contextual shape priors from seene completion. In Proceedings of the AAAI Conference Artificial Intelligence, volume 35, pages 3101-3109, 2021.
[6] Xuemeng Yang. Hao Zou, Xin Kong, Tianxin Huang. Yong Liu, Wanlong Li, Feng Wen, and Hongbo Zhang. Semantic segmentation assisted scene completion for lidar point clouds. In 2021 IEEE/RS/ International Conference on Intelligent Robots and Systems (IROS), pages
IEEE, 2021.
[7] Jiahui Zhang, Hao Zhao, Anbang Yao, Yurong Chen, Li Zhang, and Hongen Liao. Efficient semantic scene completion network with spatial group convolution. In Proceedings of the European Conference on Computer Vision (ECCV), pages 733-749, 2018
[8] Hao Zou, Xuemeng Yang. Tianxin Huang, Chujuan Zhang, Yong Liu, Wanlong Li, Feng Wen, and Hongbo Zhang. Up-to-down network: Fusing multi-scale context for 3d semantic scene completion. In 202/ JEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 16-23. IEEE, 2021.
[9] Martin Garbade, Yueh-Tung Chen, Johann Sawatzky, and Juergen Gall. Two stream 3d semantic scene completion. In Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recognition Workshops, pages 0-0, 2019.
[10] Yu Sun, Xiaolong Wang, Zhuang Liu, John Miller, Alexei Efros, and Moritz Hardt. Test-time training with self-supervision for generalization under distribution shifts, In International conference on machine learning, pages 9229-9248. PMLR, 2020.
[11] Yuejiang Liu, Parth Kothari, Bastien Van Delft, Baptiste Bellot-Gurlet, Taylor Mordan, and Alexandre Alahi Tu++: When does self-supervised test-time training fail or thrive? Advances in Neural Information Processing Systems, 1808-18811188
2021.
[12] laaxing Huang. Dayan Guan, Aoran Xiao, and Shijian Lu. Model adaptation: Historical contrastive learning for unsupervised domain adaptation without source data. Advances in Neural Information Processing Systems, 34:3635-3644 2021.
[13] M Jehanzeh Mirza, Inkyu Shin, Wei Lin, Andreas Schriebl, Kunyang Sun, Jaesung Choc, Mateusz Kozinski, Horst Possegger, In So Kweon, Kuk-Jin Yoon, et al. Mate: Masked autoencoders are online 3d test-time learners. In Proceedings of the IEFF/CVF International Conference on Computer Vision, pages (6709-16718, 2023.
[14] Ahmed Hatem, Yiming Qian, and Yang Wang. Point-lta: Test-time adaptation for point cloud registration using multitask meta auxiliary learning. Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 16494-16504, 2023.
[15] Jens Behley, Martin Garbade, Andres Milioto, Jan Quenzel, Sven Behnke, Cyrill Stachniss, and Jurgen Gall. Semantickitti: A dataset for semantic seene understanding of lidar sequences. In Proceedings of the IEEE/CVF international conference on computer vision, pages 9297-9307, 2019.
[16] Ran Cheng, Christopher Agia, Yuan Ren, Xinhai Li, and Liu Binghing, S3enet: A sparse semantic scene completion network for lidar point clouds. In Conference on Robot Learning, pages 2148-21666 PMLR, 2021.
[17] Yiming Li. Sihang Li, Xinhao Liu, Moonjun Gong, Kenan Li, Nuo Chen, Zijun Wang. Zhiheng Li, Tao Jiang, Fisher Yu, et al. Ssebench: A large-scale 3d semantic scene completion benchmark for autonomous driving. arXiv preprint 2306 230.000,, 2023.

---

[18] Xiaoyu Tian, Tao Jiang, Longfei Yun, Yucheng Mao, Huitong Yang, Yue Wang. Yilun Wang. and Hang Zhao. Occ3d: A large-scale 3d occupancy prediction benchmark for autonomous driving. Advances Neural Information Processing Systems, 36, 2024.
[19] Anh-Quan Cao and Raoul De Charette. Monoscene: Monocular 3d semantic scene completion. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 3991-4001, 2022.
[20] Yining Shi, Jiusi Li, Kun Jiang, Ke Wang, Yunlong Wang, Mengmeng Yang. and Diange Yang. Panosse: Exploring monocular panoptic scene reconstruction for autonomous driving. In 2024 International Conference on 3D Vision (3DV), pages 2219-1228. IEEE, 2024.
[21] Yiming Li, Zhiding Yu, Christopher Choy, Chaowei Xiao, Jose M Alvarez, Sanja Fidler, Chen Feng, and Anima Anandkumar. Voxformer: Sparse voxel transformer for camera-bant based 3d semantic scene completion. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 9087-9098,
2023.
[22] Yunpeng Zhang, Zheng Zhu, and Dalong Du Oceformer: Dual-path transformer for vision-based 3d semantic occupancy prediction. In Proceedings of the IEEE/CVF International Conference Computer Vision, pages 9433-9443 2023.
[23] Yuanhui Huang, Wenzhao Zheng, Yunpeng Zhang, Jie Zhou, and Jiwen Lu. Tri-perspective view for vision-based 3d semantic occupancy prediction. In Proceedings of the IEEF/CVF conference computer vision and pattern recognition, pages 9223-9232, 2023.
[24] Hao Shi, Song Wang, Jiaming Zhang, Xiaoting Yin, Zhongdao Wang. Zhijian Zhao, Guangming Wang. Jianke Zhu, Kailun Yang. and Kaiwei Wang. Ocefiner: Oftboard occupancy refinement with hybrid propagation. arXiv preprint arXiv:2403.00 085504 2024.
[25] Dequan Wang, Evan Shelhamer, Shaoteng Liu, Bruno Olshausen, and Trevor Darrell. Tent: Fully test-time adaptation by entropy minimization. arXiv preprint arXiv:2006. 1077: 2020.
[26] Jian Liang. Dapeng Hu. and Jiashi Feng. Do we really need to access the source data? source hypothesis transfer for unsupervised domain adaptation. In International conference on machine learning, pages 6028-6039. PMLR, 2020.
[27] Shuaicheng Niu, Jiaxiang Wu, Yifan Zhang, Zhiquan Wen, Yaofo Chen, Peilin Zhao, and Mingkui Tan. Towards stable test-time adaptation in dynamic wild world. orXiv preprint arXiv:2302. 12400, 2023.
[28] Qin Wang, Olga Fink, Lue Van Gool, and Dengxin Dai. Continual test-time domain adaptation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 7201-7211, 2022.
[29] Francois Fleuret et al. Uncertainty reduction for model adaptation in semantic segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 9613 9623,, 2021.
[30] Devavrat Tomar, Guillaume Vray, Behzad Bozorgtabar, and Jean-Philippe Thiran. Tesla: Test-time self-lcarning with automatic adversarial augmentation. In Proceedings of the IEEF/CVF Conference on Computer Vision and Puttern Recognition, pages 20341-20350, 2023.
[31] Inkyu Shin, Yi-Hsuan Tsai, Bingbing Zhuang, Samuel Schulter, Buyu Liu, Sparsh Garg. In So Kweon, and Kuk-Jin Yoon. Mm-tta: Multi-modal test-time adaptation for semantic segmentation. Proceedings of the IEFF/CVF Conference on Computer Vision and Pattern Recognition, pages 16928-16937, 2022.
[32] Haozhi Cao, Yuecong Xu, Jianfei Yang, Pengyu Yin, Shenghai Yuan, and Lihua Xie. Multi-modal continual test-time adaptation for semantic segmentation. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 18809-18819, 2023.
[33] Peiyun Hu, Jason Ziglar, David Held, and Deva Ramanan. What you see what you get: Exploiting visibility for 3d object detection, In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 1 10099 2020.
[34] Li Ding and Chen Feng. Deepmapping: Unsupervised map estimation from multiple point clouds. In Proceedings of the IEEF/CVF conference on computer vision and pattern recognition, pages 8650-8659, 2019.
[35] Chao Chen, Xinhao Liu, Yiming Li, Li Ding, and Chen Feng. Deepmapping2: supervup large-scale lidar map optimization. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 9306-933 2023.
12

---

[36] JE Bresenham. Algorithm for computer control of digital plotter ibm syst. J, 4()):305,,3,, 30, 1965.
[37] Maxim Berman, Amal Rannen Triki, and Matthew B Blaschko. The lovász-softmax loss: A tractable surrogate for the optimization of the interecttonn-vve--mii measure in neural networks. In Proceedings of the IEEE conference computer vision and pattern recognition, pages 4413 11.3-4421, 2018
[38] Claude Elwood Shannon. A mathematical theory of communication. The Bell system technical journal, 27(3):379-423, 1948.
[39] Yancheng Pan, Biao Gao, Jilin Mci, Sibo Geng, Chengkun Li, and Huijing 7hao. Semanticposs: A point cloud dataset with large quantity of dynamic instances. In 2020 IEEE Intelligent Vehicles Symposium (IV), pages 687-693. IEEE, 2020.
[40] Diederik P Kingma and Jimmy Ba. Adam: method for stochastic optimization. arXiv preprint arXiv:1412.6980, 2014.
[41] Hyeonseong Kim, Yoonsu Kang, Changeyoon Oh, and Kuk-Jin Yoon. Single domain generalization for lidar semantic segmentation. Proceedings of the IEEE/CVE Conference on Computer Vision and Puttern Recognition, pages 17587-17598, 2023.
[42] Inigo Alonso, Luis Riazuelo. Luis Montesano, and Ana Murillo. Domain adaptation in lidar semantic segmentation by aligning class distributions. arXiv preprint arXiv:2010. 122399 2020.
13

---

A
Implementation Details
This section provides more details about the implementation our method. We utilize a voxel size of 256 x 256 x 32, following SCPNet [3]. For cross-domain evaluation, we use the class mapping from semanticKITTI 15] to semanticPOSS [39 shown Table A.T following conducted using single NVIDIA RTX A6000.
All experiments
Meanwhile, during the test-time adaptation using TALoS, we only updated the segmentation sub- network and fixed the weights of the other modules of SCPNet. We experimentally observe that updating the whole network achieves slightly higher performance but marginal.
Table A.1: Class mapping from semanticKITTI [[] to semanticPOSS
39
KITTI POSS
bicycle
bicyelist
road, sidewalk
building
fance
vegetation plants
trunk
pole
traffic sign
bike
person
rider
ground
building
fence
trunk
pole
trafflc-sign
B Class-wise Results on SemanticKITTI Validation Set
We provide class-wise results the SemanticKITTI validation set in Table B.2 Notably, the proposed TAL outperforms the baseline (SCPNet B)) in most categories, in addition to the significant margins overall metrics, mloU and cloU, These results strongly confirm the superiority of our method.
Table B.2: Quantitative results on SemanticKITTI validation set.
motorcyele
other-vehicle
motorcyellist
other- ground
vegetation
miolU
cloU
bicyele
truck
persun
bicyelist
parking
Sidewalk
building
trnk
terrin
nole
traflic-
Methods
cur
SCPNer [3
37.6
22.9
TALo8
39.3
56.1
51.9
25.8
38.7
60.1
46.1
24.0
19.9
5.3
75.0
61.3
55.2
17.0
36.7
33.1
44.6
35.0
57.7
40.1
18.9
C Result on various baseline models
We conducted experiments using different architectures to prove that TALoS is a universally useful approach to various SSC models. As shown in Table C.3 TALoS meaningfully enhances SSC performance (mloU) across different architectures and datasets. We utilized the baseline weights trained from the training set of the dataset which validation was be performed. The results imply that TALoS can be promising solution for SSC in various settings.
Table C.3: Comparisons between the proposed TALoS and TENT 25
using various SSC models.
SSC method
Datasct
Baseline
TENT [25]
TALoS
SSCNet
KITTI-360
17.0
17.4 (+000)
(+0.4)
SSA-SC 6
SemanticKITTI
24.5
24.8 (+0.2)
25.3 (40.8)
SCPNet 3
SemanticKITTI
37.6
37.9 (+0.3)
39.3 (+1.7)
D Additional Experimental Results
All the experiments in Section are conducted on the SemanticKITTI validation set.
D.1 Impact of threshold
We verify the impact of т, the value thresholding the reliability for obtaining pseudo-GT, as in Equ. (6). As shown in Table the proposed method is quite robust to the change of т, and both mloU and cloU are saturated after a certain point (0.75). Based on this result, we the thresholding value to 0.75 by default in the main paper.
14

---

D.2 Effectiveness of noise
We verify the robustness the proposed method against the errors that possibly exist in LiDAR calibration, as shown in Table D.5] We model the eтrог by disturbing the transformation matrix in Equ. | with the noise sampled from the Gaussian distributions, which have standard deviation values listed in the first column of the table. Specifically, we add the noise to the angles of rotation and translation vectors, to acquire the disturbed projection matrices between the LiDAR coordinate systems. The results show that performance decreases as the level of noise increases, as expected. Nevertheless, we observe that the proposed TALoS achieves substantial performance even with the noise, still meaningfully higher than the baseline. These results imply the robustness and practicality of our method.
D.3 Impact of selecting moments
As the proposed TALOS explicitly leverages the observations made at various moments during adaptation, it is essential to select proper moments. To verify this, conduct experiments by varying the chosen moments, as shown in Table D.6) For example, frame difference 1 in Table selecting j
and k + moments. Here, need specially handle the boundary cases, e.g., at the start of the sequence (j<O) or the last of the sequence (k: is larger than the number of all the samples in the sequence). For these cases, simply do not use any losses relevant those moments. Furthermore, for frame difference O, we use A,, the pseudo- GT from the current moment, as the only self supervision
The results in Table D.6 show that performance decreases the selected moments are too far from the current step, the distant observation may not overlap with the current observation overly force the model to learn completely unpredictable regions, leading to bias. Considering these, we set the frame difference by default in setting.
Table D.4: Impact of changes in T, Table D.5: Effectiveness of Table D.6: Impact of selecting the thresholding for reliability.
noises conducted on semantic- different moments in TALoS.
Reliability
mloU
cloU
KITTI validation set.
Frame Diff.
mloU
cloU
0.63
39.14
53.42
Noise
mloU
cloU
38.00 66.66
54.81
0.7
39.24
55.8
0.05
38.58
54.09
39.29
56.09
0.75
39.29
56.09
0.03
38.76
54.49
2
39.21
55.94
0.8
39.28
56.05
39.29
56.09
3
39.14
55.92
39.09
55.91
D.4
Comparison with Fusion-based Approaches
To analyze the effect of F.M, we compare our experiments with temporal fusion-based approaches. Specifically, we devise naive temporal fusion methods for the previous and current timesteps in two different ways, named early and late fusion. In early fusion, we merge the raw point clouds of both the previous and current timesteps and use the fused point cloud as input for our baseline (the pre-trained SCPNet). On the other hand, in late fusion, we separately obtain the predictions of the baseline at each timestep and aggregate the results of different timesteps at the voxel level. Here, when the predicted classes differ, we trust the one with lower entropy (which means higher confidence).
Table D.7 compares the performance of these fusion-based approaches with that of TALoS. For fair comparison, we also replicate the performance of Exp. C in Table of the main paper to the 5th row. Note that this setting exclusively uses FM The results show that the cloU gain of TALoS exceeds the naive temporal fusions, both early and late. Also, for mloU, the fusion-based methods even harm the mloU performance, while TALoS achieves significant gain. Finally, is noteworthy that Exp. C. with FM only, still outperforms all the other fusion-based methods. This is strong evidence that FM indeed performs something more and better than the naive temporal fusion.
D.5 Computational overhead
We provide the required time per step of the baseline (SCPNet), conventional TTA methods, and TALoS in Table D.8 We use SCPNet baseline and conduct evaluation SemanticKITTI val set. Given the significant performance gains of TALoS, the result shows a reasonable trade-off of computational overhead and performance.
15

---

Table D.8: Time and performance for proposed TALoS and existing TTA methods 25 28
Method
Time per step (S) overhead (%)
mioU (%)
Baseline
2.26
37.56
TENT [25
4.09
37.92
CoTTA
6.14
36.55
TALoS
6.65
+194
39.29
Table D.7: Comparison with naive temporal fusion-based approaches.
Method
mioU (%)
cloU (%)
Bascline
37.56
50.24
Early fusion
52.85
Late fusion
37.11
52.49
Exp. C (with F only)
38.38
52.95
TALoS
39.29
56.09
16

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: We clearly state the main claims reflecting the contributions and scope of our paper in the abstract and introduction section.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
We discuss the limitations of methods in the conclusion section.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
17

---

Justification:
Our paper does not include theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibiliyy
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: We provide all the experiment settings or information needed to reproduce the main experimental results throughout the main paper and the supplemental material.
Guidelines:
The answer NA means that the paper does not include experiments. If the paper includes experiments, No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm. (b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome deseribe the particular way they provide for reproducibility, In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
18

---

Answer: [Yes]
Justification:
We will release the code upon acceptance.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.co cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (c.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https guwdes/dde/eeee Cooeeeoodeesssseeeeee //nips.com /ttc: ssiniioiicyp for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Does the paper specify all the training and test details (c.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary understand the results?
Question:
Answer: [Yes]
Justification:
We specify all the training and test details.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, appendix, as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification: As method is deterministic, believe that we do not have to report the statistical significance.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the enror bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
19

---

It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative enror rates).
If error bars reported in tables plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification: We clearly provide the information on the computer resources in the supple- mental material.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (c.g., preliminary or failed experiments that didn't 't make into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics http:///wwwip..ccpppp//////cceee
Answer: [Yes]
Justification:
We confirm all the guidelines the NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: believe that there no special societal impact of this work.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
20

---

The conference expects that many papers will be foundational research and not tied particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of MI..
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
This paper poses such risks.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Lieenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: We properly cite the source of the assets we used.
Guidelines:
The answer NA means that the paper does not existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
21

---

If this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: This paper does not release new assets.
Guidelines:
The answer NA means that the paper does not release new assets,
Researchers should communicate the details of the dataset/code/moodl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
This paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Ethies, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
22


Reviews:
[{"review_id": "yWHd0068PY", "reviewer": "Reviewer_MQ39", "summary": "The manuscript proposes an method for test time adaptation of semantic scene completion models. During test time the method updates two models: one causally based only on previous times and one (in a delayed way) on future times. The models are updated through harvesting freespace and occupied annotations from lidar point clouds, as well as confident semantic predictions as annotations from the other timestamps. This method improves the base pretrained model without TTA on the in-domain test set and can also be used to improve the base model on out-of-domain sequences to some degree.", "questions": "I do wonder how long that optimization takes per step? (Table 5)", "limitations": "The limitation that the method needs point cloud information is addressed in the conclusion.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The test time losses for TALoS are well motivated and sensible. Leveraging not only point but also the freespace information from LiDAR rays makes use of all available information from it. The Shannon entropy-based uncertainty measure for selecting which semantic voxels to use for TTA is well motivated in Fig 2. The idea of leveraging pose tracking to enable leveraging information from previous timesteps makes sense and seems to help (it is a bit questionable how much of a completion problem this then still is see weaknesses)\n- Overall the paper is well written and well illustrated. The figures (Fig 1 and 2 for example) are very helpful in conveying the message and ideas in the paper. The qualitative and quantitative experiments are comprehensive and clarify the choices made in designing TALoS as well as the improvement over the pretrained base model. It is good to see that other more generic TTA approaches are less performant that the proposed one (but the communication could have been more clean with a table see weaknesses)\n- It is great to see that TALoS only needs 1-3 optimization iterations to adapt to a new timestep. I do wonder how long that optimization takes per step? (Table 5)", "weaknesses": "- Approach is only shown to improve one method (SCPNet) in the experiments which weakens the (implicit) claim that TALoS is a universally useful approach for any SSC model.\n- The approaches that TALoS was compared against are not communicated well. I am not sure what I should compare the performance of the TENT and CoTTA adapted models too. Please put those into a table. Similarly the OccFiner results are just mentioned in text but not propperly reported in the table. To make the OccFiner result comparable to TALoS one could simply \"replay\" TALoS on the same sequence after TTA was performed, no? (Like in Sec 4.6)\n- Leveraging additional information from previous timestamps makes SSC less of a completion problem since it is quite possible that surfaces that are not observable in the current time are actually observed in the previous timesteps.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "### 1. Experiments using other SSC models\n\n| SSC method | Dataset       | Baseline | TALoS |\n|--------------|---------------|:----------:|:-------:|\n| SSA-SC      | SemanticKITTI | 24.5     | **25.3**  |\n| SSCNet       | KITTI-360      | 17.0     | **17.4**  |\n\nThank you for the constructive comment to clarify the general usefulness of TALoS further. In response, we conducted two additional experiments using different architectures and datasets: 1) SSA-SC with SemanticKITTI and 2) SSCNet with KITTI-360. For both, we utilized the officially released code, with baseline weights of SSA-SC and SSCNet pre-trained on the SemanticKITTI and KITTI-360 training sets, respectively. The evaluation is conducted on the validation set of each dataset. As shown in the table above, TALoS meaningfully enhances SSC performance (mIoU) across different architectures and datasets. The results imply that TALoS can be a promising solution for SSC in various settings.\n- - -\n### 2. Comparisons with TENT/CoTTA\n\n| Method | mIoU (%) | cIoU (%) |\n|----------|:-------:|:------:|\n| Baseline      | 37.56  |  50.24  |\n| TENT ($F^M$ only)  | 37.92 | 49.86 |\n| Exp. C ($F^M$ only)  | **38.38** | **52.95** |\n| CoTTA ($F^M$ and $F^G$)  | 36.55 | 50.61 |\n| Exp. E ($F^M$ and $F^G$)  | **39.29**  | **56.09** |\n\nWe conducted experiments using conventional TTA methods to elucidate the benefits of TALoS. However, we acknowledge that it could be unclear regarding which settings of our method should be compared with these techniques. We apologize for this ambiguity and would like to clarify the comparisons. All of the experiments were conducted on the SemanticKITTI validation set. \n\n**TENT**: TENT focuses on test-time adaptation on single data rather than continual adaptation on sequence. Therefore, as mentioned in lines 255-256, we exclusively used $F^M$ for this setting (without $F^G$). This setting is the same as that of **Exp. C** in Table 1 of the main paper, except using entropy minimization as its optimization goal instead of $\\mathcal{L}^{comp}$ or $\\mathcal{L}^{sem}$. The 3rd row of the table above presents the performance results of this TENT experiment. We have also replicated the performance of the Exp. C in the 4th row for convenient comparison. The results demonstrate that the optimization goal of TALoS is significantly more beneficial than TENT's entropy minimization.\n\n**CoTTA**: CoTTA aims for continual adaptation to sequential data. Therefore, as mentioned in lines 261-262, we used both $F^M$ and $F^G$ for CoTTA. However, unlike in TALoS, which is optimized by the proposed dual optimization scheme (refer to Section 3.4), in this setting, we performed a standard teacher($F^G$)-student($F^M$) optimization following CoTTA. Here, we use the prediction of the student model as the final SSC output. Also, similar to TENT, we use entropy minimization as the CoTTA setting's optimization goal. The 5th row of the table above provides the performance results of this CoTTA experiment. For a fair comparison, we have replicated the performance of **Exp. E**, which uses both $F^M$ and $F^G$, in the 6th row. Interestingly, we find that CoTTA fails to learn the continual distribution of the sequence, resulting in decreased mIoU. Conversely, TALoS effectively learns the gradual sequence, achieving significant gains in both cIoU and mIoU. This strongly supports the benefit of the TALoS optimization goal in fully leveraging information from sequential LiDAR data.\n- - -\n### 3. Comparison with OccFiner\n\nWe would like to clarify that a direct comparison between TALoS and OccFiner may not be pertinent, as our key objectives differ significantly. OccFiner is designed to refine the results of existing SSC methods in an \"offline\" manner. It first generates predictions for a LiDAR sequence using an SSC method and then fuses these predictions post-driving to refine the results. In contrast, TALoS aims to perform test-time adaptation instantly in an \"online\" manner. We assume the sequential sensing of LiDAR data during driving, and TALoS gradually enhances the model as the test-time adaptation progresses.\n\nIn our main paper and this rebuttal, we do not assert that one approach is superior to the other, as both methods have advantages in their respective practical settings. This is why we simply mention OccFiner in the text rather than directly comparing ours with it in Table 2 of the main paper. \n\nAdditionally, the purpose of the playback experiment is solely to verify whether the proposed dual optimization achieves gradual adaptation. We did not claim that the playback of TALoS is superior to OccFiner, nor did we intend to compare them directly, given that our primary goals differ. We hope this explanation helps clarify the research scope of our paper, and promise to include this discussion in the final version in more detail.\n- - -\n### 4. About leveraging additional information from previous timestamps\n\nWe understand this concern, as using observations from the previous timestep might seem to simplify the completion problem of the current timestep. As a response, we aim to demonstrate the unique contribution of this paper by showing that naively incorporating information from the previous timestep is actually less effective than TALoS.\nPlease refer to the global rebuttal above, which discusses the comparisons with the methods based on temporal fusion.\n\n- - -\n### 5. Computational overhead\n\n| Method | Time per step (s) | Overhead (%) | mIoU (%) |\n|--------------|:----------:|:-------:|:-------:|\n| Baseline   |  2.26   | -  | 37.56 |\n| TENT    | 4.09   | +81 | 37.92 |\n| CoTTA      |  6.14   | +171 | 36.55 |\n| TALoS    |  6.65   | +194 | **39.29** |\n\nThe table above provides the required time per step of the baseline (SCPNet), conventional TTA methods, and TALoS. Given the significant performance gains of TALoS, we believe this additional burden can be justified.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "vY103FR6Yk", "reviewer": "Reviewer_xDnF", "summary": "This work proposes a test-time adaptation method for 3d semantic scene completion from point clouds. Using the point cloud and predictions of past and future frames in a sequence the authors propose a self-supervision strategy to adjust the model weights during test time. They propose a completion loss based on the line of sight of each measured point and a semantic loss using pseudo labels from predictions at different points in time. Furthermore they propose to use two versions of the model, one that updates the model based on the information at one moment in time and another that gets updated continuously throughout the sequence of test images.", "questions": "The main question is how much of the TTA actually related to temporal fusion and how much is statistical fine-tuning of the model. It seems that for F_M it can only be fusion, as the model is optimized for each frame from the original model, but for F_G it would be interesting to instead of playback the same sequence use a different sequence to see if it also helps statistically.\nAs it seems to be more of a fusion method, it would be good to compare against other temporal/multi-view scene completion baselines e.g. simply fusing measurements or predictions.", "limitations": "I think the limitations were addressed.", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "The authors propose a novel test time adaptation (TTA) framework for semantic scene completion based on the temporal line-of-sight consistency.\nThe presentation of the method is clear.", "weaknesses": "The instant adaptation of F_M(oment) seems like a indirect fusion of previous point clouds. The method should then also be compared to using the fused point cloud as for the SCNet teacher and fusing the predictions at i and j directly.\nRelated to this it is also unclear what distance is used between i and j and how this effects the method or if using multiple frames is even better.\nThe fusion of p_m and p_G is not explained in detail, while the intuition of using only the static parts from p_G makes sense the actual implementation remains unclear.\nIt is only mentioned that the last \"few\" layers of the network are updated at test time. For reproducibility it would be good to provide the details.\nThe authors write \"Considering that SCPNet also involves knowledge distillation using future frames during training, this performance gain confirms that our TTA-based method is more effective for leveraging future information\". Could you elaborate on this argument? As from my point of view in SCPNet they use samples from the training data for the teacher model, here the samples could actually overlap the same spatial area and provide additional measurements.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "### 1. Comparison with temporal fusion\n\nWe deeply appreciate the reviewer's valuable feedback.\nAs a response, we would like to clarify that TALoS indeed performs \"something more\" than a naive temporal fusion. \nBesides, we also experimentally verified that this statement is still true even when using the $F^M$ only (without statistical adaptation of $F^G$).\nFor this, we devised temporal fusion methods at the level of either measurements (raw LiDAR point cloud) or predictions (SSC result of the baseline model).\nPlease refer to the global rebuttal above, which discusses temporal fusion.\n- - -\n### 2. Distance between $i$ and $j$\n\n| Frame difference | mIoU (%) | cIoU (%)| \n|:--------------:|:---------------:|:----------:|\n|0| 38.66| 54.81|\n|1| **39.29**| **56.09**|\n|2| 39.21| 55.94|\n|3| 39.14| 55.92|\n|4| 39.09| 55.91|\n\nIn the main paper, we set the distance between $i$ and $j$ to 1 by default. \nThe experimental results regarding this distance are provided in Section C.3 of the supp. material (refer to pages 12-13 of the main paper). \nFor convenience, we have replicated Table C.5 here as the table above.\nAlthough not mentioned in the paper, we also tested using multiple frames. However, considering the increased computational burdens, this approach did not yield meaningful gains.\n- - -\n### 3. Missing details for implementation\n\nWe apologize for missing some details and promise to add them to the final version of our paper.\n\n**Details about categories for fusion**: We make $F^G$ be responsible for static and large-scale categories, which are relatively stable to learn from the sequence gradually. On the other hand, more confusing movable or small objects are handled by $F^M$ via instant adaptation. For this, we consider the SemanticKITTI's official category hierarchy.\nSpecifically:\n* $F^G$: ground (road, sidewalk, parking, and other-ground), structure (building), and nature (vegetation, trunk, and terrain)\n* $F^M$: vehicle (car, truck, bicycle, motorcycle, and other-vehicle), human (person, bicyclist, and motorcyclist), and object (fence, pole, and traffic sign)\n\n**Details about architecture**: In the main paper, we use the pre-trained SCPNet as our baseline. It comprises 3 modules: 1) a stack of MLPs for extracting point features, 2) a completion sub-network, and 3) a segmentation sub-network.\nIn TALoS, we update 3) only while simply freezing 1) and 2). We have also tested updating the entire model, but the gains were trivial, considering the increased computational burdens.\n- - -\n### 4. Statement about SCPNet and future information\n\nAs pointed out by the reviewer, SCPNet uses future information to train the teacher model and distills this knowledge to the student model during the training phase. In contrast, the goal of TALoS is to leverage information from different timesteps during test time. Since we verified that TALoS achieves meaningful gains over the SCPNet baseline, we intended to convey that our approach provides a more explicit and advantageous method of utilizing future information than SCPNet's implicit approach, particularly for test-time adaptation. Nevertheless, we acknowledge that the statement in lines 237-239 is confusing and not adequately justified. Thank you for the feedback, and we will revise this statement in the final version.\n- - -\n### 5. Playback on the different sequence\n\nWe find that this is an insightful question for understanding TALoS. We deeply appreciate this feedback and the opportunity to explain this in more detail. As pointed out, we intend for $F^M$ to be responsible for temporal fusion while employing $F^G$ to perform statistical adaptation to the sequence. Here, as we explained in the first response, $F^M$ achieves more than naive temporal fusion.\n\nThe question now is, does $F^G$ indeed perform a continual adaptation of the scene over the sequence? We provided the playback experiment as an indirect answer to this. However, we acknowledge that performing playback on different sequences instead of the same sequence could be an interesting experiment. To this end, we thoroughly compared the three different settings below. Here, sequence A is the validation sequence (08) of the SemanticKITTI dataset, and sequence B is the test sequence (19). \n\n1. Baseline (SCPNet, just inference on sequence A)\n2. TALoS (first round on sequence A with $F^G$ only; refer to Exp. D in Table 1 of the main paper)\n3. Playback TALoS (first round on sequence B and playback inference on sequence A)\n\nFor setting (3), we obtained the adapted $F^G$ after the first round, as in the playback experiment in the main paper. Then, during the playback, we performed inference while fixing $F^G$.\nNote that we do not use $F^M$ in these experiments.\n\nWe provide the results as a **cumulative** mIoU plot of each setting in the attached PDF, where the x-axis is the frame number of sequence A, from 0 to 814.\nThe results suggest two interesting insights:\n\n* The gain from the gradual adaptation of $F^G$ remains even on a different sequence to some extent. For example, at the early stage of the playback on sequence A, setting (3), which adapted to sequence B, significantly outperforms both (1) and (2). This indicates that setting (3)'s first round on sequence B indeed makes $F^G$ learn the statistics of the general driving scene, and this knowledge is still useful for performing SSC on the different sequence A. Consequently, setting (3)'s performance is meaningfully better than the baseline, even without any adaptation to sequence A at all.\n\n* Despite the generalizability of the knowledge learned by $F^G$, ultimately, setting (2) outperforms setting (3). This implies that sequence-wise variance exists. Although sequences A and B share some common distributions about the driving scene, their statistics cannot be completely identical. Therefore, from the perspective of sequence A, even if $F^G$ adapted to sequence B is better than the baseline, it cannot exceed the $F^G$ adapted to sequence A itself.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "cVw0dOCoJW", "reviewer": "Reviewer_9T82", "summary": "The paper introduces TALoS, a novel test-time adaptation method for Semantic Scene Completion (SSC) that leverages information from driving environments. The key idea is to use observations and predictions from different moments as self-supervision signals to adapt the pretrained model. Specifically, LiDAR point clouds collected at various timestamps are used to create binary occupancy labels, and semantic predictions from these timestamps are fused based on their confidence levels. Additionally, a dual optimization scheme is proposed to exploit observations from both past and future frames. Experimental results on the SemanticKITTI dataset validate the effectiveness of the proposed method.", "questions": "Can the authors discuss potential extensions of their method to other sensor modalities, such as camera data, and how their approach could be adapted to handle combined sensor inputs?\n\nCan the authors provide additional details and clarification on the implementation and functioning of the dual optimization scheme (such as a pseudo algorithm)?\n\nCan the authors discuss the scalability and practicality of their approach in more detail, including potential challenges and solutions for real-world implementation?", "limitations": "The authors have addressed several limitations in their work, but there are areas where additional detail and clarity would be beneficial. Here are some suggestions for improvement:\n\n### 1. Computational Overhead\n**Current Limitation:** The method involves significant computational overhead due to the creation of binary occupancy labels and the training of two models at each timestamp.  \n**Suggestion:** Provide a detailed analysis of the computational demands, including the computation-performance tradeoff and the impact on frames per second (FPS). This would help to understand the practical feasibility of the approach.\n\n### 2. Dataset and Dynamic Objects\n**Current Limitation:** The evaluation primarily relies on the SemanticKITTI dataset, which is small-scale and does not adequately handle dynamic objects.  \n**Suggestion:** Extend the evaluation to larger and more diverse datasets such as KITTI-360, nuScenes, and Waymo in SSCBench would provide a more comprehensive validation of the method.\n\n### 3. Applicability to Different Sensor Modalities\n**Current Limitation:** The current approach is tailored to LiDAR data, which may limit its applicability to other sensor modalities or combined sensor data.  \n**Suggestion:** Discuss potential extensions to other sensor modalities, such as camera or radar data, and how the approach could be adapted to handle combined sensor inputs. This would enhance the method's applicability and robustness in real-world scenarios.\n\n### 4. Literature Survey\n**Current Limitation:** The literature survey is incomplete, missing several important SSC works.  \n**Suggestion:** Expand the related work section to include a more comprehensive review of existing SSC literature, including both datasets and methods [1-7]. This would provide better context and background for the study.\n\n### 5. Potential Negative Societal Impact\n**Current Limitation:** The potential negative societal impacts of the work have not been explicitly addressed.  \n**Suggestion:** Discuss any potential negative societal impacts, such as privacy concerns or misuse of the technology, and suggest mitigation strategies. Being upfront about these aspects would align with the NeurIPS guidelines on broader societal impacts and contribute to responsible AI research.", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The key idea of exploring the LiDAR observations and model predictions across frames during inference is novel and interesting. The Line of Sight binary occupancy and confidence-aware prediction fusion are effective and interesting. \n\n2. The proposed dual optimization scheme is also useful and interesting. Such a strategy can make sure to exploit both historical and future frames for test-time adaptation.\n\n3. The method shows improvements in both geometric and semantic segmentation tasks on the SemanticKITTI dataset, demonstrating its effectiveness. The paper includes extensive experiments, ablation studies, and cross-dataset evaluations, providing a thorough analysis of TALoS's performance and robustness.", "weaknesses": "1. My major concern lies in the evaluation. SemanticKITTI is a small-scale dataset and does not adequately address dynamic objects in the creation of ground truth SSC labels. The authors are suggested to conduct experiments on larger and more diverse datasets, such as KITTI-360, nuScenes, and Waymo, to better validate their approach. The authors are suggested to consider using SSCBench [1], which employs the same format as SemanticKITTI. Cross-dataset evaluation on SSCBench would significantly strengthen the paper. Furthermore, Occ3D [2] could also be utilized for evaluation, though it uses a different format than SemanticKITTI.\n\n2. My second concern pertains to the computational overhead introduced by the proposed method. The test-time adaptation involves multiple modules, such as creating binary occupancy labels and training two different models at each timestamp, which can significantly increase computational demands. I recommend that the authors conduct a detailed computational analysis, including an evaluation of the computation-performance tradeoff and the frames per second (FPS) achieved.\n\n3. The literature survey is notably incomplete. Several important SSC works are missing from the related work section, including both datasets and methods [1-7]. A more comprehensive review of the existing literature is necessary to provide proper context and background for your study. Meanwhile, the idea of Line of Sight has been studied in prior works, such as 3d object detection [8] and point cloud registration [9,10]. The authors are suggested to add the missing references.\n\n4. The current approach is tailored to LiDAR data, which may limit its applicability to other sensor modalities or combined sensor data. The authors are suggested to add more details about how to use their method in the camera-based SSC methods.\n\n\n[1] Li, Y., Li, S., Liu, X., Gong, M., Li, K., Chen, N., Wang, Z., Li, Z., Jiang, T., Yu, F. and Wang, Y., 2023. Sscbench: A large-scale 3d semantic scene completion benchmark for autonomous driving. arXiv preprint arXiv:2306.09001.\n\n[2] Tian, X., Jiang, T., Yun, L., Mao, Y., Yang, H., Wang, Y., Wang, Y. and Zhao, H., 2024. Occ3d: A large-scale 3d occupancy prediction benchmark for autonomous driving. Advances in Neural Information Processing Systems, 36.\n\n[3] Cao, A.Q. and De Charette, R., 2022. Monoscene: Monocular 3d semantic scene completion. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (pp. 3991-4001).\n\n[4] Shi, Y., Li, J., Jiang, K., Wang, K., Wang, Y., Yang, M. and Yang, D., 2024, March. PanoSSC: Exploring Monocular Panoptic 3D Scene Reconstruction for Autonomous Driving. In 2024 International Conference on 3D Vision (3DV) (pp. 1219-1228). IEEE.\n\n[5] Li, Y., Yu, Z., Choy, C., Xiao, C., Alvarez, J.M., Fidler, S., Feng, C. and Anandkumar, A., 2023. Voxformer: Sparse voxel transformer for camera-based 3d semantic scene completion. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition (pp. 9087-9098).\n\n[6] Zhang, Y., Zhu, Z. and Du, D., 2023. Occformer: Dual-path transformer for vision-based 3d semantic occupancy prediction. In Proceedings of the IEEE/CVF International Conference on Computer Vision (pp. 9433-9443).\n\n[7] Huang, Y., Zheng, W., Zhang, Y., Zhou, J. and Lu, J., 2023. Tri-perspective view for vision-based 3d semantic occupancy prediction. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition (pp. 9223-9232).\n\n[8] Hu, P., Ziglar, J., Held, D. and Ramanan, D., 2020. What you see is what you get: Exploiting visibility for 3d object detection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (pp. 11001-11009).\n\n[9] Ding, L. and Feng, C., 2019. DeepMapping: Unsupervised map estimation from multiple point clouds. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition (pp. 8650-8659).\n\n[10] Chen, C., Liu, X., Li, Y., Ding, L. and Feng, C., 2023. Deepmapping2: Self-supervised large-scale lidar map optimization. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (pp. 9306-9316).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "### 1. Evaluation on the other datasets\n\n| SSC method | Train dataset | Validation dataset | Baseline | TALoS |\n|--------------|--------------|---------------|:----------:|:-------:|\n| SSCNet       | KITTI-360 | KITTI-360  | 17.0     | **17.4**  |\n| SCPNet       | SemanticKITTI | KITTI-360   | 14.0     | **14.7**  |\n\nWe deeply appreciate the reviewer's constructive comment and for informing us about the excellent benchmark. Unfortunately, we found that the official repository of SSCBench does not provide evaluation GT for the nuScenes and Waymo datasets. Therefore, we conducted an experiment on the KITTI-360 dataset instead.\n\nSpecifically, we used the SSCNet weights pre-trained on the KITTI-360 training set as our baseline. We adapted the baseline model to the KITTI-360 test sequence using TALoS. The second row of the table above shows that TALoS provides a meaningful mIoU gain even on the KITTI-360 dataset.\n\nAdditionally, we performed an additional cross-dataset evaluation from SemanticKITTI to KITTI-360. In this experiment, we used the SCPNet pre-trained on SemanticKITTI as our baseline. Then, we applied TALoS to adapt the baseline model to the KITTI-360 test sequence. The third row of the table above demonstrates a reasonable performance gain in mIoU, supporting the usability of the proposed TALoS.\n\n- - -\n### 2. Computational overhead\n| Method | Time per step (s) | Overhead (%) | mIoU (%) |\n|--------------|:----------:|:-------:|:-------:|\n| Baseline   |  2.26   | -  | 37.56 |\n| TENT    | 4.09   | +81 | 37.92 |\n| CoTTA      |  6.14   | +171 | 36.55 |\n| TALoS    |  6.65   | +194 | **39.29** |\n\nThe table above provides the required time per step of the baseline (SCPNet), conventional TTA methods, and TALoS. Given the significant performance gains of TALoS, we believe this additional burden can be justified.\n- - -\n### 3. Incomplete literature survey\n\nWe apologize for the incomplete literature survey. We promise to revise the related works section to include the mentioned SSC studies and various methods based on the LoS concept. Thank you for the constructive feedback.\n- - -\n### 4. Extension to camera-based setting\n\nSince this paper's primary focus is LiDAR-based SSC, we did not extensively explore camera-based settings. However, we agree that extending our framework to camera-based or multi-modal settings would be both interesting and valuable. In response, we devised and evaluated the proposed TALoS under a camera-only setting.\nSpecifically, we employed MonoScene, an SSC method using a monocular camera, as our baseline. Unfortunately, our main idea of using Line of Sight (LoS) cannot be extended to the camera-only setting, as it assumes LiDAR input. Therefore, for this camera-based TALoS, we exclusively used our entropy-based pseudo-labeling and a dual optimization scheme.\n\nThe experiments were conducted on the SemanticKITTI validation set, with the baseline MonoScene model pre-trained on the training set. Interestingly, TALoS still enhances the mIoU of the baseline model from 11.3% to **11.8%**, even in the camera-only setting. This further supports the practicality of TALoS as a viable option for the SSC field.\nAlthough we could not conduct experiments under a multi-modal setting due to limited time, combining our original LiDAR version with the camera-based attempt could be even more promising. Thank you for suggesting constructive research directions.\n- - -\n### 5. Additional details about the dual optimization scheme\n\nWe provide an algorithm table of the proposed dual optimization scheme in the form of the pdf file attached to the global rebuttal above. For better understanding, we set the number of iterations per sample as 1 in the algorithm table.\n- - -\n### 6. The various limitations\n\n* Computational overhead: Please refer to the 2nd response.\n* Dataset and dynamic objects: Dynamic objects present a challenge for the proposed TALoS, as simply transforming such objects from one timestep to another cannot ensure their actual position. Therefore, in TALoS, we exclude these unreliable points while sampling the line of sight (refer to lines 119-121). We believe that the integration of the trajectory prediction model could be an interesting research direction. Besides, please refer to the 1st response for experiments on the KITTI-360 dataset.\n* Applicability to different sensor modalities: Please refer to the 4th response.\n* Literature survey: Please refer to the 3rd response.\n* Potential negative societal impact: One possible issue is that TALoS will sense surrounding data and use it to enhance the model in an online manner. As private data might be gathered and exploited through this process without agreement, it could present a potential problem in practice. This may be mitigated by private filtering. While this issue is beyond our current research interest, we will mention it as a societal impact in the final version if necessary.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We deeply appreciate the reviewer's valuable feedback and will incorporate the suggestions into our paper revision. 
We also sincerely thank the reviewer for acknowledging the following strengths of our paper: 
* Novelty and soundness of TALoS (MQ39, xDnF, and 9T82)
* Comprehensive qualitative and quantitative experiments (MQ39 and 9T82)
* Good writing quality and helpful illustrations in conveying the ideas (MQ39). 

In the reviewer-wise rebuttal below, we address the weaknesses and questions raised by each reviewer point by point. We hope this clarifies the contribution of the proposed TALoS.

Finally, the attached PDF file includes the following items:
* Experimental results of the playback on the different sequence (please refer to the 5th response for reviewer xDnF)
* Algorithm table of our dual optimization scheme (please refer to the 5th response for reviewer 9T82)
- - -

Meanwhile, in this global rebuttal, we address the concern commonly raised by two reviewers (MQ39 and 9T82). 
This concern is mainly about the difference between the proposed TALoS, especially $F^M$, and the temporal fusion.
Is the role of $F^M$ identical with a simple temporal fusion of the observations from multiple timesteps or something more?
To answer this question and demonstrate this paper's unique contribution, we experimentally verified that naively incorporating information from the previous timestep is actually less effective than TALoS.

Specifically, we devised naive temporal fusion methods for the previous and current timesteps in two different ways, named early and late fusion. In **early fusion**, we merge the raw point clouds of both the previous and current timesteps and use the fused point cloud as input for our baseline (the pre-trained SCPNet). On the other hand, in **late fusion**, we separately obtain the predictions of the baseline at each timestep and aggregate the results of different timesteps at the voxel level. Here, when the predicted classes differ, we trust the one with lower entropy (which means higher confidence).

| Method | mIoU (%) | cIoU (%) |
|--------------|:----------:|:-------:|
| Baseline     | 37.56 | 50.24  |
| Early fusion | 35.86 | 52.85  |
| Late fusion | 37.11  | 52.49  |
| Exp. C (with $F^M$ only) | 38.38 | 52.95  |
| TALoS | **39.29**  | **56.09**  |

The table above compares the performance of these fusion-based approaches with that of TALoS. All of the experiments are conducted on the SemanticKITTI validation set. For a fair comparison, we also replicated the performance of Exp. C in Table 1 of the main paper to the 5th row. Note that this setting exclusively uses $F^M$. The results show that the cIoU gain of TALoS exceeds the naive temporal fusions, both early and late.
Also, for mIoU, the fusion-based methods even harm the mIoU performance, while TALoS achieves significant gain. 
Finally, it is noteworthy that Exp. C, with $F^M$ only, still outperforms all the other fusion-based methods.
This is strong evidence that $F^M$ indeed performs something more and better than the naive temporal fusion.

To sum up, naive temporal fusion suffers from identifying semantically correct predictions. On the contrary, TALoS effectively aggregates the reliable observations from the multiple timesteps and leverages them to enhance the model instantly.
We believe that this advantage is the unique contribution of our TALoS, a TTA-based approach for the SSC task.

Meta-review: This paper proposes a novel method for semantic scene completion. The paper initially received all borderline ratings (BR, BR, BA). During the discussion stage, the additional insight and results discussed by the authors in their rebuttal appeared to have solved most of the concerns from the reviewers, who unanimously suggested to weak accept this work. The final recommendation from the AC is thus to accept the work - congratulations!
Decision: Accept (poster)