Self-Improving Semantic Perception for Indoor Localisation
Authors: Hermann Blum
Francesco Milano
René Zurbrügg
Roland Siegwart
Cesar Cadena
Abel Gawel
Venue: CoRL2021 Poster (2021)
Keywords: continual learning
self-supervised learning
online learning

Abstract: We propose a novel robotic system that can improve its perception during deployment. Contrary to the established approach of learning semantics from large datasets and deploying fixed models, we propose a framework in which semantic models are continuously updated on the robot to adapt to the deployment environments. By combining continual learning with self-supervision, our robotic system learns online during deployment without external supervision. We conduct real-world experiments with robots localising in 3D floorplans. Our experiments show how the robot's semantic perception improves during deployment and how this translates into improved localisation, even across drastically different environments. We further study the risk of catastrophic forgetting that such a continuous learning setting poses. We find memory replay an effective measure to reduce forgetting and show how the robotic system can improve even when switching between different environments. On average, our system improves by 60% in segmentation and 10% in localisation accuracy compared to deployment of a fixed model, and it maintains this improvement while adapting to further environments.

Self-Improving Semantic Perception for Indoor Localisation
Hermann Blum ASL, ETH Zürich blumh@ethz.cm ch
Framcesco Milano ASL, ETH Zürich fmilano@ethz.cm ch
René Zurbrüge ASL, ETH Zürich zrene@ethz.co ch
Roland Siegwart ASL, ETH Zürich rsiegwart@ethz.co ch
Cesar Cadena ASL, ETH Zürich Resarc@ethz.c. ch
Abel Gawel ASL, ETH Zürich gawelaßethz. ch
We propose novel robotic system that improve its perception during deployment. Contrary to the established approach of learning semantics from large datasets and deploying fixed models, we propose framework in which semantic models are continuously updated on the robot to adapt to the deployment environments. By combining continual learning with self-supervision, our robotic system learns online during deployment without external supervision. We conduct real-world experiments with robots localising in 3D floorplans. Our experiments show how the robot's 's semantic perception improves during deployment and how this translates into improved localisation, even across drastically different environments. We further study the risk of catastrophic forgetting that such a continuous learning setting poses. We find memory replay an effective measure to reduce forgetting and show how the robotic system can improve even when switching between different environments. On average, our system improves by 60% in segmentation and 10% in localisation accuracy compared to deployment of fixed model, and it maintains this improvement while adapting to further environments.
Abstract:
Keywords: continual learning, self-supervised learning, online learning
Introduction
Learning-basee -bsed systems enable robots to partially understand the environment through, e.g., object detection or semantic classification [1, 2]. Such understanding is a key requirement to enable many complex robotic applications such as autonomous driving mobile manipulation [3, 4]. However, mobile robots are deployed in increasingly unstructured environments, where learning- -based systems often fail [5, 6). Anticipation of wide variety of environmental conditions is required for safe operation, however difficult if not impossible. Thus, robotic actors with high degree of autonomy required to adapt unexpected and changing conditions for robust operation. Yet, deployment of bearning-based systems typically means training a model on a variety of data and then using this fixed model during deployment. Since it is unattainable for fixed datasets model all environmental conditions and objects, robustness of current approaches to novel situations is severely limited.
In this work, we explore how learning can be used as a means to self self-improve semantic perception during exploration of the environment. Enabling robots to adapt on- on-the-job poses three main chal- lenges: (i) Models need to be efficiently (re)trained to incorporate new data (incremental / continual learning). (ii) Acquired knowledge should be kept while adapting to new tasks and environments (avert forgetting). (iii) Training signals are required during deployment, i.e. automatically harvested without manual human supervision in the loop self-suuervisson)) For the first two points, we build upon recent advancements in the field of continual learning (CL) [7, 8), also referred to incremental learning [9, 10], and lifelong learning [11, 12, 13J. This learning paradigm deals with settings where tasks or classes are presented incrementally, in which the data distribution changes over time [7].
Key to our proposed system is the combination of CL and self-supervision. In the literature, CL is usually evaluated by consecutively presenting a model with parts of existing datasets. However,
5th Conference on Robot Learning (CoRI. 2021), London, UK.

---

in order to continually learn during deployment, robots require streams of training signals without humans in the loop. We make use of self-supervised pseudo labels, which can be generated from multi-sensor [14] or multi-task systems [15]. Our results show that even noisy, imperfect pseudo labels generate useful training signals and what to our knowledge has not been discovered before- also work well with CL methods that were developed for perfect labels.
To validate our approach, we conduct experiments on real physical system. We deploy robots in diverse environments with existing 3D floorplans, in which the robot is required to localize. Floorplans are ubiquitous and therefore enable easy deployment into any indoor environment, without the need to generate a map before or during the mission. In applications like construction robotics, the robot's tasks are also defined in these plans. However, robot localisation in floorplans challenging. While the plan only represents static building structure (background), the actual environments contain large amounts of un-modelled objects and elutter (foreground), potentially obstructing localization performance [16]. With our system, the robot adapts to each environment by learning to tell apart localisation in all environments. We further observe
background from foreground and improving catastrophic forgetting when the robot switches between environments, and we identify CL methods that mitigate such forgetting. Finally, our experiments show positive knowledge transfer where the robot improves in novel environments based upon learning in earlier environments. In summary, our contributions are:
A framework for self- improving perception;
A pseudo label generation method for background-foreeeee foregroomn--eegmentttioo
Validation a real-world robotic system;
Evaluation of a range of methods to prevent forgetting;
Evidence that CL with noisy pseudo labels yields significant improvements over fixed models.
All experimental data and code is available at btttp://ggthhee cratthuu..oo//etiii com/etth--as//bcckrroom setggroondssggenttttaeeoe and summary video can be accessed at https://youte be/ awaynhkkFpk.
2 Related Work
Self-improving robotic agents is both old and underexplored idea. One framework in which agents are self-improving is reinforcement learning (RL). With RL, robots have been learning to walk [17], grasp objects [18], or fly [19]. All these systems indeed learn by self-improving over time, often failing in the beginning of the learning process. Usually these learned models are fixed once they acquire the necessary skills, unlike life-long learning. This is because they require supervision signals, e.g. from simulators that are not available during deployment. However, online adaptation of model-based RL has been shown for example in [20].
Self-improving robotic systems have also been described outside of RL. For example, [21] and [22] describe online parameter optimisations for model predictive control. The adaptive stereo vision of Tonioni et al. [23] has been a particular inspiration for this work. Very related is also Sofman al. [24], who learn a probabilistic model for terrain traversability in an online and self-supervised fashion. [25] is a parallel work that uses self-supervised learning to improve localisation in a robotic map. Interestingly, the mentioned previous works often do not explicitly address the problem of forgetting, which is a more prevalent problem in our semantic domain adaptation.
In continual learning, models are trained from non-stationary data distributions over series of different tasks [8]. The main objective of CL is to optimize for the performance on each task or domain with which the network is presented at any given time, while achieving positive knowledge transfer between the tasks, and preventing performance on the previous tasks from decreasing. This decrease on previous tasks is commonly referred to as catastrophic forgetting [26, 7]. Multiple techniques have been proposed to mitigate such forgetting, ranging from architectural modifications [27, 28] regularization techniques [29, 30] and methods based on memories [31, 32, 33, 34] or generative models [35, 36). A number of works have explored the use of CL techniques in the context of semantic segmentation [37, 38, 39, 40, 41]. They however experiment on a class-incremental setting, whereas we are interested in the scenario in which the deployment environment of the agent changes. In this sense, the problem we aim at tackling is also related to a line of works that focus on domain adaptation [42, 43, 44]. However, these works generally assume that both the source and target domain are known at training time, while we tackle the problem of adapting to any (unknown) environment on the fly.

---

Self-supervision is often used to learn useful image features in convolutional neural networks (CNNs). These techniques include learning to (re)color images [45], to (un)rotate images [46], or relatively position random crops [47]. However, these features require further (supervised) processing to relate them to e.g. classes. In mobile agents, egomotion was found be a promising self-suuprvision signal for range of tasks [48]. Photometric consistency between video frames can be used to jointly learn camera calibration, visual odometry, depth estimation, and optical flow [15].
A different line of works produces pscudolabels for segmentation by leveraging models trained on more available data. Class activation maps of image classifiers [49] can be used to identify object regions as a segmentation proxy [50, 51, 52]. Furthermore, segmentation predictions be refined by optimizing over Mask R R-CNN predictions, room layout estimation, and superpixels [53], by tracking and optical flow [54]. or by aggregating predictions in 3D space and projecting them back our proposed method builds up on similar ideas to observable characteristics of the environment produce a learning signal for the target task.
onto images [14]. While there is no direct prior work for
3 Proposed System
We propose self-improving perception system that interlinks localisation within map and se- mantic segmentation of the scene. We define the semantics of the scene not as arbitrary class la- bels, but by the observable affordance that some parts of the scene are mapped (background) and some are not (foreground). Therefore, we cre- ate pseudo labels based on the localisation in the map to train the semantic segmentation, and we use the segmentation into foreground and back- ground to inform the localisation. This creates feedback loop that can yield improvements in
lahelled data
(pre)training
LiDAR
iiiitialise
cameras
segmentation
localisation
feedback loop
continual learning
sulf-supervised pseudo labels
both parts, as can be seen in Figure 1.
The proposed self-improving system. The segmentation still incorporates existing train- ing data as a prior, but is not fixed during deploy- ment, as would be the established approach. In- stead, our segmentation is updated during deploy- ment based on CL methods. The continual learn- ing uses self-supervise pseudo labels, which are available during deployment without manual la- belling. We mark signals from the deployment environment in orange and from the pretraining domain in green.
Figure 1:
3.1 Semantically Informed Localisation
We localize the robot based aligning 3D Li- DAR scans with the given floorplan in the form of 3D mesh as in [16]. Given the floorplan mesh M, a pointcloud of the LiDAR scan P, and an initial alignment mesh lidar find subsequent robot poses as
(' - lidar ICP(MM, _r(lt-)) + lidar
We use point-to-plane ICP [55]' and filter out points of large distance and other criteria. Complete parameters reported in Appendix A.5.
To further divide the scan P into foreground and background points, we use additional information from a camera system mounted on top of the LiDAR. Once camera images are semantically segmented, we filter Panckgrondd S P as those points E P whose reprojected pixel in image frame is segmented as background and localise with ICP(M, ((t)
1)
background
mesh lidar
3.2 Pseudolabel Generation
We generate pseudo labels for each camera based on the LiDAR pointcloud that is localised w.r.t. the floorplan. First, for each point of the localised LiDAR scan, we calculate the distance to the closest plane of the mesh using fast intersection and distance computation [57]. We then check if the distance surpasses a given threshold 8. so, the point is assigned the foreground class, otherwise the
'The proposed framework does not require specific registration method. For registrations with large overlap, point-to-plane ICP was found sufficient solution in [56] and subsequently used in similar settings [16].
3

---

(b) Projected Pointcloud (c) Oversegmented Image Figure 2: Generation of pseudo labels. After localising the LiDAR sean, points are labeled by comparing the distance the floorplan with threshold ó (red is foreground, green is background). They are then projected into the image and aggregated into superpixels.
(a) Mesh and Pointeloud
(d) Pseudo Labels
Tracking
Prisan
Camerna
LiDAR
(a) Multi-Sensor Sctup
(b) Parking Garage
(c) Construction Site
(d) Office Environment
Figure 3: Overview of our experimental setup.
background class, In the second stage, we project each point onto the respective camera frames and refine the projection using SLIC superpixels [58]. In particular, we first oversegment the image into superpixel set S. superpixel 8 contained projected labels. We further improve the segmentation by discarding superpixels whose depth variance surpasses given threshold. An overview of the approach is depicted in Figure
then assigned a class according to a majority voting of the
These pseudo labels are not optimal". The goal however is not to generate perfect labels, but a useful training signal that can be generated on-the-fly without requiring any external supervision. As our experiments demonstrate, even this noisy learning signal substantially boosts performance.
3.3 Domain Adaptation with Continual Learning
To solve the task of Sanground-foregroumm segmentation, incrementally train neural network architecture on different data sources. We pre-train a CNN on the NYU-Depth v2 dataset [59), which contains 1449 images extracted from video sequences of indoor scenes, each with per-pixel semantic annotations. We map the classes wall, ceiling, and floor to background and regard everything else as foreground. This (pre-)training step allows the model to acquire prior knowledge as inductive bias to perform the same segmentation task on subsequent environments that the agent is presented with.
During deployment, the network is then updated with self-supervision through the pseudolabels generated on the current scene. With reference to the nomenclature often used in CL [8], we consider each new environment a task, and assume task boundaries be known, since for each task, the robot is given a different floorplan. Every time the robot is moved to a new environment, the same scheme is applied, i.e., the network trained the previous environments is updated by training the pseudolabels from the current environment.
To prevent forgetting of information learned on the previous tasks, we adopt a method based on memory replay buffers. When adapting to new environment, each training batch is filled with the frames collected in the current environment, along with small fraction of images collected in the previ- ous environments. Therefore, the model jointly optimizes over current and previous environments. However, storing all obser- vations from past environments in memory would come at huge costs. Instead, memory buffer for each previous environment only contains a random subset of all images and (pseudo-)Jas labels. Details of the buffer implementation and comparison with other CL frameworks are described in Section 4.4.
Algorithm 1: Memory Replay
memory_ buffer = RandomChoicot 10% out of previous _ccnnc) train train_data= ShuftleTogetherr(
current sccee, memory buffer)
while training not converged
foreach batch in train data do
randomly augment batch train on batch
end
validate on 10% hold-out of current.scene
end
Experimental Evaluation
We test and verify the applicability of the proposed framework in different steps of increasing complexity. We first validate that our robot can self-improve by deploying it into different unknown
For example, for any clutter object standing on the ground, all points below a height of ó may be labelled as background in the pseudolabels, dependent on the boundaries of the superpixel.

---

eavironment
mean/mediann'tt translation error [mm]
segmentation quality 0% mloU] sell-iipgoiing
DO segrentatiin
trained on NYU
sell sell-amproving
trained on NYU
Garage
50/ 37
58 48/41/73
43/35/31
33.9
62.8
Construction
488/ 183 199*
126 / 129
104/ 68 105
27.6
48.2
225
150 138
46.5
53,9
Table 1: Deployment to a single novel environment. We observe that segmentation in general is advantageous and the CL on self-supervised pseudolabels improves in all environments compared to deploying a fixed model. marks ICP failures. uses adapted parameters, see Appendix A.5.
environments and measure the gained improvement. We then evaluate the effects of forgetting and knowledge transfer when switching deployment between different environments. Finally, we conduct an experiment in which the robot learns online during the mission. For each experiment, we measure the localisation error in the x-y plane' in mean, median and standard deviation. We also measure the segmentation quality in mean Intersection over Union (mloU).
Robotic Platform. We conduct all our experiments on a wheeled robotic ground vehicle [60] with the sensor setup shown in Figure 3a. We calibrate all cameras to IMU that we attach to the sensor system and then align trajectories from visual-inertial odometry and LiDAR to find the extrinsic calibration between the LiDAR and the cameras. The tracking prism is used to gather ground-truth with a total station for our evaluation and is mounted on the sensor system to be aligned with the optical center of the LiDAR. For time synchronisation, we trigger all cameras simultaneously. From images captured at 20 Hz we then take those closest to the timestamp of the LiDAR scans, which are captured 5Hz. Between the external total station and the robot, we correct time-offsets manually.
Evaluation Environments. We deploy our proposed system into three different environments: a construction site, a parking garage, and an office floor. Architectural researchers provided us 3D meshes, constructed from dense 3D scans (Construction) and existing 2D floorplans supplemented with additional measurements (Garage and Office). Figure 3 shows these meshes together with the experimental setup. In each environment, we steer the robot through multiple independent trajectories of 2-3 min while tracking the robot with a total station. The coordinate system of the total station is always initialised to the origin of the building mesh, which we therefore set at corners visible to the total station. To evaluate the learned segmentation models, we sample and annotate around 30 ground-truth segmentations per environment. We then evaluate the segmentation predictions against ground-truth annotations within a static field-of-view (FoV) mask of the LiDAR
Learning Setup. We use a lightweight architecture based on Fast-SCNN [61] that we train for bockground-foregronn
segmentation. We resize input images to a common size (480 x 640 pixels). We first train the model on the NYU dataset. Then, when we deploy the robot into a new environment (ef. Sec. 4.1), we fill a replay buffer with samples from NYU in addition to training from the current environment. When we evaluate the transfer from a first environment to a second one (cf. Sec. 4.2), replay images from both NYU and the first environment. We set the replay fraction to 10%, but evaluate different replay regimes and strategies in Section 4.4. Before feeding images to the network, augment them with left-right flipping and random perturbation of brightness and hue. In each experiment, we hold out 10% of the training samples for validation and train our model using Adam optimizer. See Appendix A.2 for all training parameters and Appendix A. for runtimes.
the pseudolabels
4.1 Deployment in new environment
To test the effectiveness of the pseudolabel training and the localisation based on filtered pointclouds, we deploy the robot in first new environment. There, the robot collects information in the form of pscudolabels, which are used to train the segmentation. Afterwards, we deploy the robot over a different trajectory in the same environment and measure the performance of both segmentation and localisation.
In Table 1 we compare the performance obtained from our self self-supervised segmentation (iaoiionnnsssiii ('self-improvingg) with segmentation obtained from the network pre-trained on NYU
'The ground truth from the total station only provides translation measurements and does not enable evaluation of correctly estimated orientation.
*The FoV mask has two reasons: Because we are primarily interested good localisation, comparing segmentation quality in the region that can be reprojected to the LiDAR relates better to localisation performance. Additionally, pseudolabels are also only available in image regions where the LiDAR provides information, therefore expect the segmentation learn mostly the semantics of the scene visible in these regions.
5

---

environment sequence NYU - A B (-+
method
mean/median''td translation ertor [mm]
segmentation mloU]
A
C
A
B
replay
/
98/66// 98
610.8
48.6
Garage++ Construction
fincuniig
33/ 29
/7 22
55.1
49.4
Garage++ Office
replay
39/31 /31
168/1377 109
62.6
47.2
finetuning
37/30/33
196/ 118 1118/22 267
61.0
47.4
replay
105 68 108
40/ 33/ 29
49.3
62.2
Construction Garage
finctuning
549 / 8/4 1500*
40 31/
42.3
62.0
replay
125 128
Construction Office
finctuning
514 191 /914*
146 123 19
45.4
49.4
replay
153 131 995
44/ 34/ 36
47.8
62.1
Office + Garage
finetuning
182 151 114
38 32 27
40.2
61.0
replay
171/ 161 786
/ 85
47.5
49.9
Office Construction
finctuning
168 159
1217 70.7 128
33.3
49.1
replay
105 /05//2/99 92
39. /9
167 129 113
50.4
64.7
45.2
Constraction
Garage Office
finetuning
385
8688
/
145 114 130
37.2
Office Garage
Construction
replay
157 132 102
41/31/ 32
105 70/ 100
43.9
63.2
50.3
finetuning
1487145/85
45/35/30
112/82/92
33.8
57.2
48.2
Table 2:
Evaluation of forgetting and knowledge transfer when switching between deployment environments. The perception system is subsequently on pseudolabels of the environments in the given order and then evaluated on all. Bold marks cases where one method reduces forgetting compared to the other method. Underlined metrics are better than single-enviromment deployment from Table 1. Finetuning leads three cases (marked with star) where forgetting causes ICP failures.
('trained on NYU"), and with baseline that does not semantically filter the pointcloud ('no segmenta- tion'). We note that segmentation general important for good localisation and can prevent failure, as the construction site. The results also confirm the effectiveness of the pseudolabels, as training on these yields improvements in both segmentation quality and localisation error in all metrics. We conclude that the self-immroving setup is working as expected and that through the feedback loop indeed the localisation improves the segmentation, and the segmentation improves the localisation.
4.2 Transfer into multiple environments
In a second stage of experiments, we evaluate the ability of our system to retain knowledge and still adapt to new domains when moved from first environment a second (and third) one. In particular, we iteratively train the same model on pseudolabels from a series of environments. We then evaluate the extent of forgetting on both the pretraining data (NYU) and any environment before the last one, comparing our adopted method based on replay buffer with simple fine-tuning.
As shown in Table 2, using replay buffers improves the segmentation performance on the past tasks w.r.t. the case in which no replay is adopted. This indicates successful mitigation of forgetting and is even more prominent when measured on the pseudolabels and measuring forgetting on the NYU data, which we analyse in more detail in Table in Appendix A.3. Table further shows that the forgetting of finetuning can cause localisation failure successfully. In the CL literature, memory replay is usually studied with high- segmentation masks covering all pixels in the image [7]. Yet, from our observations we can conclude that memory replay is also effective when the replayed labels are noisy and imperfect.
the source environment. Replay prevents such failure
We note that finetuning can result in better adaptation the target environment, especially with regard to localisation. This is expected, since the learning process of finetuning is fully tailored to the new environment. This effect is known can inhibit learning of new knowledge, but increasing plasticity can in turn lead to forgetting. In our experiments, the relative improvements of finetuning over memory replay that observe on the target environment are marginal, suggesting that the chosen 10% replay finds a good balance.
dilemma [62], i.e. old knowledge
Table 2 also highlights cases in which deployment in multiple environments is better than improving only towards one specific environment (Section 4.1,, both with and without memory replay. This indicates positive knowledge transfer between our evaluation environments. For self- celf-improving robotic system this is a promising finding, showing that the robot not only adapts to the target environment, but generally improves at its task with every deployment.
Finally, we observe that segmentation quality and localisation error are sometimes inconsistent, where a drop in segmentation quality in a past environment does not necessarily transfer into worse localisation. This indicates potential for advanced methods which only keep in memory those observations that are important for the task at hand.

---

4.3
Online Learning
online localisation
0.6
RMSE [m]
The previous experiments have been conducted in multi-mission fashion, in which first mis- sion gathers data of the environment and the
0.4
Lum
0.2
0
robot learns from it to improve in subsequent missions. Figure now shows case study how the system can learn online during a mission by learning from the pseudolabels directly as they are generated. We initialise the model pretrained on NYU and observe that over time segmen- tation quality increases and localisation error decreases. Notably, already a few seconds of online learning increase the segmentation qual- ity significantly. However, it takes longer until measure notable effect on the localisation. Due to the limited time until the end of the tra- jectory, we are therefore unable see if there is a feedback effect where the improved locali- sation would create better pseudolabels that can in turn increase the segmentation quality further. For this, further investigation on longer deploy- ments is necessary. We present similar studies for the other environments in Appendix A.7.
segmentation quality
0.5
mloU
0.4
0.3
localisation on reference trajectory
RMSE [m]
0.14
mean
0.12
modian
0.1
50
100
150
200
time [s]
Figure 4: Online Learning on the construction site. We measure how the robot improves online (row 1), but also evaluate snapshots of the model on reference data (rows 2 and 3). Areas in green/red show changes (better/worse) with respect to a fixed model. We observe that segmentation quality in- creases over time and localisation error decreases.
4.4 Ablation Studies
Since CL methods have not been explored in real-world robotic context, and even less in combination with noisy pseudolabels as training signal, we perform an extensive ablation of different methods and parameters to validate our use of replay buffers.
We first evaluate two different strategies for memory replay. In the first strategy, which is the one that we adopt in the main experiments, on each source-to-targed experiment (e.g., NYU-+Garage), we fill the replay buffer with fraction of samples from the source dstaset(s) (NYU in the example), which we select randomly. We then fill training batches from the replay buffer and target dataset according to their relative sizes. In the second strategy, we fill the replay buffer with the full source dataset but fill training batches with pre-defined target-source ratio. For instance, ratio Garage NYU = 4 with batch size of 10 indicates that batches on average contain images from Garage and 2 images from NYU. As shown in Table 3, milder replay regimes (larger targef-source ratios, smaller replay fractions) achieve higher performance on the target domain, but cause the amount of information retained from the source domain to drop. This forgetting phenomenon is particularly evident in the adaptation tasks in which the semantic gap between source and domain is larger. Indeed, for instance, in the NYU->Garage experiment we observe a drop of 31.9% in mIoU on the NYU labels between a replay strategy with a fraction of 10% and simple fine-tuning, while the same decrease in performance when the target domain is Office semantically closer to the indoor dataset NYU is 14.8%. At the same time, the segmentation quality on the pseudo- labels of the target dataset follows an inverse trend, generally increasing for smaller amounts of replay. This highlights the trade-off between the accuracy on the target and on the source domain.
Memory replay comes at the cost of storing images and labels of past environments. which does not scale well to more than a few environments. We therefore further compare regularization techniques literature, which try to prevent forgetting without storing much data.
from the continual-learning In particular, we investigate the use of a distillation approach, in which a regularization term is added to the cross-entropy loss, encourage the network to retain the knowledge from previous tasks. Similarly to [37], we apply this distillation either on the output logits produced by the network (Output distillation) on the intermediate features (Feature distillation). Furthermore, we evaluate Elastic Weight Consolidation (EWC) [29]. We present further details in Appendix A.4.
As shown in Table 3, in our experiments replay buffers prove to be the most effective among the examined methods minimizing the amount of forgetting on the NYU dataset, and generally allow attaining good trade-off with the segmentation quality the pseudo-labels from the target domain.
7

---

NYU-GGange
NYU DYU-CCustusionn
NYU-
Gumge
Conseriction
Pseudo
GT
Pscudo
GT
Pseudo
GT
Pinetaning
36.4
96.3
61.8
36.6
79.5
48.9
66.2
70.9
51.2
1:1
87.2
46.1
57.8
49.9
Replay butfer
4: 1
3: 1
81.1
79.8
91.7
92.4
62.3
60.7
78.7
79.8
73.1
73.1
17.8
48.8
82.0
81.4
65.8
68.1
51.7
52.2
target: NYU
73.4
75.3
75.6
47.6
73.7
71.2
52.1
repp leylay
20: 1
67.5
95.3
62.0
72.4
76.3
48.3
76.0
69.1
52.0
200: 1
53.9
96.1
61.6
53.2
77.2
48.7
74.8
68.7
50.9
Replay butler
10%
68.3
95.4
62.8
78.6
77.0
48.2
$1.0
69.7
53.9
with Fraction replay
5%
65.0
95.9
62.0
76.2
76.9
48.5
79.9
69.3
51.5
0.5
35.4
96.1
61.9
33.2
77.1
50.3
65.3
71.1
50.7
Fenture distiillation
A
34.8
95.9
61.2
30.6
76. 9
50.8
58.6
78.9
49.1
37.4
18.3
72.2
50
33.6
92.7
61.1
32.6
63.1
46.5
44.0
64.5
46.8
0.5
33.1
63.3
76.5
47.8
68.4
regréartsanionn
Ouiput distillation
1
32.4
85.3
61.4
38.2
59
46.6
$3.0
60.4
44.3
10
37.8
40.8
47.5
37.9
32.9
37.1
45.3
35.8
36.1
39.0
$3.0
28.7
31.1
30.5
35.9
À 0.5
36.1
96.4
61.5
34.4
76.5
47.9
65.7
69.2
51.6
EWC [29]
1
36.2
96.3
61.4
37.0
76.4
18.0
66.2
71,0
50.8
37.9
35.4
76.2
48.0
69.1
A 50
37.9
95.9
61.5
35.0
75.6
47.9
63.5
73.2
51.D
Table 3: Ablation study over different CL methods. After adapting from the source domain (NYU) to the different deployment environments, we measure segmentation quality [% mloUJ on the source data as well as the self-supervised pseudolabels (Pseudo) and our ground-truth annotations (GT). Forgetting is therefore indicated by low performance in the gray columns. We compare the finetuning with memory replay and regularisation methods. We observe that memory replay is more successful than regularisation at preventing catastrophic forgetting in our application.
We also note that, with limited exceptions, both regularization approaches fail to maintain good performance on the source dataset NYU. We believe that for distillation methods this can be ascribed to the fact that, as opposed to related works that explored similar techniques in a class Maas--nnremmnnta setting [37, 38], we conduct domain adaptation. More importantly, unlike [37, 38] our supervision signal does not consist of accurate ground-truth annotations available at all pixels, but of noisy pseudo-labels that often cover limited region of the image. Finally, while similar considerations also apply for EWC, we believe that the limited effectiveness of this technique in our setting is also related to the method being designed for classification as opposed to image segmentation.
4.5 Limitations
Our system required some manual intervention in the sense that few parameters were changed across environments (ICP and superpixels, see Appendix). While coming close, the system also did not run in real-time. We are confident that both points can be tackled with better software implementations.
This study is limited to binary segmentation. However, our framework itself is agnostic of the self-supervision method and can be directly extended to more classes based multi- [14] or multi-task systems [15], or temporal consistency [54]. More affordances can be observed through manipulating objects [63], temporal change [25], context and spatial co-occurrence [64, 65).
5 Conclusion & Outlook
In this work we propose framework for gelf-improving perception by combining CL with self- supervision. We study this on a real robotic system that localises 3D floorplans. Our experiments validate the gains of the self-improving systems in diverse environments. In particular, we analyse the effects of knowledge transfer and forgetting when switching between environments. We find that memory replay is an effective solution that can mitigate forgetting, and observe that the system can transfers knowledge from previously seen, distinct environments.
even further improve
The main finding from our evaluations of knowledge transfer and forgetting is that self-supervision and CL is an effective combination, even if the self-supervision is noisy. This opens up exciting questions for future research. Through the self-supervision approaches that we describe in Sections 2 and 4.5, the same framework can be transferred to other perception tasks such as more semantic classes or perception for manipulation. For some tasks, task boundaries may be less clear, and further research is necessary to lift this assumption. Finally, we identify long-term dynamics and stability of self-improving systems as an interesting direction for future research.

---

Acknowledgments
This work was partially funded by the Hilti Group. Eberhard Unternehmungen kindly allowed us to conduct experiments on their premises.
Furthermore, we thank Selen Ercan for creating the building models, Florian Tschopp and his VersaVIS board for all the multi-sensor calibration, Shen Kaiyue for initially implementing and testing various regularisation based continual learning methods, and Andrei Cramariue for GPU cluster support.
References
[1] McCormac, R. Clark, M. Bloesch, Davison, and S. Leutenegger. Fusion++: Volumetric Object-Level SLAM. In Intl. Conf. on 3D Vision (3DV), pages 32-41. IEEF, 2018.
[2] A. Rosinol, M. Abate, Y. Chang, and L. Carlone. Kimera: Open-Source Library for Real-Time Metric-Semantic Localization and Mapping. In IEEE Intl. Conf. on Robotics and Automation (ICRA), pages 1689-1696. IEEE, 2020.
[3] L. Kunze, N. Hawes, T. Duckett, M. Hanheide, and T. Krajník. Artificial intelligence for Long- Term Robot Autonomy: A Survey. IEEE Robotics and Automation Letters, 3(4):4023-4033 2018.
[4] E. Yurtsever, Lambert, A. Carballo, and K. Takeda. A Survey of Autonomous Driving: Common Practices and Emerging Technologies. IEEE Access, --444999566, 2020.
[5] H. Blum, P.-E. Sarlin, J. Nieto, R. Siegwart, and C. Cadena. The fishyscapes benchmark: Measuring blind spots in semantic segmentation. Int J Comput Vis, 2021. doi:10. 1007// 2263-021-01511-6
[6] O. Zendel, K. Honauer, M. Murschitz, D. Steininger, and G. Fernandez Dominguez. Wilddash- creating hazard-aware benchmarks. In Proceedings of the European Conference on Computer Vision (ECCV), pages 402-416. openaccess.thecvf.com 2018. 1et.11.1007997---30000000 25.
[7] G. I. Parisi, R. Kemker, Part, C. Kanan, and S. Wermter. Continual lifelong learning with neural networks: A review. Neural Networks, 1 113:54-71, 2020.
[8] T. Lesort, V. Lomonaco, A. Stoian, D. Maltoni, D. Filliat, and N. Díaz-Rodríguez. Contin- ual Learning for Robotics: Definition, Framework, Learning Strategies, Opportunities and Challenges. Information Fusion, 58:52-68, 2020.
[9] R. Camoriano, G. Pasquale, C. Ciliberto, L. Natale, L.. Rosasco, and G. Metta. Incremental Robot Learning of New Objects with Fixed Update Time. In IEEE Intl. Conf. on Robotics and Automation (ICRA), 2017.
[10] K. Shmelkov, C. Schmid, and K. Alahari. Incremental Learning of Object Detectors without Catastrophic Forgetting. In Intl. Conf. on Computer Vision (ICCV), 2017.
[11] Z. Chen and B. Liu. Lifelong Machine Learning. Synthesis Lectures Artificial Intelligence and Machine Learning, 12(3):1-207, 2018.
[12] S. Thrun and T. M. Mitchell. Lifelong robot learning. Robotics and Autonomous Systems, 15: 25 46, 1995.
[13] D. L. Silver, Q. Yang, and L. Li. Lifelong Machine Learning Systems: Beyond Learning Algorithms. In AAAÍ Conf. Artificial Intelligence (AAAI), 2013.
[14] W. Sun, J. Zhang, and N. Barnes. 3D Guided Weakly Supervised Semantic Segmentation. CoRR, abs/2012.00242, 2020.
[15] Y. Chen, C. Schmid, and C. Sminchisescu. Self-Supervised Learning With Geometric Con- straints Monocular Video: Connecting Flow, Depth, and Camera. In Intl. Conf. on Computer Vision (ICCV), pages 7063-7072, 2019.

---

[16] H. Blum, J. Stiefel, C. Cadena, R. Siegwart, and A. Gawel. Precise robot localization in architectural 3d plans. arXiv preprint 000:2000.00137,, 2020.
[17] T. Haarnoja, S. Ha, A. Zhou, J. Tan, G. Tucker, and S. Levine. Learning to Walk via Deep Reinforcement Learning. arXiv preprini arXiv:
2018.
[18] M. Q. Mohammed, K. L. Chung, and C. Chyi Review of Deep Reinforcement Learning- Based Object Grasping: Techniques, Open Challenges, and Recommendations, IEEE Access, 8: 178450-178481, 2020.
[19] J. Hwangbo, I. Sa, R. Siegwart, and M. Hutter. Control of Quadrotor With Reinforcement Learning. IEEE Robotics and Automation Letters, 2(44:2096-2103, 2017.
[20] J. Fu, S. Levine, and P. Abbeel. One-shot learning of manipulation skills with online dynamics adaptation and neural network priors. In IEEE/RSJ Intl. Conf. on Intelligent Robots and Systems (IROS), pages 4019 0019-4026. IEEE, 2016.
[21] M. Lorenzen, M. Cannon, and F. Allgöwer. Robust MPC with recursive model update. Auto- matica, 103:461-44 471,, 2019. ISSN 0005-1098.
[22] J. Kabzan, L. Hewing, A. Liniger, and M. N. Zeilinger. Learning-Based Model Predictive Control for Autonomous Racing. IEEE Robotics and Automation Letters, 4(4):3363-3370, 2019. doi: 100.1000RA..21...0000.
[23] A. Tonioni, O. Rahnama, T. Joy, L. D. Stefano, Ajanthan, and P. H. S. Torr. Learning to Adapt for Stereo. In IEEE Conf. on Computer Vision and Pattern Recognition (CVPR), pages 9661-9670, 2019.
[24] B. Sofiman, E. Lin, J. A. Bagnell, J. Cole N. Vandapel, and A. Stentz. Improving robot navigation through self-supervised online learning. Journal of Field Robotics, 23(11-12): 1059-1075, 2006.
[25] H. Thomas, B. Agro, M. Gridseth, J. Zhang, and T. D. Barfoot. Self-Supervised learning of lidar segmentation for autonomous indoor navigation. In ICRA, 2021.
[26] R. M. French. Catastrophic forgetting in connectionist networks. Trends in Cognitive Sciences, 3(4):128-135, 1999.
[27] A. A. Rusu, N. C. Rabinowitz, G. Desjardins, H. Soyer, J. Kirkpatrick, K. Kavukcuoglu, R. Pascanu, and R. Hadsell. Progressive Neural Networks. arXiv preprint arXiv:1606.04400 2016.
(28) Schwarz, W. Czarnecki, J. Luketina, A. Baawskk.Barwinska, Y. W. Teh, R. Pascanu, and R. Hadsell. Progress & Compress: A scalable framework for continual learning. In Intl. Conf. on Machine Learning (ICML), 2018.
[29] J. Kirkpatrick, R. Pascanu, N. Rabinowitz, J. Veness, G. Desjardins, A. A. Rusu, K. Milan, D. Hassabis, C. Clopath, D. Kumaran, and
J. Quan, T. Ramalho, A. R. Hadsell. Overcoming catastrophic forgetting in neural networks. Proceedings of the National Academy of Sciences, 114(13):521-33566 2017.
[30] Z. Li and D. Hoiem. Learning without Forgetting. IEEE Trans. Pattern Anal. Machine Intell., 40(12):2935-2944 2018
[31] S.-A. Rebuffi, A. Kolesnikov. G. Sperl, and C. H. Lampert. iCaRL: Incremental Classifier and Representation Learning. In IEEE Conf. on Computer Vision and Pattern Recognition (CVPR), 2017.
[32] D. Lopez-Paz. and M. Ranzato. Gradient Episodic Memory for Continual Learning. In Conf. on Neural Information Processing Systems (NIPS), 2017.
[33] L. Hayes, Cahill, and Kanan. Memory Efficient Experience Replay for Streaming Learning. In IEEE Intl. Conf. on Robotics and Automation (ICRA), 2019.
10

---

[34] S. Zhang and R. S. Sutton. A Deeper Look at Experience Replay. In Conf. on Neural Information Processing Systems (NIPS) Deep Reinforcement Learning Symposium, 2017.
[35] H. Shin, J. K. Lee, J. Kim, and J. Kim. Continual Learning with Deep Generative Replay. In Conf. on Neural Information Processing Systems (NIPS), 2017.
[36] C. Wu, L. Herranz, X. Liu, Y. Wang, J. v. d. Weijer, and B. Raducanu. Memory Replay GANs: Learning to Generate Images from New Categories without Forgetting. In Conf. on Neural Information Processing Systems (NIPS), 2018.
[37] U. Michieli and Zanuttigh. Incremental Learning Techniques for Semantic Segmentation. In Intl. Conf. on Computer Vision (ICCV), Workshop on Transferring and Adapting Source Knowledge in Computer Vision (TASK-CV) 2019.
[38] U. Michieli and P. Zanuttigh. Knowledge distillation for incremental learning in semantic segmentation. Computer Vision and Image Understanding, 205:103167, 2021.
[39] F. Cermelli, M. Mancini, S. Rota Bulð, E. Ricci, and B. Caputo. Modeling the Background for Incremental Learning in Semantic Segmentation. In IEEE Conf. on Computer Vision and Pattern Recognition (CVPR), 2020.
[40] A. Douillard, Chen, A. Dapogny, and M. Cord. PLOP: Learning without Forgetting for Continual Semantic Segmentation. CoRR abs/2011. 1.101.11119.,,, 2020.
[41] L. Yu, X. Liu, and J. van de Weijer. Self-Training for Clase-nncrementtl Semantic Segmentation. CoRR 0bss2001.03362, 2020.
[42] Y. Zou, Z. Yu, B. V. K. Vijaya Kumar, and J. Wang. Unsupervised Domain Adaptation for Semantic Segmentation via Class-Balanced Self-training. In European Conf. on Computer Vision (ECCV), 2018.
[43] A. Saporta, T.-H. Vu, M. Cord, and P. Pérez. ESL: Entropy-guided Self-supervised Learning for Domain Adaptation Semantic Segmentation. In IEEE Conf. on Computer Vision and Pattern Recognition (CVPR), Workshop Scalability in Autonomous Driving, 2020.
[44] Y. Li, L. Yuan, and N. Vasconcelos. ESL: Entropy-guided Self-supervised Learning for Domain Adaptation Semantic Segmentation. In IEEE Conf. on Computer Vision and Pattern Recognition (CVPR), 2019.
[45] R. Zhang, P. Isola, and A. A. Efros. Colorful Image Colorization. In European Conf. on Computer Vision (ECCV), pages 649-666. Springer, 2016.
[46] S. Gidaris, P. Singh, and N. Komodakis. Unsupervised representation learning by predicting image rotations. In Intl. Conf. on Learning Representations (ICLR), 2018.
[47] C. Doersch, A. Gupta, and A. A. Efros. Unsupervised Visual Representation Learning by Context Prediction. In Intl. Conf. Computer Vision (ICCV), pages 1422-1430, 2015.
[48] Agrawal, J. Carreira, and Malik. Learning to See by Moving. In Intl. Conf. on Computer Vision (ICCV), pages 37-45, 2015.
[49] B. Zhou, A. Khosla, Å. Lapedriza, A. Oliva, and A. Torralba. Learning Deep Features for Discriminative Localization. CoRR, abs/1512.041555 2015.
[50] Y.-T. Chang, Q. Wang, W.-C Hung, R. Piramuthu, Y.H. Tsai, and M.-H. Yang. Mixup- CAM: Weakly-supervisee Semantic Segmentation via Uncertainty Regularization. CoRR, abs/2008.01201, 2020.
[51] Lee, E. Kim. S. Lee, J. Lee, and S. Yoon. FickleNet: Weakly and Semi-supervised Semantic Image Segmentation using Stochastic Inference. CoRR, abs/1902. 2019.
[52] Z. Huang, X. Wang, J. Wang, W. Liu, and J. Wang. Weakly-Supervised Semantic Segmentation Network With Deep Seeded Region Growing, In IEEE Conf. on Computer Vision and Pattern Recognition (CVPR), June 2018.

---

[53] M. A. Reza, A. U. Naik, K. Chen, and D. J. Crandall. Automatic Annotation for Semantic Segmentation Indoor Scenes. IEEE/RSJ Intl. Conf. on Intelligent Robots and Systems (IROS), pages 4970-4976. IEFE, 2019.
[54] L. Porzi, M. Hofinger, I. Ruiz, J. Serrat, S. R. Bulo, and P. Kontschieder. Learning Multi-Object Tracking and Segmentation From Automatic Annotations. In IEEE Conf. on Computer Vision and Pattern Recognition (CVPR), 2020.
[55] Y. Chen and G. Medioni. Object modelling by registration of multiple range images. Image and Vision Computing, 1063):145-155, 1992.
[56] M. Magnusson, N. Vaskevicius, T. Stoyanov, K. Pathak, and A. Birk. Beyond points: Evaluating recent 3D scan-matching algorithms. In 2015 IEEE International Conference on Robotics and Automation (ICRA), pages 3631-3637. jeeexplore.ieee.og May 2015. doi: 10. 1199/CRA..005. 7139703.
[57] Alliez, S. Tayeb, and Wormser. 3D fast intersection and distance computation. In CGAL User and Reference Manual. CGAL Editorial Board, 5.2 edition, 2020. URL https: / //doc. coal. cgal..rg//........... 2/ ///aauallpacaagee html #PggAABBTee..
[58] R. Achanta, A. Shaji, K. Smith, A. Lucchi, P. Fua, and S. Süsstrunk. SLIC Superpixels Compared to -fate-of-the-Art Superpixel Methods. IEEE Trans. Pattern Anal. Machine Intell., 34(11):2274-228) 2012.
[59] N. Silberman, D. Hoiem, P. Kohli, and R. Fergus. Indoor Segmentation and Support Inference from RGBD Images. In European Conf. on Computer Vision (ECCV). 2012.
[60] A. Gawel, H. Blum, J. Pankert, K. Krämer, L. Bartolomei, S. Ercan, F. Farshidian, M. Chli, F. Gramazio, R. Siegwart, et al. A fully-integrated sensing and control system for high-accuracy mobile robotic building construction. 2019 IFFE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 2300 IEFF, 2019.
[61] R. P. K. Poudel, S. Liwicki, and R. Cipolla. Fast-SCNN: Fast Semantic Segmentation Network. In British Machine Vision Conf. (BMVC), 2019.
[62] M. Mermillod, A. Bugaiska, and P. Bonin. The sability-plaasiicis dilemma: investigating the continuum from catastrophic forgetting age-limited learning effects. Frontiers in Psychology, 4:504, 2013.
[63] C. Castellini, T. Tommasi, N. Noceti, F. Odone, and B. Caputo. Using Object Affordances to Improve Object Recognition. IEEE Trans. Autonomous Mental Development, 33) 2011.
[64] J. Bachmann, K. Blomqvist, J. Förster, and R. Siegwart. Veints2Vec: Unsupervised Object-level Feature Learning from Point Clouds. CoRR abs/2102.04136, 2021.
[65] C. Galleguillos, A. Rabinovich, and Belongie. Object categorization using co-occurrence, location and appearance. In JEEE Conf. on Computer Vision and Pattern Recognition (CVPR), pages 1-8, 2008
[66] Y. Wu and K. He. Group Normalization. In European Conf. on Computer Vision (ECCV), 2018.
[67] S. Ioffe and C. Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. Intl. Conf. on Machine Learning (ICML), 2015.
[68] R. Giraud, V. Ta, and N. Papadakis. SCALP: superpixels with contour adherence using linear 07149.
path. CoRR, abs/ 903.07 149. 2019. URL http: http://xxxvvvv...bba/
12


Reviews:
[{"review_id": "pO_TNcES5Cv", "reviewer": "Reviewer_Ncrf", "summary": "The paper combines self-supervision with methods from continual learning to enable a robot to improve its localization during deployment. More concretely, the proposed method automatically labels images that are collected during deployment based on localization information and uses them to continuously update a segmentation module, which in turn is used for localization. The authors show in a suite of real-world environments that this feedback loop between segmentation and localization modules significantly improves both the localization capabilities and the quality of generated segmentations. They also note that updating the segmentation module in an online fashion comes with the risks of catastrophic forgetting of previously learned features, and show that a simple memory buffer can prevent this from happening.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "### Strengths\n\n- The main idea of enabling continual learning for robot localization via self-supervised segmentation labels is both interesting and well-executed.\n- The paper is presented in a clear and concise fashion.\n- The approach is applied to different real-world environments, showcasing its versatility and making for a convincing experimental section.\n- A series of ablation experiments is conducted to show the influence of different hyperparameters and design decisions on the overall framework. Different methods of continual learning are explored and compared to the memory buffer chosen for the final model.\n\n### Weaknesses\n\n- The automatic segmentation labeling requires a ground truth floor-plan that is compared to LiDAR scans. This limits the applicability of the approach to indoor environments with precisely known layouts.\n- While the use of a sample buffer helps prevent catastrophic forgetting, the ablations show that it comes with the cost of reduced performance on new environments. As such, the question of how many old samples to use becomes a hyperparameter that needs to be carefully tuned for a downstream application. Note however that this is well-addressed in the ablation studies.\n- As stated in the \"Limitations\" section, some parameters need to be tuned depending on the environment. This raises the question of how well the method generalizes to different and potentially more complex environments.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "The paper explores the idea of improving semantic segmentation during robot deployment by means of continual learning and self supervision. This is an interesting contribution on its own, and naturally opens up various directions for future work.\n\nPresentation-wise, the main ideas of the paper are thoroughly motivated and explained clearly. The experimental section clearly shows the benefit over the approach over a baseline that does not use it, and a number of ablations give insight into the importance of individual components. The main shortcoming of the experiments seems to be the need for manual parameter tuning. Adding some measure of confidence (e.g., standard deviations) would help to make a better case for the approach.\n\nOverall, the paper introduces and experimentally supports an interesting novel approach, and presents everything in a clearly understandable manner. I would thus recommend to accept it.\n\n\n**Update**\nThe authors have addressed most of my concerns during the revision period. I am still not entirely satisfied with the scope and presentation of Section 4.3 (as well as the related Appendix 7), and agree with the other reviewers that the paper would benefit greatly from a pseudo-label baseline. However, the pros of the paper far outweigh the few problems I have with it, causing me to keep my score of \"Strong Accept\".", "technical_quality": "Very Good:  Convincing.", "clarity_of_presentation": "Excellent: Enjoyable to read.", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "4: The work contains interesting new ideas that will potentially have major impact in robotics or machine learning.", "reviewer_expertise": "Good: General knowledge of the area"}, "extra_text": {"issues": "- Figure 1 does not really seem to add much to the paper. I would much rather see a qualitative comparison, e.g., in the form of a suitable frame from the summary video in the supplement.\n- The images in Figure 4 have pretty high resolution, which can lead to some lag when loading the page in a .pdf reader. Down-sampling them would help, and also decrease the overall size of the file.\n- In the \"Robotic Platform\" subsection of Section 4, it says that images are captured at 20Hz and then matched to the 5Hz LiDAR scans, with offsets being corrected by hand. Is there a reason for the images to be captured at 20Hz instead of using 5Hz as the common denominator?\n- The section on online learning (Section 4.3) is rather short. While the paper states that further investigation on this topic is needed, it would still be nice to present the existing results more clearly. For example, Figure 5 Row 3 shows that the localization results get worse for some time when online learning, before eventually improving upon the baseline. It would be interesting so see why this is the case. Similarly, Figure 5 Row 1 shows relative changes to a baseline. Adding some general trend, e.g., an exponential moving average or cumulative histogram of the changes, could help readability. Finally, it would be nice to see how statistically significant these changes are, though this may be out of scope for the revision.\n\n### Misc.\n\n- Line 43 \"... has not been discovered before\" seems like a rather strong statement. Adding some variant of \"to the best of our knowledge\" would alleviate this. A similar argument can be made in Line 277.\n- In Table 3, the \"61.5\" in the last row is written in bold, which seems to be unintentional."}}, {"review_id": "nXuPt7jx4Zt", "reviewer": "Reviewer_XC6X", "summary": "This paper proposed a self-supervised continual learning strategy on scene perception where the perception task is a background/foreground binary classification. Self-supervised learning is based on generated pseudo-labels on localised LiDAR scan, where each point are labelled based on how close to the nearest floorplan via a defined threshold. By further applying a memory buffer storing on 10% of previous scene data, it achieves continual learning and shows that training on new scenes could help learn a better representation improving performances of the previous scenes.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The general concept of this paper is appealing, and I appreciate the video attached shows how this robot with the proposed method performs in a real-world environment. \n\nSome additional comments:\n\n-- Limited novelty: The proposed method is based on components that are standard and commonly used in the community, for example, the random buffer is typically used in perception-based continual-learning systems, and in most RL-based methods. Though I agree the design of pseudo-labeling is new and interesting, but limited to the task of only performing binary segmentation is a bit too easy. The background class of the indoor room has a very clear geometric structure and thus transferring across different tasks would be simple and straightforward. Considering background and foreground segmentation as \"perception\" seems to be an over-claim to me.\n\n-- Quality of visualisation: In the video, It's very hard for me to identify the ground-truth tracking and predicted tracking. \n\n-- Compared to other CL baselines: To the best of my knowledge (but happy to be corrected if wrong), EWC and other presented regularisation-based CL methods are designed with the assumption that we don't have access to any of the data in the previous tasks. While in the replay-based methods, the data are stored in the memory buffer and thus would be still trained when deploying in the new scenes. So it's a bit unfair to directly compare to these regularisation methods without properly justifying the differences.\n\n=======================================================\n\nThe authors have clarified all my questions in the rebuttal phase. And I decide to raise my final rating to Weak Accept.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "I like the overall system to be shown and work in a real-world environment. However, the paper is a bit overclaim on the impact and the novelty of the method.", "technical_quality": "Good: Generally convincing, but some additional experiments or analyses needed.", "clarity_of_presentation": "Fair: Major revision needed to improve organization or clarity.", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed.", "issues": "See weaknesses.", "reviewer_expertise": "Good: General knowledge of the area"}, "extra_text": {}}, {"review_id": "CmsTJRXQjcy", "reviewer": "Reviewer_bspZ", "summary": "The authors present a self-improving perception framework in the context of robot localization against 3D models/floor plans of changing environments. The localization is aided by a DNN, which segments the current scene into foreground (novel objects, not used for scan matching) and background (consistent with the 3D model, used for scan matching) points. The focus of this work is on how to train this network in a self-improving manner while adapting to new environments. To this end, authors propose a way to generate noisy pseudo labels from the current robot pose estimate, the current scan and the available 3D model. These pseudo labels are then used to train the DNN and thus improve its performance in the new environment. In turn, the better segmentation improves the localization itself. To avoid forgetting of already acquired knowledge during the deployment in other environments, authors use continual learning techniques such as memory replay. The experiments on a real robotic platform demonstrate that the approach is able to improve both localization and segmentation in novel environments, transfer acquired knowledge across environments and avoid catastrophic forgetting. \n", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Robots are deployed in a large variety of environments and the ability to adapt to new domains is a sought-after skill. Therefore, the paper contributes to a very relevant topic for the robotics community. It is well written and contains comprehensive experiments with valuable insights. I especially appreciate that the experiments were conducted on a real-world robotic platform. \n\nWhile the self-improving and continual learning framework is interesting, I find its demonstration/application scenario to be less appealing. Potentially, it is because the importance of the removal of the foreground points for the task at hand is not stressed enough in the introduction (unfortunately, the related citation is anonymized). I think one or two sentences elaborating on this matter could be beneficial for the reader. The chosen localization baseline with point-to-plane ICP seems to be quite basic and I was left wondering if with more robust scan-matching (Beyond points: Evaluating recent 3D scan-matching algorithms, Magnusson et al., ICRA'15) or a SLAM method (with e.g. lidar odometry which could even leverage the existence of new objects) would perform better at the localization task and in turn in foreground/background segmentation (with the same approach used for the generation of pseudo labels). This is neither discussed in the text nor investigated in the experiments. Also, if the authors could suggest more applications of the learnt binary segmentation, it could further highlight the importance of this work.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "I simply think that the topic addressed in this work is very relevant to the robotics community and this work provides some valuable contributions.", "technical_quality": "Very Good:  Convincing.", "clarity_of_presentation": "Very Good:  Well organized and clear.", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed.", "reviewer_expertise": "Fair: Some knowledge of the area"}, "extra_text": {"issues": "* For a less informed reader (like myself) on continual learning, could you please expand on the significance of the finding that CL works with noisy pseudo labels? Given the nature of memory replay and that noisy pseudo labels have been successfully used in self-training settings, I don’t find this as a surprising or very insightful outcome. This is a genuine question, without trying to reduce your contribution.\n* For the localization, do you use robot’s odometry? \n* How would your approach scale in terms of accuracy, forgetting and computational efficiency when the robot is deployed in more than two environments e.g. garage -> construction -> office, etc.?\n* How does your work compare to *Self-Supervised Learning of Lidar Segmentation\nfor Autonomous Indoor Navigation, Hugues et al., ICRA'21*? I understand that it is a very recent and parallel work, but I think it should be discussed in the related work.\n* The online learning experiment (Sec. 4.3) seems to be inconclusive and the description lacks details e.g. in which environment the experiment is performed? Is the network initially pre-trained on the NYU dataset? Do you observe similar outcomes in the rest of the environments?\n* In general, for most of the experiments one could think of applying several iterations (like it is usual in self-training) i.e. improved segmentation -> improved localization -> improved pseudo labels -> improved segmentation -> ... Have you tried that?\n"}}, {"review_id": "5aU8Wuir5Vd", "reviewer": "Reviewer_jkw9", "summary": "This paper presents a self-supervised and continual learning framework for improving indoor foreground-background segmentation and robot localization with known maps/models. The self-supervised learning approach involves generating pseudo segmentation labels from LiDAR scans and using them to fine-tune a segmentation model pre-trained with an offline segmentation dataset (NYU-Depth). The paper also extends this to continual learning where the robot acquires pseudo-labels on the fly and fine-tunes the model in new environments. This involves using a memory replay buffer to avoid catastrophic forgetting of previous experiences. Experimental results show that both the self-supervised and continual learning lead to significant error reductions in foreground-background segmentation.", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n- In general, self-supervised and continual learning are both quite relevant for real-world robotic systems. While data-driven paradigms have dominated vision methods, in robotics, systems have to rapidly adapt to new environments. And learning could be an interesting solution here.\n- The paper presents a real-robot setup with real-world evaluations. \n\nWeaknesses:\n- The self-supervised learning approach uses pseudo-labels generated by thresholding LiDAR scans to mask out foreground from background, but there is no baseline that investigates how good these pseudo-labels themselves are. What if you don't learn anything (self-supervised or continual) and directly use these pseudo-labels for segmentation? How well does this perform? The pseudo-labels might be noisy, but this depends on what sort of accuracy is necessary for the end-task, which is undefined. I am not sure if this pseudo-label result is already in Table 1; I had a hard time connecting the descriptions in Section 4.1 to items in Table 1.\n- The results emphasize 60% gains in foreground-background segmentation from the proposed approach, but the paper does not mention a strong use-case for why foreground-background is necessary or important for a specific task. The current setup assumes access to LiDAR+vision and maps/models of the environment, so I am not sure what foreground-background segmentation offers in this scenario. Obstacle avoidance and localization can be done with LiDAR, without vision. Further, the segmentation labels do not classify object categories, so they cannot be used for task-planning. \n- While the real-world evaluations are really great, they are hard to reproduce. Would it be possible to benchmark the performance gains in a simulated setup? Compared to the previous concerns, this is relatively minor, but nonetheless important. It’s hard to generate photorealistic simulations of construction and parking sites, but equivalent scenes from NVIDIA Isaac Sim or just setting-up a Gazebo environment might be feasible. While the 10% improvements in localization is great, this result can be substantially strengthened if it can be replicated in a reproducible benchmark. Similarly, the experiments in Section 4.3 need to be empirically more rigorous. What happens after 250 secs? Why does the plot stop exactly when the segmentation quality peaks?\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "While the proposed self-supervised and continual learning approaches are interesting, the evaluation is missing an important baseline and a strong motivating use-case for foreground-background segmentation.\n\n**Post Rebuttal**  \nI thank the authors for the detailed follow-up responses. Without the pseudo-label baseline, I find it hard to appreciate the value of the proposed learning-based methods. \"we never claim that different solutions to the application would not be possible\" -> I understand that there is merit to working on approaches that have other viable alternatives, but without empirical evaluations against simpler baselines, it's hard for the community to figure out what is the best way to _solve the problem_ (i.e. localization), even if the novel approach might be promising for some future applications. So I am keeping my original score of Weak Reject.", "technical_quality": "Fair: Significant additional evidence required to support the claims.", "clarity_of_presentation": "Fair: Major revision needed to improve organization or clarity.", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed.", "issues": "Major\n- See comments above on evaluation.\n\nMinor\n- L60: These contributions can be summarized succinctly. For example, the pseudo-label generation can be part of the self-improving framework.\n", "reviewer_expertise": "Good: General knowledge of the area"}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper proposes a continual learning perception framework for indoor robot localization on floor plans. All the reviewers find the problem being addressed in this work of importance and a required skill for robots. The paper is well written and clearly organized with real-world robotic experiments that are appreciated by the reviewers. Most of the concerns raised by the reviewers have been thoroughly addressed in the rebuttal. I thank the authors for the engaging discussions during the rebuttal. Some minor concerns still exist, primarily including a  pseudo-label baseline. Nevertheless, I agree with the reviewers that the paper is definitely an interesting contribution.
Decision: Accept (Poster)