# Change Detection and Model Update from Limited Query Data for Accurate Robot Localization

Stefan Larsen<fnref n="1" />, Ezio Malis<fnref n="1" />, El Mustapha Mouaddib<fnref n="2" />, Patrick Rives<fnref n="1" />

**Abstract**— The ability to maintain consistent 3D representations of changing environments remains a key challenge for long-term robotic systems in fields like surveillance, agriculture and autonomous vehicles. Visual localization and monitoring with robot agents require an accurate and up-to-date reference model of the scene, but the acquisition of such models is typically a costly task. Instead of performing specific missions to update the 3D representation of dynamic environments, this paper proposes a more efficient approach that uses limited query image data obtained by agents during previous generic missions in the area. The main contribution of the novel framework proposed in this paper is the ability to segment and locate both new and missing objects from only a few observations, to provide consistent updates to a dense reference model. Experiments with a new changing-outdoor dataset demonstrate the effectiveness of the model update framework, and show how model updates can be used to improve the accuracy of state-of-the-art visual localization over time.

## I. INTRODUCTION

Accurate digital representations of large and complex environments have many crucial applications for autonomous localization and monitoring. Recent methods using high-end sensors can create large, dense 3D representations [1], but this is typically a costly task that can not be done frequently due to time and budget constraints. However, many robotic tasks require accurate and updated reference models to perform over time, like localization [2][3]. This becomes a critical problem in environments like (peri)-urban areas, which are constantly affected by periodic changes from weather, vegetation and illumination, dynamic objects like cars and pedestrians, and construction work. Without consistent updates of the environment representation, scene changes may have significant impacts on the long-term performance of autonomous systems.

To solve the problem of updating reference models, research looks to detect significant changes for more efficient and consistent updates. These methods have many direct applications besides localization. Monitoring systems for surveillance or environmental purposes rely on change detection (CD) for precise scene understanding [4]. Robots in agriculture must deal with changing vegetation and scenery on a regular basis [5], either for navigation or tasks like monitoring of plant-growth or deforestation [6].

The ability to detect changes and update a 3D model depends on the available sensor data. Typically, explor-

ing robots are equipped with systems of multiple high-end sensors like IMU, GPS-RTK, RGBD and LiDAR. The robots often perform thorough mapping of the scene, via teleoperation or pre-defined trajectories, until the entire scene is captured. Although this enables dense and advanced CD and model update methods, it is not a realistic scenario for all autonomous, low-cost robotic systems in the near future. For example, a robot equipped with only a subset of low-cost sensors is asked to deliver a package or maneuver towards its target location. Although the sensors enable it to navigate using a pre-build 3D map or image database, it will not take the time to monitor in detail the entire scene, and its sensors are not able to update the model with the same quality in any case. However, the robot might still get glimpses of scene changes through a sensor, like a monocular RGB camera. This data could be used for continuous and consistent model updates, without the need of specific mapping operations.

The main objective of this paper is to address the problem of using limited query image data to detect changes and update a dense, pre-made 3D model. The data can be acquired during a generic mission, not necessarily dedicated to model update. An existing image-based method for 3D change detection based on geometrical ambiguities is used to detect scene change areas [7]. Then, this paper proposes to combine mesh segmentation and voxel grids to remove missing objects, and add new objects using an image-to-mesh network [8].

While previous work generally have dense query input data from RGBD video streams or LiDAR scanners, this paper considers the case where only the original environmental model is collected with high-resolution sensors. The query data comes from few samples of inexpensive sensors like monocular RGB camera. In this case, it is counterproductive to continuously update the entire model to incorporate changes, as this would lead to inconsistencies caused by the different quality of data. Instead, timely and precise local change updates will provide consistent model updates minimizing error propagation. The main contributions of this paper are (i) the segmentation and localization of changes in the 3D model using limited query image data and (ii) an approach to use model updates in order to improve long-term accuracy of vision based localization.

## II. RELEVANT WORK

Existing works with simultaneous localization and mapping (SLAM) already show impressive performance in challenging environments. Short-term dynamics are typically dealt with by ignoring dynamic objects or image pixels

<footnote n="1">¹Stefan Larsen, Ezio Malis and Patrick Rives are with ACENTAURI team at Inria, Sophia-Antipolis, France stefan.larsen@inria.fr, ezio.malis@inria.fr, patrick.rives@inria.fr</footnote>

<footnote n="2">²El Mustapha Mouaddib is with MIS lab, Amiens, France mouaddib@u-picardie.fr</footnote>
susceptible to change. However, less works have looked into
the problem of long-term spatio-temporal representations [9],
and how it can facilitate model-based localization. Mean-
while, reviews show that many pose estimation methods
struggle over time from changes in viewing conditions be-
tween reference and query models [10], which motivates the
need for consistent and accurate reference model updates.

To avoid costly remapping of the entire scene, an intuitive
approach for model update starts with detecting changes in
the scene. While some recent CD works try to detect signifi-
cant geometric change areas directly in 2D images using deep
neural networks (DNNs) [11], methods incorporating 3D
data show improved robustness to viewpoint and illumination
differences [4]. Some of these works find inconsistencies
between images and 3D models [7][12], while others use
grids for point cloud comparisons [13]. Unfortunately, most
methods only give rough estimates of change locations, like
change ellipsoids [7][12], which can not be used directly to
update a dense 3D model.

Some works present new mapping or classification tech-
niques to better locate, segment and track important changes.
One initial approach uses map differentiating and object
segmentation learning [14]. Signed distance functions can
reconstruct dense scenes from point clouds and images
[15][16][17], while neural descriptor fields can perform ro-
bust object-level CD indoors [18]. Others classify or segment
maps using random forest classifiers or DNNs [19]. While
they show promising results, it remains a challenge to
precisely distinguish new, moved and missing objects, both
indoors and outdoors, regardless of object sizes. Additionally,
these methods require large streams of RGBD data, taken
from several different points of view, to construct and update
the model of the environment. This is not feasible if only a
few images are available from previous generic missions.

Other methods have shown how only sparse change in-
formation can be enough to improve long-term localization.

Dynamic occupancy grids can estimate the robot pose and
environment state [20], while updated feature-based maps
can reduce localization error over time [3][21]. Semantic
scene graphs can relocate the robot in changing indoors
environments [22], which seems like a promising direction,
but would require more available change-data for training.

### III. PROPOSED FRAMEWORK

The proposed 3D model update framework, with steps to
locate and segment changes in the scene from only a few
query images, is described in Fig. 1. The initial 3D model is
reconstructed as a surface mesh (Fig. 1.1), to enable query
images to detect significant change areas for both new and
missing objects (Fig. 1.2). For improved change estimation,
this paper proposes 3D reconstruction from change masks
to generate newly discovered objects (Fig. 3.-4.), and mesh
segmentation and voxel grids to remove missing objects (Fig.
5.-8.). The framework is designed to run offline, in-between
robotic tasks. For model sizes of ~10 million points, the
process times are 5-10 min for mesh reconstruction, 2-10
min for mesh segmentation, 5-10 min for mesh voxelization
and 5-10 min for point occupancy checks. Meanwhile, both
the CD process and new object reconstruction are working
close to real-time for all models. The updated model can be
used to localize robots online during future missions.

#### A. Input Data

The initial dense 3D point cloud of an urban or peri-urban
scene is typically acquired using cameras, possibly with
depth estimates, or using LiDAR scanners [1]. The proposed
scheme for model update is designed for flexibility, and
works for any dense point clouds. Then, the point cloud must
be reconstructed into a mesh model with surfaces, preferrably
into two different models: dense for CD and sparse for
segmentation. The CD method used in this paper relies on
accurate image-to-model correspondences, and gets more

<figure><img src="image_1.png" /><figcaption>Fig. 1. Complete model update framework with numbered functions and visualization of data. From an initial 3D model and a set of query images, the steps are: 1. Surface mesh reconstruction. 2. Image based 3D change detection (new (blue) and missing (yellow) change ellipsoids). 3. Change mesh projection for new objects (arbitrary color). 4. 3D change localization of new object mesh. 5. Geometric mesh segmentation (arbitrary colors). 6. Pairing of change ellipsoids and segments for missing objects (paired object (pink) with ellipsoid (yellow)). 7. Voxelization of change segments. 8. Point occupancy collection (missing points are colored yellow). 9. Detected changes. 10. Model update as points or mesh.</figcaption></figure>
accurate image reprojections with a dense mesh. Meanwhile, the implemented segmentation method utilizes distinct shape differences, which are more apparent in sparse meshes.

Model updates are given by images from later inspections of the same scene, using RGB cameras. The camera intrinsic parameters and poses (with uncertainty) relative to the initial 3D model are needed, either directly from measurements, or from estimates using any visual localization method with initial registration. For the CD scheme used in this paper, at least three images observing the change area is needed, typically obtained from any previous, generic robot mission.

## B. Image-Based 3D Change Detection

While there exists different approaches for 3D CD, as presented earlier, this paper chooses to apply methods for finding geometrical ambiguities [7][12]. These two methods use pairs of query images to detect significant changes in a scene represented by an outdated 3D mesh. The images should be acquired within a narrow time frame to avoid structural or lighting changes between them. Objects that are moving during the acquisition should be filtered out using other methods, as the proposed framework only considers objects that have changed in-between missions. Typical changes are from static pedestrians, parked cars and scooters, and changes to vegetation and construction sites.

For detecting inserted objects, 2D change masks are estimated for each query image as proposed by [12]. Each query image is compared with reprojections of every other query image from its own point of view, via the outdated 3D model. Changes in the foreground scene cause color ambiguities between each query image and the reprojected images. The differences are stored and combined to reduce noise and improve accuracy of detected changes. Change location in 3D is estimated by triangulating the change masks, resulting in 3D *new-change* ellipsoids, up to scale with the model.

This CD method was later extended for removal detection by [7]. In that work, the problem was reverted, to focus on areas that should be occluded during the image reprojection process. After creating textured shadows using back projection between different viewpoints, areas in the textured shadows with the least change from the query images signify a previously occluded area, indicating foreground objects that are missing. By combining estimates from multiple images, missing objects are detected in images and localized as 3D *missing-change* ellipsoids.

While these two methods locate changes, the 3D change ellipsoids, both for new or missing objects, are not directly usable for consistent long-term model update and localization. After some iterations, the updated model would become crowded with overlapping ellipsoids. Instead, the framework proposed in this paper goes further, to use the change masks and change ellipsoids to segment and update the changed objects in the model.

## C. Locating and Segmenting New Objects

Significant scene changes may occur from new objects such as parked vehicles, or from construction work. Assuming new objects will remain in the scene for the near future,

they must be added to the model. While it is possible to obtain dense 3D reconstructions directly from images, this usually requires multiple high quality images from multiple viewpoints. However, the proposed scheme should be flexible enough for only a few images depicting each change.

Using the image data available, segmented images depicting only the new objects are obtained from the 2D change masks created during the CD method [12], see Fig. 1.2.

For new-object localization, this paper proposes to directly insert the cut-out change images into an image-to-mesh network [23]. This end-to-end deep learning architecture produces a 3D mesh model directly from a few RGB images. Using a graph-based DNN with a coarse-to-fine strategy, further improved by leveraging cross view information, this network shows high 3D shape estimation accuracy. As with all DNN-based methods, this requires training data of similar objects. The quality of the results also depends on the quality of the object images, in this case from the size and accuracy of the change masks, and the number of different viewpoints.

Another key step for locating new objects is to obtain the correct scale and 3D position of newly created object meshes. Without available dense maps, this must be done through triangulation from the camera viewpoints. As the CD method already estimates the position of each 3D *new-change* ellipsoids, the newly created object meshes are centered at the corresponding ellipsoid. The size of the ellipsoids should match the size of the new objects, but with queries with inaccurate pose estimates, the ellipsoids from [12] tend to be too small, only representing a part of the object. In that case, the new object mesh size is increased with a scalar factor based on the pose uncertainty, see Fig. 1.4.

## D. Locating and Segmenting Missing Objects

Other typical scene changes occur from objects missing from the initial model. Assuming missing objects will not return to the same location in the scene, they must be removed from the model. Here, the solution is to infer which objects the *missing-change* ellipsoids represent, and extract the objects from the model. This is done by segmenting the mesh model, and pairing each *missing-change* ellipsoids with candidate mesh segments. While well-trained DNNs show impressive semantic segmentation performance [24], geometry-based methods are often easier to implement with similar results. This paper uses a geometric, clustering-based method [25], see the result (assigned with random colors) in Fig. 1.5. For change pairing, each *missing-change* ellipsoid is converted into 3D voxelgrids. Mesh segments that overlap with a voxelized ellipsoid correspond to missing objects. This step requires an accurate mesh segmentation and few close objects, to avoid extracting nearby, unchanged segments.

To complete this change localization step, the corresponding changed points in the initial 3D model must be determined. Many surface reconstruction methods simplify the model, either by filtering nearby points, or by interpolating new points for smoother surfaces. Thus, deleting the change segment vertex points directly from the 3D model may not be a straightforward task, at least not to remove all the changed
points. Here, from the idea of voxelization for occupancy
checks, the change segments are also voxelized, and the
initial 3D points contained within them can be marked as
change, as seen as the yellow-colored points in Fig. 1.8 and
Fig. 1.9.

E. Long-Term Model Update

Finally, the initial 3D model can be updated with any
new and missing objects. The new objects can be added as
vertex points, with or without triangle faces, and the points
of missing objects from the previous occupancy check can
be subtracted. However, removing points directly from a
3D model may create holes, depending on the model and
accuracy of the segmentation. If an application requires a
solid and refined update model, any small gaps could be
filled instantly with interpolation or hole filling algorithms
[25], or with new data during the next acquisitions.

For long-term tasks with multiple model update iterations,
the proposed framework must also be robust to initial models
with gaps or various densities. By design, any small gap
in the initial 3D model will be covered during the first
mesh reconstruction step, to provide a solid model for the
following detections using image reprojections, seen on the
right of Fig 1.10. This effect motivates the use of surface
meshes for image reconstruction, compared to image render-
ing from point clouds which are more prone to gaps or sparse
data. For applications like image-based localization, the gaps
can simply be ignored using depth masks. Alternatively,
Gaussian splatting techniques [26] may create dense image
reprojections, but they require optimization with multiple
images from around the scene.

IV. EXPERIMENTS AND DISCUSSION

Experiments with a new changing-outdoor dataset demon-
strate how the proposed model update framework achieves
accurate model updates from limited query data, and im-
proves state-of-the-art visual localization over time.

A. *Experimental Setup and OCA Dataset*

Due to the lack of datasets for long-term change detection
in outdoor scenes, the OCA dataset has been created. It
contains one large, dense point cloud with RGB values,
obtained with a Leica RTC 360 LiDAR scanner with internal

<figure><img src="image_2.png" /><figcaption>Fig. 2. Query images from *parking* (top) and *dome* (bottom) at the OCA.</figcaption></figure>

RGB camera, at the Observatory of Nice (OCA). It is divided
into two parts, *parking* and *dome*, corresponding to different
areas of the OCA site. Query images are multiple sequences
of RGB images obtained with a mono camera setup on top of
a car, which is equipped with a GPS RTK and an IMU sensor.
The car was driven around manually through the site, and
captured data under shifting weather conditions at different
times of the day and the year. Each query set in the *parking*
area contains images in two directions, denoted *outbound*
*return*. The ground truth trajectory and rotation is computed
from a fusion of the GPS and IMU data, and manually
aligned with the model frame using fixed GPS points.

For presentation of the performance over time, a selection
of three iterations at both the *parking* and the *dome* are
chosen to present the evolution of the scene. The query
images are illustrated in Fig. 2. First, model update results
are presented for both scenes, and then the *parking* images
are also used for visual localization experiments.

B. Model Update Results

The model update results from the first iteration with
query *Q1p* at the *parking* are shown in Fig. 3. The proposed
framework is able to detect the new car and the missing car,
segment them and finally update the model, see Fig. 3.8 and
Fig 3.9. This provides a more detailed model update with
respect to the initial change ellipsoids [7] seen in Fig. 3.3.

While the missing car is accurately detected and removed
by the framework, the recreation of the new car object can be
difficult when only using a few images. In this case, three
masks depicting the side of the car were used to recreate
the new mesh object using [23]. Although the pre-trained

<figure><img src="image_3.png" /><figcaption>Fig. 3. Model update results from *parking* area. 1. Initial 3D point cloud. 2. A query image from *Q1p*. 3. Mesh model with change ellipsoids. 4. A change mask. 5. Segmentation. 6. New object mesh. 7. Pairing of missing objects. 8. Updated model point cloud. 9. Updated mesh reconstruction.</figcaption></figure>
<figure><img src="image_4.png" /><figcaption>Fig. 4. Model update results from dome area, three iterations. As in Fig. 3, the results from each step of the model update framework is presented. The second iteration starts with the updated model from the output of the first iteration, and the third starts with the output of the second iteration.</figcaption></figure>

network was trained on car shapes with excellent results, the obtained object shape here is not very accurate. This is due to the slightly inaccurate change masks obtained, and the lack of viewpoints all the way around the car. A possible improvement is to train the network on similar rough change masks from sparse viewpoints. However, since the focus of this paper is mainly on CD for model update, only the pretrained reconstruction model from [23] was used. Nevertheless, even using a rough mesh, stored in the position of the change ellipsoid, see Fig. 3.6, is an improvement to the ellipsoid, and it is sufficient in most cases to improve visual localisation results, as demonstrated in the next subsection and Table I. Additional acquisitions from future missions in the area can further refine the new reconstruction, if better change masks are obtained.

Notice that removing the missing car creates gaps in the model, which might cause troubles for some applications. Here, the holes were covered by a geometric-based mesh technique [25], as seen in Fig. 3.9.

Model update results from iterations with $Q1_d-Q3_d$ at the *dome* are presented in Fig. 4. In the cases shown, various objects like cars, benches and fences are appearing and disappearing from the scene between missions, and the framework is able to detect all the changes and iteratively update the model. Similar to the results in the *parking* area, the removal of missing points is more accurate than the reconstruction of the new objects. However, it is shown how the framework can be iterated to consistently detect changed objects and maintain an up-to-date model of the environment.

Another source of error for the pixel-to-mesh method is the estimation of query image poses. Here, the query poses are given by a state-of-the art robust visual local-

ization method [27] described in the next subsection. As seen in the localization experiments in the next subsection, the accuracy was typically 10-50 centimeters under normal conditions, which proved sufficient to detect and locate changes without false positive or false negative detections. During the experiments in [12], the poses were obtained through direct georeferencing fusing GPS, IMU, and visual odometry, which gives more accurate change localization and change masks. The triangulation process of the pixel-to-mesh method would also be improved with more accurate poses. This indicates that better reconstructions can be obtained with more comprehensive localization techniques or retraining the pixel-to-mesh network with noisy change masks.

### C. Visual Localization Results

Experiments have been conducted to show how the proposed model update framework improves visual localization over time. Specifically, how it improves the estimation of 6 degrees-of-freedom (DOF) poses with respect to a reference model, which has many critical applications within autonomous navigation and monitoring [2][13]. Here, four sets of query images $Q1_p-Q4_p$ will be localized using both the initial model, denoted $M1$, and iteratively updated version of the model, denoted $M2-M4$, as reference models. Each updated model is obtained by detecting significant changes using a few of the previous query images. $M1$ localizes $Q1_p$ and is updated to $M2$. $M2$ is then used to localize $Q2_p$ and obtain $M3$ etc. Since the query images contain changes from the initial reference model, the updated models should provide more accurate and robust references over time. The ground truth trajectories obtained from the GPS and IMU fusion are used to estimate the absolute and relative errors.
A state-of-the-art hierarchical visual localization method using Structure-from-Motion (SfM) and feature-based DNN is used as-is [27]. While any visual localization method could be used, this method proved as one of the best in changing outdoor environments, although with significant errors when the scene changes too much from the initial model [10].

The localization method takes reference images as input to create a database of features, matched image pairs and a sparse SfM reconstruction. For both *M1* and *M2-M4*, the reference images are taken as renders from the dense point cloud models, along a trajectory similar to the path of the query images. The first set of queries $QI_p$ is located using *M1*, while the last three $Q2_p-Q4_p$ are located using both *M1* and the updated models, for comparison. The setup of iterative model updates chained together with pose estimation of new query images becomes a complete model update and localization system for long-term autonomous operation using only mono RGB sensors.

Three metrics are used to determine the accuracy of the pose estimation. The first one is inspired by a recent review [10] that measures the percentage of query images localized within a position and orientation threshold of their ground truth. Here, the thresholds are set as: Very-high ($VH$) (0.1m, 1°), high ($H$) (0.25m, 2°), and coarse ($C$) (0.5m, 5°). They are chosen slightly stricter than in [10] to satisfy the high demands from real-world robotic applications. The other two metrics are the mean absolute error of translation $MAE_t$ and rotation $MAE_r$ with respect to the ground truth trajectory, and the mean relative error of translation $MRE_t$ and rotation $MRE_r$ by comparing the relative transformations between nearby poses with the ground truth relative transformations. The standard deviations for the absolute and relative metrics are also included in the table.

Table I presents the localization results for $QI_p-Q3_p$ (with number of images) in *outbound* and *return* direction, for *M1* and *M2-M4*. As marked in bold, the updated models significantly reduce the pose error both in absolute and relative error. The biggest improvements are seen in the *return* direction, where it almost halves the absolute errors for all three query sets. This can be explained by the presence of features in the SfM model created by [27] for each reference model. In Fig. 5 it is shown how *M1* still contains features of a car parked on the right side, which is no longer present in the $QI_p$ images. Not only does this confuse the feature matching step, it also occludes the background. The *outbound* queries also include important features from the big stone wall on the left side, so the loss in performance from this occlusion is less critical. However, the *return* queries eventually lose sight of the big wall, and must rely on the smaller stone fence for feature matching. Because of this, the occlusion of parts of the fence in *M1* causes significant errors compared to *M2-M4*. These results motivate the need to remove outdated objects from the reference model, as (peri)-urban scenes often include sections with sparsity of good visual features.

The drop in accuracy with $Q3_p$ for *M3* compared to *M1* in the *outbound* direction is caused by the setup of query

<table><caption>TABLE I
LOCALIZATION RESULTS FROM OCA DATASET</caption><thead><tr><th>Outbound</th><th>Metric</th><th>Q2<sub>p</sub> (140)</th><th>Q3<sub>p</sub> (160)</th><th>Q4<sub>p</sub> (200)</th></tr></thead><tbody><tr><td rowspan="5">M1</td><td>VH/H/C</td><td>0.7 / 80.0 / 98.6</td><td>0.0 / 52.5 / 98.8</td><td>0.0 / 0.0 / 2.0</td></tr><tr><td>MAE<sub>t</sub> [m]</td><td>0.2 ± 0.12</td><td><b>0.25 ± 0.12</b></td><td>0.95 ± 0.37</td></tr><tr><td>MAE<sub>r</sub> [deg]</td><td><b>0.83 ± 0.37</b></td><td>1.84 ± 0.49</td><td>2.46 ± 0.76</td></tr><tr><td>MRE<sub>t</sub> [m]</td><td>0.08 ± 0.11</td><td>0.09 ± 0.14</td><td>0.25 ± 0.3</td></tr><tr><td>MRE<sub>r</sub> [deg]</td><td>0.35 ± 0.43</td><td>0.42 ± 0.52</td><td>0.97 ± 1.18</td></tr><tr><td rowspan="5">M2-M4<br>(Ours)</td><td>VH/H/C</td><td><b>8.6 / 82.1 / 100.0</b></td><td>0.0 / 33.1 / 77.5</td><td>0.0 / 0.0 / 2.5</td></tr><tr><td>MAE<sub>t</sub> [m]</td><td><b>0.17 ± 0.11</b></td><td>0.35 ± 0.21</td><td><b>0.9 ± 0.33</b></td></tr><tr><td>MAE<sub>r</sub> [deg]</td><td>0.84 ± 0.28</td><td><b>1.47 ± 0.53</b></td><td><b>2.05 ± 0.75</b></td></tr><tr><td>MRE<sub>t</sub> [m]</td><td><b>0.06 ± 0.08</b></td><td><b>0.07 ± 0.11</b></td><td><b>0.16 ± 0.22</b></td></tr><tr><td>MRE<sub>r</sub> [deg]</td><td><b>0.27 ± 0.21</b></td><td><b>0.27 ± 0.23</b></td><td><b>0.55 ± 0.75</b></td></tr><tr><td rowspan="5">Return</td><td>Metric</td><td>Q2<sub>p</sub> (200)</td><td>Q3<sub>p</sub> (140)</td><td>Q4<sub>p</sub> (120)</td></tr><tr><td rowspan="5">M1</td><td>VH/H/C</td><td>0.0 / 21.0 / 62.0</td><td>0.0 / 0.7 / 31.4</td><td>0.0 / 1.7 / 45.0</td></tr><tr><td>MAE<sub>t</sub> [m]</td><td>0.56 ± 0.58</td><td>0.88 ± 1.17</td><td>0.71 ± 0.66</td></tr><tr><td>MAE<sub>r</sub> [deg]</td><td>2.45 ± 2.05</td><td>6.06 ± 4.83</td><td>3.87 ± 3.08</td></tr><tr><td>MRE<sub>t</sub> [m]</td><td>0.31 ± 0.38</td><td>0.57 ± 0.8</td><td>0.41 ± 0.39</td></tr><tr><td>MRE<sub>r</sub> [deg]</td><td>1.06 ± 1.46</td><td>2.22 ± 4.01</td><td>1.55 ± 2.32</td></tr><tr><td rowspan="5">M2-M4<br>(Ours)</td><td>VH/H/C</td><td>0.0 / 23.5 / 71.0</td><td>0.0 / 0.7 / 54.3</td><td>0.0 / 1.7 / 91.7</td></tr><tr><td>MAE<sub>t</sub> [m]</td><td><b>0.43 ± 0.28</b></td><td><b>0.39 ± 0.18</b></td><td><b>0.36 ± 0.11</b></td></tr><tr><td>MAE<sub>r</sub> [deg]</td><td><b>1.99 ± 1.65</b></td><td><b>3.32 ± 2.0</b></td><td><b>1.72 ± 0.92</b></td></tr><tr><td>MRE<sub>t</sub> [m]</td><td><b>0.14 ± 0.18</b></td><td><b>0.1 ± 0.14</b></td><td><b>0.1 ± 0.13</b></td></tr><tr><td>MRE<sub>r</sub> [deg]</td><td><b>0.58 ± 0.92</b></td><td><b>0.8 ± 0.77</b></td><td><b>0.8 ± 0.86</b></td></tr></tbody></table>

images. Although an assumption for the proposed model update framework is that all changes to the environment will remain in the scene for a significant period of time, it is not always the case in urban environments with dynamic objects. In this case, while $QI_p$ and $Q2_p$ both depicted a new car in front of the stone wall, this car is missing again from $Q3_p$.

<figure><img src="image_5.png" /><figcaption>Fig. 5. Results from visual localization in the *parking* area with the SfM model of *M1* on the left and *M2* on the right for query $Q2_p$, in *outbound* direction on the top and *return* direction at the bottom. The arrows in red indicate the points from the missing car contained in *M1*, in blue indicate the query trajectory direction. The red and cyan frames depict the position of the reference poses and estimated query poses, respectively. Green points are matched features. The white squares indicate the areas that are occluded in *M1* but not in *M2*, which is most noticeable for the *return* direction.</figcaption></figure>
and $Q4_p$. However, after $Q1_p$ and $Q2_p$, the updated model M3 includes a mesh object in this location, representing the car that was previously seen here. This becomes an "inserted obstacle", which blocks some features on the big wall, similar to how the outdated M1 blocks features from the small fence. These results indicate that although the model update framework is able to detect and update a dense 3D representation using limited query data, future work should look into better ways to incorporate new changes to the scene. In the current framework, the new object meshes are not directly useful for visual localization over time, as they do not provide new features for localization, and may cause problems if the scene is changing too much too fast. However, the removal of missing objects from reference updates have a significant impact on a state-of-the-art visual localization method over time.

## V. CONCLUSIONS AND FUTURE WORK

A novel framework for 3D model update using limited query data is proposed for long-term autonomous robotic operations like localization and monitoring in (peri)-urban environments. It demonstrates how surface reconstructions, model segmentations, voxel grids and image-based reconstructions can be chained together to successfully update a model with both new and missing objects. It can be seen as an extension on previous change detection methods [7][12]. While they only indicate change areas, the proposed framework enables consistent segmentation and localization of changed points in the initial, dense model. The usefulness of this approach is demonstrated through localization experiments, where a state-of-the-art visual localization method improves accuracy over time thanks to iterative updates to the reference model.

Improvements for more robust and accurate model updates could be to include semantic classification for better mesh segmentation and object tracking. Future work could also improve the 3D CD step, to better handle multiple objects and query pose inaccuracy. The reconstruction of new objects from a few images could be improved with training on data from similar challenging points of view.

## ACKNOWLEDGMENT

This project is funded by the ANR SAMURAI project (ANR-21-CE33-0014).

## REFERENCES

[1] T. Kühner and J. Kümmerle, "Large-scale volumetric scene reconstruction using lidar," in *IEEE ICRA*, 2020.

[2] M. Bürki, L. Schaupp, M. Dymczyk, R. Dubé, C. Cadena, R. Siegwart, and J. Nieto, "Vizard: Reliable visual localization for autonomous vehicles in urban outdoor environments," in *IEEE IV*, 2019.

[3] J. S. Berrio, S. Worrall, M. Shan, and E. Nebot, "Long-term map maintenance pipeline for autonomous vehicles," *IEEE Transactions on Intelligent Transportation Systems*, 2021.

[4] A. Kharroubi, F. Poux, Z. Ballouch, R. Hajji, and R. Billen, "Three dimensional change detection using point clouds: A review," *Geomatics*, 2022.

[5] M. Hussain, D. Chen, A. Cheng, H. Wei, and D. Stanley, "Change detection from remotely sensed images: From pixel-based to object-based approaches," *ISPRS Journal of photogrammetry and remote sensing*, 2013.

[6] R. Qin, J. Tian, and P. Reinartz, "3d change detection–approaches and applications," *ISPRS Journal of Photogrammetry and Remote Sensing*, 2016.

[7] O. Roupin, M. Fradet, C. Baillard, and G. Moreau, "Detection of removed objects in 3d meshes using up-to-date images for mixed-reality applications," *Electronics*, 2021.

[8] N. Wang, Y. Zhang, Z. Li, Y. Fu, W. Liu, and Y.-G. Jiang, "Pixel2mesh: Generating 3d mesh models from single rgb images," in *IEEE ECCV*, 2018.

[9] L. Schmid, M. Abate, Y. Chang, and L. Carlone, "Khronos: A unified approach for spatio-temporal metric-semantic slam in dynamic environments," *arXiv preprint arXiv:2402.13817*, 2024.

[10] C. Toft, W. Maddern, A. Torii, L. Hammarstrand, E. Stenborg, D. Saffari, M. Okutomi, M. Pollefeys, J. Sivic, T. Pajdla, et al., "Long-term visual localization revisited," *IEEE Transactions on Pattern Analysis and Machine Intelligence*, 2020.

[11] W. Shi, M. Zhang, R. Zhang, S. Chen, and Z. Zhan, "Change detection based on artificial intelligence: State-of-the-art and challenges," *Remote Sensing*, 2020.

[12] E. Palazzolo and C. Stachniss, "Fast image-based geometric change detection given a 3d model," in *IEEE ICRA*, 2018.

[13] W. Ma, Z. Song, Y. He, and S. Shen, "Change detection and update of 3d sparse map by merging geometry and appearance," *International Journal of Applied Earth Observation and Geoinformation*, 2023.

[14] R. Finman, T. Whelan, M. Kaess, and J. J. Leonard, "Toward lifelong object segmentation from change detection in dense rgb-d maps," in *IEEE European Conference on Mobile Robots*, 2013.

[15] M. Fehr, F. Furrer, I. Dryanovski, J. Sturm, I. Gilitschenski, R. Siegwart, and C. Cadena, "Tsdf-based change detection for consistent long-term dense reconstruction and dynamic object discovery," in *IEEE ICRA*, 2017.

[16] J. Fu, C. Lin, Y. Taguchi, A. Cohen, Y. Zhang, S. Mylabathula, and J. J. Leonard, "Planesdf-based change detection for long-term dense mapping," *IEEE Robotics and Automation Letters*, 2022.

[17] L. Schmid, J. Delmerico, J. L. Schönberger, J. Nieto, M. Pollefeys, R. Siegwart, and C. Cadena, "Panoptic multi-tsdfs: a flexible representation for online multi-resolution volumetric mapping and long-term dynamic scene consistency," in *IEEE ICRA*, 2022.

[18] J. Fu, Y. Du, K. Singh, J. B. Tenenbaum, and J. J. Leonard, "Robust change detection based on neural descriptor fields," in *IEEE IROS*, 2022.

[19] E. Langer, T. Patten, and M. Vincze, "Robust and efficient object change detection by combining global semantic information and local geometric verification," in *IEEE IROS*, 2020.

[20] G. D. Tipaldi, D. Meyer-Delius, and W. Burgard, "Lifelong localization in changing environments," *The International Journal of Robotics Research*, 2013.

[21] E. Derner, C. Gomez, A. C. Hernandez, R. Barber, and R. Babuška, "Change detection using weighted features for image-based localization," *Robotics and Autonomous Systems*, 2021.

[22] J. Kabalar, S.-C. Wu, J. Wald, K. Tateno, N. Navab, and F. Tombari, "Towards long-term retrieval-based visual localization in indoor environments with changes," *IEEE Robotics and Automation Letters*, 2023.

[23] C. Wen, Y. Zhang, Z. Li, and Y. Fu, "Pixel2mesh++: Multi-view 3d mesh generation via deformation," in *IEEE ICCV*, 2019.

[24] J. Wald, K. Tateno, J. Sturm, N. Navab, and F. Tombari, "Real-time fully incremental scene understanding on mobile platforms," *IEEE Robotics and Automation Letters*, 2018.

[25] The CGAL Project, *CGAL User and Reference Manual*, 5.1 ed. CGAL Editorial Board, 2020.

[26] B. Kerbl, G. Kopanas, T. Leimkühler, and G. Drettakis, "3d gaussian splatting for real-time radiance field rendering," *ACM Transactions on Graphics*, 2023.

[27] P.-E. Sarlin, C. Cadena, R. Siegwart, and M. Dymczyk, "From coarse to fine: Robust hierarchical localization at large scale," in *IEEE ICCV*, 2019.