Scalable Benchmarking and Robust Learning for Noise-Free Ego-Motion and 3D Reconstruction from Noisy Video
Authors: Xiaohao Xu
Tianyi Zhang
Shibo Zhao
Xiang Li
Sibo Wang
Yongqi Chen
Ye Li
Bhiksha Raj
Matthew Johnson-Roberson
Sebastian Scherer
Xiaonan Huang
Venue: ICLR 2025 Poster (2025)
Keywords: Benchmarking
Robustness
Neural 3D Reconstruction
Ego-Motion Estimation
SLAM
Primary area: datasets and benchmarks
TL;DR: We reframe ego-motion estimation and photorealistic 3D reconstruction under generalized noisy sensing conditions as model-data feedback loop framework.

Abstract: We aim to redefine robust ego-motion estimation and photorealistic 3D reconstruction by addressing a critical limitation: the reliance on noise-free data in existing models. While such sanitized conditions simplify evaluation, they fail to capture the unpredictable, noisy complexities of real-world environments. Dynamic motion, sensor imperfections, and synchronization perturbations lead to sharp performance declines when these models are deployed in practice, revealing an urgent need for frameworks that embrace and excel under real-world noise.
To bridge this gap, we tackle three core challenges: scalable data generation, comprehensive benchmarking, and model robustness enhancement. First, we introduce a scalable noisy data synthesis pipeline that generates diverse datasets simulating complex motion, sensor imperfections, and synchronization errors. Second, we leverage this pipeline to create Robust-Ego3D, a benchmark rigorously designed to expose noise-induced performance degradation, highlighting the limitations of current learning-based methods in ego-motion accuracy and 3D reconstruction quality. Third, we propose Correspondence-guided Gaussian Splatting (CorrGS), a novel method that progressively refines an internal clean 3D representation by aligning noisy observations with rendered RGB-D frames from clean 3D map, enhancing geometric alignment and appearance restoration through visual correspondence.
Extensive experiments on synthetic and real-world data demonstrate that CorrGS consistently outperforms prior state-of-the-art methods, particularly in scenarios involving rapid motion and dynamic illumination. We will release our code and benchmark to advance robust 3D vision, setting a new standard for ego-motion estimation and high-fidelity reconstruction in noisy environments.

Published as a conference paper at ICLR 2025
SCALABLE BENCHMARKING AND ROBUST LEARNING FOR NOISE-FREE EGO-MOTION AND 3D RECONSTRUC- TION FROM NOISY VIDEO
Xiaohao Xu', Tianyi Zhang", Shibo Zhao", Xiang Li,, Sibo Wang', Yonggi Chen', Ye Li', Bhiksha Raj². Matthew RobersnnR,b,rson,, Sebastian Scherer Xiaonan Huang'
1 University of Michigan, Ann Arbor, Carnegie Mellon University
ABSTRACT
We aim to redefine robust ego-motion estimation and photorealistic 3D reconstruc- tion by addressing a critical limitation: the reliance on noise-free data in existing models. While such sanitized conditions simplify evaluation, they fail to capture the unpredictable, noisy complexities of real-world environments. Dynamic motion, sensor imperfections, and synchronization perturbations lead to sharp performance declines when these models are deployed in practice, revealing an urgent need for frameworks that embrace and excel under real-world noise. To bridge this gap, we tackle three core challenges: scalable data generation, comprehensive bench- marking, and model robustness enhancement. First, we introduce scalable noisy data synthesis pipeline that generates diverse datasets simulating complex motion, sensor imperfections, and synchronization errors. Second, we leverage this pipeline to create Robust-Fgo3il a benchmark rigorously designed to expose noise-induced performance degradation, highlighting the limitations of current learning-base based methods in ego-motion accuracy and 3D reconstruction quality. Third, we propose Correspondence guided Gaussian Splatring (CorrGS), a novel method that progressively refines internal clean 3D representation by aligning noisy observations with rendered RGB-D frames from clean 3D map, enhancing geometric alignment and appearance restoration through visual correspondence. Extensive experiments on synthetic and real-world data demonstrate that CorGG consistently outperforms prior state-of-the-ar methods, particularly in scenarios involving rapid motion and dynamic illumination."
INTRODUCTION
The pursuit of reliable ego-motion estimation and photorealistic 3D reconstruction remains a fun- damental challenge in computer vision and autonomous systems, particularly in the presence of real-world complexities such as sensor noise and dynamic motion. The current landscape of dense Neural SLAM (Simultaneous Localization and Mapping) research has made significant strides in noise-free, controlled settings using advanced neural representations like Neural Radiance Fields (NeRF) (Rosinol et al., 2022) and Gaussian Splatting (Kerbl al., 2023). Despite their promise, the generalization of these methods to noisy, unstructured environments-reffee reflecting realistic deployment conditions remains largely unexplored (Tosi al., 2024), highlighting the critical importance of studying the noise resilience of models to ensure robust performance in real-world scenarios.
We envision autonomous systems capable of robustly navigating and reconstructing 3D scenes even under adverse conditions, such as sensor degradation, unpredictable disturbances, or rapid camera motion. This vision demands overcoming critical limitations of existing methods that assume clean, synthetic datasets (Straub et al., 2019; Dai et al., 2017a). which often ignore practical sensing imperfections. Real-world datasets, while valuable, face issues of scalability, annotation quality, and capturing diverse degraded scenarios (Sturm et al., 2012; Schöps et al., 2019; Zhao et al., 2024). To bridge this gap, we propose a fundamental shift toward understanding and benchmarking models under generalized noisy conditions, enabling systematic evaluation of their robustness.
Project repo: https:: / /g/ wttgggtthhhh.og com
wwt.ww.w./...........ooe www.o///ww.ooooo///ooooooooo...a Xu www.S////S/.. -LA- -Perturbat on

---

Published as a conference paper at ICLR 2025
(a) Prior Perfect Data Synthesis
Slow & Stable Motion
Prior Benchmarking on Prior Models
Noise-free Video Stream
Noise-free 3D
(d) CorrGS (Our Model)
Fast & Dynamic Motion
Perturbations
Dogacitt
(b) Noisy Data Synthesks (Ours)
(c) Benchmarking Prior Models
Noisy Video Stream
Nolsy
Figure 1: Towards robust ego-motion and photorealistic 3D reconstruction. (a) Previous ap- proaches rely on synthetic datasets with perfect conditions (noise-free and smooth motion). (b) Real-world data introduce inherent noise and complexities. We present a customizable noisy data synthesis pipeline evaluate methods under realistic noisy conditions. (c) Our Robust-Ego3i benchmark reveals that existing methods produce noisy 3D reconstructions from noisy, sparse-view videos, whereas (d) our proposed CorrGS achieves photorealistic, noise-free 3D reconstruction.
To advance this vision, we pose three key research questions: 1) How to synthesize noisy data at scale? 2) How well do current SOTA models perform under noisy conditions? 3) How can we enhance model robustness against complex perturbations? To address these questions, present a comprehensive framework for robust ego-motion estimation and photorealistic 3D reconstruction in generalized noisy conditions, with the following contributions:
1) Scalable noisy data generation pipeline. We first present a comprehensive taxonomy of RGB-D sensing perturbations for mobile systems and develop scalable noisy data synthesis pipeline that transforms clean 3D meshes into challenging datasets (see Fig. lb), supporting sensor configurations from single to distributed multi-sensor setups. The datasets offer precise 3D map and trajectory ground-truth. scalability, and customizable perturbations. Our pipeline supports more thorough and cost-effective evaluation than noise-free benchmarks, bridging the gap to real-worde testing.
2) Robust-Ego.3D benchmark for generalized noisy conditions. Utilizing our noisy data syn- thesis pipeline, we instantiate Robust-Ego31), large-scale benchmark that supports extensive and customizable RGB-D perturbations, offering 124 perturbed settings for comprehensive evaluation. Unlike existing benchmarks that focus on noise-free conditions, Robust-Ego provides a chal- lenging testbed for assessing model robustness. Our extensive experiments and theoretical analyses reveal that the most common challenges of existing models are: pose tracking failure under dynamic motion, and degradation of 3D reconstruction fidelity under imaging perturbations due to the lack of restoration mechanisms (see Fig. Ic). Several key insights emerge: i) no model demonstrates consistent robustness across all perturbations, ii) individual perturbations have similar effects in both isolated and mixed settings, and iii) highly correlated perturbations can be leveraged as proxies for efficient benchmarking. We hope Robust-Ego3D prompts researchers rethink model robustness under generalized noisy conditions and question existing dataset and model assumptions.
3) CorrGS: Correspondence-guides Gaussian Splatting. To tackle robust pose tracking under complex motion and consistent color reconstruction in noisy conditions identified in our Robust- Ego3D benchmark, we propose ConGG. CorrGS employs Gaussian Splatting 3D representation. swiftly rendered into RGB-D frames. By comparing these rendered images with noisy observations, establishes correspondences that enhance geometric alignment for robust pose learning. These correspondences also support online appearance restoration learning, enabling noise-free 3D recon- struction from noisy videos. CorGGS significantly outperforms prior state-of-the-at methods under fast motion, achieving accurate ego-motion and photorealistic 3D reconstruction from both synthetic (see Fig. ld) and real-world noisy videos, even with dynamic illumination and rapid motion.
This work rethinks robustness in dense Neural SLAM by moving beyond ideal-conditinn accuracy to true resilience under noisy sensing. By challenging existing benchmarks and assumptions, we pave the way for developing adaptive models that consistently perform in diverse, degraded conditions.
2

---

Published as a conference paper at ICLR 2025
2
RELATED WORK
Dataset synthesis for robustness benchmarking. Most robustness evaluations rely on syn- thetic datasets due to the difficulty of controlling real-world perturbations. A pioneering bench- mark (Hendrycks & Dietterich, 2019) assessed image classification under common corruptions, and subsequent research extended this approach to various tasks, such as 2D/3D object detec- tion (Michaelis et al., 2019; Kong et al., 2023), segmentation (Kamann & Rother, 2020; Li et al., 2023; Xu et al., 2022), and embodied navigation (Chattopadhyay al., 2021; Yokoyama et al., 2022). In ego-motion estimation and photorealistic 3D reconstruction, the challenges go beyond image-level corruptions, also involving temporal variations sensor noise and pose transformations.
Robustness benchmarking. Robust localization and mapping are crucial for reliable performance in dynamic environments (Cadena al., 2016). Real-world datasets (Schubert al., 2018 8; Helmberger et al., 2022; Tian et al., 2023; Ebadi et al., 2023) evaluate classical SLAM under conditions like low illumination and motion blur. While existing benchmarks like TartanAir (Wang et al., 2020) and SLAMBench (Bujanca et al., 2021) incorporate some noise injection for robustness evaluation, none comprehensively address the full range of RGB-D perturbations encountered by mobile agents. Our benchmark, Robust-Ego3ID introduces diverse, controllable perturbations for systematic evaluation of dense SLAM systems under varied noise, including sensor noise, motion deviations, and synchro- nization issues. Unlike prior work focused on localization accuracy in classical SLAM, assess dense Neural SLAM models considering both localization and photorealistic mapping, pioneering robustness evaluation to drive future research in this evolving field.
Dense Neural SLAM model. Classical SLAM models, e.g., ORB-SLAM3 (Campos et al., 2021). 1) excel at estimating ego-motion but produce only sparse maps. In contrast, dense non-neural SLAM models (Newcombe et al., 2011; I; Dai et al., 2017b) provide dense reconstructions but struggle to capture fine appearance details and textures. Dense Neural SLAM models overcome these limitations by reconstructing both geometry and appearance, enabling high-fidelity photorealistic 3D rendering. Implicit neural representations, e.g., NeRF (Rosinol et al., 2022), have been integrated for high- quality textured reconstruction (Sucar et al., 2021), with significant advancements in representation techniques ( (Wang et al., 2023; Johari et al., 2023; Zhu et al., 2022; Yang et al., 2022; Sandström et al., 2023). Additionally, some methods decouple Cearning-based pose estimation from dense mapping (Teed & Deng, 2021; Zhang et al., 2023). Building on the success 3D Gaussian Splatting (Kerbl et al., 2023), recent approaches have adapted this technique SLAM (Keetha et al., 2024; Matsuki et al., 2024), enabling efficient RGB-D rendering from reconstructed maps. Despite these advancements, most Neural models focus primarily on roise-free assumptions, highlighting gap in pose tracking and 3D scene restoration under challenging conditions.
3 NOISY DATA SYNTHESIS WITH CUSTOMIZABLE PERTURBATIONS
3.1 PROBLEM FORMULATION
We address the challenge of robust ego-motion estimation and photorealistic 3D reconstruction from noisy data using a structured approach involving three interconnected components: noisy data synthesis, model estimation, and evaluation with feedback (see Fig. 2).
Noisy data synthesis (from clean 3D to noisy video). Perturbed observations 21:1 and trajectories X1.t are generated from a clean 3D representation m, a clean trajectory xf... and perturbations {1:
p(Zi: X1x m, xf.r-1:a)) E1:))
(1)
m, X Ext
Model estimation (from noisy video to noise-free ego-motion and 3D reconstruction). From noisy ob- servations 21z,, the model estimates the 3D scene in and the trajectory &
7
m,
Noisy Data
Model Estimation
Synthesis
Feedback conditioned on perturbation (SL.
((ñ &1s
(2)
m, M,
Evaluation
Evaluation and feedback. The estimates m and &1.0 are compared with ground truth m and X1ss revealing perturbation-specffic weaknesses caused by Ei.--
Figure 2: Framework for noisy data syn- thesis, model estimation, and evaluation.
3

---

Published as a conference paper at ICLR 2025
tx2
3D Obsrrvatio
2D Piud
Clean Depth
Trajectory
10-
10
RGB Stream
Spachronfpod
Translation
Deviation
[0]
10
10 tx2
1+2
Environment Interference
Lens Distoctions
Semsor Notses
Pestpracess
Noise in Depth Imaging
Depth Streems
[+]
1=2
t+3
Rotation
I+1
Noisy
RGB
t+]
1+2
Deviation
Sensing Rango I Limitation
Desuuhronized
Mation
1+1
Natse
Brightness
Out Range
Degii Stream
[+1
E+2
(a) Sensor Poses
(b) RGB Sensor Imaging
(c) Depth Sensor Imaging
(d) Synchronisation
Figure 3: Perturbation taxonomy for RGB-D sensing.
3.2 PERTURBATION TAXONOMY FOR RGB-D SENSING SYSTEMS
Perturbation sources. As shown in Fig. 3, perturbations for RGB-D sensing stem from pose deviations, imaging inaccuracies, and sensor desynchronization All perturbations are constructed using fundamental physical and kinematic modeling, e.g. motion deviations are derived from rigid- body transformations. We briefly illustrate perturbations here; details are in Appendix Sec. A.
(a) Perturbations on sensor pose motion. Real-world mobile platforms can exhibit diverse and dynamic motions that challenge model robustness. As shown in Fig. 3a, we categorize motion perturbations into Motion Deviations (by applying combination of rotation AR SO(3) and translation At E R perturbations) and Faster Motion Effect
(b) Perturbation on RGB sensor imaging. Imaging corruptions, such as motion blur and high illumination, are common in real-world data collection, affecting image quality. As shown in Fig. 3b, sixteen RGB imaging perturbations are modeled mimic error sources that arise throughout the entire imaging procedural-five 3D scene capture to final image output. These perturbations stem from environmental interference affecting light transmission, lens distortions leading to blurring, sensor-induced noise, and artifacts introduced during post-p processro...ng..
(c) Perturbation on depth sensor imaging. We observed a significant discrepancy in depth distri- bution between the simulated SLAM benchmark (Replica (Straub et al., 2019)) and the real- world dataset (TUM-RGBL (Schubert et al., 2018)), with TUM-RGBD missing 25% of depth data com- pared to just 0.39% in Replica. To address this, propose of depth perturbation operations (see Fig. 3c): Gaussian Noise to mimic depth noise; Edge Erosion and Random Missing Depth for missing data; Range Clipping for limited depth sensor perception range.
(d) Perturbation on multi-sensor synchronization. To emulate sensor delays when multiple sensors within RGB-D sensing system are well-synchronized (e.g., due to varied signal sampling frequency), we introduce temporal misalignment between multiple sensor streams.
Perturbation propagation and composition. For RGB-D sensing system, the specific composi- tion is an ordered transformation derived from the generic formulation (see Eq. 1):
(essyne Finaging Frender F Fmmstion) (m, xx E1:e);
Zi:t, X1:t
where Fmotion introduces motion perturbations, Frender generates clean observations via photorealistic rendering, Fimaging adds imaging perturbations, and Fdesyne simulates multi-sensor desyncronization.
Perturbation mode and severity. Perturbations are examined in two modes: static, with constant severity, and dynamic, with frame-to-frame variations. Severity levels are based on real-world data (e.g., real depth distribution) or physics- based models (Kong et al., 2023; Hendrycks & Dietterich, 2019), and can be adjusted to progressively evaluate model performance under varying difficulties.
3.3 SCALABLE NOISY DATA SYNTHESIS PIPELINE
We present a scalable noisy data synthesis pipeline integrating customizable RGB-D perturba- tions (see Fig. 4). The key innovation lies in its customizability and physically-based model- ing, enabling the generation of photorealistic noisy data with controllable perturbations. The initial phase involves configuring the sensor setup, original trajectory, and 3D scene. Next, the clean and stable trajectories are processed by the motion perturbation composer to introduce de- viations and faster motion effects. The render generates clean but unstable sensor data streams
4

---

Published as a conference paper at ICLR 2025
from the 3D scene mesh, perturbed tra- jectory, and sensor configurations. Sub- sequently, sensor-related perturbations, in- cluding imaging corruptions and multi- sensor desynchronization, are composed, resulting in noisy and unstable sensor streams. This noisy data, with perturbed sensor streams inputs and clean 3D and perturbed trajectories as ground truth, al- lows for perturbation-condiiioeee model performance evaluation.
(Clsun) Trajectory
Perturbed
Trajectory
Motion Perturb.
Clean, Linstalee
Noley, Unstable Senser Stream
Compoaet
Senser Stream
Sensor Configuration
Sensur Perturb.
Render
Composer
(Clean) Scete
Figure 4: Noisy data synthesis pipeline.
4 RETHINKING MODEL ROBUSTNESS UNDER PERTURBATIONS
Leveraging our noisy data synthesis pipeline, we instantiate the Robust-Ego3D benchmark, designed as the first compre- hensive benchmark that supports model evaluation under diverse noisy sensing con- ditions. Robust-Ego3D surpasses existing synthetic benchmarks in both diversity and scope, offering unprecedented granularity (see Table 1). It offers editable perturbation capabilities and covers 124 distinct RGB-D perturbation settings across 1,000 videos. This level of granularity in systematic evaluations. See Appendix
Table 1: Synthetic SLAM benchmark comparison.
Benchmark
#See
#Perturb Setting
Perturbation Category
Editable
RGB Motion Depth Sync.
Replica
Tartan/Air
30
8
RobusttEggo3D
1,000
perturbation settings is exceptional and opens the door Sec. B for detailed Robust- Ego31) benchmark setup, hyperparameters, and statistics.
Next, re-evaluate representative and top dense Neural SLAM models including iMAP (Sucar et al,, 10 Nice-SLAM (Zhu et al., 2022), CO-SLAM (Wang et al., 2023), GO-SLAM (Zhang et al., 2023), and SplaTAM (Keetha al., 2024)-under perturbations (detailed in Appendix Sec. B.3). While we focus on dense Neural SLAM methods which support photorealistic 3D reconstruction, we include ORB-SLAM3 (Campos al., 2021), a strong DRB-fcature-basee model, for reference (see Appendix Fig. D13). The primary evaluation metric is Absolute Trajectory Error (ATE) (Prokhorov al., 20,9), supplemented by PSNR, Depth I.1 Loss, and qualitative comparisons on 3D mesh and rendered RGB-D for textured 3D quality evaluation. To ensure robustness, each experiment was repeated three times across eight scenes, averaging results over 24 trials per perturbation.
4.1 BENCHMARKING UNDER INDIVIDUAL. PERTURBATIONS
Performance under RGB imaging corruptions (Table 2). 1) Clean vs. perturbed. The models evaluated generally show robustness to most RGB imaging corruptions. Even for iMAP, which uses a simple fully-connected network to implicitly encode the scene, the trajec- tory estimation performance under most perturbations ditions, though it still degrades. 2) Neural models' robustness partly stems from the lo- cal denoising property of neural representation. For example, iMAP, with NeRF-based rep- resentation, exhibits an intriguing local denoising property under Spatter Effect, producing RGB images and textured mesh nearly free of spatter noise (please see Sec. D.3 for details). 3) Comparison across perturbations categories. Different perturbations impact performance to varying degrees, with environmental effects posing the most significant challenge, followed by sensor noise, while image post-processing per- turbations like JPEG Compression have a minor influence. 4) Dynamic vs. static mode. Dynamic perturbations are more challenging than static ones, complicating model de- ployment in dynamic environments. Static perturbations, with consistent intensity across frames, simplify the opti-
comparable to that under clean con-
No Parturhatioo
Shot Parturbation
Trajectory
Pred. Trajectory
Figure 5: Effect of Shot Noise on 3D) mization of Neural methods, whereas dynamic perturba- reconstruction of Nice-SLAM.
tions require cross- dooss- pose and map optimization. 5) Effect of depth. By comparing GO-SLAM (Mono) and GO-SLAM (RGB-D), DD),D,, find that adding depth information increases robustness to RGB imaging perturbations. While the monocular version GO-SLAM exhibits slightly lower error in clean settings, the RGB-D version outperforms it under perturbed conditions, 6) 3D reconstruction quality.
5

---

Published as a conference paper at ICLR 2025
Table 2: Performance (ATE) (m)) under static (Top) and dynamic (Bottom) RGB imaging perturba- tions. The best- pestorerformig methods are highlighted with first second. and third
Method
Clon
Perturbed
Basinummentd Interfoncee
Poll-grccosing
Moon
May
[Mution
Defocts
Goanndon
Gibex
Shoe
Impaater
Speckle
Fog
Frai
Sam Spalter Bright
Controd
Ipeg
Piceber
(Mooo)
MAP RGB DJ
0.120
0.1565
0.3631
0.1424
0.1671
0.1811
0.0072
0.0278
0.0779
0.1710
0.1067
0.1913
D.3316
0.1665
D.1433
0,100
0,5831
0.1584
0.1600
Nip (XGES-12)
0.1477
(10253
0.0654
0.0057
0.0111
0.0161
0.015
010254
0,0827
0.0253
00151
0.0186
(0010))
00125
06330
0,0654
0.0161
00150
00145
CO (RGB-D)
0.000
0.025
100155
010356
010097
000001
0.0080
0010
00118
D011
0/0060
0.0003
01011
0.080
0/096
SpuTAM IRGB-D)
0.0100
(00160
0.0053
0.0949
0.0045
010050
0,0065
0.003
IMAP DJ
0.120
111755
0.2573
0011333
0
0.1042
0.3149
0.122
01351
0.117
0,1967
0.157
0.2279
0.3673
0,3412
0.344
0.2576
0.1607
0,9955
Nia AM (RGIT-I)
0.1177
01014
0.000
0.055
01253
0,0119
0.011
00255
010439
00007
0:0111
00197
0.00M
010155
00146
0017)
CO-SLAM
0.0077
0.0955
0.003
00116
0.0005
00065
00000
SpATAM-S (RGB-DD)
0.0400
(10151
(UU953
(10073
B4062
(MAP(RGBD)
CO-SLAM (RGBD)
SylaTAM-S (
CO SLAM
SPeTAM 5 S/ROBD
SLAM (RGBD)
GO-ST (RGRD)
NaasSLAN (RGRD) CO-SLAN (RGBD)
GD-SLAM (Maun))
Trantaiion
Gieercal
Obsarved Depth
Bandared Depth
GT
1.0
Severdl Deg
0.0
Pree: Faik
0.5
Conserge
GT&
ATB3
0.6
Estimated
0.4
0.2
0.0
Clean
Naise
Edge Eroeion
Masity
Dauge Clip
Speed up
Figure 6: Effect of depth perturbations.
Figure 7: Effect of fast motion.
Fig. 5 shows that trajectory estimation accuracy does not always correlate with 3D reconstruction quality. While Nice accurately reconstructs 3D geometry and trajectory under Shot Noise RGB perturbation, the noise severely degrades color reconstruction. This highlights the need for restoration mechanisms to ensure noise-tolerant and noise-free 3D reconstruction. Furthermore, Appendix Theorem A demonstrates that restoration mechanisms can reduce gradient variability from noisy inputs, supporting more robust optimization. 7) Learning-based vs. heuristic representation. Learning- based models show strong robustness to RGB imaging perturbations for pose tracking (Table 2). while ORB-SLAM3, relying on ORB features, suffers severe tracking loss.
Performance under depth missing or noises. Fig. 6 illustrates the effect of depth perturbations. 1) Depth missing effects. Dense Neural SLAM models exhibit minimal performance degradation when faced with partial depth missing. This robustness can be explained by Appendix Theorem B, which demonstrates that missing depth data does not introduce noise into the optimization process. By excluding missing depth from the gradient calculation, these models maintain stable pose updates, which prevents significant errors from propagating. 2) Depth noise effects. Noise introduced to depth significantly impacts all models, directly interfering with the depth loss term and disrupting optimization. Appendix Theorem C demonstrates that noisy depth leads to biased gradients, causing misaligned pose updates. This instability results in substantial performance degradation, evidenced by the considerable increase in ATE and the failure of depth reconstruction, as shown in the qualitative depth rendering results of SplaTAM- in Fig. 6. 3) RGB vs. RGB & noisy depth. GO-SLAM's ATE rises from 0.46 em to 3.78 em with depth noise, making RGBD-version GO-SLAM perform worse than its monocular counterpart. This highlights the need for selective disregard of noisy inputs.
Performance under fast motion. Fig. 7 shows significant performance degradation in most models under fast motion. 1) Robust Neural models. The Neural model GO-SLAM performs robustly under fast motion, thanks to global optimization techniques like bundle adjustment and loop closure. 2) Fragile Neural models. In contrast, other models like Nice-SLAM, CO-SLAM, and SplaTAM-SS which rely on RGB-D reconstruction loss for differentiable pose and map optimization, struggle, leading to failed pose optimization and degraded map reconstruction (see Fig. 7). As Appendix Theorem D demonstrates, large pose changes amplify gradients, causing inefficient optimization and potential divergence for differentiable pose optimization. 3) Neural vs. non-neural. Overall, pure NeRF-basee and Gaussian-Spp -Spltt based models evaluated show lower robustness to rapid motion changes compared to the non-neural model ORB-SLAME (whose results are in Appendix Fig. D13d), which leverages ORB descriptors for tracking, making it more resilient to such effects. Thus, to
6

---

Published as a conference paper at ICLR 2025
CO-SLAM
1,0
SLAM (RCIBO)
NO-SLAMGGRRD
0.8
Static
Dynamic
Rolotion Dedittinn (dlog)
Mode
Mode
0.-
0.5
on
-0.6
0.5
0.5
CO (K6BD
GO (B.60)
SpliTAM S (8083)
ATE
0
-0.4
(40)
0.3
1 6.45 AMM 83515 5.540
4.779 0.273 03111
0.2
0.2
-02
0403
8,532
6577)
NOW
1040
L000
18M
1.M
1.00 LONO
0.1
0.3:
100
1.WW
0.0
0.0
5-0
A-5
4-10
10
4-
20
5
5-0
A-5
--
10
20
0
1.25 2.5
5
1.25
2.5 5
1.25
2.5
5
RGB-D Desynchrorization Frame Interval (A)
Translation Deviation (cm)
Figure& 8: Effect of motion deviations.
Figure 9: Effect of RGB-D desynchronization.
improve robustness, combining the learnable components of Neural methods with ORB-SLAM3's resilience to fast motion could offer a more balanced approach.
Performance under unstable sensor motion with deviations. 1) Small deviations cause great challenge. As shown in Fig. 8, even minor pose deviations (e.g.. 2.5 cm in translation) significantly degrade trajectory estimation across most models. Combined translation and rotation deviations am- plify errors, leading to complete pose tracking failure. 2) Failure due to invalid linear motion assumptions under dynamic motion. By leveraging the assumption of smooth, linear sensor motion, models such SplaTAM-S and CO-SLAM perform well on standard datasets that exhibit this data bias. However, this assumption does not hold in cases with abrupt movements. Table 3 demonstrates significant performance drops when this assumption is violated. Increasing pose optimization iterations further degrades performance. highlighting the sensitivity of these methods to the linear motion assumption. Appendix Theorem E shows that assuming linear motion under linear conditions leads to misaligned gradients and convergence failures in differentiable pose optimization, as the models fail capture higher-order dynamics.
Table 3: Effect of linear motion assumption on
ATEL (cm)) of SplaTAM on original-speed (1x) clean videos. 'Iter': pose optimization iteration.
Assump Bler.
0-0
0-1
0-2
0-3
0-1
R-2
R-2
40
0.47
0.27
0.29
0.32
0.55
0.31
0.40
0.29
0.36
3.59
2.48
0.58
2.39
25.82
4.83
2000
16.09
0.62
15.09
0:40
17.21
4.27
65.05
4,87
15.30
Performance under poorly-synced RGB-D streams. Fig. 9 shows that while increased desyn- chronization leads to performance drops, the degradation less severe for methods like CO-SLAM,, SplaTAAM-S, and GO-SLAM. The relative performance ranking of top-performing methods remains consistent under both well-synced and desynchronized sensor settings, which suggests that enhancing performance in clean settings could potentially improve robustness under poorly-synced settings.
Remark. 1) No universally robust model. No single model handles all perturbations. The inconsis- tency between performance in clean and perturbed settings highlights the need for evaluating systems across diverse degraded conditions. 2) Synergy across methods. Neural models excel in pose tracking under RGB imaging perturbations, while -SBM with feature matching is more robust to fast motion. This inspired our CorrGS method, marrying correspondence-hann representation. 3) Caution with model and dataset assumptions. Assuming linear sensor motion can lead to overfitting to dataset biases, impairing the model's 's ability to generalize to diverse world scenarios. Our customizable noisy data generation pipeline helps reveal such model vulnerabilities by introducing varied perturbations, ensuring robustness beyond standard dataset biases.
matching with neural
4.2 FROM INDIVIDUAL TO MIXED PERTURBATIONS
We conduct case studies (see Table 4) on scene Office - 0 under static mode perturbations.
Mixed perturbations generally lead to de-
Table 4: Mixed perturbation effect (ATE) (m)).
graded performance, though not always. We
Metheel
Clean
find that some perturbation combinations like Snow Effect and Motion Blur (ii) amplify errors, leading to higher ATE for GO-SLAM. However, other combinations, like adding JPEG Compres-
GU-SLAM
0,004
0.095
01512
0.356
0.397
AATE
0.000
-0.90
+6.417
40. 108
40 0.766
+0041
+0.200
SpaTAMMS
0,005
0,00
0011
0,008
0,00
0211
0.200
SATE
0.000
+0.002
40.004
-0.003
00001
+0.264
40.058
Laft Right We progosiselly sld (0 RGB Sumw Affeez (i RGG Mation Hlar.
(60 RGB Ganrrion Nitir, (iv) MGB JPEG, & (v) Deysh Moise, RGB-D Dasjac:
sion (iv), produce an effect similar to the sintle-perturbation setting and do not degrade performance.
7

---

Published as a conference paper at ICLR 2025
Individual perturbations often yield similar performance degradation in isolated and mixed settings. Introducing additional Depth Noise (v) significantly impacts SplaTAM-S, increasing ATE by 0.204 m and degrading rendered RGB quality (see Fig. 10). Likewise, coupled RGB corruptions (ii) strongly affect GO-SLAM. This suggests ana- lyzing individual perturbations can expose model vulnerabilities in multi multi-perturrbation scenarios.
Renderal Observei
(iv) Mixed RGB
(v) Mixed RGB-D
Figure 10: Rendering under mixed perturbations (same as Table 4) for SplaTAM-S.
4.3 TOWARDS STREAMLINED BENCHMARKING VIA PERTURBATION PROXIES
Optimizing robustness benchmarking through correlation analysis. Evaluating model robustness
under large-scale perturbations can be desource-intensive requiring numerous runs across varying perturbation types, severity levels, scenes, and models. However, by under- standing correlations between different perturbations, we can streamline the model evaluation process.
(1)
Tmaging Porturbations:
(2)
(1) Metion Blur, (2) Deforus Blur, (3) Gaustan Blar,
(4) Glase Blat, (5) Geussion Noise, (6) Shot Noiss, teppliin Noise, (8i Specite, (9) Fag.
(3)
(10) Fooet, Snow. (12) Spellee
[5)
Bright (14) Contrott
EM EEB oze
LiW
(13) JF2G, (16) itselltt.
(7)
Streamlined benchmarking using representative pertur-
(0)
[.0
bations. We use Spearman's 's rank correlation (Spearman, 1961) to identify perturbations that closely mirror overall performance degradation, allowing efficient evaluation by focusing on representative perturbations. Fig. 11 shows that perturbations like Gaussian Noise, Shot Noise, Impulse Noise, and Spatter have high correlations with others, mak- ing them effective proxies for broader testing. While they may not capture every specific effect, they provide a rough estimate of performance under degraded RGB.
(0)
100 o
1:94
0.0
(10)
1:54
LAN
(22)
0.5
8 c:4 026
3:5
0.1
LE
(10)
(15)
E56
L1W
8:4 050
409
8:
154
8:34
(3)
121
(1)
(4)
(5)
16
07
(6)
(90
301
33X
Figure 11: Correlation of performance
(ATE) across static RGB perturbations.
CORRGS: ROBUST LEARNING UNDER NOISY VIDEO
Our benchmarking reveals that existing methods struggle with large motions and suffer color degrada- tion in 3D reconstructions under RGB imaging perturbations. These insights motivate us to propose a more robust model, i.e., Correspondence- -ugiddd Gaussian Splatting (CorrGS). to achieve reliable ego-motion estimation and noise-free dense 3D reconstruction, even from noisy, fast-motion videos.
5.1 METHOD
Overview. CorrGS employs Gaussian Splatting combined with perrespondence-guideer differentiable optimization deliver robust ego-motion estimation and photorealistic 3D reconstruction. Using the Gaussian- map representation maintained to be free, CorrGS renders RGB-D images at estimated poses. then conducts differentiable optimization by calculating the differences between these rendered images and the observed noisy RGB-D frames. The integration of correspondence guidance pivotal, as enhances geometric alignment for accurate ego-motion estimation and facilitates appearance restoration for reconstructing noise-free, photorealistic 3D. Please see Appendix Sec. G for details on CorrGS, including schematic pipeline (Fig. G31) and pseudocode (Alg. 1).
Pose Learning (CPL) uses correspondeecce-iitttaiii the initialization for differentiable pose parameter learning, enabling robust ego-motion estimation. E {1..... .N. by lifting 2D visual
Correspondence-guidee
poses to improve
1) We first calculate N 3D correspondences. indexed matches between rendered and observed RGB frames into 3D using the rendered and observed depth, respectively. This process establishes correspondences between observed noisy 3D points po. and rendered 31) points pr. 2) Then, we calculate the relative pose Prat (Ryat= tret) } between rendered and observed points via Eq. 4. 3) The pose initialization via correspondence obtained by multiplying the previous pose estimate with the relative pose Pret-
N Σn (IRnaaPrss + tre! - po.il
Pied = arg min R....
where || denotes the Euclidean norm, and p(-) is the soft L1 loss function. The computational complexity of this optimization process is bounded by the image resolution, enabling low time cost.
8

---

Published as a conference paper at ICLR 2025
Table 5: Comparison (ATE)) (em)) on sparse- video.
CPL
w CPL
Method
LC|
C-0
0-1
0-2
0-3
0-6
R-0
R--
B-2
GLAL
[V]
20002
0.44
Fall
0.40
0.67
0.71
0.50
0.43
Paal
CD-SLAM
99.38
60.77
176.1
162.02
223.55
170.5
140.4
15
146.45
SplTAA
115.55
76.55
196.95
126.66
150.66
1006
CoreGS Defanl
1,50
2,11
1.96
0.99
1.36
0.84
17.73
1.72
3.60
CD-SLAM-L
55.52
6.78
3.80
10.37
11.
104.03
3.91
25.08
Hen-SLAM-L
0.77
1.10
1.97
1.33
K2.3
22.5
1.83
0.90
14.08
SplaTAM-L
1.59
0.39
1.52
0.93
81.
1.73
CoeeGS-L (Defaultt
0.39
0.33
00.22
0.33
11.49
0.76
0.59
0.71
0.46
Figure 12: Ablation of CPL on 3D re-
Cancis-4, (odl PQV)
1.50
0.83
0.43
0.20
0.45
1:06
10,
664
1.70
11:11
construction from sparse-view video.
DC (Cindicaees using koop
"L" indicales using the posa
(200)
4) However, the initialization from Eq. 4 may fail when the two viewpoints are very close together. In such scenarios, it is challenging to accurately estimate the relative pose by optimizing the transforma- tion between the point clouds of the two viewpoints, leading to higher RGB-D rendering error. To mitigate this, we propose a Pose Quality Verification step that rejects the initialization from Eq. 4 if it results in higher rendering loss than a naive pose propagated from previous frames. 5) The final pose is further refined using the differentiable rendering- based optimization.
perreppondene-guuddee Appearance Restoration Learning (CARL). 1) CARL mitigates color degradation by learning a restoration model f(; 0) that maps noisy colors C.,; of observed points to their clean counterparts Cr rendered from historical map which is maintained to be noise-free:
0"
arg min
Σ (I((C.,;; 1
2) The learnt model is then applied to the observed image. It ensures 3D consistency and enhances pose tracking via a second round of CPL with the restored image.
Theoretical insights the benefits of CorrGS. 1) CPL enhances pose learning by generating more accurate pose estimates that are closer to the ground-truth. This accuracy reduces the magnitude of gradients and improves the conditioning of the Hessian matrix, thereby facilitating more stable and robust tracking. 2) CARL employs learned restoration mechanism to denoise perturbed color inputs, effectively mitigating noise- induced residual variability, which in turn stabilizes gradients and accelerates optimization convergence. See Appendix Sec. I for more details.
5.2 PILOT STUDY ON SYNTHETIC NOISY SPARSE-VIEW VIDEO
Metries. Performance quantified using 1) ATE for ego-motion accuracy, 2) PSNR for simultaneous color restoration and reconstruction quality, and 3) depth LI loss for depth reconstruction accuracy.
Noisy data synthesis setup. We focus on illumination changes, leaving other perturbations for future work. 1) To evaluate CPL's robustness under fast motion, we synthesize sparse-view videos (10x faster than (Straub et al., 2019)). 2) To test CARL, we create sequences with partial brightness reduction, where the first half of each video is clean and the latter half has lower brightness.
Implementation. 1) CPL uses the RGB-based matcher from (Sun et al., 2021) to produce 2D correspondence, which is lift to 3I) correspondence via depth. 2) CARL uses linear model for restoration, optimized using the Adam optimizer over 100 iterations with a learning rate of 0.2. 3) CPL and CARL each add a total of 0.1 seconds (for 1200x680 input images), resulting in minimal overhead while enhancing robustness. 4) Hyperparameters follow the baseline (Keetha et al., 2024).
Effect of CPL: Accelerated convergence and enhanced robustness under dynamic motion. 1) As shown in Table 5, our ConGG and CorrGS-L Connsssty achieve the lowest ATE across all sequences, outperforming top pure Neural models like SplaTAM, SplaTAM-L. and Hero-SLAM (Xin al., 2024), which target extreme motion scenarios, while also demonstrating greater robustness compared to GO-SLAM, which utilizes an additional loop closure mechanism. 2) CPL leads to notable acceleration in pose learning convergence (see Appendix Fig. H37), significantly improving trajectory estimation under fast motion. 3) Pose Quality Verification (PQV) step safeguards against potential failures in CPL, as seen in the performance drop without PQV (CorrGS-L w/o PQV) in Table 5. 4) Fig. 12 shows that CPL enables robust tracking, resulting in high-quality 3D reconstruction, whereas the baseline fails. 5) Our coupled data-model approach- uusng senchmark-identiinn to guide model refinements advances robustness. Specifically, motion perturbation benchmarks exposed pose tracking issues, leading to the development of CPL. to address these limitations.
weaknesses

---

Published as a conference paper at ICLR 2025
Table 6: Comparisons on noisy sparse-view video. X:
w/ CARL--M
W CARL-T
w CARL-T&M
tracking failure. CARL-T&M is default ConGGS-L model.
Offico4
Fallure to Reconstruct
Setting
0-0
O-2
D-2
D-3
C-4
B-0
B-1
B-2
Arg
Trajectory Extimation Erroe (ATE]
Baeline
x
0.51
CARL
3.44
0.65
0.96
0443
CARL-M
5.35
Tuxnno
CARL--T&M (Delault)
0.30
0.12
0.23
0.31
0.41
0.35
1.08
1.55
0.54
RGll Resteration Quality (PSNR" (dD))
Beuilin:
12.67
CARL-T
12:53
CARL M
24.53
Failure to Reconstract
(Dedault)
Depth
Qualitt (Dpth (cm])
CARL-T
1.80
1010
146
0.76
Figure 13: Effect of CARL on 3D recon- struction from sparse-view noisy video.
CARL M
X
2.66
CARI-T&M (Badanlt)
0.25
0.19
0.55
0.81
0.52
0.58
0.52
0.63
Effect of CARL: Online restoration learning enables consistent noise-free 3D reconstruction. Table 6 quantitatively evaluates CARL's effect on ego-motion estimation, RGB restoration, and depth rendering quality under partial degraded illumination, with qualitative 3D reconstruction results in Fig. 13. 1) CARL significantly improves pose accuracy and enables noise-frree 3D reconstruction from noisy video, increasing tracking success from 1/8 (baseline) to 8/8, with an ATE of 0.54 cm, close to the noise-free value of 0.45 cm. 2) Omitting the restored image in mapping (CARL-T) reduces tracking success from 8/8 4/8 and results in inconsistent maps with regional severe artifacts. 3) Using CARL solely for mapping (CARL-M) improves RGB quality compared to the baseline (24.53 dB vs. 12.67 dB in PSNR on C-1), but loses track due to pose optimization relying on unrestored RGB-D. 4) Thus, the full CARL method (CARL-T&M) is essential for robust pose tracking and noise-free 3D reconstruction, achieving a high average restored RGB PSNR of 35.48 dB.
5.3 PILOT STUDY ON REAL-WORLD NOISY SPARSE- VIEW VIDEO
To demonstrate CorrGS's practical applicability, we conducted a pilot study using noisy, sparse- video captured under real-world conditions, including extreme lighting variations (e.g., darkness Noise-free Reconstruction
to overexposure), fast motion, and depth noise, recorded with RealSense D435i sensor. Fig. 14 shows that CorrGS effec- tively handles sensing degradation from dynamic illumination and fast motion, ren- dering noise-free RGB and reconstructing a clean, photorealistic 3D map from noisy RGB-D. Noisy depth in some frames (see Appendix Fig. H44) suggests an oppor- tunity enhance the model with depth restoration alongside color restoration.
Light
Rondeeet Olterved Hastard
T:#6 (Dark)
T-77 (Uvestsposir))
Figure 14: RGB-D rendering and 3D) reconstruction by CorrGS on real, fast video with dynamic illumination.
CONCLUSION AND FUTURE WORK
Conclusion. We presented structured approach to tackle robust ego-motion estimation and photo- realistic 3D reconstruction from noisy video. Our contributions are threefold: 1) We developed scalable noisy data synthesis pipeline incorporating comprehensive RGB-D perturbations for mobile agents, enabling realistic and diverse noisy datasets. 2) We introduced the Robust-Ego311 benchmark to systematically expose performance bottlenecks and guide targeted improvements. 3) Building on insights from our benchmark analyses, we proposed CorrGS, which significantly enhances robustness ego-motion estimation and 3D reconstruction from noisy, sparse-view video. This work sets a new standard for assessing and advancing the robustness of dense Neural SLAM.
Future work. We hope to inspire exploration to embrace imperfect observations and turn them into clarity in 3D and ego-motion, the road ahead offers opportunities: 1) refining the approach to adaptively select perturbations based on real-world conditions, ensuring efficiency and relevance; 2) leveraging generative models to produce richer, more diverse perturbations that better reflect real- world challenges; 3) expanding Robust- Ego3D to encompass outdoor and cross- environments, broadening the scope of model evaluation; and 4) optimizing CorrGS by implementing adaptive processing for key frames, further enhancing its performance and efficiency.
10

---

Published as a conference paper at ICLR 2025
REFERENCES
Cameron A Aubin, Ronald H Heisser, Ofek Peretz, Julia Timko, Jacqueline Lo, Farrell Helbling. Sadaf Sobhani, Amir D Gat, and Robert F Shepherd. Powerful, soft combustion actuators for insect-scale robots. Science, (6666)):2112-1111 2023.
Mihai Bujanca, Xuesong Shi, Matthew Spear. Pengpeng Zhao, Barry Lennox, and Mikel Luján. Robust slam systems: Are we there yet? In 2021 IEEF/RS/ International Conference on Intelligent Robots and Systems (IROS), pp. 5320-5327. IEEE, 2021.
Michael Burri, Janosch Nikolic, Pascal Gohl, Thomas Schneider, Joern Rehder, Sammy Omari, Markus W Achtelik, and Roland Siegwart. The EuRoC micro aerial vehicle datasets. The International Journal of Robotics Research, 2016.
Cesar Cadena, Luca Carlone, Henry Carrillo, Yasir Latif, Davide Scaramuzza, José Neira, lan Reid, and John J. Leonard. Past, present, and future simultancous localization and mapping: Toward 2016.
the robust-perceptin age. IEEE Transactions on Robotics,
Carlos Campos, Richard Elvira, Juan J. Gómez Rodríguez, José M. M. Montiel, and Juan D. Tardós. Orb-slam3: An accurate open-source library for visual, visual- ineetial, and multimap slam. IEEE Transactions Robolics, (7(6): 1890 2021.
Angel X Chang, Thomas Funkhouser, Leonidas Guibas, Pat Hanrahan, Qixing Huang, Zimo Li, Silvio Savarese, Manolis Savva, Shuran Song, Hao Su, al. Shapenet: An information-rich 3d model repository. arXiv preprint arXiv: 2015.
Prithvijit Chattopadhyay, Judy Hoffman, Roozbeh Mottaghi, and Aniruddha Kembhavi. Robustnav: Towards benchmarking robustness in embodied navigation. In IEEE/CVF International Conference on Computer Vision, pp. 1-5000,,,,,, 2021.
Israel Cohen, Yiteng Huang, Jingdong Chen, Jacob Benesty, Jacob Benesty, Jingdong Chen, Yiteng Huang, and Israel Cohen. Pearson correlation coefficient. Noise reduction in speech processing.
pp. 1-4, 2009.
Angela Dai, Angel X Chang, Manolis Savva, Maciej Halber, Thomas Funkhouser, and Matthias Nießner. Scannet: Richly-annotated 3d reconstructions of indoor scenes. In IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 5828-5822 2017a.
Angela Dai, Matthias Nießner, Michael Zollhöfer, Shahram Izadi, and Christian Theobalt. Bundlefu- sion: Real-time globally consistent 3d reconstruction using on-the-fly surface reintegration. ACM Trans. Graph., 36(4), jul 2017b.
Alexey Dosovitskiy, German Ros, Felipe Codevilla, Antonio Lopez, and Vladlen Koltun. Carla: An open urban driving simulator. In Conference on robot learning, pp. 1-16. PMLR, 2017.
Kamak Ebadi, Lukas Bernreiter, Harel Biggie, Gavin Catt, Yun Chang, Arghya Chatterjee, Christo- pher E. Denniston, Simon-Pierre Deschénes, Kyle Harlow, Shehryar Khattak, Lucas Nogueira, Matteo Palieri, Pavel Petráček, Matëj Petrlík, Andrzej Reinke, Vít Krátky, Shibo Zhao, Alí-akbar mogh-mohhammaii, Kostas Alexis, Christoffer Heckman, Kasra Khosoussi, Navinda Kottege, Ben- jamin Morrell, Marco Hutter, Fred Pauling, François Pomerleau, Martin Saska, Sebastian Scherer, Roland Siegwart, Jason L. Williams, and Luca Carlone. Present and future of slam in extreme environments:
The darpa subt challenge. IEEE Transactions on Robotics, pp. 1-20, 2023.
Siyuan Feng, Eric Whitman, X Xinjilefu, and Christopher G Atkeson. Optimization based full body control for the atlas robot. 2014 IEEE-RAS International Conference on Humanoid Robots, pp. 120-127. IEEE, 2014.
Timnit Gebru, Jamie Morgenstern, Briana Vecchione, Jennifer Wortman Vaughan, Hanna Wallach, Hal Daumé III, and Kate Crawford. Datasheets for datasets. Commun, ACM, 64(12):86-9.. nov 2021.
Ruben Grandia, Espen Knoop, Michael A. Hopkins, Georg Wiedebach, Jared Bishop, Steven Pickles, David Muller, and Moritz Bacher. Design and control of bipedal robotic character. In Robotics: Science and Systems 2024, Delft, Netherlands, July 15-July 19 2024. Disney Research, Switzerland and USA; Walt Disney Imagineering R&D, USA.
11

---

Published as a conference paper at ICLR 2025
Samuel W Hasinoff. Photon, poisson noise. Computer Vision, A Reference Guide, (016);1,0044
Michael Helmberger, Kristian Morin, Beda Berner, Nitish Kumar, Giovanni Cioffi, and Davide Scaramuzza, The hilti slam challenge dataset. IEEE Robotics and Automation Letters, 7(3):
7518-7525, 2022.
Dan Hendrycks and Thomas Dietterich. Benchmarking neural network robustness to common corrup- tions and perturbations. Proceedings of the International Conference on Learning Representations. 2019.
Xiaonan Huang. William R Johnson, Joran Booth, and Rebecca Kramer- -oanner--tttilloo.. Live demonstration: Tensegrity state estimation. In 2022 IEEE Sensors, pp. 1-1. IEEE, 2022.
Mohammad Mahdi Johari, Camilla Carta, and François Fleuret. Eslam: Efficient dense slam system based on hybrid representation of signed distance fields. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 17408-17419, 2023.
Christoph Kamann and Carsten Rother. Benchmarking the robustness of semantic segmentation models. In Proceedings of the IEEE/CVE Conference on Computer Vision and Puttern Recognition, pp. 8828-8838, 2020.
Elia Kaufmann, Leonard Bauersfeld, Antonio Loquercio, Matthias Müller, Vladlen Koltun, and Davide Scaramuzza. Champion-level drone racing using deep reinforcement learning. Nature, 620 (7976):982-988 2023.
Nikhil Keetha, Jay Karhade, Krishna Murthy Jatavallabhula, Gengshan Yang, Sebastian Scherer, Deva Ramanan, and Jonathon Luiten. Splatam: Splat, track map 3d gaussians for dense rgb-d slam. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2024.
Bernhard Kerbl, Georgios Kopanas, Thomas Leimkühler, and George Drettakis. 3d gaussian splatting for real-time radiance field rendering. ACM Transactions Graphics, 42(4), 2023.
Tae Kyun Kim. T test parametric statistic. Korean journal of anesthesiology, 68(6))540-546 2015.
Victor Klemm, Alessandro Morra, Ciro Salzmann, Florian Tschopp, Karen Bodie, Lionel Gulich, Nicola Küng, Dominik Mannhart, Corentin Plister, Marcus Vierneisel, et al. Ascento: A two- wheeled jumping robot. In 2019 International Conference on Robotics and Automation (ICRA), pp. 7515-7521. IEEE, 2019.
Lingdong Kong, Youquan Liu, Xin Li, Runnan Chen, Wenwei Zhang, Jiawei Ren, Liang Pan, Kai Chen, and Ziwei Liu. Robo3d: Towards robust and reliable 3d perception against corruptions. In Proceedings of the JEEE/CVF International Conference on Computer Vision, pp. 19994-20006, 2023.
Xiang Li, Jinglu Wang, Xiaohao Xu, Xiao Li, Bhiksha Raj. and Yan Lu. Robust referring video object segmentation with cyclic structural consensus. In Proceedings of the IEEE/CVF International Conference Computer Vision, pp.
2023.
Hidenobu Matsuki, Riku Murai, Paul HJ Kelly, and Andrew J Davison. Gaussian splatting slam. In Proceedings of the IEEE/CVE Conference on Computer Vision and Puttern Recognition, pp. 18039-18048, 2024.
Claudio Michaelis, Benjamin Mitzkus, Robert Geirhos, Evgenia Rusak, Oliver Bringmann, Alexan- der Ecker, Matthias Bethge, and Wieland Brendel. Benchmarking robustness in object detection: Autonomous driving when winter is coming. arXiv preprint arXiv: 2019.
Raúl Mur-Artal and Juan D. Tardós. Orb-slam2: An open-source slam system for monocular, stereo, and rgb-d cameras. IEEE Transactions on Robotics, 3355): 2017.
Richard A. Newcombe, Shahram Izadi, Otmar Hilliges, David Molyneaux, David Kim, Andrew J. Davison, Pushmeet Kohli, Jamie Shotton, Steve Hodges, and Andrew W. Fitzgibbon. Kinectfusion: Real-time dense surface mapping and tracking. In 10th IEEE International Symposium on Mixed and Augmented Reality, ISMAR 2011, Basel, Switzerland, October 26-29, 2011, pp. 127-136. IEEE Computer Society, 201 1.
12

---

Published as a conference paper at ICLR 2025
Dinesh K Patel, Xiaonan Huang, Yichi Luo, Mrunmayi Mungekar, M Khalid Jawed, Lining Yao, and Carmel Majidi. Highly dynamic bistable soft actuator for reconfigurable multimodal soft robots (adv. mater. technol. 2/2023). Advanced Materials Technologies, 8(2):2370066 2023a.
Manthan Patel, Marco Karrer, Philipp Bänninger, and Margarita Chli. Covins-g: A generic back-end for collaborative risual-inertial slam. In 2023 IEEE International Conference on Robotics and Automation (ICRA), pp. 2086-2082, 2023b.
David Prokhorov, Dmitry Zhukov, Olga Barinova, Konushin Anton, and Anna Vorontsova. Mcasuring robustness of visual slam. In 2019 16th International conference machine vision applications (MVA), pp. 1-6. IEEE, 2019.
Antoni Rosinol, John J Leonard, and Luca Carlone. Nerf-slam: Real-time dense monocular slam with neural radiance fields. arXiv preprint arXiv:2210. 11222100166,,, 13641, 2022.
Andrew Sabelhaus, Jonathan Bruce, Ken Caluwaerts, Pavlo Manovi, Roya Fallah Firoozi, Sarah Dobi, Alice M Agogino, and Vytas SunSpiral. System design and locomotion of superball, an untethered tensegrity robot. In 2015 IEEE international conference on robotics and automation (ICRA), pp. 2867-2873, IEEE, 2015.
Erik Sandström, Yue Li, Luc Van Gool, and Martin R. Oswald. Point-slam: Dense neural point cloud-based slam. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), 2023.
Thomas Schöps, Torsten Sattler, and Marc Pollefeys. BAD SLAM:: bundle adjusted direct RGB-D SLAM. In IEEE Conference Computer Vision and Pattern Recognition, CVPR 2019, Long Beach, CA, USA, June 16-20, 2019, pp. 134-144, Computer Vision Foundation IEEE, 2019.
David Schubert, Thore Goll, Nikolaus Demmel, Vladyslav Usenko, Jörg Stückler, and Daniel Cremers. The tum vi benchmark for evaluating visual-inertial odometry. In 2018 IEEE/RSJ International Conference on Intelligent Robots and Systems, pp. 1680-1687, 2018
Charles Spearman. The proof and measurement of association between two things. 1961.
Julian Straub, Thomas Whelan, Lingni Ma, Yufan Chen, Erik Wijmans, Simon Green, Jakob J Engel, Raul Mur-Artal, Carl Ren, Shobhit Verma, al. The replica dataset: digital replica of indoor spaces. arXiv preprint
2019.
Jürgen Sturm, Nikolas Engelhard, Felix Endres, Wolfram Burgard, and Daniel Cremers. benchmark for the evaluation of rgb-d slam systems. In 2012 IEEE/RSJ International Conference on Intelligent Robots and Systems, pp. 573-580, 2012.
Edgar Sucar, Shikum Liu, Joseph Ortiz, and Andrew J Davison. imap: Implicit mapping and positioning in real-time. In Proceedingy of the IEEF/CVF International Conference on Computer Vision, pp.
2021.
Jiaming Sun, Zehong Shen, Yuang Wang, Hujun Bao, and Xiaowei Zhou. Loftr: Detector-free local feature matching with transformers. In Proceedings of the IEEF/CVE conference on computer vision and pattern recognition, pp. 8922-899 29318 2021.
Zachary Teed and Jia Deng. Droid-slam: Deep visual slam for monocular, stereo, and rgb-d cameras. Advances in neural information processing systems, 01111658-166666 2021.
Yulun Tian, Yun Chang, Long Quang, Arthur Schang, Carlos Nieto-Granda, Jonathan P How, and Luca Carlone. Resilient and distributed multi-robot visual slam: Datasets, experiments, and lessons learned. In 2023 JEEE/RS/ International Conference on Intelligent Robots and Systems (IROS), pp. 11027-11034, IEEE, 2023.
Fabio Tosi, Youmin Zhang, Ziren Gong, Erik Sandström, Stefano Mattoccia, Martin R Oswald, and Matteo Poggi. How nerfs and 3d gaussian splatting are reshaping slam: a survey. arXiv preprint arXiv:2402. 13255, 4, 2024.
13

---

Published as a conference paper at ICLR 2025
Pauli Virtanen, Ralf Gommers, Travis E Oliphant, Matt Haberland, Tyler Reddy, David Coumapeau, Evgeni Burovski, Pearu Peterson, Warren Weckesser, Jonathan Bright, al. Scipy 1.:: fundamental algorithms for scientific computing in python. Nature methods, 27(3):211-27,, 2020.
Alexander Von Bernuth, Georg Volk, and Oliver Bringmann. Simulating photo-realistic snow and fog on existing images for enhanced can training and evaluation. In 2019 IEEE Intelligent Transportation Systems Conference (ITSC), pp. 41 41-46. IEEE, 2019.
Hengyi Wang, Jingwen Wang, and Lourdes Agapito. Co-slam: Joint coordinate and sparse parametric encodings for neural real-time slam. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pp. 13293-13302, 2023.
Wenshan Wang, Delong Zhu, Xiangwei Wang, Yaoyu Hu, Yuheng Qiu, Chen Wang, Yafei Hu, Ashish Kapoor, and Sebastian Scherer. Tartanair: A dataset to push the limits of visual slam. In 2020 JEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pp. 4909-4916. IEEE, 2020.
Zhe Xin, Yufeng Yue, Liangjun Zhang, and Chenming Wu. Hero-slam: Hybrid enhanced robust optimization of neural slam. In 2024 IEEE International Conference on Robotics and Automation (ICRA), pp. 8610-8616. IEEE, 2024.
Xiaohao Xu, Jinglu Wang, Xiang Ming, and Yan Lu. Towards robust video object segmentation with adaptive object calibration. In Proceedings of the 30th ACM International Conference on Multimedia, pp. 2709-2718 2718, 2022.
Xingrui Yang. Hai Li, Hongjia Zhai, Yuhang Ming, Yuqian Liu, and Guofeng Zhang. Vox-fusion: Dense tracking and mapping with voxel-based neural implicit representation. In Henry B. L. Duh, Ian Williams, Jens Grubert, J. Adam Jones, and Jianmin Zheng (eds.), IEEE International Symposium on Mixed and Augmented Reality, ISMAR 2022, Singapore, October 17-21, 2022, pp- 499-507. IEEE, 2022.
Justin K. Yim and Ronald S Fearing. Precision jumping limits from flight-phase control in salto- 1p. In 2018 IEEE/RSJ international conference on intelligent robots and systems (IROS), pp. 2229-2236. IEEE, 2018.
Naoki Yokoyama, Qian Luo, Dhruv Batra, and Sehoon Ha. Benchmarking augmentation methods for learning robust navigation agents: the winning entry of the 2021 igibson challenge. In 2022 JEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pp. 1748-1755. IEEE, 2022.
Youmin Zhang, Fabio Tosi, Stefano Mattoccia, and Matteo Poggi. Go-slam: Global optimization for consistent 3d instant reconstruction. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), October 2023.
Shibo Zhao, Yuanjun Gao, Tianhao Wu, Damanpreet Singh, Rushan Jiang, Haoxiang Sun, Mansi Sarawata, Warren C Whittaker, Ian Higgins, Shaoshu Su, Yi Du, Can Xu, John Keller, Jay Karhade, Lucas Nogueira, Sourojit Saha, Yuheng Qiu, Ji Zhang, Wenshan Wang, Chen Wang, and Sebastian Scherer. SubT-MRS dataset: Pushing slam towards all-weather environments. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2024.
Zihan Zhu, Songyou Peng, Viktor Larsson, Weiwei Xu, Hujun Bao, Zhaopeng Cui, Martin Oswald, and Marc Pollefeys. Nice-slam: Neural implicit scalable encoding for slam. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 12786-12796, 2022.
14

---

Published as a conference paper at ICLR 2025
TABLE OF CONTENTS OF THE APPENDIX
A. More Details about Perturbation Taxonomy
17
A.1 Perturbation on Sensor Poses
18
A.2 Perturbation on RGB Sensor Imaging
21
A.3 Perturbation on Depth Sensor Imaging
25
A.4 Perturbation Multi-sensor Synchronization
26
B. More Details about Robust-Ego3D Benchmark
27
B.1 Assumptions for Benchmarking Setup
27
B.2 Robust-Ego3D Benchmark Setup Details and Statistics
28
B.3 More Details about Baseline Models for Benchmarking
30
B.4 Details about Hardware Setup for Benchmarking Experiments
31
B.5 Uniqueness and Advantages of Our Robust-Ego3D Benchmark
31
Datasheet for Robust-Ego3D Benchmark
33
C.1 Motivation
33
C.2 Composition
33
C.3 Collection Process
C.4 Preprocessing, Cleaning, and/or Labeling
C.5 Uses
36
C.6 Distribution and License
.37
C.7 Maintenance
37
D. Additional Results and Analyses on Robust-Ego3D Benchmark
39
D.1 More Benchmarking Analyses
39
D.2 Classical SLAM ORB-SLAM3 Results (for Reference)
43
D.2 More Qualitative Analyses of Model under Perturbations
44
E. Theoretical Insights on Robust-Ego3D Benchmarking Results
57
E.1 Potential Benefits of Image Restoration on Optimization
57
E.2 Effect of Missing and Noisy Depth Data
57
E.3 Effect of Large Pose Changes Optimization
58
E,4 Invalid Linear Motion Assumption under Complex Motion
58
F. Detailed Tables of Results Robust-Ego3D Benchmark
60
G. More Details about Correspondence-guidee Gaussian Splatting (CorrGS)
65
G.1 Schematic Pipeline Overview and Pseudocode
65
G.2 Preliminaries
67
G.3 Correspondence-guidee Pose Learning (CPL)
68
G.4 Correspondence-guidee Appearance Restoration Learning (CARL)
73
G.5 Reactive Strategy for Handling Correspondence Calculation Failures
74
More Qualitative Results on CorrGS
75
H.1 More Qualitative Results on Synthetic (Clean) Sparse-view Video
75
H.2 More Qualitative Results on Synthetic Noisy Sparse-view Video
77
H.3 More Qualitative Results on Real-world Noisy Sparse-view Video
79
H.4 Robustness of CorrGS to Pose Initialization Errors
81
H.S Additional CorrGS Results in Outdoor Settings
83
I. Theoretical Insights on CorrGS
85
L.1 Pose Optimization in Gaussian Splat Representation
85
L.2 Impact of Fast Motion on Gradients and Hessian
85
I.3 Inefficiency and Potential Failure in Pure Differentiable Pose Optimization
86
I.4 Effect of Currespondenee-Guüdee Pose Learning
87
15

---

Published as a conference paper at ICLR 2025
1,5 Effect of Correspondence-Guidee Appearance Restoration Learning
87
1.6 Combined Impact of CPL and CARL
88
J. Other Potential Directions to Explore
89
J.1 Robustness Evaluation for More Diverse SLAM Systems
89
J.2 Additional Future Directions
90
K. Broader Impact
91
L. Social Impact
91
M. Availability and Maintenance
92
N. License
O. Public Resources Used
92
16

---

Published as a conference paper at ICLR 2025
A MORE DETAILS ABOUT PERTURBATION TAXONOMY
Cleen Back Seesor
Claan Senecr Street
Spotter Effect
Mitton Dler
Shot Nouse
Contrest
Trasiatioo Devaatio
Fog Lifect
Olr
Ingulao
Urgihtas
Sensee Stoare
Effect
Gremden
Speckle Nosin
Pitelin
tel
Observation
Pizel
Paster Niobon
Pattubeed Senser Straes
Olocred
Envivvamental Interforence
Disturtiens expeciially lllri
Post: procestng
(a) Sensor Poses
(bb Imaging of RGB Sensor
Source and Propagation Order of Perturbations for RG Sensing
(c) Imaging of Depth Sensor
(d) Sensor Synchronisation
Class Depti
RG3 Image lhe formmmemooo
L-3
RG3
Suesur
Synchronize
Gaussio Nalse
Random Mistin
Dept
Edde Emneson
O
Bange Clippina
RGS Secwor
Misteggh
beyth
achronicdd
Depth
map.
Dick riginns mededd nage
Scnsor
H+]
Figure A1: Taxonomy of perturbations for RGB-D sensing. The sources of perturbations include: (a) sensor pose errors, (b) RGB and (c) depth imaging corruptions, and (d) RGB-D sensor synchronization errors. Dashed arrows illustrate the propagation order of individual perturbations.
As shown in Fig. A1, perturbations affecting RGB-D sensing systems for mobile agents orig- inate from sensor pose deviations, inaccuracies within the RGB-D imaging processes, and de- synchronization issues between RGB and depth sensors. Within RGB-D sensing system, the order (dashed arrows of Fig. A1) in which perturbations occur and interact follows the sensing and processing procedure.
17

---

Published as a conference paper at ICLR 2025
A.1 PERTURBATION ON SENSOR POSES
(a) Robotswith Motion
(b) Catogories Motion-relatod Parturrations
Translation Deviation (from Low stiffness Structur)
Disney Bipedal Robot (Versatile Motion)
Micro-soft Crawling Robot
als
SuperBall (Rolling Motion)
Tensegrity Robot
Rotation Deviation (from Rolling Locomotion)
(Versatile Motion)
o 0.24
Zom
Ascento 2 (Hybrid Motion)
Salto-1P 1P (Jumping Motion)
Insect-scale Soft Robot with Combustion chuators
Fast Motion (from Jumping Locomotion)
Figure A2: Real-world mobile platforms can have versatile motion. (a) Autonomous robots exhibit highly dynamic and versatile motions, pushing the boundaries of traditional control systems. These diverse movement types challenge SLAM systems, which must operate in these dynamic contexts. (b) To address these challenges. we categorize motion perturbations into three key types: translation deviations (e.g., caused by low-stiffness structures in soft robots (Patel et al., 2023a)), rotation deviations (e.g., from rolling locomotion in tensegrity robots (Huang et al., 2022)), and fast-motion deviations (e.g., arising from high-speed or jumping movements (Aubin et al., 2023)). These categories capture the full spectrum dynamic disturbances that perception systems must handle for reliable operation.
Motivation from real-world mobile agents. Deploying sensing systems on autonomous robots requires addressing the diverse and dynamic motions these robots experience, as shown in Fig. A2a. Examples include the Disney bipedal robot (Grandia et al., 2024) and Atlas humanoid robot (Feng al., 2014), both of which exhibit complex full-body control, as well the SuperBall tensegrity robot (Sabelhaus et al., 2015), which rolls, and the hybrid jumping and wheel-based locomotion of Ascento (Klemm et al., 2019) and Salto-1P (Yim Fearing, 2018), which excels in rapid jumping maneuvers. While these versatile motions allow for more advanced robot manipulation and locomotion, they present significant challenges when sensors are placed on the robots for dense SLAM. In real-world scenarios, autonomous agents frequently face external disturbances such as platform vibrations, uneven terrain, and dynamic movements, complicating sensor pose estimation and degrading SLAM performance. Existing benchmarks like Replica (Straub al., 2019) and ShapeNet (Chang et al., 2015) typically assume smooth, stable sensor trajectories, which fail to capture the complexities and instabilities of real-world environments. For example, visual SLAM system operating legged robot navigating rough terrain is prone to significant motion deviations and vibrations, negatively affecting both sensor pose estimation and the quality of visual data.
To better emulate real-world challenges, we introduce rigid-body motion perturbations that simulate disturbances by perturbing the sensor's 's trajectory in both rotational and translational axes. Grounded in rigid-body transformation theory, this method uses precise mathematical models to inject dynamic perturbations that mirror the unpredictable conditions encountered in realistic environments, providing more rigorous test for SLAM systems. As shown in Fig. A2b, we categorize these motion perturbations into three main types: translation deviations (e.g., from low-stiffiess structures in soft robots (Patel et al., 2023a)), rotation deviations (e.g., from rolling locomotion in tensegrity robots (Huang et al., 2022)), and fast-motion deviations (e.g., arising from high-speed movements or jumping (Aubin al., 2023)). These cafegories capture the full spectrum of dynamic disturbances
18

---

Published as a conference paper at ICLR 2025
Modion (2x)
Vertical lines rnderenc)))
Orkgioal
Fostr
Oberrved
Thanzaato (0.0125mm))
(29
(0.223m))))
(3
Toursbtee (0.6600
(87)
Figure A3: Rendered RGB image streams under trajectory-level perturbations, including
translation deviations (Translate), rotation deviations (Rotate), and the faster motion effect.
that SLAM systems must handle, ensuring a more robust and comprehensive evaluation of their performance and reliability in real-world scenarios.
rigid-body transformation in 3D space consists of both a rotational and translational component. For each sensor pose, represented by a rotation matrix R E (3(3) and a translation vector t E IR33, we introduce perturbations that simulate physical disturbances in both orientation and position.
(a) Motion deviations using rigid-body transformations.
Rotation deviations. (AR E (3(3)): The rotation of the sensor is modeled by a 3x3 orthogonal matrix R. which belongs to the special orthogonal group SO(3). To simulate realistic rotational deviations, we introduce a perturbation AR, also a member of SO(3), which represents small rotational offset. The perturbed orientation R' is then given by:
R'=RAR
(A1)
where AR is rotation matrix corresponding to small angular deviations around the principal axes (x, y, z). These deviations are sampled from zero-mean multivariate Gaussian distribution (0, Ex). with covariance Σạ governing the severity of the perturbation. Specifically, AR can be computed as:
AR = exp ([w)x)
(A2)
where [w]x is the skew- symmettitiic matrix representation of the angular velocity vector [We, Wy: wa] T, and w is sampled from (0,, Σx). This formulation ensures the perturbed rotation
matrix remains valid element 50(3), preserving the properties of rigid-body rotations.
Translation deviations. (At E ) The translational position of the sensor is represented by vector which defines the sensor's s position in the world frame. To model translational perturbations, we add random perturbation At, sampled from zero-mean Gaussian distribution A ((0, Σ,), resulting in the perturbed position:
t At
(A3)
where E, is the covariance matrix that controls the magnitude and directionality of the perturbations. These translational deviations reflect small shifts caused by external physical factors such as terrain- induced vibrations mechanical imperfections in the robot platform.
Together, the perturbed sensor pose (R', t') models the real-world instability experienced by mobile platforms, where both rotational and translational movements affected by external disturbances.
In Fig. A3, we present the rendered sensor streams under varying severity levels of trajectory- level perturbations, encompassing translational deviations, rotational deviations, and faster motion effects. Although the rotational and translational deviations we examined result in minor changes in observations between adjacent frames, these perturbations lead to significant performance degradation across the majority of benchmarking SLAM models. As depicted in Fig. A4, even slight trajectory- level deviations have substantial impact trajectory estimation performance.
19

---

Published as a conference paper at ICLR 2025
Ground-trus -ruth Trajectory
Estimated Trajectory
0.50
Original (Cloun)
Rotation Deviation (1')
0.50
Trandalion
0.25
0.50
Deviation
0.00
0.00
0.00
(0.0125m))
00.50
0.30
-0.50
-0.75
-1.00
-1.25
ATE: 0.1247
ATE: 0,9856
-1.25
ATE: 0.2077
3.5
2.0
-15
[m0]
Faster Effect (2x)
1.00
0.50
Transiation
0.50
0.30
Deviation (9)
0.00
Deviation
(0095am)
6.00
0.00
E
-0.25
.0.00
-0.50
00.00
-1.00
-1.00
-0.75
ATR:
0,7970
-1.25
ATE: 0,2704
3.5
3.0
-2.5
220
-1.5
-10
-4.0
-3.0
-2.0
-1.0
0.0
-35
-3.0
-25
-2.0
-15
-1.0
-0.5
Figure A4: Illustrations of motion deviations and the faster motion effect. We present the synthesized ground-truth trajectories (in black) and the estimated trajectories (in blue) obtained using the CO-SLAM (Wang et al., 2023) model, which shows that slight trajectory deviations can have significant impact on the trajectory estimation performance.
(b) Faster motion effect. addition to introducing pose perturbations, we evaluate the robustness of sensing systems by simulating higher-speed motion scenarios. To do this, we down-sample the sensor's observation stream along the time axis, effectively increasing the perceived motion between consecutive frames. Let the original sensor stream have time step i, and introduce faster motion effect by down-sampling at intervals kt, where & > I the down-sampling factor. This creates larger inter- inter-frame motion, challenging SLAM algorithms to maintain accurate pose estimation despite fewer temporal observations.
20

---

Published as a conference paper at ICLR 2025
Blimp Robot
Severe Depth Missing due to Range L Limitation
High Illumination
Motion Blu
Figure A5: Corruption in RGB-D imaging is a frequent issue in real-world data collection. Challenges like motion blur and extreme illumination are common in the RGB sensor stream collected by mobile platforms (Left), while depth data loss due to range limitations significantly affects modeling (Right)
A.2 PERTURBATION ON RGB SENSOR IMAGING
Imaging corruptions is common in real-world data collected by mobile systems. In real-world deployments, the perception model of autonomous systems frequently encounters imaging corruptions that degrade sensor data quality. These include motion blur, high illumination conditions, and depth data loss due range limitations, as seen in the examples Fig. A5. For instance, the Blimp robot experiences motion blur and high illumination during operation, while the depth collection via RealSense D435 RGB-D sensor could suffer from severe depth missing due to depth perception limitation, highlighting the challenges sensing systems face in maintaining accurate sensor pose estimation and environmental mapping under such conditions.
The perturbations introduced for RGB imaging are designed to model potential error sources across the entire image formation and processing pipeline, spanning from the physical 3D scene to final 2D image. These perturbation sources encompass environmental interferences affecting light propagation, lens-related optical distortions, sensor noise due to imperfections in image acquisition hardware, and post-processing effects applied after image capture. This comprehensive approach simulates real-world image degradations a physically grounded manner to test the robustness of perception systems under diverse visual conditions.
(a) Environmental Interference Environmental interference, such as adverse weather conditions, affects light transmission through scattering, absorption, and occlusion. To simulate these effects, employ alpha blending techniques that mix layer representing the environmental interference with the clean image, generating composite that simulates the perturbed conditions. This method aims approximating real-world phenomena where environmental factors interfere with direct light paths.
Snow introduces random, localized variations in light intensity due to snowflakes scattering light. To model this, we generate random noise layer with white spots simulating snow and blend with the original image (Von Bernuth al., 2019).
Snow Effect:
Frost simulates translucent overlay that partially occludes the scene. This effect modeled by applying a semi-transparent whitening filter to the image using a weighted combination of the clean image and whitened version of it (Michaelis et al., 2019).
Frost Effect:
Fog Effect: Fog scatters light uniformly, reducing contrast and blurring distant objects. We simulate fog by linearly interpolating between the original image and a constant gray-value image to simulate the effect of reduced visibility (Hendrycks & Dietterich, 2019).
21

---

Published as a conference paper at ICLR 2025
Spatter simulates droplets streaks the camera lens, obstructing part of the image. This effect is modeled using a semi aani-tranpparent mask applied to the image, with regions randomly designated perturbed (spattered) transparent. The spatter mask controls the distribution and severity of the distortion.
Spatter Effect:
(b) Lens-Related Distortions (Blur) Lens imperfections or motion during image capture can introduce blurring effects, which we simulate by convolving the input image with specific blur kernels. These kernels approximate various physical processes responsible for image blur, such as defocusing rapid motion during capture.
Defocus Blur:
Caused by the camera lens being out of focus, this effect simulated by convolving the image with a circular disc kernel (bokeh) to model the blurring of regions outside the camera's 'ss depth of field.
Glass Blur: This effect emulates viewing image through textured or distorted glass. We model this by convolving the image with an irregular blur kernel, simulating the scattering of light by the glass surface.
Motion Blur:
When the camera objects in the scene move rapidly during image capture, it results in motion blur. We simulate this effect by convolving the image with a linear kernel oriented in the direction of motion model the smearing caused by the movement.
To approximate image smoothing caused by lens imperfections, we apply Gaussian blur, which convolves the image with Gaussian kernel. The degree of blurring is controlled by the standard deviation of the Gaussian function, simulating varying levels of softening.
Gaussian Blur:
(c) Sensor Noise RGB image sensors are inherently imperfect, introducing noise during image acquisition that affects image quality. We model several types of sensor noise, each representing different physical process that affects image formation.
Simulated by adding random pixel-wise noise sampled from a Gaussian distribution with zero mean, this models sensor imperfections where pixel intensities are corrupted by random fluctuations.
Gaussian Noise:
Shot Noise:
Shot noise arises from the quantum nature of light, caused by the random arrival photons at the sensor. is modeled using a Poisson distribution (Hasinoff, 2014) to account for the discrete nature of photon interactions during image capture.
noise, this effect randomly corrupts indi- vidual pixels. For each pixel, a random value is sampled, and if it falls within a specified range, the pixel is either set to the minimum or maximum intensity value, while other pixels remain unchanged.
Impulse Noise: Also known as salt-and-peppa
Speckle noise is modeled as multiplicative noise, commonly seen in coherent imaging systems like radar and ultrasound. The intensity of each pixel in the image is perturbed as follows:
Speckle Noise:
(( v) I(x, y) (1+ p 17)
(A4)
where p controls the noise intensity, and 17 is a Gaussian noise term.
(d) Post- Post-Processing Perturbations Post-processing effeects modify the image after it has been captured, often degrading quality through compression or global adjustments.
Adjusting the brightness of the image involves adding a constant value to every pixel. This models exposure adjustments or lighting variations that affect the overall intensity of the image.
Brightness:
Contrast: Contrast changes affect the variance in pixel intensities. We model contrast
adjustments by linearly scaling pixel intensities about the mean intensity J:
I' v) = 3 (1 J +
(A5)
where 3 controls the level of contrast.
22

---

Published as a conference paper at ICLR 2025
Bigghteess
Contrast
Defocus Alur
Fug Hffort
Frost Rffoct
Ganasian Blur
Gausiaa Notse
Glass Blur
Lewo
Modile
High
Impulse Nosse
Jag Comppession
Motion Blar
Pixelate
Spatter
Speckle
Noise
Effect
Noise
Figure A6: Illustration of RGB imaging perturbations under different severity levels. We consider 16 common image corrpption types (Hendrycks & Dietterich, 2019) from 4 main categories of perturbations for robustness evaluation: (1) noise-based distortions: Gaussian Noise, Shot Noise, Impulse Noise, and Speckle Noise; (2) blur-based effects: Defocus Blur, Glass Blur, Motion Blur, and Gaussian Blur; (3) environmental interferences: Snow Effect, Frost Effect, Fog Effect, and Spatter Effect. (4) post-processing manipulations: Brighiness, Contrast, Pixelate, and JPEG Compression. Each perturbation type is further split into severity levels (low, middle, and high) for benchmarking evaluation, which corresponds to Level 1, Level 3, and Level 5 of Table Al.
JPEG Compression: JPEG compression introduces artifacts during lossy image compres- sion. These artifacts, such as blocking effects and loss of detail, are simulated by reducing the image quality and reintroducing cumpression-ndduce distortions.
This effect reduces image resolution by dividing the image into pixel blocks and replacing each block with its average value, simulating drop in resolution commonly seen in low-quality video feeds.
Pixelation:
Implementation of RGB imaging perturbations. We illustrate RGB imaging perturbations under different severity levels in Fig. A6. As shown in Table A1, we define five severity levels for each type of RGB imaging perturbation, following established robustness evaluation literature (Hendrycks & Dietterich, 2019).
23

---

Published as a conference paper at ICLR 2025
Table A1: Specific configurations of the RGB imaging perturbations.
Perturbation
Parameter
Level
Level
Level
Level
Level
(Mean, Uneshold, blur radius, blur std, blending ralio)
0.55,
0.55,
2.5,
Snow Effect
0.5, 10.0, 4.0, 088
05, 12, 0.7
09, 12, 0.7
085, 12, 0.65
0.85, 12, 12, 0.55
(Frost intensity,
influence)
0.40)
(0.80, 0.60)
0.70)
(0.65, 0.70)
0.75)
Fog Effect
(Thickness, smoothhess)
(1.5.20)
(2.0, 2.0)
(2.5, 1.7)
(2.5, 1.5)
(3.0, 1.4)
(mean, standard deviation,
Spatter Effect
scaling factrr, complexity effect)
sigma, threshold,
(0.65, 0.3.4,
0.69, 0.6.0) 0)
(0.65. 0.3,
0.68, 0.6, 0)
(0.65,
0.68, 0.5,0)
(0.65, 0.3, I.
0.65, 1,5, I)
(0.67. 0.4, 0465, 1.5, I)
Defocus Blur
(Kernel radius, alias blur)
(3.0, (3.0,0.1)
0.1)
(4,0, 0.5)
(6.0, 0.5)
(8.0, 0.5)
(10.0, 0.5)
(Sigma, max
(0.9,.0. L.))
2.0,
2.0)
Musion Blur
(Radius, sigma)
(10.3)
(15.5)
(15,8)
(15,
(20, 15)
Gunssian Blur
Sigma
3
6
Shot Noise
Photon number
25
Impulse Noise
Noise amount
0.03
0.06
0.09
0.17
0.27
Spocklc Noise
Noise scale
0.15
0.2
0.35
0.15
0.6
Brightmes
Adjustment
Contrast Decresse
Adjustment of pisel
0.40
0.30
0.20
0.10
0.05
JPEG Compression
Compression quality
23
18
13
10
7
24

---

Published as a conference paper at ICLR 2025
(a) Depth Distribution
(b) Depth Missing Rate
10.50
TUM-RGBD
TUM RGBD
Repliss
Replica
24.719
8
Missing Rate [%)]
Depth [m]
10
9.571
2.42
2
1,22
0.42
0.18
0.41 0.56
0.394
1.440
Depth Min Depth Max Depth Mean Depth Std
Mean
Std
Discrepancy in depth characteristics between simulated Replica (Straub et al., 2019) and real TUM-RGBD (Sturm et al., 2012) datasets..
Figure A7:
A.3 PERTURBATION ON DEPTH SENSOR IMAGING
Motivation. As illustrated in Fig. A7, there exists noticeable disparity between simulated clean depth data, such as those from the Replica (Straub et al., 2019) SLAM benchmark, and real-world noisy depth data obtained from the TUM-RGED (Schubert al., 2018) SLAM benchmark. For example, while the Replica dataset has a minimum depth of approximately 0.18 m, the TUM-RGBD data exhibits minimum depth of 0.4 m, reflecting the range limitations typical of real-world depth sensors. Additionally, real-world depth sensors tend to have a significantly higher rate of missing data, with TUM-RGBI showing approximately 25% missing depth values compared less than 0.4% in Replica. These discrepancies highlight the need to introduce perturbations that model the limitations of real depth sensors, bridging the gap between simulated and weal-world data.
In this section, we propose four key depth perturbations that emulate the sensing limitations and environmental constraints faced by real-world depth sensors, thereby offering a more realistic evaluation of SLAM systems.
(a) Gaussian Noise Real- world depth sensors, such time-ff-light (ToF) and structured light systems, are subject to random measurement noise due to factors like photon counting errors and sensor precision limits. This noise is often modeled map. To simulate this, we perturb each pixel (ær, y) in the depth map / by adding Gaussian noise v AO0 o²), where the standard deviation of the noise distribution, dependent the sensor's characteristics:
naussian-distribune
errors across the depth
D' v) = D(a, v) +
(A6)
This models the stochastic fluctuations in depth measurements, capturing the inherent noise that arises during the depth sensing process.
(b) Edge Erosion Depth sensors, particularly ToF systems, often suffer from multi-path interference and sensor limitations near depth discontinuities, such as object edges. This results in inaccurate depth estimates along these regions. We simulate this by performing edge detection to identify the boundary pixels in the depth map and subsequently eroding a fraction of these edge pixels, representing measurement failures near complex geometries:
VOID if (z. v) EP D(,,y)
D' v)
(A7)
otherwise
where P represents the set of detected edge pixels, and VOID signifies missing data. This emulates the failure of depth sensors capture accurate data near edges due signal interference.
25

---

Published as a conference paper at ICLR 2025
Table A2: Specific configurations of the depth imaging perturbations.
Perturbation
Parameter
Level 1
Level 2
Level 3
Level 4
Level 5
Gaussian Noise
Noise scale
0.1
0.2
0.3
0.4
0.5
Edge Erosion
Erosion rate
0.015
0.020
0.025
0.03
0.035
Random missing depth data
Missing rate (%)
10
15
20
25
30
Range clipping
(Min depth, Max depth)
(0.2, 4.4)
(0.3, 4.2)
(0.4, 4.0)
(0.5, 3.8)
(0.6, 3.6)
(c) Random Missing Depth Data Real-worid depth sensors frequently encounter occlusions,
reflective surfaces, or environmental conditions that block the sensor's line of sight, leading to missing or incomplete depth measurements. We simulate this phenomenon by applying binary mask M to the depth map, randomly occluding portions of the image reflect areas where depth data is not captured:
D' y) D(x, M M(r
(A8)
where M (x.U) y) is binary mask generated by randomly sampling rectangular patches. Pixels within the occluded regions are set to a VOID value, representing missing data due to sensor limitations.
(d) Range Clipping Depth sensors, especially consumer-grann grade systems like Kinect or RealSense, have limited depth ranges, with measurements becoming unreliable beyond specific maximum distance or too noisy below minimum distance. We simulate this range limitation by clipping depth values outside a predefined range [Dmin, Dmaz]. Depth values that fall outside this range are replaced with VOID value:
VOID
if D(z, v) < Dmi or D(x, y) > Dmax otherwise
D' v)
(A9)
(zr v)
This perturbation reflects the operational range limitations of real-world depth sensors, where objects too far or too close to the sensor result in unreliable or missing depth data.
Implementation of Depth Imaging Perturbations As shown in Table A2, better reflect real- world depth sensing conditions, we apply varying severity levels for each of the proposed perturba- tions, inspired by the real depth distribution of datasets such TUM-RGBD (Schubert et al., 2018), 8). The specific implementation of these depth perturbations is available through our Depth Imaging Per- turbation Synthesis Toolbox. The toolbox defines five severity levels for each perturbation, ensuring comprehensive evaluation of SL.AM systems across different conditions.
A.4 PERTURBATION ON MULTI-SENSOR SYNCHRONIZATION
Motivation. While integrated RGB-D sensors like RealSense, Kinect, and ZED minimize desyn- chronization issues due to their tightly coupled design, many real-world applications require more flexibility in sensor selection. In this work, we address a more generalized scenario where RGB and depth sensors operate independently, rather than assuming they synchronized. This approach pro- vides greater adaptability for sensor confi configurations in autonomous systems. For instance, lightweight ToF-based depth sensors, such as the Arducam ToF, paired with compact RGB cameras, offer ad- vantages in weight and energy ceriiency-crtticee factors for SLAM systems deployed on platforms with striet payload and power constraints, such as micro UAVs. However, using distributed RGB and depth sensors introduces potential inconsistencies between data streams. De-synchronization, caused by differences in signal frequencies, can lead temporal misalignment, resulting in inaccuracies in the combined RGB-D data. Addressing these challenges is crucial for enabling robust performance in resource-constrained environments.
To emulate sensor delays in cases where multiple sensors within an RGB-D sensing system are not synchronized, we introduce temporal misalignment between sensor streams (see Fig. 2d of the main paper). Consider two initially synchronized sensor streams, denoted as S (t) and (t). We simulate a delay in the second stream by shifting its sensor sequence by a frame interval A. This creates perturbed streams S (t) (t) and (t) St A). While one sensor stream is shifted, the poses associated with each sensor reading remain unchanged. This ensures the system is operating on data grounded in the past, reflecting the real-worke scenario of misaligned sensor information.
26

---

Published as a conference paper at ICLR 2025
B
MORE DETAILS ABOUT Robust-Ego3D BENCHMARK
B.1 ASSUMPTIONS FOR BENCHMARKING SETUP
While our customizable noisy data synthesis pipeline allows for noisy data generation from any clean 3D scene, incorporating any composition of perturbation types, with any number and configuration of RGB and depth sensors, we initialize the Bobust-Ego.33 benchmark for model robustness evaluation under the following assumptions:
Task. We focus on the standard (passive) SLAM setting, assuming the absence of active decision- making processes.
Model. Our analysis is centered on vision-oriented SLAM scenarios, specifically targeting monocular and RGB-D settings. We assume the use of dense depth representation as opposed to sparse depth data obtained from a LiDAR scanner. addition, the SLAM system presumed have known motion and observation models.
Perturbation. Although our noisy data synthesis pipeline is capable of generating SLAM bench- marks with multiple heterogeneous perturbations, we concentrate on investigating the performance degradation caused by individual sensor trajectory perturbations. This focused approach is de- signed to dissect the system's 's response to isolated perturbations, allowing precise quantification of their specific impacts on SI.AM performance. By analyzing the degradation induced by individual perturbations, we can effectively assess the system's robustness in a controlled manner and identify the root causes of performance degradation. This knowledge is crucial for developing targeted mitigation strategies that address the most vulnerable aspects, i.e., Achilles' Heel, of the whole SLAM system. Also, we model these perturbations using simplified linear models (e.g.. Gaussian noise assumptions), in line with precedent set by established literature (Hendrycks & Dietterich, 2019; Wang et al., 2020; Chattopadhyay et al., 2021). While these simplified perturbations may not fully capture the complexity of real- world scenarios, they offer interpretability and facilitate analysis across different perturbation types.
3D scene. We assume that the environment is static, meaning there are moving dynamically changing objects within the scene. Also, the scene is bounded, typically referring to an indoor setting with predefined boundaries limits.
27

---

Published as a conference paper at ICLR 2025
H400 beed 6000
mett: taech 01
nedd ttas 1155
wed tas B5te
mec1 tras doCte
met taa 1250
tra cotte
merT the coure
10000
ne the 5300
trad
baad
mes5 tradd
ReCDI a00lO110 (VC)
HM tas (05%
HN.7 Ansd 01230
HIPP; cas 80000
roufi coan 100000m
na5 co0 00900
rad cand
1465 trasd
nest: mrant
1000 cas
Figure B8: Motion statistics of trajectory distribution under varying combinations of translation and rotation deviations. Assuming a frame rate of 20 frames per second for the SLAM system, i.e. a time interval of 0.05 seconds between neighboring pose frames, we present the motion distribution of perturbed trajectories the proposed Robust-Egoss benchmark. The figures show the distribution of translation speed (Top Left), translation acceleration (Top Right), rotation speed (Bottom Left), and rotation acceleration (Bottom Right). We report the mean value of each setting.
B.2 Robust-Ego3D BENCHMARK SETUP DETAILS AND STATISTICS
Data source for rendering. We render the RGB-D sensor streams using 3D scene models sourced from the Replica dataset (Straub et al., 2019), which comprises real 3D scans indoor scenes. We select the same set of eight rooms and offices as the (clean) Replica-SLAN dataset (Sucar et al., 2021) for consistent comparison. Each sequence has 2,000 frames at 1200x680 resolution.
Benchmark sequence number distribution. Using our established taxonomy of perturbations for SLAM and the noisy data synthesis pipeline, we have created a large-scale SLAM robustness benchmark called Bobust-Egoo:1 to evaluate the robustness of monocular and RGB-D SLAM methods by incorporating various perturbations that mimic real-world sensor and motion effects.
Each perturbed setting of our benchmark is rendered in eight scenes from the 3D indoor scan dataset Replica (Straub et al., 2019). This process generates eight sequences from single trajectory. For each perturbed setting, we calculate the average result from 24 experimental data points (eight sequences, each repeated three times). Then, we report the averaged result for each perturbed setting. Specifically, for the RGB imaging perturbation, present the averaged result across three severity levels, under both static and dynamic perturbation modes.
We provide details about the specific distribution and setup of each perturbed setting as follows:
8 original clean sequences: These sequences replicate the quality and the sequence number of the original Replica SLAM dataset (Sucar et al., 2021).
768 sequences with RGB imaging perturbations: We apply 16 different types of image- level perturbations at severity levels (Level 1, Level3, and Level 5 of Table A1), both under static and dynamic conditions.
28

---

Published as a conference paper at ICLR 2025
32 sequences with depth imaging perturbations: This category consists of 4 types of perturbations. For the depth noise, we adopt the hyperparameters of the Gaussian noise distribution specified in previous literature (Hendrycks & Dietterich, 2019). Moreover, we set the depth missing rate to 20% and establish the depth clipping range based on the real-world depth distribution of the TUM-RGBD dataset (Sturm al., 2012). The severity strength of each depth perturbation is shown in the Level 3 column of Table A2.
These sequences involve faster speed than the original sequences, with variations of two, four, and eight times the original speed.
24 sequences with faster motion effects:
120 sequences with motion deviations: This category includes pure rotation deviation, pure translation deviation, and combined transformation matrix deviation. We define three severity levels for both rotation and translation deviations, and sample the deviation from Gaussian distribution. Specifically, for rotation deviation, we introduce random deviations in rotation around the x, y, and axes, with mean values of zero and standard deviations I, 3, and degrees at each pose frame. For translation deviation, we introduce random deviations in the y, and axes, with mean values of zero and standard deviations of 0.01 25, 0.025, and 0.05 meters at each pose frame. In Fig. B8, we show the motion statistics of the perturbed trajectory sequences under varying combinations of translation and rotation deviations. This category of trajectory-devinet of motion speeds and accelerations, enabling a progressive evaluation of the robustness of SLAM models against increasingly challenging motion types. These insights are especially valuable for evaluating the implementation of SLAM systems high-speed scenarios or on agile robot platforms exposed to significant vibrations.
sequences encompasses a broad spectrum
48 sequences with RGB-D sensor de-synchronization: We consider both static and dynamic perturbation models for multi-sensee misalignment. In the static mode, a constant time delay is synthesized between the two sensor streams, while in the dynamic perturbation model, there varying time delay between the streams. Specifically, the multi-sensor misalignment perturbation sequences consist of 24 sequences with a fixed cross-sensor frame delay interval (A) of 5, 10, and 20 frames, as well as 24 sequences with dynamic perturbation where A deviates by frame from the fixed intervals of 5, 10, and 20 frames.
Overall, this benchmark dataset enables comprehensive evaluation of existing SLAM algorithms under simulated perturbations, providing a thorough assessment of the robustness of multi-modal SLAM systems in wide range of challenges.
29

---

Published as a conference paper at ICLR 2025
Table B3: RGB-D SLAM methods for evaluation.
Method
Type
Modality Monn/RGBDD
Map Representation
Loop Closare
External Data
Speed
Processing
Year
ORB-SLAME
Classical, Sparie
Koyfrim-ORB Explicit
X
Real-time
CPU
2020
Neanl, Dense
NoRF-based"i Implicit
X
Quasi Real-time
CPL4GIPU
2021
SLAM
Neural Donse
X
baRR--bsed. Implictt
X
Quasi Real-time
CPU+GPU
2022
CO-SLAM
Neural Dense
X
NeRF-bused, Implicit
Real-fime
CPU+GPU
2023
SOAS
Neural, Donse
NeRF- based, Implicit
(
Quasi Real time
CPU+GPU
2023
Sp4TAM-S
Neural, Dense
Gaussian Splat, Explacit
Quasi Real-time
CPU+GPU
2024
TT NNRRF-bused incicatiss methods that loverage implicii neural newworks to encode the 3D sceme, Following the philosophy of NcR3F (Roxinul ei all,, 20233.
(2) GO-SLAM iniializes the model parameters from the PROID-SLAM. model (Tood & Deng, 2021) which levengess exiemaa data for mode pre-tmiining.
B.3 MORE DETAILS ABOUT BASELINE MODELS FOR BENCHMARKING
Additional descriptions about benchmarking models. While previous SLAM robustness evalua- tions primarily focused on classical methods (Wang et al., 2020; Bujanca et al., 2021), our benchmark encompasses both classical and learning- -aaedd SLAMM systems. As shown in Table B3, in addition to the classical SLAM model erence, we mainly evaluate top-performing dense Neural SLAM models on standard benchmarks with diverse architecture design. iMAP(Sucaa et al., 2021) and Nice-SLAMCC et al., 2022) are representative methods with implicit neural representations. GO-SLAMMZZaan on global pose optimization, while CO-SLAM(Wang et al., 2023) utilizes hybrid neural encoding. SplaTAMMSS (Keetha et al., 2024) employs Gaussian- (Kerbl et al., 2023) map representation. The hyperparameters are set based on the recommendations given in the original papers or use default settings otherwise. Below, we offer additional descriptions of these models.
(Campos et al., 2021), whose results are used for ref-
et al., 2023) focuses
ORB-SLAM3 (Campos et al., 2021): An extension of ORB-SLAM2 (Mur-Artal & Tardós, 2017) that incorporates a multi-map system and visual-inertial odometry, enhancing robust- ness and performance.
iMAP (Sucar et al., 2021): A neural RGB-D SLAM system that utilizes the MLP represen- tation to achieve joint tracking and mapping.
Nice-SLAM (Zhu al., 2022): neural RGB-D SLAM model that employs a multi-level feature grid for scene representation, reducing computational overhead and improving scalability.
CO-SLAM (Wang al., 2023): An advanced neural RGB-D SLAM system with hybrid representation, enabling robust camera tracking and high-fidelity surface reconstruction in real time.
GO-SLAM (Zhang al., 2023): A neural visual SLAM framework for real-time opti- mization of poses and 3D reconstruction. It supports both monocular and RGB-D input settings.
SplaTAM (Keetha et al., 2024): A neural RGB-D SLAM model that follows Gaussian Splatting (Kerbl et al., 2023) to construct an adaptive map representation based on Gaussian kemnlls. Due to time and computational constraints, we evaluate the relatively more efficient SplaTAM-S model variant in our benchmark.
Remark: Some dense Neural SLAM models are inherently 'trained' on the testing distribution. Neural SLAM models are optimized (i.e., 'trained') on perturbed RGB-D observations, enabling continuous adaptation and updating of internal representations based on incoming data at each timestamp. This inherently includes 'training with introduced perturbations', as the models adjust to variations during online operation. Neural SLAM methods like Nice-SLAM (Zhu al., 2022), CO-SLAM (Wang et al., 2023), and GO-SLAM (Zhang et al., 2023) leverage Neural Radiance Field (NeRF) as the map representation, updating the parameters of NeRF network and the parameters of poses for each frame when new observations arrive during testing; SplaTTMM (Keetha et al., 2024) leverages explicit Gaussian Splats (Kerbl et al., 2023) as the map representation, updating the parameters of Gaussian kernels well as the parameters of poses for each frame during testing. We find that this test-time online learning mechanism provides better robustness compared to non-Neural
SLAM methods without adaptation capabilities, allowing Neural SLAM models to be robust to static RGBD imaging perturbations by continuously refining their environment understanding for optimal performance.
30

---

Published as a conference paper at ICLR 2025
B.4 DETAILS ABOUT HARDWARE SETUP FOR BENCHMARKING EXPERIMENTS
Our experiments were primarily conducted on a GPU server equipped with two NVIDIA A6000 GPUs, each featuring 48 GB of memory. These resources were utilized for synthesizing perturbed noisy data and evaluating the robustness of RGB-D SLAM models. The operating system used was Ubuntu 22.04. Additionally, we tested the compatibility of our benchmarking code on a GPU server with four NVIDIA RTX6000 Ada GPUs, each with 48 GB of memory, and on GPU server with two NVIDIA A100 GPUS, each with 40 GB of memory.
It is important to note that the memory requirements of different SLAM methods vary based on the complexity of the perturbed RGB-D video sequences used for evaluation and the specific memory cost of each method. For instance, the CO-SLAM (Wang et al., 2023) model can run on a GPU with 12GB of memory. Meanwhile, only GPU required for the SLAM methods evaluated in our study under each perturbed setting.
B.5 UNIQUENESS AND ADVANTAGES OF OUR Robust- Ego3D BENCHMARK
Our instantiated benchmark Robust-EgooD for RGB-D SLAM robustness evaluation offer several distinct advantages that can advance scalable SLAM evaluation and more robust SLAM model:
Unparalleled diversity and controllability. With 124 perturbation settings and an extensive dataset comprising 1,000 long video sequences and nearly million image-depth pairs, our tool offers unmatched diversity and controllability. Researchers can create highly customized and challenging test scenarios, exploring wide range of real-world conditions and pushing the boundaries of SLAM algorithms. This extensive collection of perturbations allows for comprehensive assessment of SLAM systems under diverse environmental conditions and sensor noise profiles.
Scalability and fair comparison. The large size of our dataset enables statistically significant evaluations and fair comparisons between different SLAM algorithms under diverse conditions. This scalability is crucial for robust benchmarking and identifying the strengths and weaknesses of various approaches. By providing a large and diverse testing ground, our tool facilitates unbiased comparisons and promotes the development of more reliable and generalizable SLAM solutions.
Decoupled perturbation study. Our pipeline facilitates the decoupled study of individual and mixed perturbations, providing valuable insights into the isolated and combined effeets of various noise sources. This granular understanding is essential for developing targeted strategies to enhance SL.AM robustness in complex environments. By disentangling the impact of individual noise sources, researchers can gain a deeper understanding of their specific effects on SLAM performance and design algorithms resilient specific types of perturbations.
Standardization. Our pipeline introduces a systematic and standardized approach to generating noisy environments, ensuring consistency and reproducibility across different studies. This standardization is crucial for facilitating meaningful comparisons and advancing the field of SLAM research. By establishing common framework for generating and evaluating SLAM datasets with perturbations, our tool promotes collaboration and accelerates the progress of the entire research community.
31

---

Published as a conference paper at ICLR 2025
B.6 UNIQUENESS OF SYNTHETIC DATASETS COMPARED TO REAL- WEAL-WORLD DATA
Synthesized datasets offer unique and substantial advantages over real-world datasets, particularly when it comes SLAM evaluation under noisy or complex conditions. While real-world datasets like TUM-RGBD (Sturm et al., 2012) capture certain real-world complexities, such as environmental noise and dynamic motion, the flexibility, scalability, and precision provided by synthesized datasets make them invaluable for comprehensive dense Neural SLAM evaluation and beyond.
Accurate and complete 3D ground-truth for photorealistic mapping evaluation. Real-world datasets, though offering realism, often lack precise ground-truth 3D scans needed for detailed, quan- titative mapping evaluation. Specifically, the absence of high-fidelity 3D geometry and appearance data limits the ability to assess mapping quality in depth. For example, datasets such as SubT (Ebadi et al., 2023) and TUM-RGBD (Sturm et al., 2012) provide odometry data but fail to deliver accurate 3D scans required for photorealistic mapping evaluations. In contrast, synthesized datasets, such as those rendered from high-quality 3D assets like Replica (Straub et al., 2019) and ScanNet (Dai et al., 2017a), provide nearly perfect 3D ground-truth data for both odometry and geometry, making them indispensable for evaluating mapping performance.
Scalability and diversity for evaluating (and training) generalizable models. Real-world SLAM datasets are limited in size and diversity due to the challenges of data collection, such as manual setup and difficulty capturing complex scenarios like sensor noise, motion blur, or dynamic motion. As a result, these datasets are often insufficient for training robust, generalizable SL.MM models. In contrast, synthesized datasets offer a scalable solution by generating large, diverse data under controlled conditions. This allows exploration of rare and costly scenarios, such as dynamic lighting changes and sensor degradation. Synthesized data not only serves as a comprehensive evaluation platform but also helps SLAM models generalize across real-world environments. The ability to generate unlimited data under varied conditions is key to developing SLAM systems that perform reliably in dynamic settings. We argue that scalable synthesized datasets are essential for training models that generalize to real-world applications.
Customization and control over perturbations. One of the most significant advantages of synthe- sized datasets is their ability to inject specific, customizable perturbations in a controlled manner. Real-world datasets are inherently limited by the conditions present during data collection, which restricts the types of perturbations they can capture. Factors such as sensor noise, environmental variation, and lighting conditions are often incidental, making it difficult to systematically test how different perturbations affect SLAM performance. Synthesized datasets, however, offer greater flexibility in customizing and injecting perturbations, such noise, motion artifacts, or sensor degradation, in controlled and systematic way. This allows researchers to isolate specific effects and systematically evaluate model performance under a wide variety of challenging conditions. Such flexibility is invaluable for stress-festing models and understanding their limitations, especially in real-world conditions where similar noise patterns might occur but in less predictable ways.
Bias management and enhanced explainability. While all datasets, real or synthetic, have inherent biases, the controllability of synthesized datasets makes it easier to identify, manage, and mitigate these biases. Real-world datasets are often subject to hidden biases introduced by uncontrolled envi- ronmental factors, sensor characteristics, or specific collection methodologies, which can skew model performance in unanticipated ways. In contrast. synthesized datasets allow for greater transparency and control over biases, as perturbations can be systematically introduced and analyzed. This not only enhances the explainability of model behavior but also allows for better tuning and optimization, leading to models that are more robust to bias-related challenges in real-world deployment.
Bridging the gap between noise-free and real-world conditions. Finally, synthesized datasets play a critical role in bridging the gap between existing noise-free SLAM datasets and the complexities of real-world environments where noise, imperfect sensing, and challenging agent motions common. While current real-world datasets offer valuable insights, the level of control and precision offered by synthesized datasets, particularly when simulating degraded sensor data and complex motion scenar- ios, makes them an essential tool for SLAM evaluation. As such, our Robust-Ego3D benchmark synthesizes realistic noise and perturbations under imperfect sensing and motion conditions, offering a more comprehensive evaluation platform that brings SLAM closer to real-world robustness.
32

---

Published as a conference paper at ICLR 2025
C DATASHEET FOR Robust-Ego3D BENCHMARK
We document the necessary information about the proposed datasets and benchmarks following the guidelines of Gebru et al. (Gebru et al., 2021).
C.1 MOTIVATION
Q1 For what purpose was the dataset created? Was there specific task in mind? Was there specific gap that needed to be filled? Please provide description.
Our benchmark created to holistically evaluate the robustness of dense Neural SLAM models under diverse perturbations. Prior to our work, dense Neural SI.AM models were typically evaluated under noise-free settings. With our customizable perturbation synthesis pipeline, we evaluate the models across a wide range of RGB-D perturbations crucial for real-world deployment, including RGB imaging perturbations, depth imaging perturbations, motion-related perturbations, and RGB-D desynchroniza- tion.
Q2 Who created the dataset (e.g., which team, research group) and on behalf of which entity (e.g., company, institution, organization)?
This benchmark is presented by [Anonymous Author]². Our aim is to advance the benchmarking, development, and deployment of more reliable and robust ego-motion estimation and photo realistic 3D reconstruction.
Q3 Who funded the creation of the dataset? If there is associated grant, please provide the name of the grantor and the grant name and number.
This work was partially supported by [Anonymous Author].
Q4 Any other comments?
No.
C.2 COMPOSITION
Q5 What do the instances that comprise the dataset represent (e.g., documents, photos, people, countries)? Are there multiple types of instances (e.g.. movies, users, and ratings; people and interactions between them; nodes and edges)? Please provide description.
Our initialized Robust-Ego3l) benchmark includes RGB-I video sequences rendered from scanned 3D scenes, the 6D trajectory at each timestamp, and the 3D scene point cloud.
Q6 How many instances are there in total (of each type, if appropriate)?
The Robust-Ego3i benchmark includes 1,000 perturbed settings, each with 2,000 RGB-D video sequences. Detailed statistics for each scenario are provided in Sec. B.2 of the Appendix.
Q7 Does the dataset contain all possible instances or is it sample (not necessarily random) of instances from larger set? If the dataset is sample, what is the larger set? Is the sample representative of the larger set (e.g., geographic coverage)? so, so, please describe how this representativeness was validated/verifee.. If it is not representative of the larger set, please describe why not (e.g.. to cover a more diverse range of instances, because instances were withheld or unavailable).
The 3D scenes in our benchmark are sourced from the existing 3D scanned indoor scene dataset Replica (Straub et al., 2019), and we use all possible instances from these datasets.
Q8 What data does each instance consist of? "Raw" data (e.g. unprocessed text or images) features? In either case, pleuse provide description.
Each instance includes RGB-D images, trajectories, perturbation categories, and ground-truu truth 3D scene meshes.
Author and repository in information have been anonymized in compliance with the double-blind policy and will be provided upon acceptance.
33

---

Published as a conference paper at ICLR 2025
Q9 Is there label or target associated with each instance? If so, please provide description.
RGB-D video sequences for each perturbed setting are rendered from 3D scans in the Replica dataset (Straub et al., 2019).
Q10 Is any information missing from individual instances? If so, please provide description, explaining why this information is missing (e.g.. because it was unavailable). This does not include intentionally removed information, but might include, egg,, reducted text.
No.
Q11 Are relationships between individual instances made explicit (e.g., users' movie ratings, social network links)? If so, please describe how these relationships are made explicil.
Each RGB-D video sequence is explicitly linked to a specific 3D scene in the Replica dataset, defined by a unique trajectory and set of perturbations.
Q12 Are there recommended data splits (e.g., training, development/validation,, testing)? If so, please provide a description of these splits, explaining the rutionale behind them.
No, our benchmark is primarily intended for evaluation, as RGB-D SLAM models use online catpiiiiatiooadaptattttii and do not require separate training stage. However, it can be extended for model training if needed.
Q13 Are there any errors, sources of noise, or redundancies in the dataset? If so, please provide description.
No.
Q14 Is the dataset self-contained, or does it link to or otherwise rely on external resources (e.g., websites, tweets, other datasets)? If it links or relies external resources, a) are there guarantees that they will exist, and remain constant, over time; b) are there official archival versions of the complete dataset (i.e., including the external resources they existed at the time the dataset was created); c) are there any restrictions (e.g.. licenses, fees) associated with any of the external resources that might apply to a future user? Please provide descriptions of all external resources and any restrictions associated with them, as well as links or other access points, as appropriate.
The benchmark is self-contained. We provide all the details and instructions [Anony- mous Repo].
Q15 Does the dataset contain data that might be considered confidential (e.g., data that is protected by legal privilege or by dactor-patient confidentiality, data that includes the content of individuals' non-public communications). If so, please provide a description.
No. The 3D scans used in our Bobust-Egool benchmark are sourced from existing open-source datasets.
Q16 Does the dataset contain data that, if viewed directly, might be offensive, insulting, threatening, or might otherwise cause anxiety? If so, please describe why.
No.
Q17 Does the dataset relate to people? If not, you may skip the remaining questions in this section.
No. This dataset does not relate to people.
Q18 Does the dataset identify any subpopulations (c.g., by age, gender)?
N/A.
Q19 Is it possible identify individuals (i.e., one more natural persons), either directly or indirectly (i.e., in combination with other data) from the dataset? If so, please describe how.
N/A.
Q20 Does the dataset contain data that might be considered sensitive in any way (c.g., data that reveals racial or ethnic origins, sexual orientations, religious beliefs, political opinions union memberships, or locations; financial or health data; biometric or genetic data; forms of government identification, such as social security numbers; criminal history)? If so, please provide description.
34

---

Published as a conference paper at ICLR 2025
No.
Q21 Any other comments?
We caution discretion on behalf of the user and call for responsible usage of the benchmark for research purposes only.
C.3 COLLECTION PROCESS
Q22 How was the data associated with each instance acquired? Was the data directly observable (e.g.. raw text, movie ratings), reported by subjects (e.g., survey responses), or indirectly inferved/derived from other data (e.g., part-of-speech tags, model-based guesses for age language)? If data reported by subjects indirectly inferreul/eeriee from other data, was the data validated/verified?! If so, please describe how.
The 3D scenes used for SLAM data generation in our benchmark are sourced from the existing open-source dataset Replica (Straub al., 2019). The details of the benchmark construction are provided in the Experiment section of the main paper, with further details in Sec. the Appendix.
Q23
What mechanisms or procedures were used to collect the data (e.g., hardware apparatus or sensor, manual human curation, software program, software API)? How were these mechanisms or procedures validated?
No additional raw data was collected. Our contribution is the development of perturbation taxonomy and toolbox to transform existing clean SLAM datasets and 3D scenes into noisy datasets for robustness evaluation.
Q24 If the dataset is a sample from a larger set, what was the sampling strategy (e.g., deterministic, probabilistic with specific sampling probabilities)?
No new raw data was collected. RGB-D sensor streams were rendered using 3D scene models from the Replica dataset (Straub et al., 2019), consisting of real indoor scene scans. We used the same eight rooms and offices as the (clean) Replica-SLAM dataset (Sucar al., 2021) to ensure consistency in comparisons.
Q25 Who was involved the data collection process (e.g., students, crowdworkers, contrac- tors) and how were they compensated (e.g., how much were crowdworkers paid)?
N/A. Our benchmark does not include new raw data collection.
Q26 Over what timeframe was the data collected? Does this timeframe match the creation timeframe of the data associated with the instances (e.g., recent crawl of old news articles)? If not, please describe the timeframe in which the data associated with the instances was created.
N/A. Our benchmark does not include new raw data collection.
Q27 Were any ethical review processes conducted (e.g., by an institutional review board)? If so. please provide a description of these review processes, including the outcomes, as well as link or other access point to any supporting documentation.
N/A. Our benchmark does not include new raw data collection.
Q28 Does the dataset relate to people? If not, you may skip the remaining questions in this section.
No.
Q29 Did you collect the data from the individuals in question directly, or obtain it via third parties or other sources (e.g,, websites)?
N/A. Our dataset does not relate to people.
Q30 Were the individuals in question notified about the data collection? so, please describe for show with screenshots other information) how notice was provided, and provide a link or other access point to, or otherwise reproduce, the exact language of the notification itself.
N/A. Our dataset does not relate to people.
35

---

Published as a conference paper at ICLR 2025
Q31 Did the individuals in question consent to the collection and use of their data? so, please describe (or show with screenshots or other information) how consent was requested and provided. and provide a link other access point otherwise reproduce, the exact language to which the individuals consented.
N/A. Our dataset does not relate to people.
Q32 consent was obtained, were the consenting individuals provided with a mechanism to revoke their consent in the future or for certain uses? If so, please provide description, as well as a link or other access point to the mechanism (if appropriate).
N/A. Our dataset does not relate to people.
Q33
Has am analysis of the potential impact of the dataset and its use on data subjects (e-g.. a data protection impact analysis) been conducted? If so, please provide a description of this analysis, including the outcomes, as well as a link or other access point to any supporting documentation.
We discuss the limitations of our current work the Conclusion and Future Work section of the main paper, and we plan to further investigate and analyze the impact of our benchmark in future work. We acknowledge the potential data biases and limitations of our initial benchmark and have detailed the assumptions made during benchmark construction Sec. B.1 of the Appendix.
Q34 Any other comments?
No.
C.4
PREPROCESSING, CLEANING, AND/OR LABELING
Q35
Was any preprocesingddeannngllbbee ing, tokenization, part-of-speech tagging, SIFT feature extraction, removal of instances, processing of missing values)? If so, please provide description. If not, you may skip the remainder of the questions in this section.
of the data done (e.g., discretization or bucket-
No preprocessing or labeling was performed.
Q36 Was the "raw" data saved in addition the preproessal/deanddhhheel data (e-g., to support unanticipated future uses)? If so, please provide link or other access point to the "raw" data.
N/A. No preprocessing or labeling was performed for creating the scenarios.
Q37 Is the software used to preprocess/clenn/lbell the instances available? If so, please provide a link or other access point.
N/A. No preprocessing or labeling was performed for creating the scenarios.
Q38
Any other comments?
No.
C.5
USES
Q39
Has the dataset been used for any tasks already? If so, please provide a description.
Not yet. We present a new benchmark.
Q40 Is there a repository that links to any or all papers systems that use the dataset? If so, please provide a link or other access point.
We will provide links to works that use our benchmark at [Anonymous Repo].
Q41 What (other) tasks could the dataset be used for?
Our benchmark primarily studies the robustness of dense dense Neural SLAM models under perturbations.
Its customizable nature allows for future research on the robustness of various SLAM systems and other RGB-D sensing-related tasks.
36

---

Published as a conference paper at ICLR 2025
Q42 Is there anything about the composition of the dataset or the way it collected and prepoocesed/cleene//bbbeee anything that future user might need know to avoid uses that could result in unfair treatment of individuals or groups (e.g,, stereotyping. quality of service issues) or other undesirable harms (e.g.. financial harms, legal risks)? so, please provide description. Is there anything a future user could do to mitigate these undesirable harms?
that might impact future uses? For example, is there
No.
Q43 Are there tasks for which the dataset should not be used? If so, please provide description.
No.
Q44 Any other comments?
No.
C.6
DISTRIBUTION AND LICENSE
Q45 Will the dataset be distributed third parties outside of the entity (e.g., company, institution, organization) on behalf of which the dataset was created? If so, please provide a description.
Yes, this benchmark has been open- ppen-sourced.
Q46 How will the dataset be distributed (e.g., tarball on website, API, GitHub)? Does the dataset have digital object identifier (DOI)?
Our benchmark and the code used for evaluation are available at [Anonymous Repo].
Q47
When will the dataset be distributed?
Sep, 2024, and onward.
Q48 Will the dataset be distributed under a copyright or other intellectual property (IP) license, and/or under applicable terms of use (ToU)? so, please describe this license and/or ToU, and provide link or other access point to, or otherwise reproduce, any relevant licensing terms or ToU, as well as any fees associated with these restrictions.
The 3D scene dataset Replica and the benchmarking methods used in our benchmark sourced from existing associated with them is followed accordingly.
source repositories, illustrated in Sec. O. The license
Our code is released under the Apache-2.0 license.
Q49
Have any third parties imposed IP-based or other restrictions on the data associated with the instances? If so, please describe these restrictions, and provide a link or other access point otherwise reproduce, any relevan! licensing terms, well any fees associated with these restrictions.
We release it under the Apache-2.0 license.
Copyright for the original 3D scenes used for rendering is not owned by us.
Q50 Do any export controls or other regulatory restrictions apply to the dataset or to individual instances? so, please describe these restrictions, and provide link or other access point to, or otherwise reproduce, any supporting documentation.
No.
Q51 Any other comments? No.
C.7 MAINTENANCE
Q52 Who will be supportinghasttngmmiiaaaiinngg the dataset?
Anonymous Author
Q53 How the owner/curaaoor/manager of the dataset be contacted (e.g., email address)?
Anonymous Author
37

---

Published as a conference paper at ICLR 2025
Q54 Is there erratum? If so, please provide a link or other access point.
There no erratum for initial release. Errata will be documented future releases on the benchmark website.
Q55 Will the dataset be updated (e.g,, to correct labeling errors, add new instances, delete in- stances)? If so, please describe how often, by whom, and how updates will be communicated to users (e.g., mailing list, GitHub)?
Yes, benchmark will be updated. We plan to expand scenarios, metrics, and models to be evaluated.
Q56 If the dataset relates to people, are there applicable limits on the retention of the data associated with the instances (e.g., were individuals in question told that their data would be retained for fixed period of time and then deleted)? If so, please describe these limits and explain how they will be enforced.
N/A. Our dataset does not relate to people.
Q57 Will older versions the dataset continue to be supporte/hnstelmaattannee? If so, please describe how. If not, please describe how its obsolescence will be communicated to Nsers.
We will host other versions.
Q58 If others want to extend/augmeet/huill on/contribute to the dataset, is there mech- anism for them to do so? If so, please provide a description. Will these contributions be validated/verified? so, please describe how. If not, why not? Is there a process for gonmuunicuuing//ittrbbuuuu description.
these contributions to other users? If so, please provide
Users may contact us by reporting an issue on our benchmark GitHuub page or directly contacting the author of this project ([Anonymous Author]) to request adding new scenarios, metrics, or models.
Q59 Any other comments?
No.
38

---

Published as a conference paper at ICLR 2025
D ADDITIONAL RESULTS AND ANALYSES ON Robust-Ego3D BENCHMARK
D.1 MORE BENCHMARKING ANALYSES
(a) Perturbation Mode: Static
Stpps shows sudisteelly sguufiiant 1p (p-alar-s2.111 parfomaaneo degradttion
Log (ATE RMSE)
petarfetice for al
Ganssion Blur
Close, Blur
Caussion Nukee
Shol, Noise
Impuise Nois
Speakie, Noise
Fog
Snow
Spatter
Be Beighnes
Contrast
JUSE Comgression
Pixdute
None
Maiin,
Detees
Perturbation Type
Perturbation Mode: Dynamic
12types shons staistically signiciont
Log (ATE RMSE)
(p-aalas-0.0 21) perfomanno degiasation
No pettriation inberpartle range jarmleney
Motinn Blur
Delocu Blur
Guassian Blor
Closs Blu
Ciussiinn Nokee
Noise
Impuise Noise
Sporkle Noive
Snow
Spatter
Brighmoss
JpsR Comprossion
Phoeltt
Nooe
Perturbation Type
Figure D9: Effect of each RGB imaging perturbation type on the trajectory estimation performance of SplaTAM-S (Keetha et al., 2024) model, which shows the best overall performance under RGB imaging perturbations among all the benchmark methods. The t-test (Kim, 2015) is performed to compare the performance distribution between each perturbed setting and the perturbation-free setting (which denoted as None in the last column of each sub-figure). ** and * indicate a significant distribution difference between the pair at the 0.01 and 0.05 significance level, respectively.
More analyses on the most robust SLAM model under RGB imaging perturbation, i.e., SplaTAM-S. Even the top-perforning SplaTAMMS model, which achieves the best average per- formance under RGB imaging perturbations, experiences a more substantial decrease in trajectory estimation accuracy under dynamic conditions, with statistically significant differences observed for most of the tested perturhation types (see Fig. D9). Interestingly, increased brightness, while slightly beneficial under static conditions, leads to significant errors under dynamic conditions. In Fig. D10, we examine the impact of perturbation strength on SAM-TAM-S. While the trajectory estimation error and geometry rendering quality (measured by Depth LI) remain mostly stable, the rendered RGB image quality (measured by PSNR, MS-SSIM, and LPIPS) degrades significantly as perturbation severity increases.
39

---

Published as a conference paper at ICLR 2025
MS-SSIM
LPIPS
Def8 MeDlont
6.30
6.20
6.22
2.0
122
1.5
0.56
0.48
13.5
0.40
31.5
SpecN ImpuN Show GAVe GleB Ceud
056
040
040
0.365
6.30
052
0.4
0.66
0.00
0.0
04
0.6
30
186
0.00
5200
Frest
Contra Bright Spatter Show
0.00
0.66
88
0.55
024
0.52
0.88
0.56
10.2
76.5
6.2
27.3
Rieel 10eg
29.6
0.5t
25
$3.0
0.540
0.335
Low
Mid
High
Low
Mid
High
Low
Mid
High
Low
Mid
High
Low
Mid
High
Figure D10: Effect of varied perturbation severity of static RGB imaging perturbations on perfor- mance of trajectory estimation (measured by ATEJ (cm)), RGB rendering quality (measured by PSNR1. MS-SSIM", and LPIPSJ), and depth rendering quality (measured by Depth LI loss (cm)) for SpM.TAM-S. The fitted trend lines illustrated in blue.
40

---

Published as a conference paper at ICLR 2025
Table D4: 3D metries for evaluation of mesh reconstruction quality of the reconstructed 3D mesh P when given the ground-truth 3D mesh Q (in the scale of meter [m]). We follows the 3D reconstruction metrics defined the CO-SLAM (Wang et al., 2023) paper.
3D Reconstruction Metric
Definition
Accuracy (ACC)
Σ meP (minge Ilp 9113)
Completion (Comp.)
Σyeq (minper || - g/12)
Completion Ratio (Comp. R.)
1 [Q]
geq (minper ||p a/12 S 0.05)
Table D5: Effects of RGB imaging perturbation on mapping for CO-SLAM.
Clean Mean
Low Severity
Middle Severity
High Severity
Perturb. Mean
Metrics
ACCJ [cm]
2.08
2.11
2.12
2.39
2.21
Comp- [cm]
2.17
2.19
2.20
2.89
2.43
Comp. R. [%]
93.13
93.07
93.04
92.34
92.82
Effect of RGB imaging perturbations on the geometry quality of mapping. We follow the mapping quality evaluation protocol in (Wang et al., 2023) to assess 3D reconstruction using Accuracy (ACC) [em], Completion (Comp.) [cm]. and Completion Ratio (Comp. R.) [%] with 5 cm threshold. Table D4 details the definition for each of these metrics. Note that only certain dense SLAM models can produce 3D reconstruction results for further evaluation of mapping quality. In Table D5, we evaluate the impact of image perturbations on the mapping quality of the CO-SLAM (Wang et al., 2023) model, which shows strong robustness to most of the image-level corruptions. The results reveal direct correlation between perturbation severity and both 3D reconstruction error and completion error. Specifically, the clean setting achieves the highest accuracy (2.08 cm) and the lowest completeness score (2.17 cm), while the high perturbation severity setting exhibits the highest errors in ACC (2.39 cm) and completion (2.89 cm). Overall, our analysis shows that increasing severity levels of perturbation lead to larger errors in the reconstructed 3D map.
41

---

Published as a conference paper at ICLR 2025
Ground truth (for reference)
Original Speed (1x)
Faster Motion (2x)
Faster Motion (4x)
RCB Image
PSNR: 29.10
PSNR: 28.33
PSNR: 21.54
Recondmicttri Calture
Mapth
L1: -0.00
L1: 0.01
L1: 0.02
Rocasttattio Falure
Figure DI1: Effect of faster motion on the 2D reconstruction losses of RGB images (Left) and depth maps (Right), which are measured via PSNR and Depth LI loss, for SplaTAM-S (Keetha et al., 2024).
8X
2x
Log (ATE RMSF)
Log (ATE RMSE)
1X
@:oogoo
-2
8X
2X
1x
Correlation: -0.36
Correlation: 0.89
10
15
20
25
30
35
40
-3.0
-25
-2.0
-1.5
1.0
0.5
Image PSNR
Log (Depth LI)
Figure D12: Correlation between ATE (logarithm form) and the 2D reconstruction losses of RGB images (Left) and depth maps (Right), which are used for model optimization, under faster motion effects for SplaTAM--S (Keetha et al., 2024). Pearson correlation coefficient (Cohen et al., 2009) is reported in the bottom-right corner.
There exists SLAM models that can perceive perturbed observations. We conduct case study to explore the ability of SplaTAM-S model to perceive the severity of perturbations. Notice that SplaTAM-S optimizes the map and 3D pose by minimizing the 2D reconstruction loss for the RGB and depth maps during inference. In Fig. D11 and Fig. D12, we observe a strong correlation between the accuracy of the final trajectory estimation and the RGB-D reconstruction loss. This indicates that when the model produces a larger reconstruction loss for certain sensor stream, it is likely that the trajectory estimation is also inaccurate. While this doesn't t provide exact localization, serves valuable indicator of potential observation degradation and model failure. This suggests SLAM systems could self-mot monitor performance using internal indicators, enabling real wal--imme failure mitigation in safety critical applications.
42

---

Published as a conference paper at ICLR 2025
(a) Perturbations on RGB Imaging
Motion Deviations on Sensor Pose
OCTS uARte Cide
ORI-EEAND Niteol -1
DES-SLAND (Urntt
1211
(c) Perturbations Depth Imaging
Porturbations
Synchrosbaaton
Dre
606
Rale
Figure D13: Performance (measured by ATEJ (m), RPE, (m), and SR1) of ORB-SLAM3 (Campos et al., 2021) under diverse perturbations. For visualization, sequences resulting in failure are assigned an ATE/RPE value of 1.0 and a Success Rate of O.
D.2 CLASSICAL SLAM ORB-SLAME RESULTS (FOR REFERENCE)
To better understand the strengths and weaknesses of learning Barning-based dense SLAM models, we also include results from the classical SLAM model and comparison. We evaluate its performance using Absolute Trajectory Error (ATE), Relative Pose Error (RPE), and Success Rate (SR). SR is defined the ratio of the cumulative sum of Euclidean distances between consecutive estimated poses to the cumulative sum of Euclidean distances between consecutive ground truth poses. Formally, the SR metric is defined as:
(Campos al., 2021) for reference
Vi=1 lpsst p Di=1
SR
(D10)
10$
gt
where || denotes the Euclidean distance.
Lower ATE&RPE (L) and higher SR (f) indicate better performance, with ORB-SLAMB' results under various perturbations presented in Fig D13. Since ORB-SLAME lacks photorealistic 3D
mapping capability, it is excluded from the main comparison, which focuses on dense Neural SLAM models designed for high-fidelity reconstructions.
43

---

Published as a conference paper at ICLR 2025
D.3 MORE QUALITATIVE ANALYSES OF MODEL UNDER PERTURBATIONS
Nice-SLAM excels in 3D geometry reconstruction under shot noise but fails in color detail re- construction and loses tracking under rapid motion. 1) Fig. D14 showcases the 3D reconstruction and trajectory estimation results of the Nice-SLAM (Zhu et al., 2022) model under varying levels of shot noise perturbation on the RGB image. Nice-SLAM consistently produces high-quality geometry reconstructions even when subjected to high severity levels of shot noise, which we attribute to the nearly error-free, free, unperturbed depth map aiding geometry reconstruction. However, we observe that the model struggles to accurately predict and reconstruct appearance details. Consequently, as the noise in the RGB images intensifies, color reconstruction quality diminishes. 2) Fig. D15, Fig. D16, Fig. D17, and Fig. DI8 demonstrates the complete failure of the Nice-SLAM model in reconstructing 3D geometry and maintaining tracking under rapid motion.
SplaTAM-S shows strong robustness in 3D reconstruction under motion blur but fails under severe contrast reduction, losing tracking and reconstruction capabilities. 1) Fig. D19 presents the qualitative results of the SplaTAM-S (Keetha et al., 2024) model under different severity levels of motion blur image-level perturbations. The trajectory estimation reveals that, in the absence of perturbation with low levels of motion blur, the model produces smooth trajectories. However, as perturbation severity increases to a moderate or high level, the predicted trajectory exhibits more deviations. Notably, the 3D reconstruction consistently maintains high quality despite the increasing blurring caused by observation degradation. 2) Fig. D20 and Fig. D21 depict failure instances of the SplaTAM-S (Keetha al., 2024) model. These failures occur when subjected varying levels of contrast decrease image-level perturbation under both static and dynamic perturbation modes. Higher severity levels of perturbation result in complete tracking loss and reconstruction failure.
NeRF as a locally noise-tolerant representation (for certain perturbations like Spatter Effect) for iMAP and Nice-SLAM. 1) The RGB-D rendering and 3D reconstruction results depicted in Fig. D22, Fig. D24, Fig. D23, and Fig. D25 demonstrate the impressive local de-noising capabilities of iMAP (Sucar et al., 2021) and Nice-SLAM (Zhu et al., 2022) under severe spatter noise RGB imaging. Both methods generate RGB images and reconstruct 3D meshes that nearly free of spatter, showcasing their robustness in handling local disturbances. 2) iMAP, as a NeRF-based SLAM system, leverages the power of neural representations effectively capture scene features despite significant noise. Its feed-forward network (FFN) learns compact and noise-robust representations, allowing adaptive refinement and selective artifact suppression. This capability highlights the importance of neural representation in achieving effective local de-noising. 3) Similarly, Nice-SLAM also shows strong local noise-tolerant capabilities, benefiting from the strengths of the iMAP model. Both methods exemplify the role of neural representations in maintaining clean visual data under challenging conditions.
ORB-SLAM3's reliance on handcrafted ORB feature detection makes it highly sensitive to image perturbations, leading increased trajectory estimation error (ATE) compared to the more robust, learned features used by Neural SLAM methods. While ORB-SLAM3 demonstrates resilience to certain types of image corruption such as brightness changes and defocus blur (Fig.D27), its performance significantly degrades under perturbations that impair ORB feature detection. As shown in Fig. D28, increasing levels Gaussian blur result in a noticeable reduction in the number of detected ORB features. This reduction correlates with a rise in ATE, as depicted in Fig. D29, indicating that the degradation of feature descriptors directly impairs trajectory estimation accuracy. Noise-related perturbations have an even more pronounced effect, often causing complete tracking failure due to the susceptibility of handcrafted keypoints to noise (Fig. D30). In contrast, dense Neural SLAM methods employ learned features that adapt better to noise and maintain robustness under challenging visual conditions, highlighting key limitation of traditional feature-based SLAM models like ORB-SL RRB-SLAM3.
44

---

Published as a conference paper at ICLR 2025
No Perturbation (clean)
Shot Noise (Low Severity)
Dynamic Perturbation Mode
Shot Noise (Mid Severity)
Shot Noise (High Severity) Dynamic Perturbation Mode
Dynamic Perturbation Mode
Poor appearance recosstuction.
Good geometry mocnstraction.
Ground-truth Trajectory
Trajectory
Figure D14: Nice-SLAM (Zhu al., 2022) excels in 3D geometry reconstruction under Shot Noise RGB imaging perturbations but fails in color detail reconstruction under high-severity of perturbations.
45

---

Published as a conference paper at ICLR 2025
Original Speed (Ix)
Faster Motion (2x)
Faster Motion (4x)
Fail to reconstruct!
Fail reconstruct!
Ground-truth Trajectory
Estimated Trajectory
Figure D15: Qualitative failure results of Nice-SLAM (Zhu et al., 2022) under fast motion.
Original Speed (Ix)
Faster Motion (2x)
Faster Motion (4x)
Degraded trajectory estimation!
Fail to reconstruct!
Ground-truth Trajectory
Estimated Trajectory
Figure D16: Qualitative failure results of Nice-SLAM (Zhu et al., 2022) under fast motion.
46

---

Published as a conference paper at ICLR 2025
Original Speed (1x)
Faster Motion (2x)
Faster Motion (4x)
Degraded trajectory estimation!
Fail to reconstruct!
Ground-truth Trajectory
Estimated Trajectory
Figure D17: Qualitative failure results of Nice-SLAM (Zhu et al., 2022) under fast motion.
Original Speed (Ix)
Faster Motion (2x)
Faster Motion (4x)
Fail to reconstruct!
Fail reconstruct!
Ground-truth Trajectory
Estimated Trajectory
Figure DI8: Qualitative failure results of Nice-SLAM (Zhu et al., 2022) under fast motion.
47

---

Published as a conference paper at ICLR 2025
No Pertarbation (Clean)
Motion Blur (Low Severity)
Static Penturbation Mode
Smoother trajectnry prediction
Smoother trajectnry prediction
Motion Blur (Mid Severity)
Motion Blur (High Severity) Static Perturbation Mode
Static Perturbation Mode
More deviated trajectory prediction
More deviated trajectory prediction
Camera Pose
Estimated Trajectory
Figure D19: Qualitative results successful cases of SplaTAM-S model (Keetha et al., 2024)
with RGB-D input.
48

---

Published as a conference paper at ICLR 2025
No Perturbation (Clean)
Contrast Decrease (Low Severity) Dynamic Perturbation Mode
Contrast Decrease (Mid Severity)
Contrast Decrease (High Seyetity)
Dynumic Perturbation Mode
Dynamic Perturbation Mode
Fail track and reconstruct!
Camera Pose
Estimated Trajectory
Figure D20: Qualitative results of the failure cases of SplaTAM-S model (Keetha et al., 2024)
with RGB-D input.
49

---

Published as a conference paper at ICLR 2025
Perlarbation (Clean)
Contrast Decrease (Low Severity) Static Perturbation Mode
Contrust Decreas (MId (areii
Contrast Decreuse (High Severity)
Static Perturbation Mode
Static Perturbation Mode
Fail track and reconstruct!
Fail to track and reconstruct!
Camera Pose
Estimated Trajectory
Figure D21: Qualitative results of the failure cases of SplaTAM-S model (Keetha et al., 2024)
with RGB-D input.
50

---

Published as a conference paper at ICLR 2025
Input Depth
Generated Depth
Depth Residual
Input RGB
Generated RGB
RGB Residual
Input Depth
Generated Depth
Depth Residual
Input RGB
Generated RGB
RGB Residual
Figure D22: Intriguing local de-noising capability of iMAP (Sucar et al., 2021) under severe Spatter effect perturbations in RGB imaging. The generated RGB, rendered from the implicit representation, is nearly free of spatter noise.
51

---

Published as a conference paper at ICLR 2025
No Perturbation (for reference)
Brightness Change RGB Perturbation
Spatter Effect RGB Perturbation
Low
Middle
High
Figure D23: Local de-noising capability of iMAP (Sucar et al., 2021) under different severities of Spatter Effect perturbations. The reconstruction is nearly spatter-free (Left), but global perturbations like brightness changes (Right) cause color shifts in the reconstructed 3D mesh.
52

---

Published as a conference paper at ICLR 2025
Input Depth
Generated Depth
Depth Residual
Input RGB
Generated RGB
RGB Residual
Input Depth
Generated Depth
Depth Residual
Input RGB
Generated RGB
RGB Residual
Figure D24: Intriguing local de-noising capability of Nice-SLAM (Zhu et al., 2022) (similar to iMAP) under severe Spatter effect perturbations in RGB imaging. The generated RGB, rendered from the implicit representation, is nearly free of spatter noise.
53

---

Published as a conference paper at ICLR 2025
No Perturbation (for reference)
Spatter Effect RGB Perturbation
Brightness Change RGB Pertrbation
Low
Middle
High
GT Trajeccyyyy
Pred Trajectory
Figure D25: Local de-noising capability of Nice-SLAM (Zhu et al., 2022) (similar iMAP) under different severities of Spatter Effect perturbations. The reconstruction is nearly spatter-free (Left), but global perturbations like brightness changes (Right) cause color shifts in the reconstructed 3D mesh.
RGB: Show Effect
RGB: Motion
RGB: Gaussian Noise
RGB: JPE
Depth: Gaussian Noise
Observed
Rendered
Figure D26: Intriguing local de-noising capability of SplaTAMSS (Keetha et al., 2024) under mixed RGB and depth imaging perturbations (composed from left to right). The RGB frames rendered from the explicit neural representation, i.e,, 3D Gaussian Splatting (Kerbl al., 2023), is with less noise than the original noisy observations.
54

---

Published as a conference paper at ICLR 2025
Shot, Noise (Mid Severity)
Defncns Blur (Low Severity)
Brightness (High Severity)
Defiiess Blur (High Severity)
Figure D27: Qualitative results of the successful cases of ORB-SLAM3 model (Campos et al., 2021) with RGB-D input.
Shot Noise (Mid Severity)
Effect (High Severity)
Lose track!
Lose track!
Impoiie Noise (High Severity)
Vog Effect (High Severity)
Lose track!
Lose track!
Figure D28: Sensitivity of ORB features under perturbations in ORB-SLAM3 (Campos et al., 2021), which can lead to complete tracking loss and failure.
55

---

Published as a conference paper at ICLR 2025
Original Clean Image
Geussian Blur Severity)
Gaussian Blur (Mid Severity
Gaussian Blur (High Severity)
ATE RMSE: 0r062778
ATE RMSE. 0.06295
ATE R 0:00630
ATE RMSE. 0:04496
#ORB Peahur:: 1005
#OBB Feature: 1004
ORB Feature: 965
# ORB Fealure: 445
Figure D29: Effect of Gaussian Blur image-level perturbation under different severity (Top) on the quality of detected ORB features (Bottom), which are marked as green dots, for the classical SLAM model ORB-SLAM3 (Campos al., 2021). We report the average trajectory accuracy via ATE RMSE and the average number of ORB features detected in various perturbed settings.
Sevenity Low
Severity Low Middle
Middle
(ATE RMSB)
High
Log RMSE)
High
-3
Correlation: -0,48
Correlation: "0.42
400 500
600
700
500
900
1000
150
260
250
300
350
400
Average Detected ORB Feature Number
Average Tracked Feature Number
Figure D30: Correlation between trajectory estimation accuracy and the average number of detected (Left) and tracked (Right) ORB features for ORB-SLAM3 (Campos et al., 2021) model (RGB-D setting) under different severity level of Gaussian Blur image-level perturbation. We report the Pearson correlation coefficient (Cohen et al., 2009) at the bottom right corner. While the correlation coefficient does not indicate a significant linear correlation, there is a noticeable trend of increased trajectory estimation when fewer ORB features are detected tracked.
56

---

Published as a conference paper at ICLR 2025
E THEORETICAL INSIGHTS ON Robust-Ego3D BENCHMARKING RESULTS
E.1 POTENTIAL. BENEFITS OF IMAGE RESTORATION ON OPTIMIZATION
We first analyze how restoring noisy RGB images affects the optimization process in differentiable rendering-baan basnn SLAM.
Theorem A. Restoring noisy RGB images reduces the variability in the residuals used in differentiable handering-based SLAM, leading more stable gradient computations and more efficient convergence.
Proof. Let P. = {Rc, }t} represent the camera pose at time step t. The photometric loss for the RGB image is:
(c)P:) = [, - 1,1.
(BI1)
where I, is the rendered image from the map M given the pose Pe, and I, is the observed image. Assume the observed RGB image is corrupted by additive Gaussian noise:
I 1 No. N, OO
(E12)
where Ii is the true image without noise, and 0² is the variance of the noise.
The gradient of the RGB loss with respect to the pose is:
Vp.Cc = 2(1, VPIL.
(E13)
The noise N₂ affects the residual (f, I,,, introducing variability into the gradient, which can hinder optimization due to the stochastic nature of the noise.
By applying restoration mechanism to obtain denoised image I, reduce the noise component:
I Iuw
(E14)
Using the denoised image, the gradient becomes:
VPiCo 2(I, 1. VPI..
(E15)
Since (i, I,) has reduced noise, the gradient is more stable and accurately reflects the discrepancy between the rendered and true images. This leads to more efficient convergence during optimization.
E.2 EFFECT OF MISSING AND NOISY DEPTH DATA
Next, we examine how missing and noisy depth data affect the gradient computation and the stability of the optimization.
Theorem B. differentiable rendering based optimization for RGB-D SLAM, missing depth data does not introduce noise into the gradient computation and thus does not lead to instability in the optimization procexs.
Proof. Let Ω be the set of pixels where depth measurements available. The depth loss is computed over Q:
1
CD(P))
[2 -D,))?
(E16)
where Dr the rendered depth pixel and Dr. is the observed depth.
The gradient of the depth loss with respect to the pose is:
2 Pp Ω Σ Dp,
(E17)
Since missing depth data is excluded from Ω. it does not contribute to the loss its gradient. Therefore, the absence depth measurements does not introduce any noise or instability into the gradient computation, allowing the optimization process to remain stable.
57

---

Published as a conference paper at ICLR 2025
Theorem C. In differentiable maddring-basedd optimization for RGB-D SLAM, noisy depth inputs introduce variability into the residuals used for gradient computation, which can lead to misaligned pose updates and instability in the optimization process.
Proof. Assume the observed depth map is corrupted by additive Gaussian noise:
D: D + No, N, ~ ~o<,N0,,
(B188
where Di is the true depth map, and of is the variance of the depth noise.
The gradient of the depth loss with respect the pose is:
Vp, = 2(D, - D.)
(E19)
The noise N₂ affects the residual (D: - D.), introducing variability into the gradient. This noisy gra- dient misalign pose updates because the optimization is guided by inaccurate depth discrepancies.
When AD is large, the optimization relies heavily on the depth loss. The variability in the gradient due to noisy depth can lead incorrect pose updates, causing instability and possibly divergence in the optimization process.
E.3 EFFECT OF LARGE POSE CHANGES ON OPTIMIZATION
We analyze how large pose changes impact the efficiency and convergence of gradient-based opti- mization in differentiable rendering.
Theorem D. Large pose changes differentiable rendering-basedd optimization can lead large residuals and gradients, causing inefficiencies in gradient-based optimization and increasing the risk of divergence.
Proof. Consider a camera moving with large pose change AP = (AR, At]. The rendered images I and D depend the pose P and the map M:
i = Fremter ( (M,P) D Frender (M,P).
(F20)
With large pose changes, the projections of the Gaussians onto the image plane can change drastically, leading large discrepancies between the rendered and observed images. This results in large residuals in the loss function:
C(P) = + -
(E21)
Consequently, the gradients of the loss with respect to the pose parameters become large:
VpL = [>o A D) Vp
(E22)
If the optimization uses fixed learning rate y/. large gradients can cause the pose update step to overshoot the minimum:
Pa+1 = Ps - - (PPt))
(F23)
This overshooting can lead to divergence or oscillations in the optimization process. To handle large pose changes efficiently, adaptive step sizes robust optimization methods (e.g., using line search or second-order methods) are required ensure convergence.
E.4 INVALID LINEAR MOTION ASSUMPTION UNDER COMPLEX MOTION
We now introduce theorem to support the observation that differentiable rendering optimization fail to tackle dynamic, non-lineat motion due to invalid linear motion assumptions.
Theorem E. Differentiable rendering -baeed optimization methods that rely on linear motion assump- tions can fail to converge under dynamic. non-linear motion due the inability of linear models to accurately capture higher-ordes order motion dynamics. leading to large residuals and misaligned gradients.
58

---

Published as a conference paper at ICLR 2025
Proof. In differentiable cendering-basee SLAM, many methods assume that the camera motion between consecutive frames is approximately linear and smooth. This assumption simplifies the motion model and is reasonable for slow and steady movements. The pose at time t is often predicted using a first-order approximation:
P. Pa-1 AP;
(H24)
where @ denotes the pose composition, and AP is small incremental motion estimated under the linear motion assumption.
Under dynamic, non-linear motion, the actual motion APie includes higher-order dynamics such as acceleration and jerk, which are not captured by the linear model:
A Irue
Pinner
linear
Ponnineerr:
(E25)
where AP nonlinear represents higher-order motion components.
The linear motion assumption leads to an inaccurate initial pose estimate:
Pt Pt-1 1 B A lineur-
(E26)
The discrepancy between the estimated pose Pest and the true pose Ptue introduces large residuals in the loss function:
[(P: Ac (P::*) I.l² ADID(P(tt) D.12.
(E27)
These large residuals result in large gradients:
vv = (
(E28)
Due to the significant pose discrepancy, the gradients may point in incorrect directions because the linear approximation does not align with the true motion trajectory. The optimization process, relying on these misaligned gradients, can fail to converge even diverge.
Therefore, under dynamic, non-linear motion, differentiable rendering-bant -baedd optimization methods that rely linear motion assumptions prone to failure because they cannot accurately capture higher-orde motion dynamics, leading to ineffective optimization.
59

---

Published as a conference paper at ICLR 2025
F DETAILED TABLES OF RESULTS ON Robust-Ego3D BENCHMARK
To mitigate potential randomness, for each perturbed setting, we conduct each experiment three times on eight 3D scenes (totaling 24 experiments per perturbed result) and report the averaged results. Specifically, for the RGB imaging perturbation, we present the averaged result across three severity levels, under both static and dynamic perturbation modes. This approach reduces the impact of randomness while maintaining computational efficiency, striking balance between mitigating randomness and ensuring feasibility.
To facilitate future quantitative comparison on our benchmark, we have provided additional detailed benchmarking tables of RGB-D SLAM methods in this section, which includes:
Neural SLAM methods under depth imaging perturbation (Table F6).
Neural SLAM methods under the faster motion effect (Table F7).
Neural SLAM methods under motion deviations (Table F8).
Neural SLAM methods under RGB-D desynchromization (Table F9).
ORB-SLAM3 under RGB imaging perturbation (Table F10,Table F11, Table F12).
ORB- BRBB-LLA33 under depth imaging perturbation (Table F13, Table F14, Table F15).
ORB- BRAM under the faster motion effect (Table F16, Table F17, Table F18).
ORB-SLAM3 under the motion deviations (Table F19, Table F20, Table F21
ORB-SLAM3 under RGB-D desynchronization (Table F22, Table F23, Table F24).
Furthermore, we have noticed large performance deviations with each perturbed setting for the ORB-SLAM3 model. To reflect potential randomness, we have included the performance standard deviation of this model our experiment results. We acknowledge these potential performance deviations, which indicate low model robustness phenomenon work aims highlight encourage the SLAM community to develop more robust and stable SLAM systems.
Table F6: Performance (ATE [m]) of dense Neural SLAM models under depth imaging perturbation.
Gaussian Noise
Edge Erosion
Random Missing
Range Clipping
Method
Clean
iMAP (RGB-D) (Sucar et al., 2021)
0.1209
0.0307
0.1083
0.2438
Nice-SLAM (RGB-D) (Zhu et al., 2022)
0.0147
X
0.0149
0.0154
0.1183
CO-SLAM (RGB-DD -D) (Wang al., 2023)
0.0090
0.5794
0,0096
0.0094
0.0122
GO-SLAM (RGB-D) (Zhang et al., 2023)
0.0046
0.0378
0.0046
0.0046
0.0045
SplaTAM-S (RGB-D) (Keetha et al., 2024)
0.0045
X
0.0042
0.0042
0.0048
X indicates completely unacceptable, Le., performance (ATE 2 1.0 [m])
60

---

Published as a conference paper at ICLR 2025
Table F7: Performance (ATE [m]) of dense Neural SLAM under faster motion.
Speed-up Ratio
1x
2x
4x
8x
GO-SLAM (Mono) (Zhang et al., 2023)
0.0039
0.0042
0.0046
0.0048
iMAP (RGB-D) (Sucar et al., 2021)
0.1209
0.4675
0.9445
1.0000
Nice-SLAM (RGB-D) (Zhu et al., 2022)
0.0147
0.1702
1.0000
1.0000
CO-SLAM (RGB-D) (Wang al., 2023)
0.0090
0.1062
0.9510
1.0000
GO-SLAM (RGB-D) (Zhang et al., 2023)
0.0046
0.0046
0.0046
0.0050
SplaTAM-S (RGB-D) D) (Keetha ct al., 2024)
0.0045
0.0184
1.0000
1.0000
Table F8: Performance (ATE [m]) of dense Neural SLAM models under motion deviations.
Rotate
Clean
Translate [vo]
0,0225
GO-SLAN (Mono)
0.0030/0.0004 0.0077 0. 0.0001
10.083
0.0070 0.0882 0.0094
X
X
X
X
X
X
(RGB-D)
0.120)000034
0.1386 0.0422
0.2136
0.5754 0,2801
X
X
CO-SLAM (RGB-D)
0.5088
0,7632
0.5772
0.8457
0.7986
GO-SLAM (RGB-D)
0.045 0L0022 0.0082 0.00S1 0.0000
0.0080 OUOOTS 0.0077
X
X
X
X
SplaTAM-S (RCIB-D)
0.0045 0.00550.0545 0.0990 0 2064 2.757P 0.2272 0.2313
X
X
X
X
X
X
Notation represents settings that include failure sequenoes where final trajectory genersted duc to tracing less. The number in front of repressens the value Notation indicatis completely unscceptable trajectory estimation performance, Le. ATE 2 (m).
average ATE fuilure sequences
Table F9: Performance (ATE [m]) of dense Neural SLAM models under sensor de-synchronizzttin..
Clean
Static Mode
Dynamic Mode
Method
A 0
A =
A
10
A
20
A
5
A
10
A 20
iMAP (RGB-D) (Sucar et 2021)
0.1209
0.4672
0.5344
0.6345
0.5104
0.6803
0.6745
Nice-SL.M (RGB-D) (7hu et al., 2022)
0.0147
0.3820
0.4062
0.5216
0.5433
0.5548
0.7020
CO-SLAM (RGB-D) (Wang al., 2023)
0.0090
0.0320
0.1005
0.1939
0.0740
0.1164
0.2108
GO-SLAM (RGB-D) (Zhang et al., 2023)
0.0046
0.0148
0.0292
0.0646
0.0151
0.0297
0.0650
SplaTAM-S (RGB-D) (Keetha et al., 2024)
0.0045
0.0554
0.0629
0.0880
0.0402
0.0645
0.0850
denotes the misaligned frame interval between RGB and depth streams.
61

---

Published as a conference paper at ICLR 2025
Table F10: ATE [m] metric of URB-SLAM3 (Campos et al., 2021) under RGB imaging perturbations.
Perturk
Input
Fffect
Naise Rifect
Ravirummental
Interfeeence
Post-prueessigg Fffret
Mode
Modality
[aassian
Impulse Speckle]
Pog
Sputter
Bright
Contra, JPEG
Pinelate
Static
Mena
-0.028
0.014
+0.985
0.891
+-04467
0.585
+0.506
0.505
+0.104
0.797
+0.281
0.917
+:0
0,969
+00000
1,000
+0.201
0,528
+0.400
0.829
+0.250
0.917
+0.000
1,000
+0.610
0.715
+0009
0.027
+0.478
0.612
-0.525
0.177
+0.410
0.76N
RGB
20,177
0.082
#0.066
0.300
20.408
0.380
40.425
0.202
40.180
0.11
$0,488
0.470
4:0,400
0.433
100,00
0.591
40.4600
0.351
0.297
40.474
0.600
20,400
0,798
30.500
0.508
20012
0.005
40.492
0.527
-0.301
0.132
200.354
0,00
Dyuamic
Moni
-0.028
0.014
+0.351
0.886
+0.528
0.506
+0.455
0,200
+0.000
1.000
+0.00
1,000
+0.000
1,000
+0/000
1,000
+0.505
0.885
+0.10]
0.751
+0.000
1.003
+0.000
1.000
+0.000
1.000
H0.02
0.062
+0.000
1.000
-0.511
0.160
+0.412
0.718
RGB
-0,179
0.082
20,030
0.279
20 483
0.305
10.080
0102
10.240
0. 105
230220
0.306
200,20
0,513
11.450
0.755
20.406
0.352
2:0476
0.609
40.400
0.266
20,322
0.876
20.05s
0.733
20.145
0.006
20.402
0.51
-01,01
0.615
200.216
0,736
Table F11: RPE [m] metric of ORB-SLAME (Campos et al., 2021) under RGB imaging perturbations.
Perturh
Input
Clean
Effect
Nuise Effect
Environmental Interference
Post-proessing Effect
Mode
Modality
Motion
Delocas
Gaussian
[Gasssian
Impulse
Speckle]
Spatler
Beight
JPEG
Pisclate
Static
Mono
00003
0.197
100,335
0.863
1.0 00.400
0.805
400410
0:401
10:376
0.311
2.1.247
0.927
4.0.194
0,960
1,00000
0.100
10.:200
0.002
1.0425
0.680
100244
0.907
20,000
1.000
10.380
0.700
1:0028
0.102
110,373
0.009
-0,244
0.217
1.0.480
0.525
RGB-D
-0.023
+0.350
01238
+0.100
0.223
+0.397
+0.096
0.081
+04477
0,49%
+0.479
0,445
+0.482
0
+0.450
01377
+0.159
0,221
+0.170
0.617
+0.400
0,738
+0.08
0.529
H00022
0,078
+0.442
0.570
-0.078
0,080
+0.000
Dynomic...
Meno
-0,030
0.197
10,342
0.879
10.459
0.572
10. 20.403
0.782
10.000
1.000
10,000
1,000
10,000
1,000
10,000
1,000
10.284
11889
10,308
0.785
10,000
1.00
10,000
1,000
10,000
1.000
100044
0.344
10.000
1.000
-0.307
0,312
10.440
0,257
RGB-DD
-0.023
0.114
+0.021
0U043
+0 157
0.295
+0007
8008
+0001
0.045
+0.517
0.517
+0.511
0.528
+0.453
0.755
+0.445
0,234
+0.094
0.642
+0.050
0.272
+0.350
0,876
+0.488
0.758
H0.016
0.074
+0.300
0.792
-0.474
0.656
+0.011
0,031
Table F12: Success rate (SR) of tions.
(Campos et al., 2021) under RGB imaging perturba-
Perturls
Isput
Clean
Blur Elfrct
Noise Efflect
Ravironmental Interfernnce
Post-proeesing Elfeet
Mode
Modality
Mecion
Defocus Guussian
Glass
|Gausian
Shot
Impulse Speckle]
Fog
Frost
Snow
Spatter
Bright Contra,
JPEG
Pixclate
Static
Mono
-0.140
0.854
#0.152
0.650
#0.385
0.331
+0.415
0.382
+0.30
0. 12222
#0.186
0,055
+0.076
0,010
#00000
0,000
+0.185
0,052
+00362
0.229
#0
0.057
197
=0,000
0,000
#0.352
0.211
+0.142
0.915
#0.400
0.520
=11.54
0.712
=00.69
0.061
RGB-D
-0,040
0,990
10.423
0.621
10.443
0.605
10
0.617
500
10:019
0.778
10.408
0,388
10.475
0,391
100.00
0,219
10.457
0,443
10.421
0.276
10.282
0.111
10.220
0,081
0.239
303
10,00
0.971
0.412
100461
-0,284
0,818
10.202
0,361
0.851
0,008
0:907
0.368
0.000
0.006
0.000
0.708
Dynamic
270
40
+0.000
+0.277
+00046
RGB-D
00,00
0.900
10.364
0.871
0.354
10
0.301
0.901
1.0378
0,307
10.044
0,040
10.44
0,303
100045
0.027
10,000
0.005
10,000
0,000
10,020
0.009
1.0007
0.975
10.000
0.003
-0.509
0,370
10.218
0,430
Table F13: ATE metric [m] of ORB-SLAM3 (Campos et al., 2021) with RGB-D input under depth perturbations.
Gaussian Noise
Edge Erosion
Random Missing
Range Clipping
Clean
0.082+0.111 0.179
0.803 + 0.301
0.807 + 0.365
0.756 + 0.397
0.994 + 0.283
Table F14: RPE metric [m] of perturbations.
(Campos et al., 2021) with RGB-D input under depth
Gaussian Noise
Edge Erosion
Random Missing
Range Clipping
Clean
0.114 + 0.023
0.064 + 0.018
0.154 + 0.342
0.054 + 0.022
0.047 + 0.017
Table F15: Success rate (SR) of ORB-SLAM3 (Campos et al., 2021) with RGB-D input under depth perturbations.
Gaussian Noise
Edge Erosion
Random Missing
Range Clipping
Clean
0.960 + 0.046
0.421 + 0.331
0.379 + 0.281
0.322 + 0.354
0.286 + 0.181
62

---

Published as a conference paper at ICLR 2025
Table F16: ATE metric [m] of ORB-SLAM3 (Campos et al., 2021) under fast motion.
Sueed-up Ratio
1
2x
4x
8x
Monocular
0.014 + 0.028
0.009 + 0.009
0.023 + 0.051
0.023 : 0.053
RGB-D
0.082 + 0.179
0.019 0.023
0.064 0.156
0.077 = 0.125
Table F17: RPE metric [m] of ORB-SIAMM (Campos et al., 2021) under fast motion.
Speed-up Ratio
1
2x
xx
8x
Monocular
0.197 + 0.030
0.194 0.035
0.227 + 0.035
0.289 + 0.064
RGB-D
0. + 0.023
0.152 + 0.032
0.190 + 0.030
0.268 + 0.059
Table F18: Success rate (SR) of
(Campos et al., 2021) under fast motion.
Speed-up Ratio
8x
Monocular RGB-D
0.854 + 0.149
0.893 0.081
0.909 + 0.049
0.837 + 0.115
0.960 + 0.046
0.964 + 0.012
0.938 + 0.029
0.866 + 0.129
Table F19: ATE metric [m] of ORB-SLAM3 (Campos et al., 2021) under motion deviations.
Rotate [ong]
Clean
3
Translate
0.0125
0.615
0.6125
0.025
0.05
0.0/25
0.025
0.05
0
0.0125
0.025
Monocalte
+0.028
0.014
+0.027
0.017
-0.349
0.136
+0.330
0.100
+0.047
0.063
+0. 196
0.146
+0.378
0.348
+0.355
0.200
+0.338
0.170
-0.057
0.055
+0.061
0.063
+000
0.061
+0.
0.073
135
+0.030
0.025
+0.022
0.041
+0.108
0.090
RGB-D
0.082
0.191
0.065
10,33
0,171
10.067
0
0.163
0.984
0.228
0.148
10.100
0,158
10.07
0.055
0.264
10.100
0.072
0.060
0.082
Table F20: RPE metric [m] of ORB-SLAM3 (Campos et al., 2021) under motion deviations.
Rotale [6gg]
Clean
3
Translate [m]
0.0125
0.025
0.05
0.0125
0.025
0.05
0.0125
0.025
0.05
0
0.0125
0.025
0.05
0.355
20000
+0.007
-0.280
40.297
#0.071
10.000
#0.294
20,05
10.270
20,86
40.100
20.087
40.002
40.083
20,07
$0.000
RGB-D
+0.033
0.114
+0.313
0.220
-0.08
0.119
+0.322
0.208
+0.039
0.114
+0.088
0.116
+0.046
0.153
+0.313
0.231
+0.046
0.107
0.230
00027
+0.036
0.203
+0.035
0.220
+0.230
0.417
+0.047
0.313
+0.022
0.298
+0.042
0.334
Table F21: Success rate (SR) of
(Campos et al., 2021) under motion deviations.
Rotate [[og]
Clean
Translate
0.0125
0.025
0.05
0.0125
0.025
0.05
0.0125
0.025
0,05
0
0.0125
0.025
0.05
Menocalur
#0.149
0.864
+:0.107
0.489
=0.
0.247
129
+0.091
0.128
#0.329
0.662
+0.
0.340
189
0.221
20,009
0.120
+40320
11377
0.006
154
0,085
#0001
0,88
#0000
0000
00000
0:410
+0.418
#6.160
0.105
#0.112
0.214
+00020
0.144
RGB-D
20,066
0.960
+00.13
0.162
0.5
0.321
0.152
0.506
40.218
0.345
0.228
10.116
6.114
0.325
200
0.259
0.146
0.118
0.422
40.158
0.158
40.148
0.160
0.129
63

---

Published as a conference paper at ICLR 2025
Table F22: ATE metric [m] of ORB-SLAM3 (Campos et al., 2021) with RGB-D input under desynchronization.
Clean
Misaligned Frame Interval (A)
Perturb
Mode
A
A 10
A 20
Static
0.069 + 0.168
0.066 + 0.154
0.065 = 0.163
0.082 = 0.179
Dynamic
0.070 + 0.157
0.077 + 0.161
0.083 + 0.178
Table F23: RPE metric [m] of ORB-SLAM3 (Campos et al., 2021) with RGB-D input under desynchronization.
Clean
Misaligned Frame Interval (A)
Perturb
Mode
À 0
A 5
A = 10
A 20
Static
0.123 + 0.024
0.114 + 0.024
0.115 + 0.023
0.114 + 0.023
Dynamic
0.116 + 0.025
0.112 + 0.019
0.117 + 0.027
Table F24: Success rate (SR) of ORB-SLAM3 (Campos et al., 2021) with RGB-D input under desynchronization.
Clean
Misaligned Frame Interval (A)
Perturb Mode
A
A
A
Static
0.960 + 0.036
0.958 1 0.029
0.954 + 0.030
0.960 = 0.046
Dynamic
0.955 + 0.039
0.942 = 0.050
0.948 = 0.041
64

---

Published as a conference paper at ICLR 2025
G MORE DETAILS ABOUT COREESPONDENGE-QUIEDD GAUSSIAN SPLATTING (CORRGS)
G.1 SCHEMATIC PIPELINE OVERVIEW AND PSEUDOCODE
(a) Correspondence-gudded Pose Learning (CPL)
Learnable
Parameter
Rendered RGB-D
Observed RGB-D
Map
Render Loss (iter >1)
Differentiable
Render
Pose
Pose Update
M Matcher (iter
(leer >1)
Pose Update (tte=11 1)
I. Legend:
Quality Verification
Lift Paired 2D Points 3D
Forward
Gradient Badeprrpegattonn
2D Correspondence
Correspondence- based Pose Init.
(b) Correspondence-muidee Appearance Restoration L Learning (CARL)
Noise-free Dense 3D Representation
Rendered
Observed RGB
Observed RG B
Correspondence
Learnable Restorer
Restoration Loss
Learnable Restorer
Restored RGB
Map Update via Restored Observation
Restored Pixels
Corresponded Observed Pixels
Rundered Pixels
from observed
Image Restoration via Learnt Restorer
Carresoondenceguiieeed Online Restoration Learning
Figure G31: Main components of Correspondence-guide forrespondence-guide. Pose Learning (CPL) for robust tracking. (b) Correspondence- guided Appearance Restoration Learning (CARL) for noise-free and consistent 3D reconstruction.
Gaussian Splatting (CorrGS). (a)
65

---

Published as a conference paper at ICLR 2025
Algorithm CorrGS: Correspondence-Cuuddee Gaussian Splatting SLAM System(pseuudocode)
Require: Noisy RGB image L, depth map D,, camera intrinsics K, historical poses P<t. historical
map Ma (Gaussian Splat representation)
Ensure: Refined camera pose Pe, updated map M₁
1: 1: Naive Pose Initialization
2: Initialize camera pose P. by propagating from historical poses P<t.
3: 2: Correspondence-basee Pose Initialization
4: Render synthesized RGB-D images I, D, using initial pose P, and historical map Mar-
5: Compute 2D correspondences between I, and observed image L,
6: Lift 2D correspondences to 3D correspondences using D,, D,, and K.
7: Update pose P via non-linear optimization over 3D correspondences using prior pose Pe as initialization.
8: 3: Correspondence-basee Pose Quality Verification
9: Compute rendering losses for poses Př and P..
10: Select the pose with the lower loss for further processing.
4: Correspondence-
Appearance Restoration Learning
12: Learn restoration model [( 0) using the color data from the correspondences.
Generate restored RGB frame I.R ().
14: Update observed image I, + 1.
15: Repeat Steps 2-4 for iteration for further refinement.
16: 5: Differentiable Rendering-basee based Pose Refinement
17: Refine pose P. using differentiable rendering- based optimization.
18: 6: Map Update
19: Integrate (restored) observed RGB image I, depth map D,, and refined pose P. into the historical map M <e produce the updated 3D map Mr.
20: return Refined pose Pt. updated foise-free map Mr.
66

---

Published as a conference paper at ICLR 2025
G.2 PRELIMINARIES
Gaussian map representation. We represent the 3D environment using a set of oriented 3D Gaussians, capturing both geometry and appearance. Each Gaussian G; is parameterized its position X; E R&3, covariance matrix Σg E R2x3 (which we simplify to isotropic Gaussians), opacity A, E [0, 1], and RGB color C; E R The simplified map M is defined as:
M (G, (X, A,, c;) = I.....N...
(G29)
Each Gaussian G; contributes to point in 3D space x E IR3 according to the Gaussian function weighted by its opacity:
x X,
f frond(x) Ai exp
(G30)
2r²
where is the isotropic Gaussian radius. This representation supports efficient and differentiable rendering of the scene.
Differentiable moddering-based pose optimization. Our approach enables differentiable rendering of color, depth, and silhouette images from the Gaussian Splat Map into any camera frame. By lever- aging this differentiable process, we compute gradients with respect to both the scene representation M (the Gaussians) and the camera pose P = (R, t). where R. E SO(3) is the rotation matrix and t E R the translation vector. The rendered images can be compared to input RGB-D frames to minimize the error and refine both the scene and camera parameters.
Rendering an RGB image follows the process Gaussian Splatting (Kerbl et al., 2023); given collection of Gaussians, first sort them from front to back and then render the image by alpha- compositing the splatted 2D projections of each Gaussian. The color of a pixel (u, v) is computed as:
N
((p) Σ
c.ffeedd
II
(1-
(G31)
j=1
where fp( computed by projecting the 3D Gaussians into pixel space:
P.X;
2D fr; d.
X,20
d. (P,X,)²
(G32)
d;
Here, K is the known camera intrinsic matrix, P. is the estimated pose, i.e., the camera extrinsic matrix frame f focal length, and d, depth the ith Gaussian primitive.
Similarly, we render the depth image:
D(p) Σ d.f. rend
Il (1 ((od))on(p)))
(G33)
1=1
These rendered images enable the optimization of the camera pose (P) by minimizing loss function L(P):
((P) Xc (D,
(G34)
where Ac, Ap > 0 are weighting parameters for the photometric loss terms Lo and Lp for color and depth, respectively.
The pose is iteratively updated using gradient descent:
PM+1 = Ps - (PPt))
(G35)
where 7) > the learning rate.
67

---

Published as a conference paper at ICLR 2025
Rendered RGB
Observed RGB
Degraded Image Quality
Incomplete & Wrong Region
Figure G32:
Comparison of rendered and observed image quality. The rendered image exhibits degradation due to pose noise, impacting correspondence calculation.
G.3
CORRESOONDECCE-GUDDDD POSE LEARNING (CPL)
G.3.1 NAIVE POSE INITIALIZATION
Pose learning starts by propagating the camera pose from historical poses Pet as the naive initializa- tion P.. Synthesized RGB-D images I, and D, are then rendered using P. and the historical map Mer- These rendered images are used establish correspondences with the observed RGB-D data (I, and D2), refining the pose for more accurate tracking.
G.3.2 2D CORRESPONDENCE CALCULATION
We utilize an off-the-shelf feature matcher (e.g., LoFTR (Sun et al., 2021)) to compute 2D correspon- dences between the rendered İ, and the observed RGB images. These correspondences consist of pairs of 2D pixel coordinates, (u,, v,) for the rendered image and (u,, v.) for the observed image. However, due to feature matching inaccuracies, such as degraded quality in the rendered image (see Fig. G32), errors E2D arise:
(ur, v,) (u,, v²) + E2D.r. (u,, vo) -u', vo) E2D.o>
(G36)
where (u,, v;) and (u",, vo) represent the true correspondences, and e2D,rr 6DD are the 2D matching errors for the rendered and observed images, respectively.
G.3.3 LIFTING 2D CORRESPONDENCES TO 3D
Given the observed depth map D, and rendered depth map D,,, we lift the corresponding 2D pixel coordinates into 3D points via:
u.
pr (, D.
(G37)
P. Do K-1
Vo
(G38)
where K the camera intrinsics matrix, and p, and Pa are the corresponding 3D points in the rendered and observed point clouds, respectively.
Due to errors depth measurements cD, and cD,o, the 3D points become perturbed:
1. + 62D.r
p. = = DD" + EDD K-
K-1
62Drr
(G39)
68

---

Published as a conference paper at ICLR 2025
Rendered RGB
Observed RGB
High
Conrespondence Confidence
Before
After
Low
Figure G33: Effect of one-step pose initialization via correspondence.
11' + <20 P. = DD + (D.
1 E2D.0
(G40)
where D and D are the true depths, and cD,r and cDo are the depth errors.
G.3.4 CORRESPONDENCEF BBASED POSE INITIALIZATION
To estimate the relative pose transformation between the two corresponded 3D point clouds, i.e., the rendered points p, and the observed points Po. we leverage the correspondences to align them. By minimizing the reprojection eπrоr between the two sets of points, iteratively estimate the relative pose transformation. This transformation consists of rotation matrix Rywl and translation vector frat, which describe how the rendered point cloud should be aligned with the observed one. Once the relative transformation is computed, it is used to update the camera's 's pose estimation the current frame, refining the pose incrementally based on the previous pose estimation.
Relative transformation calculation. We begin by solving the following non-linear least squares problem iteratively refine the relative pose:
Σ p CI( + trel - Po..Il").
mii Ryat
(G41)
where Ryet E SO(3) is the relative rotation matrix, tyet R is the relative translation vector, || || denotes the Euclidean norm, and p(-) is the soft.ll loss function. The optimization iteratively refines these parameters to estimate the relative pose between the two point clouds.
Fig. G33 demonstrates that, even in the case of a substantial view change, the image rendered after one step of pose initialization via correspondence closely matches the observed image.
The soft..ll loss function is defined as:
p(s) = 1+ 1 (11 2
(G42)
where ||r,||2 is the squared residual. The smooth L1 loss behaves quadratically for small residuals and transitions to linear behavior for larger residuals, reducing the influence of outliers.
At the start of the optimization, the relative pose is initialized as the identity transformation, meaning no initial rotation or translation is applied. We then iteratively refine the pose by solving for R and tre that minimize the reprojection error. The residuals at each iteration are computed as:
r Rr + tral Paçi
(G43)
69

---

Published as a conference paper at ICLR 2025
where r; is the residual for the i-th correspondence.
The Jacobian of the residuals, J (Ryet tret). is approximated using the "3-point finite-difference method. This involves numerically perturbing Ryet and trel and measuring the corresponding changes in the residuals:
J (Ryets trei) a(Rrel, Eret)
(G44)
The '3-point' method improves the accuracy of Jacobian estimation, leading to better convergence without the need for explicit analytical derivatives.
Starting with the identity transformation (Rrel I, tr! = 0), the optimizer iteratively updates the pose parameters to minimize the total reprojection error. The residuals and Jacobian are recalculated each step until the pose converges to the optimal estimate. This ensures a robust and accurate relative pose estimate, even in the presence of noisy or imperfect correspondences.
Pose update after relative transformation calculation. After estimating the relative pose (Ryet, trel), update the current estimated pose Pf by applying this relative transformation to the previous pose estimation P. The updated pose is computed as:
( RRyel trel 1
P. P
O0r
(G45)
G.3.5 COMPUTATIONAL. COMPLEXITY OF CORRESPONDENCE-GUIDDR INITIALIZATION
Let m denote the number of correspondences, which is bounded by the image resolution, i.e., w, where h and w represent the image height and width, respectively. The number of pose parameters, n, is fixed at 16, corresponding to the elements of a 4 x transformation matrix.
The computational complexity per iteration is primarily determined by two factors: evaluating the residual function and approximating the Jacobian. The total complexity can be expressed as:
(m x n) + O
(G46)
Substituting m = h x w and n 16, this becomes:
O(16
163).
(G47)
Since h >> 16 the dominant term simplifies to: O(h x w).
(G48)
Thus, the computational complexity of the correspondencc--uideed pose initialization is primarily linear with respect to the image resolution.
This method is implemented using the least function from scipy library (Virtanen et al., 2020). For a 1200 x 680 x image, the average processing time is upproximately 10ms on server, demonstrating the method's efficiency even higher resolutions. The computational complexity scales with image resolution, ensuring efficient convergence in pose estimation.
G.3.6 THEORETICAL ANALYSIS SMALL AND LARGE MOTION CASES OF CORRESPONDENCE-BANN BASED POSE INITIALIZATION
Theoretical analysis on the small motion case. When the relative motion between the rendered and observed images small, i.e., R and x 0, the optimization problem becomes highly sensitive to small perturbations in R and t. The transformed 3D points can be approximated as:
Rp + t ~ Pr. + ARp,, + At.
(G49)
where AR and At are small deviations from the true pose. The residual error then becomes:
ARR.. + Atll
(G50)
This residual error is further influenced by the errors 2D correspondences c2p and depth mea- surements eD (c.f., Eq. (G39) and Eq. (G40)) The resulting residual error, incorporating these perturbations, is:
| + (pr) + At. (Pa.i (p,))l
(G51)
70

---

Published as a conference paper at ICLR 2025
where Eprr and Cp encapsulate the combined effects of both 2D correspondence and depth errors. Thus, the small motion case, the condition number of this optimization problem increases, making it highly sensitive to these small enrors.
Theoretical analysis on the large motion case. When there is significant relative motion between the rendered and observed images, the optimization process benefits from larger gradients, making more stable and resistant to noise. The transformed points under large motion can be expressed as:
Rp +
Pr.
Trans + p,r"
(G52)
Trans includes the 2D correspondence and Trans represents the true transformed points, and fpr where Py. depth errors after transformation.
The residual error is given by:
Pras
Trans
Trans + ppr"
- (Po,i + (p.o)
(G53)
In the case of large motion, the magnitude of the gradients (derived from the residual error with respect to pose parameters) increases, providing more pronounced and clearer directionality during optimization. This stabilizes the optimization process by making it less susceptible to small perturba- tions in the input data. Thus, larger motion enhances the robustness of the optimization, leading to more reliable convergence.
G.3.7 RENDERING-BASER POSE QUALITY VERIFICATION
To address the potential failure correspondence-basee pose initialization in small motion scenarios (see Fig. G34), we reject the pose P estimated from CPL. if it produces a higher loss than the naively propagated pose P., which is estimated using a linear motion model based on historical poses. We render RGB-D images ÍCPL, DCPL, from the correspondence guided initialization and nnive,, D maive, from the naively propagated pose. The losses are then computed as:
CCPL,((PE) AcCooCCrL Io) + An (DCPL. Dt), Lnaive, (Pi) Xc (Inivee L.) + Ap naive,t Dt),
(G54)
(G55)
After computing the losses for the CPL pose Pi and the naively propagated pose Pt, we select the pose with the lower loss for further refinement. The selection criterion is defined as:
P.PPe P arg min (PCPL,((P)), Lnavve (Pi))
(G56)
G.4 DIFFERENTIABLE RENDERING-BR BASED POSE OPTIMIZATION WITH THE CORRESPODEECIEGUUDDE POSE INITIALIZATION
After verifying the quality of the initial pose Pr, is refined through differentiable rendering- pose optimization (see Sec. G.2). The synergy between the correspondence-guiie and differentiable optimization enables the system to leverage both precise correspondences and photometric consistency during optimization. This two-step process improves the robustness of the final pose estimate by first aligning point correspondences and then using the differentiable rendering to capture finer details and reduce ambiguities. As result, this combination ensures more robust pose learning, thus enhancing the accuracy of pose tracking in the dense Neural SLAM system.
initialization
71

---

Published as a conference paper at ICLR 2025
Corr. Init.
RGB-D Rendering Loss
Naive
Corr. is better
4-
2
25
50
75
100
125
150
175
200
Frame Index
Previous
Frame
Curent
Frame
Frame Index: 25
Frame Index: 52
Frame Index: 110
Figure G34: A pose initialized using correspondence in CPL demonstrates superior performance in scenarios involving rapid motion, effectively maintaining alignment and stability. However, it may fail to achieve similar accuracy under conditions of small motion, where the lack of distinct changes makes correspondence-based initialization less reliable.
72

---

Published as a conference paper at ICLR 2025
Rendered Image
Observed (Perturbed) Image
High
Before
Conresoondeecce Confidence
After
Low
Figure G35: Existing matchers (Sun et al., 2021) can produce weak but usable correspondences in cross-domain settings (e.g., varying illumination), which be leveraged for correspondence- guided pose learning to improve pose estimation accuracy. As shown in the bottom figure, the rendered image from the estimated pose is well-aligned with the observation, compared the top figure where the initial pose results in rendered image with significant view misalignment.
G.5 CORRESPONDENCE-GUDDA APPEARANCE RESTORATION LEARNING (CARL)
G.5.1 MOTIVATION AND INSIGHTS
We find that existing matchers can generate weak but usable correspondences in noisy cross-domain settings, such as varying illumination. These correspondences can be leveraged for correspondence- guided pose learning to improve pose estimation accuracy. For instance, as illustrated in Fig. G35, the rendered image from the estimated pose aligns well with the observed image (bottom figure). compared to the large misalignment seen with the initial pose (top figure).
Building on this, we further utilize these weak correspondences for appearance restoration. By training restoration model, [(; 0), CARL maps noisy colors from the observed images to clean counterparts derived from a ppoorresiively-uddacee noise-free historical map. This restored image then used to refine the camera pose, iteratively improving both image quality and pose estimation, ensuring robustness in noisy, cross-domain environments.
G.5.2 RESTORATION MODEL LEARNING
The restoration model f(=; 0) learned online, using 2D correspondences obtained from the pose initialization step. The goal is to minimize the discrepancy between the noisy colors Co,s from the noisy observed points and the clean colors C, rendered from the historical map:
0- arg min Σ II
- 2
(G57)
This optimization process ensures that the restored colors closely match the clean appearance which is maintained in the poogressivelyuuddae m3D map, providing an improved and noise-reduced version of the current frame.
G.5.3 RESTORING PERTURBED OBSERVATIONS AND IMPROVING TRACKING VIA CARL
CARL restores (perturbed) observations for noise-free mapping. Once the restoration model f(; 0) is trained, it is applied to the entire image to reduce noise and restore the appearance of the observed scene. Let I, be the observed RGB image at time t, which may contain noise or degradation, and I be the restored image. The restoration model is applied pixel-wise across the entire image:
I
(1,;0)
(G58)
73

---

Published as a conference paper at ICLR 2025
The restored image If depth map Dr, and refined pose Pa are integrated into the historical map Met to update the 3D map Mr, ensuring consistency.
CARL enables more robust pose tracking optimization. CARL also enhances tracking robustness by improving pose estimation through a second-round correspondence calculation and pose optimization. Once the image has been restored, the corresponding can be recomputed. And then, the updated observed 3D points p are then used in a second-iteration of pose optimization. The objective function for the second-round relative pose refinement (with the restored points) is:
min Ryet
Σ ([Bredrrs trel -.oill)
(G59)
By iteratively refining the pose using the restored 3D points, CARL ensures more robust tracking, especially in challenging environments where noise or dynamic elements are present.
G.6 REACTIVE STRATEGY FOR HANDLING CORRESPONDENCE CALCULATION FAILURES
In real-world cases where an incoming frame severely perturbed and the correspondence calculation fails, the system bypasses CPL and switches to the conventional differentiable rendering-based pose optimization for pose estimation. Additionally, CARL disabled for these frames to avoid degradation in the restoration process. These problematic frames also excluded from the map update stage to prevent error propagation into the 3D representation, ensuring the overall accuracy and integrity of both pose estimation and the maintained map in the dense Neural SL.AM system.
74

---

Published as a conference paper at ICLR 2025
H
MORE QUALITATIVE RESULTS ON CORRGS
H.1 MORE QUALITATIVE RESULTS ON SYNTHETIC (CLEAn) SPARSE-VIEW VIDEO
w/o CPL
w/ CPL
Office
Room-
Office-
Figure H36: Qualitative ablation study of Correspondence-guidee Pose Learning (CPL) on 3D reconstruction with sparse- view (clean) videos. CPL ensures robust pose tracking, enabling high- quality 3D reconstruction even under extreme fast motion.
75

---

Published as a conference paper at ICLR 2025
Translational Components
w/O CPL
0.0
Slow Pose Converyence
-0.5
-1.0
0
25
50
75
100
125
150
0.5
0.0
Fast Pose
w/ CPL
Convergence
-0.5
-1.0
-1.5
Iteration
Rotational Quatemions
1.0
W/CPL
Slow Pose
0.5
Canverance
0.0
25
50
75
100
125
150
w/ CPL
Fast Pose
Converyence
0.0
25
50
75
100
125
150
Iteration
Figure H37: CPL facilitates faster convergence of pose optimization, significantly reducing the number of iterations required to align the estimated pose with the ground-truth. Dotted lines represent the ground-trun -ruth poses, while solid lines denote the estimated poses throughout the optimization.
76

---

Published as a conference paper at ICLR 2025
H.2
MORE QUALITATIVE RESULTS ON SYNTHETIC NOISY SPARSE- VIDEO
Obarved RGB-D
GT Ravekee RGB-D
ROB-D
Obxrvvii
ROB-D
Rendeee RCB-D
RGB-D
Figure H38: Comparison of noise-free RGB and depth frames rendered using CorrGS with observed noisy RGB and depth frames (scene Office-/), /). Ground-truth clean RGB-D frames are included for reference, demonstrating qualitative results on synthetic data under dynamic illumination changes.
Noise-free Dense 3D Reconstruction (Baseline)
Noise-free Dense 3D Reconstruction (CorGGS, Ours)
Failure of Pose Tracking Geomelry Reconstnuction
Figure H39: Qualitative comparison of photorealistic 3D maps constructed using CorrGS and the baseline synthetic, noisy, sparse-view data (scene Office- 1). The bascline struggles to accurately reconstruct both geometry and appearance, whereas our method produces a clean 3D representation.
77

---

Published as a conference paper at ICLR 2025
Observed RGBBD
Readlre RGBD
X% SN
GT Clean RGE-D
Olserved
RGBBD
Readard RGEDD
GI Clean RGBBD
Figure H40: Comparison noise-free RGB and depth frames rendered using CorrGS with observed noisy RGB and depth frames (scene Office-O). Ground-truth clean RGB-D frames included for reference, demonstrating qualitative results on synthetic data under dynamic illumination changes. While there some small local artifacts, the overall reconstruction and restoration quality is promising.
Noise-free Dense 3D Reconstruction (Baseline)
Noise-free Dense 3D Reconstruction (CorGG, Ours)
T
%
Figure H41: Qualitative comparison of photorealistic 3D maps constructed using CorrGS and the baseline on synthetic, noisy, sparse-view data (scene Office--)). The bascline struggles to accurately reconstruct both geometry and appearance, whereas our method produces clean 3D representation.
78

---

Published as a conference paper at ICLR 2025
H.3 MORE QUALITATIVE RESULTS ON REAL-WORLD NOISY SPARSE- VIEW VIDEO
Observe
RGB-D
Randavd
RGBSD
Observed RGB-D
Renelaed RGB-D
Obsaved RGB-D
Rendaed
RGED
Renleed Observe
RGBD
RGBBD
Figure H42: Comparison of the rendered noise-free RGB and depth frames (via ConTGS) with the observed noisy RGB and depth, showcasing qualitative results on real-world data with dynamic illu- mination changes. The sequences involve fast-motion, sparse-view scenarios, transitioning between normal light, darkness, and overexposure, demonstrating our method's robustness in reconstructing noise-free, dense 3D maps under challenging conditions.
Noise-free Photorealistic 3D Reconstruction
Figure H43: Qualitative results of constructed noise-free photorealistic 3D map via CorrGS on real-world noisy sierse-view data.
79

---

Published as a conference paper at ICLR 2025
(a) Observed Depth Distribution of Each Frame (Min, Mean, Max Depth)
Depth Value (m)
Case
Case 2
200
400
600
800
Frame
(b) Degraded Rendering due to Depth Degradation
Degraded Case 1
Degraded Case 2
Observed
RGB-D
Rendered
RGB-D
Figure H44: Failure cases of degraded rendering by ConGGS real-world noisy, sparse-view data.
The highlighted regions correspond to two failure cases (Case and Case 2) where noisy depth data leads to degraded RGB rendering in specific frames, Addressing depth restoration remains valuable future direction.
80

---

Published as a conference paper at ICLR 2025
H.4 ROBUSTNESS OF CORRGS TO POSE INITIALIZATION ERRORS
To evaluate the robustness of CorrGS against pose initialization errors, we tested it on 100 overlapping RGB-D image pairs from each of three room scenes in dataset, which features diverse poses. The proposed method demonstrates robust performance against significant pose initialization errors, maintaining accuracy under angular discrepancies of up 40 and translational offsets of up 1.0 m. Evaluations across three distinct ccenes-roome room I, and room2- its resilience to both rotational and translational noise, as illustrated in Figures H45 and H46.
Fig. H45 shows that PQV maintains low rotation error across all scenes, even as absolute rotation increases. RoomO exhibits the lowest eπror with minimal degradation, reflecting the framework adaptability in simpler environments. While rooml and room2 show slightly higher errors due to more complex spatial arrangements, performance remains stable and consistent, emphasizing PQV's robustness to angular noise across varying environmental complexities.
Fig. H46 illustrates PQV's response to translational noise, where translation error remains small and gradually increases with larger offsets. Roomo again achieves the lowest error, demonstrating its effectiveness in less complex conditions. contrast, room2 exhibits higher variance, likely due to increased structural geometric challenges. Despite this, PQV consistently delivers stable performance without significant deviations, confirming its ability to handle varying translational conditions across diverse environments.
2.00
Scenes
room2
1.75
room0
room1
1.50
Rotation Error [deg]
X
1.25
1.00
0.75
0.50
0.25
X
0.00
50
Absolute Rotation [deg]
Figure H45: Rotation error under varying levels of absolute rotation, demonstrating consistent pose estimation accuracy across diverse environments despite large angular deviations.
81

---

Published as a conference paper at ICLR 2025
0.035
Scenes
room2
0.030
roomo
room1
0.025
Translation Error [m]
X
0.020
X
0.015
X
0.010
X
0.005
X
0.000
0.0
0.2
0.4 Absolute Translation [m]
0.6
0.8
1.0
Figure H46:
Translation error under increasing absolute translation, highlighting the robustness of the method across scenes even with significant translational discrepancies.
82

---

Published as a conference paper at ICLR 2025
H.5
ADDITIONAL CORRGS RESULTS IN OUTDOOR SETTINGS
Observed RGB-D
Rendered RGB-D
Observed RGB-D
Rendered RGB-D
Observed RGE-D
Rendered RGB-D
Observed RGB-D
Rendered RGB-D
Figure H47: Rendered RGB-D results from the reconstructed map in the outdoor scene during the morning. The average PSNR 28 [dB] and average depth Ll loss is 1.80 [cm].
83

---

Published as a conference paper at ICLR 2025
Observed RGB-D
Rendered RGB-D
Observed RGB-D
Raddrd RGB-D
Observed RGB-D
Renderd RGB-D
Observed RGB-D
Ronderd RGB-D
Figure H48: Rendered RGB-D results from the reconstructed map in the outdoor scene during the
night. The average PSNR is 34 [dB] and average depth LI loss is 2.17 [cm].
84

---

Published as a conference paper at ICLR 2025
I THEORETICAL INSIGHTS ON CORRGS
We begin by analyzing the causes of inefficiency and failure in differentiable pose optimization, specifically for the Gaussian Splat map representation, under fast pose motion. We then develop a theorem to demonstrate that incorporating Correspondence-Guns Guided Pose Learning (CPL) and Correspondence- Guided Appearance Restoration Learning (CARL), which constructs our CorrGS framework, enhances robustness under these challenging conditions.
Theorem F. In differentiable rendering-based SLAM systems utilizing Gaussian Splat represen- lation, incorporating Correspondence-Gutiee Appearance Restoration Learning (CARL) mitigates the inefficiency and potential failure in pose optimization under fast motion and noisy video inputs. Specifically, CPL reduces the magnitude of the gradients and improves the conditioning of the Hessian matrix by providing better pose initialization, while CARL reduces the variability the gradients caused by noise, leading to more stable and efficient convergence of the optimization process.
Pose Learning (CPL) and Correspondence-Guutie
Proof. We analyze the effects of CPL and CARL on the pose optimization problem under the Gaussian Splat representation, focusing on how they address the issues caused by large gradients and poor conditioning of the Hessian matrix under fast motion and noisy inputs.
I.1 POSE OPTIMIZATION IN GAUSSIAN SPLAT REPRESENTATION
In differentiable rendering- based SLAM with Gaussian Splat representation, the camera pose P [gr, Is: Gj- 9x}²
(R, t) is parameterized using a unit quaternion translation vector E IR$, The rotation matrix R is derived from the quaternion q:
Rd for rotation and
(o) + +
q4j 9r9k
qoak + gr
R 2 2
9:9; 9r
3 (g + v
dΙ 9-
(I60)
9:9k 9r
fj9k 9-9:
(q + 93
The loss function L(P) measures the discrepancy between the rendered images (i, D) and the observed images (I.
((P) = do [(()) 1/2 + +Ao D(P)
(161)
where Ac, Ap > 0 are weighting parameters.
The pose is updated iteratively using gradient descent:
Pk+1 1 Pe -//
(162)
where 7) >> 0 is the learning rate.
1.2 IMPACT OF FAST MOTION ON GRADIENTS AND HESSIAN
Under fast motion, the change in pose AP = AR, At between successive frames is large. This leads large residuals in the loss function and consequently large gradients and Hessian entries.
Gradients w.r.f. translation t. The projected 2D coordinates m, of a 3D Gaussian G, depend on the camera pose P. The gradient of the projected coordinates with respect to translation t is:
θm; Øt
Om,
where
X
R.X, +
(163)
OX
and X; is the 3D position of the Gaussian. The gradients are given by:
3m,
fx
0
f
8c
(164)
0
f
Zo
=2
where f, and fy the focal lengths, and (mey Hey TT) X
85

---

Published as a conference paper at ICLR 2025
Under large translations Åt, the terms involving
f.
and fyge 2g
2 become large, resulting in large
gradients dm,, This is because significant shifts in the 3D points relative the camera amplify the öt effect of translation on the projected coordinates.
Gradients w.r.f. rotation R (Quaternion q). The gradients of the transformed 3D points with respect to the quaternion components are:
0X;
gr qi Mk
=
9rble + qu- 2
(165)
dqr
9 q:yl qe
0x; 2
q:3c 9j0e 2c
9ide + Te + 2.
(166)
0qi
qiZe + 4r 1/c qu-E.
OX 2
2f + qile Ze
9jye qe 9N-
(167)
Ogj
9i + qu-le + T.
0x.
CkTC - 9rde + Q:
qulle q. O₁ Re
(168)
0qk
2x2e g.Te 9,ye
Under large rotations AR, the rotation gradients become large due to significant changes in the rotation matrix R. This leads to substantial changes in the positions of the 3D Gaussians when projected into the 2D image plane.
Impact on Hessian matrix. The Hessian matrix H of the loss function with respect to the pose parameters includes second derivatives such as:
82 8f Bq.age
8a C Og 8 C Ouf
V
(169)
Large gradients result in large entries in the Hessian matrix, including off-diagonal elements repre- senting interactions between different pose parameters. This indicates strong coupling between the parameters, leading to poor conditioning of the Hessian and making the optimization landscape more challenging.
1.3 INEFFICIECCY AND POTENTIAL FAILURE IN PURE DIFFERENTIABLE POSE OPTIMIZATION
In gradient- bradde optimization, the pose update is:
P P - (pL()))
(170)
With large gradients due fast motion, the update step 1/ P(Ps) can be excessively large, causing the optimization to overshoot and potentially diverge.
Using Taylor series expansion of the loss function around Pk:
1 "" (VPL()) ((P C(Pp) -
TTOpL((.)TTVVCCCC HV
(171)
For the loss to decrease, we require:
2 P(Pp() VPL(P)) HVPL(PP)
(172)
With large gradients and poor conditioning of the Hessian, the permissible 7) becomes very small. n is not adjusted accordingly, the optimization can diverge, convergence can be extremely slow.
86

---

Published as a conference paper at ICLR 2025
1.4 EFFECT OF CORRESPONDENCE GUIDED POSE LEARNING
CPL mitigates these issues by providing a better initial pose estimate P; through minimizing the geometric error between correspondences:
Pi = arg min
R.pr. + -Po,il
(173)
where Pr, and Po, corresponding 3D points from the rendered and observed frames, respectively. By solving this optimization problem, CPL provides pose estimate closer to the true pose Pr This reduces the initial residuals in the loss function:
i(Pt) -1 1f I(PP-1)
(174)
and similarly for D. Consequently, the magnitude of the gradients VpL(Pt) is reduced.
With a better initial pose, the Hessian matrix H is better conditioned near the true pose. The second derivatives become smaller, and the interactions between parameters are less pronounced. The condition number of the Hessian decreases:
x(((P)) < (H((P-)))),
(175)
where <(H) denotes the condition number of the Hessian matrix at pose P. A lower condition number indicates better conditioning, which enhances the convergence properties the optimization algorithm.
1.5 EFFECT OF CORRESPONDENCE- GUIDED APPEARANCE RESTORATION LEARNING
Noisy video inputs introduce variability into the observed images I T + N, where N represents additive noise (e.g., Gaussian noise).
This noise affects the residuals in the loss function:
i(P) - I= (I(P) Timee) N.
(176)
The noise term N introduces variability into the gradients:
VpLc = vic (I(P) - t -ume- -N).
(177)
The term involving N adds stochasticity to the gradient, leading to increased variance and potential misalignment in the gradient direction.
CARL mitigates this issue by learning a restoration model ((=00) to obtain denoised images i = f(I: 8) ~ Iime
Using the denoised images in the loss function:
Lc(P) dx [((P) 1
(178)
the residuals become:
i(P) - i i(P) te
(179)
eliminating the noise term N. Consequently, the gradients:
VpLc = (i(p)-1th))
-
(180)
are free from the variability introduced by noise, leading to more stable and accurate gradient computations.
87

---

Published as a conference paper at ICLR 2025
1.6 COMBINED IMPACT OF CPL AND CARL
By integrating CPL and CARL, we address both the issues of large gradients due to fast motion and the variability in gradients due noisy inputs.
CPL reduces initial residuals and gradients. Providing a better initial pose estimate PV reduces the magnitude of the gradients and improves the conditioning of the Hessian matrix, allowing for larger permissible learning rate 7 without risking divergence.
CARL stabilizes gradient computations. By denoising the observed images, CARL reduces the variability in the residuals and gradients caused by noise, leading to more stable and reliable updates during optimization.
Synergistic effect. The improved pose estimate from CPL enhances the alignment between the rendered and observed images, which in turn improves the effectiveness of CARL in restoring the appearance. Accurate pose estimation ensures that correspondences used in CARL are valid, leading better denoising performance.
1.7 OVERALL IMPROVEMENT IN OPTIMIZATION EFFICIENCY OF CORRGS
The combined effect of CPL and CARL leads to:
Reduced gradient magnitudes, mitigating the risk overshooting and divergence.
Improved conditioning of the Hessian matrix, enhancing convergence rates.
Stabilized gradient directions, ensuring consistent progress towards the optimal pose.
Enhanced robustness under fast motion and noisy video inputs.
Therefore, incorporating CPL and CARL into differentiable rendering-basee based SLAM systems utilizing Gaussian Splat representation effectively mitigates the inefficiency and potential failure in pose optimization under challenging conditions, leading more robust and efficient convergence of the optimization process.
88

---

Published as a conference paper at ICLR 2025
Table J25: Effects of perturbing the stereo camera baseline length on trajectory estimation performance of the ORBSLAM3 model with stereo (Top) and stereo-inertia (Bottom) SLAM setting. We leverage three sequences, including MH01, MH02, and MH03, on the EuRoC (Burri et al., 2016) dataset. We introduce random noise 1l along the axis of the stereo camera baseline. The noise is sampled from Gaussian distribution with zero mean and variance o², represented as 7 NN(0 o²) [m]. The stereo camera has a predefined baseline length of 0.1 meters.
MH01 Sequence
MH02 Sequence
MH03 Sequence
Metries
0.00
d 0.001
o= 0.01
x 0.00
5 0.001
5= 0.01
a 0.00
0 0.001
0.01
Stereo Setting
ATE-w/o Scalel, [m]
0.036
1.646
8.284
0.016
2 2.943
19.42
0.028
1.414
6.211
Scale
1.006
1.129
0.261
2.251
0.172
0.998
0.941
0.395
ATE-w/ Scatel, [m]
0.024
1.380
2.221
0.013
1.674
2.415
0.027
1.429
1.965
Stereo-inertia Setting
ATE-w/o Scalel
0.068
0.353
5.170
0.050
2.370
3.951
0,053
0.372
673.8
Scale
1.012
0.987
0.410
1.004
2.172
0.491
1.004
1.018
0.001
ATE-w/ Scatel, [m]
0,046
0.319
1.865
0.046
0.312
1.021
0.052
0.368
3.576
J OTHER POTENTIAL DIRECTIONS TO EXPLORE
Towards more robust SLAM for spatial perception, we present additional potential research avenues.
J.1 ROBUSTNESS EVALUATION FOR MORE DIVERSE SLAM SYSTEMS
We believe it is valuable to extend the robustness study to broader range of SLAM systems, beyond the dense RGB-D SLAM framework considered in this research. Different SLAM architectures face distinct challenges when exposed to real-world noise and perturbations.
To illustrate this, we present two case studies that demonstrate the fragility of stereo and multi-agent SLAM systems when subjected to perturbations. These findings emphasize the importance of future robustness evaluations across diverse SLAM configurations to ensure their resilience.
Robustness analysis of stereo SLAM system under perturbation. We conduct a robustness analysis of a stereo SLAM system, which utilizes two cameras to capture observations of the environment. In this study, we focus on the inter-camera transformation matrix, a crucial sensor parameter that describes the transformation between the left and right cameras, Our objective is to investigate the effects of introducing Gaussian noise as the perturbation to the inter-camera transformation matrix of the stereo SLAM system, which mimics the inaccuracy of the transformation matrix. This inaccuracy can be caused by factors such as noisy extrinsic matrix calibration, vibration due to less rigid camera holder, or errors in camera placement or assembly. Specifically, we introduce random noise v) along the stereo camera baseline direction, sampled from a Gaussian distribution with zero mean and variance o², denoted 17 N o²). ). In our preliminary robustness study, utilize the ORB-SLAM3 model with the stereo camera input setting and the stereo-inent inertia setting. We apply random perturbations the camera baseline length for every frame in three sequences (MH01, MH02, and NM03) from the FuRoC dataset (Burri et al., 2016). The results of our study are summarized in Table J25, which includes three severity levels of noises: a 0.00, 0.001, and = 0.01. Under the clean setting (o 0.00). When we introduced perturbations to the baseline axis of the inter-camera transformation matrix, increasing the noise levels (e.g., a 0.001 and 0.01) resulted in a noticeable rise in trajectory estimation error, which is quantified by ATE-w/o Scale and ATE-w/ Scale metrics. This indicates higher susceptibility errors and deviations from the ground truth trajectory during trajectory estimation. These findings highlight the significance of taking a holistic approach when considering the robustness of the whole SLAM system. To ensure comprehensive evaluation of the SLAM model's 's robustness, it is crucial to consider not only the software (SLAM model) but also the hardware, specifically the inaccuracies in predefined or estimated sensor configurations.
Robustness analysis of multi-agent SLAM system under perturbation. We investigate the robustness of the multi-agent SLAM framework COVINS-G (Patel et al., 2023b) under Gaussian noise image-level perturbation. We evaluate the performance of the model on the EuRoC (Burri et al., 2016) dataset using the ORB-SI ORB-SLAM3 (Campos al., 2021) model with monocular-inertia
89

---

Published as a conference paper at ICLR 2025
Table J26: Effects of Gaussian noise perturbation on trajectory estimation performance for the multi-agent SLAM framework COVINS-G (Patel et al., 2023b) with the ORBSLAM3 (Campos et al., 2021) model as the front-end are presented. The noise is sampled from a Gaussian distribution 7) (0, The experiments were conducted using mono-inertia input setting for each agent.
Metries
o² = 0.00
o² = 0.02
x² = 0.04
ATE/ [cm]
0.071
0.071
0.088 20.354
SSE| [cm]
11.545
12.517
input as the front-end module. The experimental setup involves introducing varying severity levels of perturbation, measured by the parameter 0, on all the agents of a multi multi-agent system. The results are reported based on running five agents in sequential order on the MH scene of EuRoC (Burri et al., 2016). Two metrics, Average Trajectory Error (ATH) and Sum of Squared Errors (SSE), are used to assess the multi-agent SLAM system's 's performance. These metrics provide a comprehensive evaluation of trajectory estimation accuracy and the accumulation of errors, respectively. As is shown in Table J26, the multi-agent SLAM system demonstrates consistent performance in the clean setting (a = 0.00) and low perturbation severity (a 0.02), indicating the system's 's ability to tolerate low random noise. However, the perturbation severity increases to 0.04, the system's 's performance degrades noticeably. The ATE experiences a significant increase, suggesting the vulnerability of the model to higher levels of perturbation. Similarly, the SSE metric rises to 20.354 cm, reflecting the accumulation of errors caused by the higher noise level.
J.2 ADDITIONAL FUTURE DIRECTIONS
Robustness evaluation in unbounded 3D scene. Our work does not encompass the robustness of SLAM systems unbounded scenes, such as outdoor environments (Dosovitskiy al., 2017). Investigating the robustness of SLAM systems in such scenarios holds significant potential for future research. It can contribute better understanding of the practical applicability and generalizability of SLAM systems in complex scenes.
Computationallly-fficee robustness evaluation. Our findings have identified discernible indicators within certain SLAM models that can reflect degraded observations, ie., the reconstruction quality of RGB images and depth maps for the SplaTAM-SS (Keetha et al., 2024) model. Future research could explore leveraging and designing robustness indicators to evaluate the robustness of SLAM systems more efficiently. By incorporating such indicators, we have the potential to enable unsupervised performance evaluation of SLAM, especially in scenarios where obtaining ground-truth annotations is challenging or costly.
Real-world robustness evaluation. While work primarily focuses synthesis-baseh robustness analysis, we recognize the value of real-world verification and validation for SLAM systems. Con- ducting extensive field tests in more challenging environments (Ebadi et al., 2023), where SI.AM systems are subjected to agile locomotion types (Kaufmann et al., 2023), would provide empirical validation of simulation results and uncover additional challenges. This real-world evaluation would bridge the gap between simulated environments and actual deployment conditions, ensuring the practical reliability and robustness of SLAM systems.
Future directions for CorrGS. While our method presents proof-of-concept approach, there is room for further enhancement. In future work, we aim to explore the scalability of ConGGS in outdoor enviromments with unpredictable motion patterns and large-scale scenes. These environments present unique challenges such as dynamic lighting and large-scale motion, and further research will focus on improving generalization to handle these complexities. Additionally, we plan to explore the trade-offs between computational efficiency and robustness, optimizing CorrGS for real-time applications with limited resources through the use of lightweight architectures and GPU-based parallel processing. Furthermore, improving CorrGS's s robustness to erratic motion and extreme lighting conditions (e.g.. sudden changes from sunlight to shadows) is another important avenue. Finally, intend to enhance computational efficiency by incorporating adaptive learning mechanisms that activate only under frame quality degradation, reducing unnecessary processing.
90

---

Published as a conference paper at ICLR 2025
K
BROADER IMPACT
The impact this work goes far beyond robotics and Dense Neural SLAM. By enhancing robust- ness under noisy sensing conditions, address a fundamental challenge faced by autonomous systems-reliable perception and localization under noise, motion, and misaligned sensors. This improvement is crucial for safety and efficiency in systems such self-driving cars and drones, enabling them to navigate and operate with greater confidence in unpredictable environments.
Central to this work is the first comprehensive taxonomy of RGB-D perturbations for mobile agents, which serves flexible tool for understanding and improving robustness across diverse applications. This taxonomy, which categorizes key disruptions like motion blur and sensor desynchronization, is broadly applicable in fields like augmented reality, 3D reconstruction, embodied AI, and surgical robotics. By providing structured framework, it enables these systems to better handle real-world imperfections.
The Robust-Egoo3 benchmark, built on this taxonomy, provides a groundbreaking platform for evaluating SLAM algorithms under real-world conditions, offering insights into model vulnerabilities and driving the development of more resilient solutions. Researchers across computer vision and AI leverage this benchmark to refine their models in complex environments, broadening the utility of SLAM technology.
Furthermore, the ConGS method demonstrates how visual correspondences can empower SLAM to maintain high-quality 3D reconstructions even from sparse, noisy inputs. This opens new possibilities for deploying autonomous systems in resource-constrained high-stakes environments, such as search and rescue, space exploration, and industrial inspection.
By pushing the frontier of robustness alongside accuracy, this work provides the tools and frameworks essential for building AI systems that can thrive in real-world, unpredictable conditions. These advances are critical ensuring that AI can be trusted in safety-critical and mission-driven domains.
L SOCIAL IMPACT
The proposed pipeline for noisy data synthesis and the Robust-Ego3D benchmark have the potential to advance the development of robust SLAM systems. By enabling the evaluation of SLAM models under diverse perturbations, this work can facilitate the creation of more resilient robotic systems capable of operating reliably in unstructured and challenging environments. Robust SLAM systems can enhance safety, efficiency, and effectiveness various domains, such as autonomous navigation, exploration, and mapping in hazardous inaccessible areas.
However, it is essential to acknowledge potential negative implications. The synthesized perturbations and noisy environments, while designed to evaluate robustness, might not fully capture the complexi- ties of real-world scenarios. Overreliance on these simulated environments could lead to overlooking unforeseen challenges, highlighting the importance of complementing simulations with real- world testing and validation. While we curated diverse perturbations, inherent biases or blind spots in the taxonomy synthesis process may remain. We emphasize the need for continuous refinement and expansion of the perturbation taxonomy to capture a broader range of real-world disturbances and mitigate potential biases. Additionally, the potential for unethical misuse of robust SLAM systems should be considered. Manipulated or biased data could be used to construct environments that present a distorted view of reality, leading to erroneous decision-making or harmful consequences. To mitigate this risk, robust validation, fact-checking mechanisms, and adherence to ethical guidelines must be integrated into the development and deployment processes.
91

---

Published as a conference paper at ICLR 2025
M AVAILABILITY AND MAINTENANCE
The code and datasets from this study will be publicly accessible. The project repository contains the following resources:
Robustness benchmark, The repository includes code for SLAM robustness evaluation
under customized perturbations.
Baseline models for benchmarking. Detailed instructions are provided for running all baseline models, facilitating the reproduction of all results presented in this paper.
Experiment reproduction. Comprehensive guidelines for reproducing all experiments found the Instructions.md file within the repository.
CorrGS: the proposed method. The repository contains the implementation of the pro- posed ConGGS method, along with detailed instructions for running experiments that demon- strate effectiveness addressing robustness challenges in SLAM systems.
These resources enable researchers to utilize and extend this work.
We encourage the community to propose more robust SLAM models to further advance the frontier of robust embodied agents, ensuring their safe and reliable deployment in real-world environments.
N
LICENSE
The benchmark and code are licensed under Apache License 2.0.
PUBLIC RESOURCES USED
We acknowledge the following public resources used in this work:
Classiiication-Roussman
Apache License 2.0
Replica
Research-only License
Nice-SLAM
Apache License 2.0
Co-SLAM
Apache License 2.0
SplaTAM'
BSD 3-Clause "New" or "Revised" License
GO-SLAM
Apache License 2.0
ORB-SLAM3
GNU General Public License v3.0
LoFTR
Apache License 2.0
"https: hhtp://///gghhhhoooo////////////
92


Reviews:
[{"review_id": "j85YaHMP9I", "reviewer": "Reviewer_ccbx", "summary": "The paper proposes a structured approach to advancing robust ego-motion estimation and 3D reconstruction in the presence of noisy video data. The authors tackle three core challenges: developing a scalable noisy data generation pipeline, creating the Robust-Ego3D benchmark to evaluate model robustness under varied perturbations, and introducing a novel model called Correspondence guidded Gaussian Splatting. The data generation pipeline is designed to synthesize large scale, customizable   datasets simulating realistic RGBD sensor noise and motion perturbations, which expose vulnerabilities in existing models. With these data, the Robust-Ego3D benchmark allows a systematic robustness evaluation of models across diverse noisy conditions.  CorrGS leverages correspondences between internally generated and observed frames to achieve photorealistic 3D reconstructions and robust pose tracking, significantly outperforming SOTA methods in both synthetic and real world experiments under rapid motion and complex noise.", "questions": "1. How does the pose quality verification step perform under different levels of noise? Did the authors experiment with varying PQV thresholds to understand its sensitivity to noise, and if so, what insights did they gain? \n2. Does CorrGS account for scenarios with high-frequency dynamic changes?\n3.Different sensor types produce different noise profiles. Did the authors experiment with adjusting CorrGS for specific types of sensor noise (such asmotion blur for RGB or range limitations for depth sensors)?\n4. The Robust-Ego3D benchmark and CorrGS are tested primarily on indoor or synthetic datasets. Do the authors envision any adjustments needed to apply these methods to outdoor environments, where conditions such as changing weather, shadows, or variable terrain could add complexity?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The paper addresses the largely unexplored domain of robustness benchmarking for ego-motion and 3D reconstruction under realistic, noisy conditions. Unlike traditional methods that assume noise-free data, the paper  introduces a customizable pipeline for generating data with varied noise levels, motion deviations, and synchronization perturbations, filling a critical gap in existing SLAM evaluation techniques. Furthermore, CorrGS’s correspondence-based method is innovative, as it maintains a noise-free internal 3D representation while dynamically aligning it with noisy external inputs, a novel approach to handling real-world sensor noise. I believe the authors employ a rigorous methodological framework, incorporating a comprehensive set of experiments across both synthetic and real-world noisy datasets. They evaluate CorrGS and other baseline models on metrics including absolute trajectory rrror, PSNR, and depth loss, providing a strong quantitative foundation to support their claims. The ablation studies and theoretical analyses of perturbation effects on model performance offer a clear view of CorrGS's robustness and justify each of the model's components. The paper is clearly structured, with a logical flow from problem definition to solution details. For example, the pipeline for noisy data generation and perturbation taxonomy is visualized in Figure 1, which helps clarify complex processes. Additionally, the explanation of each perturbation type provides readers with a clear understanding of the challenges the benchmark addresses. The ablation studies further aid in understanding the impact of individual components within CorrGS on final model performance.", "weaknesses": "While the paper addresses RGB noise effectively, it could expand on the depth perturbations. Depth noise effects are discussed, the authors could delve further into the challenges that depth noise presents in reconstruction. Additional experiments on models’ performance under severe depth noise and missing depth data would strengthen the robustness evaluation and underscore CorrGS's adaptability. The PQV step is essential for their method’s robustness under complex conditions, but the rationale behind its design choices could be explained in greater detail. I believe it would be valuable to know if different loss functions  were tested and if so, why those choices were made. Demonstrating its impact across different noise levels would give a clearer picture of its effectiveness and necessity under varied conditions. The paper primarily compares CorrGS to other dense Neural SLAM models, but additional comparisons with classical non-neural SLAM methods would highlight CorrGS’s improvements in robustness.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "y02MpGMNfK", "reviewer": "Reviewer_iqVc", "summary": "The paper addresses the challenge of ego-motion estimation and photorealistic 3D reconstruction from noisy video data. The authors present a customizable data synthesis pipeline that generates large-scale datasets with varied noise and perturbations, simulating real-world complexities such as rapid motion and sensor degradation. This pipeline powers the proposed Robust-Ego3D benchmark, which rigorously evaluates model performance under noise, revealing that current models significantly degrade under realistic conditions. Then, benchmark state-of-the-art models using the noisy data in extensive detail. Finally, the authors propose Correspondence-guided Gaussian Splatting (CorrGS), which maintains a clean internal 3D representation, aligning it with noisy observations to improve pose estimation and 3D reconstruction fidelity.", "questions": "What's the necessity of the problem formulation section? generate a dataset and then evaluate the model by calculating the evaluation matrix via the output and ground truth. This is a standard pipeline to conduct experiments. Why it was separately mentioned in an independent section? Please help me to understand what I missed if I misunderstood it.", "limitations": "", "rating": 6, "confidence": 5, "soundness": 4, "presentation": 2, "contribution": 3, "strengths": "The most novel contribution of this paper is a novel data synthesis pipeline capable of generating large-scale, customizable datasets with various noise and perturbations. This contribution fills a crucial gap in Neural SLAM research by addressing real-world challenges in ego-motion estimation and 3D reconstruction.\n\nThe experiments are extensive, meticulously designed, and methodologically sound. Each experiment is carefully aligned with the core claims of the paper, providing solid evidence to support the proposed pipeline and benchmarking framework. The evaluation spans synthetic and real-world settings, presenting a thorough analysis of model performance under diverse noisy conditions. This approach strengthens the validity and robustness of the findings.\n\nTables and figures effectively illustrate complex concepts, making it easier to grasp the data synthesis pipeline, experimental setup, and results.", "weaknesses": "The paper reads more like a technical report, with an emphasis on experimental results over theoretical analysis or in-depth discussion. While the experiments are thorough, they could benefit from additional interpretive insights that go beyond reporting outcomes. For example, rather than solely presenting quantitative improvements, the authors could analyze why CorrGS outperforms other methods under certain noise conditions, offering more theoretical or practical insights into its advantages.\n\nLack of Contextual Foundation: The abstract and introduction lack a thorough discussion of why noise-free data in current reconstruction models presents a significant limitation. While the paper briefly mentions that models degrade in noisy conditions, it could strengthen its foundation by discussing why these models struggle under real-world noise and how noise impacts reconstruction quality, ego-motion estimation, and mapping accuracy. To improve, the authors should provide an analysis of these limitations with references to studies or examples that highlight the common challenges noise introduces in such models in the introduction section. \n\nThe transition from Problem to Method: The paper transitions too quickly from briefly mentioning the limitations of noise-free models to discussing the proposed pipeline and benchmark, missing an opportunity to underscore the problem’s significance. Expanding on how noise introduces challenges for different aspects of 3D reconstruction and SLAM (such as depth estimation, alignment, or tracking stability) would ground the paper’s contributions more effectively. A focused narrative in the introduction that explains why noise resilience is critical would better support the rationale for the proposed solution.\n\nLimited Analysis of Model Vulnerabilities: Although the paper introduces a robust benchmark for noisy conditions, it lacks a detailed analysis of specific vulnerabilities of existing models across different types of noise. This analysis could enrich the understanding of how and why noise affects certain models more than others and help position CorrGS as a targeted solution to these specific gaps. Adding such an analysis would make the benchmark more actionable and informative for future model improvement.\n\nIt can be understood the authors conducted extensive experiments and have a lot of content wish to express, yet the logic of proposing a novel method to fill a gap is overwhelmed by the extensive details. I think this work would be more fit for a journal instead of ICLR.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "qsHJG0SCme", "reviewer": "Reviewer_5SrC", "summary": "The paper claims to have three-fold contributions. 1.A scalable noisy data synthesis pipeline that generates realistic RGB-D perturbations, addressing the gap between controlled and real-world environments for ego-motion estimation and 3D reconstruction tasks. 2. Robust-Ego3D, a benchmark based on the previous proposed pipeline that evaluates model robustness under diverse noisy conditions, setting a new standard for realistic model testing. 3. It also proposed CorrGS, a method that improves pose estimation and 3D reconstruction quality under noise, outperforming prior approaches in challenging scenarios.", "questions": "Q1. How does the system perform when the initial internal model is significantly misaligned or inaccurate? Does CorrGS require a reasonable starting point to work effectively, or can it handle scenarios where initial pose estimates are far off?\n\nQ2. While CPL refines the pose estimates, how sensitive is CorrGS to extremely bad initial pose estimates? Are there scenarios where the initial pose is so far off that CPL struggles to recover accurate alignment, even with PQV in place?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "This paper clearly defines the challenges posed by noise perturbations in ego-motion estimation and 3D reconstruction, offering a systematic exploration of how to evaluate and address these issues. This structured problem formulation provides a solid foundation for advancing robustness research in noisy environments and helps guide both model development and evaluation methodologies.\n\nAuthor provides a very extensive set of experiments showcasing both the benchmark quality and methodology effectiveness across numerous noisy environments, demonstrating the generalizability of the solution.\n\nThe proposed scalable pipeline for generating realistic RGB-D perturbations is a major strength, as it introduces a systematic way to create customizable noisy datasets that mirror real-world conditions\n\nThe CorrGS method is an effective solution for improving pose estimation and 3D reconstruction under noise. It outperforms state-of-the-art models, particularly in challenging scenarios like fast motion.", "weaknesses": "While the paper identifies issues related to depth sensor noise, the evaluation and solutions for depth restoration are not as exhaustive as the RGB counterparts leaving some gaps in how to handle severe depth data inconsistencies.\n\nWhile the scalable noisy data synthesis pipeline is impressive, it relies heavily on synthetic noise generation. This might not fully capture the unpredictable nature of some real-world scenarios.\n\nThis paper provides a new benchmark for evaluating ego-motion and 3D reconstruction quality in noisy environments, but it does not introduce any new evaluation metrics, relying instead on existing ones like ATE, PSNR, and Depth L1 Loss. While this is not a significant weakness, it might even be considered somewhat nit-picky to note. However, introducing new metrics specifically tailored to the unique challenges posed by noisy environments could have further strengthened the paper’s contributions.\n\nAlthough the internal clean model is progressively refined, the method starts with the baseline assumption of a rough model that is iteratively improved. In some extreme real-world scenarios, where no good initialization is available, the system might initially struggle, which could be a practical limitation. Again, I acknowledge this may be somewhat nit-picky as well.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "SdI76CQ1Gy", "reviewer": "Reviewer_nb61", "summary": "The paper presents a benchmark to evaluate dense neural SLAM systems that allow photo-realistic renderings. The benchmark starts from synthetic data (Replica) and considers several data degradation methods including trajectory perturbation, sensor brightness adjustment, motion blurs, etc. The authors compare several different neural SLAM methods including iMap, NICE-SLAM, and SplaTAM, as well as a traditional ORB-SLAM3 method. Extensive analysis and comparisons are conducted to understand the effect of pose accuracy under different noises added to Replica. The paper also presents a new method named CorrGS that leverages correspondence-initialized poses to improve the initialization of the system, which is demonstrated to work well on the benchmark and surpasses existing methods.", "questions": "I think the paper is written in a very clear way with many details already put in the appendix. I have no specific questions at the moment.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The overall contributions made by this paper are pretty extensive, with a benchmark, a thorough analysis of existing methods, and a new proposed CorrGS method that improves over the existing Neural-SLAM methods. The benchmark considers many aspects that a real capture could differ from synthetic captures that are typically overlooked by existing methods, and carefully decouples these factors into several fronts and provides their influence on the trajectory estimation accuracy.\n\n2. While many benchmarks have been provided for real-world SLAM systems, this paper focuses specifically on Neural SLAM systems where the end product is not only camera trajectories but also a neural representation that allows photo-realistic renderings. This demonstrates the novelty of the paper.\n\n3. The quality of the figures and the tables are of high-quality, with many details provided and essential details presented. One can easily tell the impact of different degradation methods to the performance of the SLAM systems.", "weaknesses": "1. I had a hard time reading through the entire paper when I was trying to understand the technical details, and I think that the papers need some refactors. There is extensive cross-referencing to the appendix in the middle of the main text, making the flow of understanding constantly interrupted by having to refer to the details later. While the benchmark, the analysis, and the new CorrGS method are all valid contributions, it would be nice to re-prioritize their importance and put more texts and the details of the most important part in the main text, while putting other parts in the appendix. For example, the entire section 3 could be largely shortened or even removed since the mathematical formulations are rather simple and do not help convey the simple idea behind the scenes.\n\n2. The noise added to the synthetic Replica dataset (e.g. perturbation on depth sensor imaging) does not authentically simulate the real-world depth images. Real-world ToF cameras usually expose systematic noise with planes being pushed to curves, but the paper's enhancement method simply adds independent Gaussian noise to the depth images. This does not align with real-world observations.\n\n3. The introduced benchmark is heavily based on Replica that only contains a limited number of indoor scenes. To further improve the utility of the benchmark it is advisable that the authors investigates into more diverse capturing settings, such as outdoor scenes (where the depth camera still works) and highly-dynamic scenes. \n\n4. While the main goal of the paper is to provide a benchmark that helps these Neural-SLAM methods perform better on real-world settings (by synthesizing real-world noises), it does not test its flagship method CorrGS on many real-world scenes. The only reconstruction results provided is Sec 6.3, where a relatively small chunk of real-world videos are reconstructed. It would be intriguing to see CorrGS being applied to more real-world datasets such as ScanNet++.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)