Open-Set Semantic Gaussian Splatting SLAM with Expandable Representation
Authors: Yucheng Yan
Chen Liang
Wenguan Wang
Yi Yang
Venue: ICLR 2026 Poster (2026)
Keywords: 3D Gaussian Splatting
Dense Semantic SLAM
3D Scene Representation
Primary area: applications to computer vision, audio, language, and other modalities

Abstract: This work enables everyday devices, e.g., smartphones, to dynamically capture open-ended 3D scenes with rich, expandable semantics for immersive virtual worlds. While 3DGS and foundation models hold promise for semantic scene understanding, existing solutions suffer from unscalable semantic integration, prohibitive memory costs, and cross-view inconsistency. To respond, we propose Open-Set Semantic Gaussian Splatting SLAM, a GS-SLAM system augmented by an expandable semantic feature pool that decouples condensed scene-level semantics from individual 3D Gaussians. Each Gaussian references semantics via a lightweight indexing vector, reducing memory overhead by orders of magnitude while supporting dynamic updates. Besides, we introduce a consistency-aware optimization strategy alongside a Semantic Stability Guidance mechanism to enhance long-term, cross-view semantic consistency and resolve inconsistencies. Experiments demonstrate that our system achieves high-fidelity rendering with scalable, open-set semantics across both controlled and in-the-wild environments, supporting applications like 3D localization and scene editing. These results mark an initial yet solid step towards high-quality, expressive, and accessible 3D virtual world modeling. Our code will be publicly released.

Published as a conference paper at ICLR 2026
OPEN-SET SEMANTIC GAUSSIAN SPLATTING SLAM WITH EXPANDABLE REPRESENTATION
Yucheng Yan Chen Liang', Wenguan Wang's Yi Yang' The State Key Lab of Brain-Machine Intelligence, Zhejjang University Shanghai Innovation Institute
Online
Conceptual Query
Long-Tail Objects
Fatimated Camem Trajecoory
RGB-D Stream
Represatttion
Opezieu Semonts SLAM
"alange
Repecsentation
Localizatin
Multi-level Semantics & Visual Cues
Text Analysis
3D Editing
Navigation
AR & VR
Figure 1: This work introduces Open-Set Semantic Gaussian Splatring SLAMM, system designed enable everyday devices (e.g., smartphones) to capture and reconstruct in-the-wild 3D scenes with rich, open-set semantics on top of SLAM frameworks.
ABSTRACT
This work enables everyday devices, egg. smartphones, to dynamically capture open-ended 3D scenes with rich, expandable semanties for immersive virtual worlds. While 3DGS and foundation models hold promise for semantic scene understanding, existing solutions suffer from unscalable semantic integration, pro- hibitive memory costs, and cross-view inconsistency. To respond, propose Open-Set Semantic Gaussian Splatting SLAM, GS-SLAM system augmented by an expandable semantic feature pool that decouples condensed scene-level semantics from individual 3D Gaussians. Each Gaussian references semantics via lightweight indexing vector, reducing memory overhead by orders of magnitude while supporting dynamic updates. Besides, we introduce a consistency-awant optimization strategy alongside Semantic Stability Guidance mechanism to en- hance long-term, cross-view semantic consistency and resolve inconsistencies. Experiments demonstrate that our system achieves high-fidelit rendering with scalable, open-set semantics across both controlled and in-the-wild environments, supporting applications like 3D localization and scene editing. These results mark initial yet solid step towards high-quality, expressive, and accessible 3D virtual world modeling. Our code will be released on our Project Page.
INTRODUCTION
3D virtual world functions collective space where avatars interact seamlessly within metric-semantic representation of 3D environments, capturing both appearance and semantics [1, 2].
"Corresoodiigg author

---

Published as a conference paper at ICLR 2026
Recent technological strides in platforms, such VisionPro [3] and Metaverse [4], have driven the transition toward richer content, larger-seale environments, and the ambitious goal of digitizing the entire Earth [5-9]. However, achieving this vision currently requires specialized and expensive hardware [10], limiting scalability and accessibility. This work aims to bridge this gap by enabling everyday devices, such as handheld smartphones, capture in-the-wild 3D scenes while identifying a broad spectrum of semantics (c.f., Fig. 1). including multi-level semantics and visual properties ( f'uiit in plastic bag'. 'yellow apple'), ) long-tail objects ('xbox'), ) etc., to support diverse applications in open-ended 3D virtual worlds.
One natural solution is to distill open-set semantics from 2D foundation models like CLIP [II] or SAM [12] into 3DGS- representations [13], then optimizing them within a simultaneous localization and mapping (SLAM) framework. Early efforts along this direction generally follow two strategies for semantic integration: (i) Directly encoding feature vectors into each 3D Gaussian, alongside color attributes, to represent semantics [14, 15.. (ii) Assigning pixel-wise semanties after rendering the scene into 2D maps, bypassing explicit 3D point-wise semanties [16, 17]. While
promising, these approaches struggle scale as environments gradually evolve as in SLAM: O Both methods are trained to fit pre-extracted semantic features of a fixed scene, making them incapable of accommodating new concepts dynamically. @ For (i), memory consumption grows prohibitively large as the number of Gaussians and feature dimensions increase. Existing solutions often compress feature dimensions [14], i.e., 3 vs. 512 (original CLIP dimension), to reduce costs, sacrificing semantic expressiveness. @ For (ii), semantics are only recoverable on 2D maps, whereas applications such as 3D localization, embodied navigation, and scene editing require fully 3D-aware semantics. ! Due to 2D nature of foundation models, ensuring semantic coherence across multiple viewpoints is challenging. hindering effective 3D semantic fields learning.
Focusing on the aforementioned challenges, we propose Open-Set Semantic Gaussian Splatting SLAM. Our system in brief is GS-SLAM system augmented by an expandable representation of open-set semantics. At its core, the system maintain an expandable semantic feature pool that stores condensed semantics across the entire scene. This pool can dynamically incorporate incoming semantics, updating existing ones, and increase its capacity to accommodate continuously evolving novel concepts. Each 3D Gaussian is associated with low-dimensional indexing vector that retrieves relevant semanties from this shared pool. This design offers four advantages: First, its expandable nature allows for continuous integration of new semantic concepts, addressing issue one. Second, the shared pool by design is significantly smaller than the number of Gaussian points, i.e., 200 vs. 5 x 10S, reducing memory overhead and enabling scalability (issue two). Third, each Gaussian explicitly aggregates 3D sponnt-specific semanties (issue three). Besides, fourth, since local Gaussians representing the same object should exhibit similar semantics, this deign diminishes local semantic redundancy and promotes field learning.
Moreover, to enhance semantic coherence (issue four), we implement a consistency-aware opti- mization strategy ensuring that semantic representations align both within frames (intra-frame and across frames (inter-frame). We employ contrastive learning to enforce these correspondences and introduce an Intra-Inter Semantic Consistency Objective during semantic field learning. Additionally, we introduce a Semantic Stability Guidance mechanism to mitigate semantic ambiguity, where different semantics may be attributed to the same object. By measuring inter-frames pixel-to-object semantic consistency through cosine similarity, we adjust the learning signal, reducing influence from inconsistent regions while amplifying signals from consistent regions, thereby enhancing the overall coherence and stability of semantic representations within our system.
For examination, we compare our framework with both NeRF-based [18] and JDGS-based [19, 20] SLAM methods on Replica [21]. Experimental results demonstrate that our system establishes a precise metric-semantic field. As seen in $4.2, we enhance current SLAM frameworks on rendering quality and tracking accuracy with 1.32 3.50 PSNR gains and over 0.07 ATE RMSE reductions. In $4.3, we show that our framework accurately capture closed-set object categories (0.13% - 9.54% mIoU improvements). In $4.4, our framework showcases the ability of 3D explicit open-set semantic reconstruction, facilitating 2D segmentation (over 7.2% mIoU improvements), 3D object localization and 3D scene editing. Notably, our method is applicable on in-the-wild scenes captured by hand- held everyday devices and able to construct open-set semantic representation (64.4).
2

---

Published as a conference paper at ICLR 2026
2 RELATED WORK
3D Scene Representation. For a long time, point clouds have served as one of the most widely used representations for 3D scenes, due to their simplicity and compatibility with classical reconstruction pipelines. Several studies [22, 23] have further attempted to enrich point clouds with semantic segmentation and object-level understanding. However, most of these approaches focus on inter- preting pre-reconstructed point clouds, rather than jointly optimizing both geometry and semantics directly from multi-view images. Recently, NeRFs [24] have emerged a pivotal technique, enabling high-quality end-to-end learning of both 3D appearance [25-27] and open-set semantic information [28-31]. However, NeRF-based methods face inherent limitations, including slow training times and difficulties in precise 3D region identification due to their implicit representations. 3DGS [13], an explicit representation, offers efficient and accurate region identification while maintaining re- construction quality and boosting speed. Although recent works [32-39] have advanced the field, these SfM-based methods struggle with scalability large scenes. response, we introduce an expandable semantic representation to current SLAM works, catering to expansion demands.
Dense Visual SLAM. Recent advancements [18, 40-47] have integrated NeRFs into SLAM frame- works, including hierarchical multi-feature grids and uncertainty estimation, yet their implicit repre- sentations still incur high computational cost. 3DGS leads to a surge in SLAM research [19, 20, 48- 60], improving reconstruction quality and efficiency. However, both NeRF- and 3DGS-based methods mainly enhance RGB mapping, neglecting the critical role of semantic representation in expanding the practical applications of SLAM. To bridge this gap, some works [61-68] incorporate semantic information into dense SLAM. Nevertheless, these methods, utilizing closed-set semantic data, lack the adaptability required for continuous integration into any scene. Some latest works [69, 70] employ a label generation approach, still confined to the label-correspondenc paradigm, exhibiting limited capability when handling open-set queries (e.g., 'fruit in plastic bag'). Our work aims to integrate precise open-set semantics into SLAM systems, thereby significantly improving their ability understand and adapt arbitrary scenes, enabling more effective applications such as autonomous driving [71] and vision-language navigation [72-75].
Semantic Pooling. Directly embedding high-dimensional semanties into 3D Gaussians incurs prohibitive memory cost, a critical issue overlooked by many works [14, 15, 76-81]. Inspired by earlier efforts [82-84], some studies [16, 17] attempt to mitigate this by employing pre-built hard-indexing codebooks with fixed sizes. However, this approach suffers from two fundamental limitations: (i) pre-built codebooks cannot dynamically adapt to incremental scene observations; (ii) hard-indexing and fixed-size codebooks fail to generalize to novel or rare objects in scalable scenes [85, 86). To address these challenges, we propose a learnable, soft-aggregated and expandable semantic feature pool, which enables adaptation to sequential SLAM inputs.
Contrastive Semantic Learning. Image-level pre-trained models (e.g., CLIP [II] fail to ensure consistent semantics across frames. While SfM-based methods average semantics across views, overlooking inconsistencies, SLAM systems, processing data sequentially, are more sensitive to such issue. In 2D self-supervised learning, instance discrinination [87] distinguishes images negative samples [88, 89], excelling in image classification [90-92]. We adopt pixel-level contrastive learning, contrasting intra-object affinities against inter-object ones [93-100], ensuring single-frame semantic consistency. In 3D, self-supervised learning [101-111] is still developing. GarField [29] applies contrastive supervision for only inconsistencies. Some other works [80] focus only on closed-set classification, lacking open-set semantic field reconstruction. Video-based methods track objects [112-114] and pixels [115-119] across frame, but they often introduce excessive complexity and computational overhead, unsuitable for SLAM. Our approach leverages 3D scene to establish frams- pixel correspondence, ensuring semantics consistency while maintaining efficiency and scalability.
consistency, overlooking the problem of inter-frame
3 METHODOLOGY
Given an RGB-D stream, our system enhances 3DGS-SLAM frameworks through an expandable semantic representation. Each Gaussian retrieves semantics via key-based aggregation from shared feature pool, which can dynamically store new semantics and update existing ones ($3.2). During mapping, we optimize semantic parameters through: (1) Intra-Inter Consistency Objectives enforcing cross-view feature stability, and (2) Semantic Stability Guidance preventing inconsistent semantics from affecting reconstructed semantic fields ($3.3). An overview is in Fig. 2.
3

---

Published as a conference paper at ICLR 2026
3DGS-Based SLAM ($3.1.2)
Ope Semantic Gaussian Splatiing SLAM ($3.2)
Intra-Inter Somandic
Semontte Stabliny
Consistency
Guidance
Beeekeer
Pool
pos sapple
update
Mac
pot porsanple
Rerdene
sataples
Festure
MaG 1
Leo
upslatic
nepeamplls
current
chspansion
Cs
Reedeeee
Sen, Prane
Fentume
CD,
Ursaadite Deys Frane
Pool
Ce
Expandable Representation (93.2.1)
Grracking Ay,ofo + An,of + Art.s (Ta)
Jestenn spacto
Tackai speat 3D Semanric Fiovsdave
Coopping Asrifo Aa
3D Gaassinn
+Aa,,6(Es Mac Goo)
KP Fy
Semantic Aggregation (33.2.1)
Framework Optimization ($3.3)
Figure Framework Overview. enhance existing 3DGS-base SLAM with an expandable semantic representation, introducing Icarnable semantic feature pool that stores condensed scene-level semantics and supports dynamic expansion. Each Gaussian retrieves its semantic feature via soft aggregation from the shared pool through lightweight key. To improve cross-view and temporal consistency, further introduce Intra-Inter Semantic Consistency Objective and Semantic Stability Guidance mechanism, enabling stable and coherent open-set set semantic reconstruction during SLAM.
3.1 PRELIMINARIES
3.1.1 3D GAUSSIAN FIELD
Gaussian field G represents an explicit 3D scene with set of 3D Gaussians:
== (9: (mi, E., 0,,0..0 c.))) =
(1)
where each Gaussian G. contains its position μ4; R3, covariance E, 3x3 opacity a; E [0.
and color C, E g³. Specifically, given a scaling matrix S, E g3x3 and rotation matrix R; E g3x3 the covariance matrix E, computed:
Σ, = R,S, R. which describes the shape, size, and orientation of G..
Color Rendering. Given a viewpoint with its camera pose Tew E R1xx4, the Gaussian Gj can be projected to 2D plane for rendering [120]:
Σ J.W.W. J.
(3)
where Jj E 2x3
3 is the Jacobian matrix of the Gaussian centroid's projection on 2D image plane in camera coordinates, and W, E 3gx3 is the rotation matrix of Tcw- The rendered color of pixel can be computed via sorting Gaussians by depth and perform front-to- back o-- a-blending rendering:
C(p)
リ Σ,,, Cjaz
N
(1= ax),
(4)
k-1
where cj E R the color of j-th Gaussian of the sorted Gaussians, ay is computed via:
aj oj exp(- - (E))
(5)
where oj, oj are the opacity and the offset between p and the center 2D projection of j-th Gaussian.
3.1.2 JDGS-BASED SLAM
Given Gaussian field constructed from frames time 1 to t 1, along with RGB and depth frames at f, JDGGSbbaeed SLAM framework performs tracking and mapping.
Tracking determines the current camera position with incoming data, optimizing only camera parameters with Gaussian parameters fixed. For each timestep. the camera pose is estimated by forward-projecting pose parameters from the camera center into quaternion space.
4

---

Published as a conference paper at ICLR 2026
Mapping generates scene representation with 3D Gaussians. With the estimated position and the input depth frame, it adds Gaussians to underdeveloped areas, with the poses fixed while Gaussian parameters updated. Large or low-opacity Gaussians are removed.
3.2
OPEN-SET SEMANTIC GAUSSIAN SPLATTING SLAM
Our framework extends existing EDGS-based SLAM with an expandable semantic representation. In 13.2.1, we detail the representation; In $3.2.2, we introduce the rendering process of our framework.
3.2.1
EXPANDABLE SEMANTIC REPRESENTATION
We enhance each Gaussian with a semantic feature. To lower memory costs while better adapting to requirements of SLAM, we propose a randomly initialized, learnable and expandable semantic pool.
Initialization. Each Gaussian Gs: from a semantic Gaussian field Gs is assigned De-dimensional key k2 E Pt. Os also contains a Dy De-dimensional key pool Kp E LL Da and a D2 -d-dimensional semantic feature pool Fp TR. LxD_, both learnable, of the same size L and D& D,:
Gs = {Gst (Ju, Σ,, 0s C,, k,), Kp, Fr] Aggregation. To obtain a semantic feature, 9s; first identifies m nearest neighbors (supp. (B.8) of k; from Kp p with cosine similarity:
(6)
NN... (k:, Kp) arg max (cuss (cos(k, Kp)),
(7)
where cos and arg max operate along the size dimension (L), and NN, (k;, Kp) Rm. determine importance based on the magnitude of similarity and convert it into weights:
We
w. = softmax(NN,,. (k; KP))-
(8)
Subsequently, Gs: aggregates a semantic feature from Fp with wn E R!::
8, w. Fy.
(9)
where E DRD, represents the semantic feature of Gsi. and FpP E RmxD, contains m semantics from Fr, the same positions of these m nearest neighbors Kr.
Expansion. At initialization, the semantic feature pool Fp and key pool Kp are randomly initialized, and we additionally store copy of the initial semantic pool, denoted F(") which serves as a reference for tracking feature utilization. When a new semantic frame is received, we extract unique semantic features and determine whether each of them corresponds to an existing semantic concept. Specifically, we compute cosine similarities between the incoming feature and all entries the current pool Fp. If the top-m similarity scores exceed a predefined threshold, the feature is considered non-novel and will be aggregated by existing pool entries; otherwise, it is marked as novel and scheduled for insertion into the pool. Before inserting novel features, we check whether sufficient capacity exists in Fp. Rather than maintaining explicit usage counter, similarity-biee criterion to identify empty slots. Concretely, for each entry in the current pool
adopt
Fp. we measure its similarity to the corresponding entry in the initial pool F(") Entries whose similarity falls below a predefined threshold are considered to have deviated substantially from their initial random state, indicating that they have accumulated meaningful semantic information and are thus occupied. Conversely, entries that remain close to their initial state are treated as empty slots available for storing new semantic features. If the number of available empty slots is insufficient to accommodate the incoming novel features, we expand both the semantic feature pool Fy and the key pool Kp by a factor of n. Specifically, we append randomly initialized and learnable segments of size ((n - L,, D.) and ((n- L,, D&) to Fp and Kp respectively, resulting in expanded pools Fp E RmLxD. D. and Kp E D., The corresponding expansion segment is also appended to F(O) to maintain a consistent reference for future empty-slot detection. This expansion operation is infrequent in practice and exhibits a self-limiting behavior the pool gradually captures the semantic diversity of the scene. Implementation details and threshold selection are provided in supp. $B.8, and the pseudocode is in Algorithm in APPENDIX.
3.2.2 METRIC AND SEMANTIC RENDERING
To render per-pixel color, employ Eq. 4. Additionally, to optimize the positions of Gaussians, render the depth pixel p with a formulation analogous to Eq. 4:
.V D(p) = Σ d,a5 (1 a2).
(10)
where d, represents the depth of the j-th Gaussian.
5

---

Published as a conference paper at ICLR 2026
To optimize the semantic field with Scr, we render semantic frame after assign semantics to Gaussians. contrast, previous codebook-base methods [16, 17] render feature maps first and then assign semantics to each pixel, which means they do not truly construct 3D semantic field. During rendering, we only select Gaussians within the current view to assign semantics and render the semantic feature of the pixel p:
S(p) Σ
j=1 By0; II ax).
(11)
where is the allocated semantic feature of the j-th Gaussian.
3.3 FRAMEWORK OPTIMIZATION
Our framework first distills semantic features with pre-trained models from RGB images (supp. $A.1).
Given field reconstructed from time I to to - 1, with new RGB, depth, semantic feature frames CGT. Der, Ser and rendered frames C, D. S at t, our framework optimizes camera poses via tracking
loss ($3.3.1) and optimize Kp, Fp and Gaussian parameters with a mapping loss ($3.3.2).
3.3.1 TRACKING LOSS
For each pixel p. camera poses optimized with:
Liracking E,, (rr.cerrc + År.DCT,D + Xr.s,r,s),
(12)
where Cr,C, LT.D. Lr.S are tracking losses for color, depth and semantics, respectively. Lr.s [[(p) - (c)(p)],, and Cr.c. Cr, are identical to those in the SLAM baseline. For example, in SplaTAM [20], L [|C(p) Ccr(p)l Cr. = D(p) Der(p)ll-
3.3.2 MAPPING Loss
Intra-Inter Semantic Consistency Objective. The semantic field is reconstructed from a temporally- ordered sequence of images. However, the semantics retrieved from 2D open-set models, which often produce viewpoint-dependerr and temporally inconsistent predictions. In particular, the same physical object may be assigned different semantic embeddings across frames, since 2D open- vocabulary features are not explicitly aligned over time. This inconsistency leads to contradictory semantic signals in the reconstructed map and hinders the effective learning of coherent 3D semantic field. To address it, we enforce semantic consistency by progressively aggregating semantic evidence from multiple observations. suppressing noisy or inconsistent predictions across viewpoints.
For pixel it should share the same semantic as other pixels from the same object in the current frame (intra-frame, S), and pixels from the same 3D locations in previous frames framer, S,). Therefore, we select Q neighboring pixels {p: ) of p from the same object in S and treat their semantics as intra positive samples. We then project into 3D space using depth and reproject it onto S, to find corresponding pixels {p,,, 1tt whose semantics serve as inter positive samples, and R represents the number of selected samples. Semanties of remaining pixels from S are treated as negative samples and added to a negative sample pool N
For an intra positive sample S(p,") ), we enhance the similarity between S(p) and and reduce the similarity between Sp) and negative samples S(p-):
exp(S(p) S(p:))
[(pt) log Estp exp -xp(S())pS(pp))
(13)
similar (p.,)) can be calculated for inter positive samples (p likewise. Thus, Intra-Inter Semantic Consistency Objective is computed:
Lco 1- E., c(p)) d E C.(P))))
(14)
R
where X, is hyper-parameter balance intra and inter objectives.
Semantic Stability Guidance. To further mitigate the adverse effects of inconsistencies between SeT and the current seman- tic field, we introduce a Semantic Stability Guidance Msc-
fige simibrity
MsG f
For a pixel in S, we use the same projection rules from the Intra-Inter Semantic Consistency Objective to locate corre- sponding pixels in previous frames chosen based on the overlap with the current frame, identifying features the correspond- ing segmentation regions that corresponding pixels belong to.
Jovsimilrriy
Msc
Figure 3: Semantic Stabiltyy Guidance.

---

Published as a conference paper at ICLR 2026
SplaTAM [2][CCPX24)
SplaTAM + Ours
Ground Truih
SplaTAM [V0224)
SplaTAM + Our
Ground Truth
Figure 4: Qualitative Rendering Quality Comparisons ($4.2) over SplaTAM [20] on Replica [21].
If p belongs to an object that appears for the first time with no information to guide it, MsG (p) is set to 1. Otherwise, calculate the cosine similarity between S(p) and the average feature of all corresponding regions, using this similarity as (p)
Overall Mapping Loss. We
We jointly optimize the color, depth and semantic feature of p:
Emapping AM,.CM. + AM.DCM.D + AM.sCM,s,
(15)
where Ca.cc LM.DD, Lx.s are mapping losses for color, depth and semanties, respectively. LM.S As [ps(p) - (cr(p)] Mp)(p) + (1 Cco. and similarly, LM.C, LM.D are identical to the SLAM baseline, egg., SplaTAM [20]. CM.C (1 Xc) Ccr(p) + Ao
(S)) Cer (p))), LMDD = ((p) - Der (p)ll.
EXPERIMENT
4.1 EXPERIMENTAL SETUP
Datasets. Our evaluation protocol strictly follows the prevailing practice (e.g., SGS-SLAM [64]). Experiments are carried out on eight scenes of Replica [21]. Results on real-world data TUM [121]
and ScanNet [122] are delivered in supp. $B.
Implementation Details. Our experiments GPU. DR, D., L, Q, R, m, are set to 3, 32, 25, 1, 5, respectively. The optimization involves nine parameters, A, 0.5, λT,C AM,c 0.5, AT AM.D 1.0, λr.s AM,S 0.05.
server with single NVIDIA GeFFrce RTX 3090
Ac = 0.2, and Àg = 0.999.
4.2 TRACKING AND APPEARANCE MAPPING RESULTS
Baselines. We apply our method on SplaTAM [20] and LoopSplat [19], and compare them with Point-SLAM [18], LS [67], SplaTAM [20], and LoopSplat [19].
Metrics. We adopt standardized set of metrics to evaluate both appearance mapping and camera pose estimation. RGB rendering performance measured by PSNR, SSIM [123] and LPIPS [124]. Camera pose tracking is assessed by the average absolute trajectory error (ATE RMSE) [121]. Geomatry reconstruction evaluated using Depth L1.
Rendering, Our method outperforms NeRF- Point-SLAN [18], shown in Table la. Compared 3DGS- based SplaTAM [20] and LoopSplat [19], the inclusion of semanties helps to better capture scene features such as object shapes, further enhancing reconstruction quality. We also surpass current SOTA closed-set semantic SLAMM system LS [67].
Camera Pose Estimation. Shown in table 1b, we outperform Point-SLAM [18]; Semantics provides frameworks with more details about the scene, leading to more precise trajectory estimation than SplaTAM [20] and LoopSplat [19].
Geometric Reconstruction. As shown Table 1, our approach outperforms Point-SI.AM [18]. We also surpass SplaTAM [20] and LoopSplat [19]. showing that our semantic reconstruction helps to better capture geometry information. Noteworthy, LL [67] did not report this result in its paper; the data comes from reproduction of the code. Although GS [67] introduces closed-set semantic reconstruction, our consistency objective and stability guidance actually enhance the system's 's sensitivity to geometry, allowing to significantly outperform it.
Qualitative Comparisons. Fig. demonstrate the superior performance of our method. Compared against SplaTAM [20), the integeration of semantics helps to capture scene details such as geometry.
7

---

Published as a conference paper at ICLR 2026
Mesics
Aug.
8-0
020
023
0.4
Poini-SLAM [18]
PSNRT
SSIMT
0.975
B517
0.974
1244
63977
18
0.982
15.50
0.883
18
0.986
16
0960
3559
0.96
15.88
0.979
35,47
hiit-SLAM [1]]
1.52
061
041
0:37
0.36
0.46
0.54
0.0
LPPS
0,124
0.113
0.116
1,100
0.1
0,156
0,132
0,142
GS'LAM [67]
0.27
0.2
0.25
0.26
0.6
0.1
0.53
0:30
065
167]
PSNK
SSIM
0.999
36.26
0,96
0.990
0.993
0.994
0.988
0.90
0,983
SplaTAM a Ours
SpetAM AM]
0.29
1.36
0.3
0.34
0:40
0.29
0.29
0.26
0.1
0.29
0.21
0.20
0.25
0.20
0.22
0.30
0.66
LPPPS
0.052
Loopspin 10
0.66
122
022
0.22
0.16
0:39
022 0.20
SphTAM (20]
FSNK
SSIM
0.97
14.11
0380
12.66
0:970
35.0
35
960
16
0.000
36
0.960
17
0.970
01.07
0.950
20.70
0.950
11.5
(b) Camera Pose Fstimation (ATE RMSEL [em]).
LompSplat
machia imp lay
0.23
Glount gataa bundle udjastme..
0.16
LPIPS
0.100
0.700
0.100
0.080
0.050
0,060
0,100
0.120
0,150
SpleTAM Ours
PSNR
SSIM
LPPS
0,990 mm
0,500
3728--3
0.48
84
0.955
0.990
0:041
27.9%
0.990
0.050
37.37
41.46
0.045
0,992
42.10
0.044
0.993
0.99M
0.053
25.70
0,047
35.59
0.9900
0.076
0,996
35:34
Point SLAM
USLAAM [6/]
Medhods
0.4
AVE
0.3
20
0:64
21
0.59
GED
0.47
0.36
021 OE2 023 024
0.48
051
0.50 0.6
Epep [19]
FSNR:
SS0
0.985
36,63
0.973
16.16
0.992
0.950
40.21
0.950
15.01
0.989
SpatAM
TAM Dam
6M
0.4
0.26
0.34
0:18
0.44
0.26
0.15
0.32
0.44 0.49
LPPSS
0:112
0.116
0.111
0.085
0.123
0.096
0.106
LompSpaa [19]
0.51
1.39
033
0.52
0.32 0.51
0.61
1.00 0.40
PSNK
37.85
34.55
4220
42221
35384
35339
37.12
LompSplat Own
0.30
0.34
0.21
0.36
0.24
0.27
0,30
0.36 0.36
LoopSplat + Ours
LPIPS
SSIM
0.033
1.89.....
0.056
0.979
0.048
0:987
0.990
0.360
0.047
0,993
0.045
0,923
0.960
0,993
0.055
0,089
0.157
(c) Reconstruction Performance (54.2) (Depth L.1, [em]).
(a) Rendering Performance (54.2).
Table 1: Quantitative Tracking and Appearance Mapping Results Replica [21].
Nesson choles
"han
Spesiik
Womee"
"oln Mike flnwes
Figure 5: Qualitative Results of Object Localization (84.4) on Replica [21].
4.3 CLOSED-SET SEMANTIC MAPPING RESULTS
Baselines, We compare our method with SNI-SLAM [63], SGS-SLAM [64], GA [67] and Hier-SLAM++ [68], using the basic framework of SplaTAM [20].
Metrics. We employ the mean Intersection over Union (mloU) as our evaluation metric.
Results. Table presents comparisons across four scenes from Replica [21]. For fair comparisons, our results are obtained with classification head on top of semantic features (supp. $A.2), super- vised by ground truth labels. Our method out- performs NeRF-based based SNI-SLAM [63]. Further-
Methods
Aug.
R:0
Closed-set
n!]
32
cro
2D mlat)
Open-set
SNL-SI [63]
SGS-SLAM [6(]
GS'LAM (67)
92.72
96.83
92.95
96.83
96.68
92.91
96.40
92.10
96.61
92.90
NIA
Hie--S [68]
90.35
91.21
90.62
89.11
90.45
SpliTAM + Ours
96.76
96.93
96.82
96.87
96.62
66.5
Table 2: Comparisons with semantic SLAM ($4.3).
more, by integrating the Intra-Inter Semantic Con- sistency Objective and Semantic Stability Guidance, our framework maintains consistent semantics across frames, and surpasses 3DGS-based frameworks. We also conduct experiments on open-set semantic segmentation ($4.4), which is not possible with previous closed-set SLAM.
4.4 OPEN-SET SEMANTIC MAPPING RESULTS
Datasets. Experiments are carried out on Replica [21] and in- thehheewld data captured by our everyday devices. Details our in-the-will data are presented in supp. $B.6.
Baselines, We apply our method based on SplaTAM [20]. Our competitors include SiM-based methods (i.e., LERF [28], LEGaussians [16] and GOI [17]).
Metrics. For segmentation, we employ mIoU as our metric, same as in GOI [17]. For localization, use acc 0.25 and 0.5 metrics. Details localization evaluations are in supp. $B.7.
Semantic Segmentation Localization. Com- parisons with SfM-based methods are in Table 3. While the baselines require COLMAP and full image sequences for camera pose estimation, we estimate poses online solely from RGB-D input. Results demonstrate that we construct a more accu-
Keplica [2]]
In-the-wiic data
Methods
2D
3D Localization
2D
3D Localization
mlooi
pco6eD 25 acce00.5
mial
$50.55555 ace480.5
LERFT88
28.2
20.6
I.Ganosians
39.4
N/A
32.1
N/A
GDI [12)
61.7
49.5
SpliTAM + Ours
66.5
54.7
45.1
60.3
$1.2
42.9
Table 3: Comparisons with SIM-hased Open-Set
rate semantic field. We also conduct experiments on 3D localization, where we match the similarity most relevant Gaussian sets. Other bidebook-based methods [16, 17] cannot construct an explicit 3D semantic field and interact with Gaussians directly, consistent with our previous discussions ($3.2.2).
Semantic Methods ($4.4).
between input text and Gaussians identify the
8

---

Published as a conference paper at ICLR 2026
Figure 8:
Visualizations ($4.4) on three in-the-wild scenes captured by everyday devices. Top: Whole scenes. Bottom: Rendered images. Details on the data of scenes are in supp. $B.6.
Open-set Semantic Visualizations. Visualiza- tions of RGB, depth, open-set and closed-set semantics on Replica [21] Room 0 are in Fig. As seen, we able reconstruct high-quality appearence with high geometric accuracy. We
Depth
Open Clased
Renderel GT Renderad Gi
are also able to reconstruct an open-set semantic
field with clear edges, which is beneficial for
downstream tasks such as 3D scene editing.
Open-Vocabine Vocabulary Object Localization. Given an open- open-vocabulary query, our method generates highly concentrated activation regions that align precisely with the ground truth shape. Moreover, as shown in Fig. 5,r field can identify specific objects when provided with both general queries (e.g., "plant ") and specific queries (e.g., "plant like flowers"), capability that closed-set methods lack.
Figure 6: Visualizations ($4.4) of RGB, depth, open- set and closed-set semantics on Replica [21] Room 0.
3D Editing. We evaluate our open-set seman- tie representation for 3D editing in Fig. 7. Our method achieves object editing while maintain- ing the integrity of the surrounding environment, showing the accuracy of our semantic field.
"Tarn cabinet red.
Everyday Devices Results. We evaluate
framework on four in-the-wild scenes captured by our hand-held everyday devices in Fig. and Figure 7: Results of 3D Editing ($4.4) on Replica [21]. 8, demonstrating the ability of constructing dense color maps and open-set semantic fields with clear boundaries noval scenes with only RGB-D data, even without any additional annotation.
flower
4.5 DIAGNOSTIC EXPERIMENTS
Experiments are conducted on Replica Room 0 based on SplaTAM.
PSNR
RMSE I mloU
ATE
Key Component Analysis. In Table 4, we validate the importance of our proposed components by attaching them one at time. The 1s row directly assigns a 3-dim semantic to each Gaussian. In the 2nd row, we utilize Fp. resulting in performance improvements. The 3rd row and the 4th row attach Lco and MsG respectively, further improving performance, which shows that our method can effectively ensure semantic consistency.
vala All
32,88
0.30
72.56
4 Coo MMa
33.67
0.25
84.23
33.81
0.24
87.29
Table 4: Key Component Analy-
sis (D.=3) ($4.5).
Semantic Feature Dimension. We examine how the se- mantic feature dimension affects semantic reconstruction in Table 5, where X = 16.0 GB. Without Fp. when D; 6, allocating feature each Gaussian incurs substan- tial memory consumption, leads to experimental failure. While with Fr, are able utilize features with larger
15 GPU Mem. Lsnge
PSNKT
miotir
w/o Fy
12.88
72.56
~(X 12.00
(X GB
33.81
87.29
w/ Fp
8
(X + GB
33.97
96.93
88
64
(X 3.7)
34.67
96.73
Table 5: Semantic Feature Dimension (84.5).

---

Published as a conference paper at ICLR 2026
dimension than those without Fp, L.e,, 64 vs. 3, showeasing the superior semantic compression capabilitiies of our pool. Additionally, low-dim features are inadequate for high-quality reconstruction, while higher dimensions provide diminishing returns.
Semantic Pool Design. We compare four pool designs: i) implicit design of sequential internal state updates through LSTM-style gates; ii fixed-iize explicit pool updated via
mloU
Traiming
Time
GPC Mem.
Baseline
72.36
Xh
G
Implill Matred
7729
[X U500h
(V +0330
8r.04
1X
1
L8TGB
gating mechanisms similar to LSTM; iii) compressive ex- plicit pool employing content-aware compression for capac- ity management; and iv) expandable explicit pool as our
Compresivv
b
Expundsbble
96.93
+ 1.99h
( 2.4)
Table 6: Semantic Pool Design ($4.5).
default setup. Baseline employs no pool with D, = 3, and X 6.0 h, Y 16.0 GB. As shown in Table 6, our expandable pool achieves superior accuracy with moderate memory and computation cost, maintaining practical resource utilization. The implicit variant achieves fastest training and minimal memory consumption, but suffers severe accuracy degradation. Fixed-size variant fails to maintain stable performance across varying scenes, and compressive pool loses fine-grained temporal information. Besides, surprisingly, expandable pool achieves faster speed compared to compressive pool, due avoided compression overhead.
Large-Scale Scenes Performance. We conduct experiments on a large-scat scale in-the-wild scene comprising 12,000 frames (supp. $B.6), comparing our method with SGS-SLAM [64] and LAM [67]. Since both SGS-SLAM and LAM require supervision from labels, we use generated pseudo- semantic labels (supp. $B.8). Shown in Table 7, both SGS-
GPU Mem. Usage
Method
Complete
SGS SLAM [5]]
>:24 GIB
GS*LAM [67]
5-24 GB
SplaTAM Ours
19.5 GB
Table 7: Large-Scale Scenes Perfor-
mance ($4.5).
SLAM and GAM, which assign 32-dim features to each Gaussian, terminate prematurely due to GPU memory overflow. contrast, our approach runs stably until full reconstruction. During this process, the size of Fp expands from 25 to 400 to accommodate additional semantics. Despite this significant expansion, our method still maintains lower memory consumption compared to direct assignment. These results demonstrate our method's effectiveness in handling large-scale environments and long sequence inputs.
5 CONCLUSION
We present Open-Set Semantic Gaussian Splatting SLAM, enabling everyday devices to reconstruct 3D scenes with rich and expandable open-set semantics. By aggregating semantics for 3D Gaussians through shared and lightweight representation, our method significantly reduces memory costs while supporting continuous and scalable semantic updates during SLAM. Furthermore, the proposed consistency-awe aaasttency-awareeeee optimization improves cross- cross-view semantic coherence, leading to more stable and reliable semantic fields over time. Experiments demonstrate high-fidelity rendering quality together with scalable semantic reconstruction across diverse scenes. This work advances accessible, high- quality, and expressive 3D world modeling, and provides practical foundation for semantic-aware applications such as open-vocabulary localization and interactive 3D scene understanding.
6 LIMITATIONS
This work focuses on introducing scalable open-set semantic reconstruction into SLAM systems, rather than redesigning the tracking mapping back-end. As result, our framework is implemented as a modular extension on top of existing SLAM pipelines, and its runtime performance as well robustness are largely bounded by the underlying baseline system (see supp. $B.4). While this design enables broad compatibility with different 3DGS- based SLAM frameworks, is also possible that further improvements in efficiency, robustness, real-time performance would likely require tighter co-design of the SLAM back-end and the proposed semantic components.
In addition, similar most existing dense SLAM systems, our approach assumes static or weakly dynamic environments and may exhibit limited robustness in highly dynamic or extremely large- scale settings, such scenes with frequent object motion or crowded environments. Although the expandable semantic pool and the proposed consistency-awant optimization provide a foundation for aggregating object-level semantics over time, explicit modeling of object motion and long- term semantic drift is beyond the scope of the current system. Addressing these challenges would require additional mechanisms, such as dynamic- mapping, motion segmentation, or temporal semantic association, which we leave for future work.
10

---

Published as a conference paper at ICLR 2026
ACKNOWLEDGEMENT
This work was supported by "Pioneer" and "Leading Goose" R&D Program of Zhejiang (No. 2024C01 161), Fundamental Research Funds for the Central Universities (226-2025-00057, CIE- Tencent Robotics XRhino-Bird Focused Research Program, China Postdoctoral Science Foundation (No. 8025T1804211 Postdoctoral Fellowship Program of CPSF (No. GZC20251066), and Postdoc- toral Science Preferential Funding of Zhejiang (No. ZJ2025001).
REFERENCES
[1] Nitin Gupta, Alan Demers, Johannes Gehrke, Philipp Unterbrunner, and Walker White. Scalability for virtual worlds. In ICDE, 2009. 1
[2] John David N Dionisio, William G Burns lii, and Richard Gilbert. 3d virtual worlds and the metaverse: Current status and future possibilittis. CSUR, 2013.
[3] Apple. Apple vision pro. http::/wwwwaappccccco///pppststs 2023.
[4] Meta. Metaverse. In
2024. 2
[5] Chengshu Li, Ruohan Zhang. Josiah Wong, Cem Gokmen, Sanjana Srivastava, Roberto Martín-Martía, Chen Wang. Gabrael Levine. Michael Lingelbach. Jiankai Sun, al. Behavior- Ik: A benchmark for embodied with 1,000 everyday activitics and realistic simulation. In CoRL, 2023, 2
[6] Wenlong Huang, Chen Wang, Ruohan Zhang, Yunzhu Li, Jiajun Wu, and Li Fei-Fei. Voxposer: Compos- able value maps for robotic manipulation with language models. CoRL, 2023.
[7] Xavier Puig, Eric Undersander, Andrew Szot, Mikael Dallaire Cote, Tsung- Yen Yang, Ruslan Partsey, Ruta Desai, Alexander William Clegg, Michal Hlavac, So Yeon Min, et al. Habitat 3.0: A co-habitat for humans, avatars and robots. CoRR, 2023.
[8] Tai Wang, Xiaohan Mao, Chenming Zhu, Runsen Xu, Ruiyuan Lyu, Peisen Li, Xiao Chen, Wenwei Zhang, Kai Chen, Tianfan Xue, et al. Embodiedscan: A holistic multi-modal 3d perception suite towards embodied ai. CVPR, 2024.
[9] Wenxiao Cai, Yaroslav Ponomarenko, Jianhao Yuan, Xiaoqi Li, Wankou Yang. Hao Dong, and Bo Zhao. Spatialbot: Precise spatial understanding with vision language models. In ICRA, 2025. 2
[10] Santhosh K Ramakrishnan, Aaron Gokaslan, Erik Wijmans, Oleksandr Maksymets, Alex Clegg, John Turner, Eric Undersander, Wojeiech Galuba, Andrew Westbury, Angel X Chang, al. Habitat-matterpor 3d dataset (hm3d): 1000 large-scale 3d environments for embodied ai. CoRR, 2021.
[11] Alec Radford. Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning transfcrable visual models from natural language supervision. In ICML, 2021. 2, 3, 18, 19, 24
[12] Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloé Rolland, Laura Gustafson, Tete Xiao, Speneer Whitehead, Alexander C. Berg, Wan-Yen Lo. Piotr Dollár, and Ross B. Girshick. Segment anything. /CCV, 2023. 2, 18,
[13] Bernhard Kerbl, Georgios Kopanas, Thomas Leimküihler, and George Drettakis, 3d gaussian splatting for real-time radiance field rendering. In SIGGRAPH, 2023. 2,
[14] Minghan Qin, Wanhua Li, Jiawei Zhou, Haoqian Wang, and Hanspeter Pfister. Langsplat: 3d language gaussian splatting. In CVPR, 2024. 2,
[15] Shijie Zhou, Haoran Chang, Sicheng Jiang, Zhiwen Fan, Zehao 7hu, Dejia Xu, Pradyumna Chari, Suya You, Zhangyang Wang, and Achuta Kadambi. Feature 3dgs: Supercharging 3d gaussian splatting enable distilled feature fields. CVPR, 2024. 2.
[16] Jin- An-CChan Shi, Miao Wang, Hao-Bin Duan, and Shao-Hua Guan. Language embedded 3d gaussians for open-vocabulary scene understanding. In CVPR, 2024. 2, 3, 6,
[17] Yansong Qu, Shaohui Dai, Xinyang Li, Jianghang Lin, Liujuan Cao, Shengchuan Zhang, and Rongrong Ji. Goi: Find 3d gaussians of interest with optimizable open-vocabulary semantic-space hyperplane. In ACMMM, 2024. 8,8
11

---

Published as a conference paper at ICLR 2026
[18] Erik Sandström, Yue Li, Luc Van Gool, and Martin R. Oswald. Point-slam: Dense neural point cloud- based SLAM. In ICCV, 2023. 2, 7, 80,00,2 26
[19] Liyuan Zhu, Yue Li, Erik Sandström, Shengyu Huang, Konrad Schindler, and Iro Armeni. Loopsplat: Loop closure by registering 3d gaussian splats. In 3DV, 2025. 2, 3. 7, 8, 20, 21, 26
[20] Nikhil Varma Keetha, Jay Karhade, Krishna Murthy Jatavallabhula, Gengshan Yang. Sebastian A. Scherer, Deva Ramanan, and Jonathon Luiten. Splatam: Splat, track & map 3d gaussians for dense RGB-D SLAM. In CVPR, 2024. 2, 3, 6, 7. 8, 20, 21, 22, 23, 26
[21] Julian Straub, Thomas Whelan, Lingni Ma, Yufan Chen, Erik Wijmans, Simon Green, Jakob Engel. Raul MrraArtal, Carl Yuheng Ren, Shobhit Verma, Anton Clarkson, Mingfei Yan, Brian Budge, Yajie Yan, Xiaqing Pan, June Yon, Yuyang Zou, Kimberly Leon, Nigel Carter, Jesus Briales, Tyler Gillingham, Elias Mueggler, Luis Pesqueira, Manolis Savva, Dhruv Batra, Hauke M. Strasdat, Renzo De Nardi, Michael Goesele, Steven Lovegrove, and Richard Newcombe. The replica dataset: A digital replica of indoor spaces. CoRR, 2019. 2, 7, 8, 9, 20, 21, 22, 24, 27
[22] Ayça Takmaz, Elisabetta Fedele, Robert W Sumner, Mare Pollefeys, Federico Tombari, and Francis Engelmann. Openmask3d: Open-vocabulayy 3d instance segmentation. In NeurlPS, 2023, 3
[23] Songyou Peng, Kyle Genova, Chiyu Jiang, Andrea Tagliasacchi, Marc Pollefeys, Thomas Funkhouser, al. Openscene: 3d scene understanding with open vocabularies. In CVPR, 2023. 3
[24] Ben Mildenhall, Pratul Srinivasan, Matthew Tancik, Jonathan T. Barron, Ravi Ramamoorthi, and Ren
Ng. Nerf: Representing scenes as neural radiance fields for view synthesis. In ECCV, 2020.
[25] Thomas Müller. Alex Evans, Christoph Schied. and Alexander Keller. Instant neural graphics primitives with a multiresolution hash encoding. In SIGGRAPH, 2022." 3
[26] Jonathan T Barron, Ben Mildenhall, Matthew Tancik, Peter Hedman, Ricardo Martin-Brualla, and Pratul Srinivasan. ner-nerf: A multiscale representation for anti-aliasing neural radiance fields. In CVPR, 2021.
[27] Jonathan Barron, Ben Mildenhall, Dor Verbin, Pratul Srinivasan, and Peter Hedman. Zip-nerf: Anti-aliased grid-based neural radiance fields. In ICCV, 2023. 3
[28] Justin Kerr, Chung Min Kim, Ken Goldberg, Angjoo Kanazawa, and Matthew Tancik. LERF: language embedded radiance fields. In ICCV, 2023. 3,8 8
[29] Chung Min Kim, Mingxuan Wu, Justin Kerr, Ken Goldberg, Matthew Tancik, and Angjoo Kanazawa. Garfield: Group anything with radiance fields. In CVPR, 2024. 3
[30] Kunhao Liu. Fangneng Zhan, Jiahui Zhang, Muyu Xu, Yingchen Yu, Abdulmotaleb Saddik, Christian Theobalt, Eric Xing, and Shijian Lu. Weakly supervised 3d open-vocabulary segmentation. In NeurIPS, 2023.
[31] Francis Engelmann, Fabian Manhardt, Michael Niemeyer, Keisuke Tateno, Marc Pollefeys, and Federico Tombari. Opennerf: Open set 3d neural scene segmentation with pixel-wise features and rendered novel views. In ICLR, 2024. 3
[32] Yang Fu, Sifei Liu, Amey Kulkarni, Jan Kautz, Alexei A Efros, and Xiaolong Wang. Colmap-free 3d gaussian splatting. In CVPR, 2024. 3
[33] Joo Chan Lee, Daniel Rho, Xiangyu Sun, Jong Hwan Ko, and Eunbyung Park. Compact 3d gaussian representation for radiance field. In CVPR, 2024.
[34] Sharath Girish, Kamal Gupta, and Abhinav Shrivastava, Eagles: Efficient accelerated 3d gaussians with lightweight encodings. In ECCV, 2024.
[35] Jaeyoung Chung, Jeongtack Oh, and Kyoung Mu Lee. Depth-regularzeed optimization for 3d gaussian splatting in few-shot images. In CVPR, 2024.
[36] Zehao Yu, Anpei Chen, Binbin Huang, Torsten Sattler, and Andreas Geiger. Mip-splatting: Alias-free 3d gaussian splatting. In CVPR, 2024.
[37] Zehao Zhu, Zhiwen Fan, Yifan Jiang, and Zhangyang Wang. Fsgs: Real-time few-shot view synthesis using gaussian splatting. In ECCV, 2024.
[38] Haoyu Guo, He 7hu, Sida Peng, Haotong Lin, Yunzhi Yan, Tao Xie, Wenguan Wang, Xiaowei Zhou, and Hujun Bao. Multi-view reconstruction sfm-guided monocular depth estimation. In CVPR, 2025.
12

---

Published as a conference paper at ICLR 2026
[39] Rui Liu, Zhibo Duan, Jianzhe Gao, Yi Yang, and Wenguan Wang. Uncertainty-aware 3d reconstruction for dynamic underwater seenes. ICLR, 2026.
[40] Edgar Sucar, Shikun Liu, Joseph Ortiz, and Andrew Davison. imap: Implicit mapping and positioning in real-time. In ICCV, 2021. 3
[41] Zihan Zhu, Songyou Peng, Viktor Larsson, Weiwei Xu, Hujun Bao, Zhaopeng Cui, Martin R. Oswald, and Marc Pollefeys. NICE-SLAM: neural implicit scalable encoding for SLAM. In CVPR, 2022. 26
[42] Antoni Rosinol, John J. Leonard, and Luca Carlone. Nerf-slam: Real-time dense monocular SLAM with neural radiance fields. In IROS, 2023.
[43] Xingrui Yang, Hai Li, Hongjia Zhai, Yuhang Ming, Yuqian Liu, and Guofeng Zhang. Vox-fusion: Dense tracking and mapping with voxel-based neural implicit representation. In ISMAR, 2022. 26
[44] Youmin Zhang, Fabio Tosi, Stefano Mattoccia, and Matteo Poggi. GO-SLAM: global optimization for consistent 3d instant reconstruction. ICCV, 2023.
[45] Hengyi Wang. Jingwen Wang, and Lourdes Agapito. Co-slam: Joint coordinate and sparse parametric encodings for neural real-time SLAM. In CVPR, 2023.
[46] Xin Kong, Shikun Liu, Marwan Taher, and Andrew J. Davison. vmap: Vectorised object mapping for neural field SLAM. In CVPR, 2023.
[47] Mohammad Mahdi Johari, Camilla Carta, and François Fleuret. ESLAM: efffccinnt dense SLAM system based on hybrid representation of signed distance fields. In CVPR, 2023. 3, 26
[48] Hidenobu Matsuki, Riku Murai, Paul H. J. Kelly, and Andrew J. Davison. Gaussian splatting SLAM. In CVPR, 2024. 3,
[49] Vladimir Yugay, Yue Li, Theo Gevers, Martin R. Oswald. Gaussian-slam: Photo-realistic dense SLAM with gaussian splatting. CoRR, 2023.
[50] Chi Yan, Delin Qu, Dong Wang. Dan Xu, Zhigang Wang, Bin Zhao, and Xuelong Li. GS-SLAM: dense visual SLAM with 3d gaussian splatting. In CVPR, 2024.
[51] Huajian Huang, Longwei Li, Hui Cheng, and Sii-Kit Yeung. Photo-slam: Real-time simultancous localization and photorealistic mapping for monocular, stereo, and RGB-D cameras. In CVPR, 2024.
[52] Tianchen Deng, Yaohui Chen, Leyan Zhang, Jianfei Yang, Shenghai Yuan, Danwei Wang, and Weidong Chen. Compact 3d gaussian splatting for dense visual slam. CoRR, 2024.
[53] Mingrui Li, Jingwei Huang, Ici Sun, Aaron Xuxiang Tian, Tianchen Deng, and Hongyu Wang. Ngm-slam: Gaussian splatting slam with radiance field submap. CoRR, 2024.
[54] Erik Sandström, Keisuke Tateno, Michael Oechsle, Michael Niemeyer, Luc Van Gool, Martin Oswald, and Federico Tombari. Splat-slam: Globally optimized rgb-only slam with 3d gaussians. CoRR, 2024.
[55] Jiarui Hu, Xianhao Chen, Boyin Feng, Guanglin Li, Liangjing Yang. Hujun Bao, Guofeng Zhang, and Zhaopeng Cui. Cg-slam: Efficient dense rgb-d slam in consistent iucertainty-awat 3d gaussian field. In ECCV, 2024.
[56] Scongbo Ha, Jiung Ycon, and Hyconwoo Yu. Rgbd gs-iep slam. In ECCV, 2024.
[57] Shuo Sun, Malcolm Mielle, Achim J Lilienthll, and Martin Magnusson. Hlghhfideling slam using gaussian splatting with sonddering-uudee densification and regularized optimization. IROS, 2024.
[58] Zhexi Peng, Tianjia Shao, Yong Liu, Jingke Zhou, Yin Yang. Jingdong Wang, and Kun Zhou. Rtg-slam: Real-time 3d reconstruction at scale using gaussian splatting. In SIGGRAPH, 2024.
[59] Chong Cheng, Sicheng Yu, Zijian Wang, Yifan Zhou, and Hao Wang. Outdoor monocular slam with global scale-consistent 3d gaussian pointmaps. In ICCV, 2025. 20
[60] Tianci Wen, Zhiang Liu, and Yongchun Fang. Segs-slam: Structure-cnhancee 3d gaussian splatting slam with appearance embedding, In ICCV, 2025. 3, 20
[61] Yasaman Haghighi, Suryansh Kumar, Jean- Phillppp Thiran, and Luc Van Gool. Neural implicit dense semantic slam. CoRR, 2023. 3
13

---

Published as a conference paper at ICLR 2026
[62] Kunyi Li, Michael Niemeyer, Nassir Navab, and Federico Tombari. Dns slam: Dense neural semantic- informed slam. In IROS, 2024.
[63] Siting Zhu, Guangming Wang, Hermann Blum, Jiuming Liu, Liang Song, Mare Pollefeys, and Hesheng Wang. Sni-slam: Semantic neural implicit slam. In CVPR, 2024, 8
[64] Mingrui Li, Shuhong Liu, Heng Zhou, Guohao Zhu, Na Cheng, Tianchen Deng, and Hongyu Wang. SGS-SLAM: semantic gaussian splatting for neural dense SLAM. In ECCV, 2024. 7,8, 10, 22
[65] Yiming Ji, Yang Liu, Guanghu Xie, Boyu Ma, and Zongwu Xie. Neds-slam: A novel neural explicit dense semantic slam framework using 3d gaussian splatting. RA-L, 2024.
[66] Siting Zhu, Renjie Qin, Guangming Wang, Jiuming Liu, and Hesheng Wang. Semgauss- Dense semantic gaussian splatting slam. In IROS, 2025.
[67] Linfei Li. Lin Zhang, Zhong Wang, and Ying Shen. Gs3lam: Gaussian semantic splatting slam. In ACMMM, 2024. 7, 8, 10, 21, 22, 26
[68] Boying Li, Vuong Chi Hao, Peter J Stuckey, Ian Reid, and Hamid Rezatofighi. Hier-slam++:: Neuro- symbolic semantic slam with hierarchically categorical gaussian splatting. CoRR, 2025. 3,8 8
[69] Tomas Berriel Martins, Martin R Oswald, and Javier Civera. Ovo-slam: Open-vocabulary online simultaneous localization and mapping. RA-L, 2025. 3
[70] Dianyi Yang, Yu Gao, Xihan Wang. Yufeng Yue, Yi Yang, and Mengyin Fu. Opengs-slam: Open-set dense semantic slam with 3d gaussian splatting for object-level scene understanding. In ICRA, 2025. 3
[71] Tuo Feng, Wenguan Wang, and Yi Yang. Gaussian-based world model: Gaussian priors for voxel-based occupancy prediction and future motion prediction. In ICCV, 2025. 3
[72] Rui Liu. Wenguan Wang, and Yi Yang. Visiun-aagguage navigation with energy- -ased policy. NeurlPS, 2024.
[73] Rui Liu, Wenguan Wang, and Yi Yang. Volumetric environment representation for lision-language navigation. In CVPR, 2024.
[74] Jianzhe Gao, Rui Liu, and Wenguan Wang. 3d gaussian map with open-sct semantic grouping for vision-language navigation. In /CCV, 2025.
[75] Jianzhe Gao, Rui Liu, Yuxuan Xu, Tongtong Cao, Yingxue Zhang, Zhanguang Zhang, Sida Peng. Yi Yang, and Wenguan Wang. Uncertainty-awwe aware gaussian map for vision-languag navigation. In ICLR, 2026.
[76] Xingxing Zuo, Pouya Samangouei, Yunwen Zhou, Yan Di, and Mingyang Li. Fmgs: Foundation model embedded gaussian splatting for holistic 3d scene understanding. IJCV, 2024. 3
[77] Ri-Zhao Qiu, Ge Yang. Weijia Zeng, and Xiaolong Wang. Feature splatting: Langunge-driven physics- based scene synthesis and editing. In ECCV, 2024.
[78] Bin Dou, Tianyu Zhang, Yongjia Ma, Zhaohui Wang, and Zejian Yuan. Cosseggaussians: Compact and swift seene segmenting 3d gaussians. CORR, 2024.
[79] Jun Guo, Xiaojian Ma, Yue Fan, Huaping Liu, and Qing Li. Semantic gaussians: Open-vocabulary scene understanding with 3d gaussian splatting. CoRR, 2024.
[80] Guibiao Liao, Jiankun Li. Zhenyu Bao, Xiaoqing Ye, Jingdong Wang, Qing Li, and Kanglin Liu. Clip-gs: Clip-informed gaussian splatting for real-time and view-consistent 3d semantic understanding. CORR, 2024.3
[81] Yanmin Wu, Jiarui Meng, Haijie Li, Chenming Wu, Yahao Shi, Xinhua Cheng, Chen Zhao, Haocheng Feng, Errui Ding, Jingdong Wang, al. Opengaussian: Towards point-level 3d gaussian-based open vocabulary understanding. In NeurlPS, 2024.
[82] Chen Huang, Walter Talbott, Navdeep Jaitly, and Joshua M Susskind. Efficient representation learning via adaptive context pooling. In ICML, 2022.
[83] Lei Zhu, Fangyun Wei, Yanye Lu, and Dong Chen. Scaling the codebook size of vqgan 100,000 with utilization rate of 99%. In NewrlPS, 2025.
[84] YoungJoon Yoo and Jongwon Choi. Topic-vq-vae: leveraging latent codebooks for flexible topic-guided document generation. In AAAI, 2024.
14

---

Published as a conference paper at ICLR 2026
[85] Yanbei Chen, Xiatian Zhu, and Shaogang Gong. Semi-supervised deep learning with memory. In ECCV. 2018.
[86] Jingyuan Sun, Shaonan Wang, and Chengging Zong. Memory, show the way: Memory based few shot word representation learning. EMNLP, 2018.
[87] Alexey Dosovitskiy, Jost Tobias Springenberg, Martin Riedmiller, and Thomas Brox. Discriminative unsupervised feature learning with convolutional neural networks. In NeurlPS, 2014. 3
[88] Devon Hjelm, Alex Fedorov, Samuel Lavoie-Marchilooe Karan Grewal, Phil Bachman, Adam Trischler. and Yoshua Bengio. Learning deep representations by mutual information estimation and maximization. CoRR, 2018. 3
[89] Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predictive coding. CoRR, 2018. 3
[90] Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In JCML, 2020. 3
[91] Xinlei Chen, Haoqi Fan, Ross Girshick, and Kaiming He. Improved baselines with momentum contrastive learning, CORR, 2020.
[92] Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. In CVPR, 2020. 3
[93] Songtao Liu, Zeming Li, and Jian Sun. Self-emd: Self-supervised object detection without imagenet. CoRR, 2020. 3
[94] Xinlong Wang, Rufeng Zhang, Chunhua Shen, Tao Kong, and Lei Li. Dense contrastive learning for self-supervised visual pre-training. In CVPR, 2021.
[95] Zhenda Xie, Yutong Lin. Zheng Zhang, Yue Cao, Stephen Lin, and Han Hu. Propagate yourself: Exploring pixel-level consistency for unsupervised visual representation lcarning. In CVPR, 2021.
[96] Olivier J Hénaff, Skanda Koppula, Jean-Baptiste Alayrac, Aaron Van den Oord, Oriol Vinyals, and Joao Carreira. Efficient visual pretraining with contrastive detection. In ICCV, 2021.
[97] Olivier J Hénaff, Skanda Koppula, Evan Shelhamer, Danicl Zoran, Andrew Jacgle, Andrew Zisserman, João Carreira, and Relja Arandjelović. Object discovery and representation networks. In ECCV, 2022.
[98] Fangyun Wei, Yue Gao, Zhirong Wu, Han Hu, and Stephen Lin. Aligning pretraining for detection via object-level contrastive learning. NeurlPS, 2021.
[99] Xin Wen, Bingchen Zhao, Anlin Zheng, Xiangyu Zhang, and Xiaojuan Qi. Self-supervised visual representation learning with semantic grouping. In NeurlPS, 2022.
[100] Jiahao Xie, Xiaohang Zhan, Ziwei Liu, Yew Soon Ong, and Chen Change Loy. Unsupervised object-level representation learning from scene images. In NewrlPS, 2021. 3
[101] Kaveh Hassani and Mike Haley. Unsupervised multi-task feature learning on point clouds. In ICCV, 2019. 3
[102] Aditya Sanghi. Info3d: Representation learning on 3d objects using mutual information maximization and contrastive learning. In ECCV, 2020.
[103] Jonathan Sauder and Bjarne Sievers. Self-supervised deep learning on point clouds by reconstructing space. In NewrlPS, 2019.
[104] Yue Wang and Justin M Solomon. Deep closest point: Learning representttesssstoo for point cloud registration. In ICCV, 2019.
[105] Ji Hou, Benjamin Graham, Matthias Nieliner, and Saining Xie. Exploring data-efficient 3d scene understanding with contrastive scene contexts. In CVPR, 2021.
[106] Siyuan Huang, Yichen Xie, Mang-Chun22 Zhu, and Yixin Zhu. Spatio-temporal self-supervised representa- tion learning for 3d point clouds. In ICCV, 2021.
[107] Xiaoyang Wu, Xin Wen, Xihui Liu, and Hengshuang Zhao. Masked scene contrast: A scalable framework for unsupervised 3d representation learning. In CVPR, 2023.
15

---

Published as a conference paper at ICLR 2026
[108] Saining Xie, Jiatao Gu, Demi Guo, Charles R Qi. Leonidas Guibas, and Or Litany. Pointcontrast: Unsupervised pre-training for 3d point cloud understanding. ECCV, 2020.
[109] Honghui Yang, Sha 7hang, Di Huang, Xiaoyang Wu, Haoyi Zhu, Tong He, Shixiang Tang. Hengshuang Zhao, Qibo Qiu, Binbin Lin, al. Unipad: A universal pre-training paradigm for autonomous driving. In CVPR, 2024.
[110] Zaiwei Zhang, Rohit Girdhar, Armand Joulin, and Ishan Misra. Self- self-super pretraining of 3d features on any point-cloud. In ICCV, 2021.
[111] Haoyi Zhu, Honghui Yang, Xiaoyang Wu, Di Huang, Sha Zhang, Xianglong He, Tong Hc, Hengshuang Zhao, Chunhua Shen, Yu Qiao, al. Ponderv2: Pave the way for 3d foundation model with universal pre-training paradigm. TPAMI, 2025.
[112] David Tsai, Matthew Flagg, Atsushi Nakazawa, and James M Rehg. Motion coherent tracking using multi-label mrf optimization. In BMVC, 2012.
[113] Wenguan Wang. Jianhing Shen, Fatih Porikli, and Ruigang Yang. Semi-supervised video object segmen- tation with super-trajectories. TPAMI, 2018.
[114] Liulei Li, Tianfei Zhou, Wenguan Wang, Lu Yang, Jianwu Li, and Yi Yang. Locality-aware inter-and intra-video reconstruction for self-supervised correspondence learning. In CVPR, 2022. 3
[115] Allan Jabri, Andrew Owens, and Alexci Efros. Space-time correspondence as a contrastive random walk. In NeurlPS, 2020. 3
[116] Xueting Li, Sifei Shalini De Mello, Xiaolong Wang, Jan Kautz, and Ming-Hsuan Yang. Joint-task self-sude learning for temporal correspondence. In NeurlPS, 2019.
[117] Xiankai Lu, Wenguan Wang. Jianbing Shen, Yu- Wing Tai, David J Crandall, and Steven CH Hoi. Learning video object segmentation from unlabeled videos. In CVPR, 2020.
[118] Wenguan Wang, Hongmei Song, Shuyang Zhao, Jianbing Shen, Sanyuan Zhao, Steven CH Hoi, and Haibin Ling. Learning unsupervised video object segmentation through visual attention. In CVPR, 2019.
[119] Ning Wang, Yibing Song, Chao Ma. Wengang Zhou, Wei Liu, and Houqiang Li. Unsupervised deep tracking. In CVPR, 2019. 3
[120] Matthias Zwicker, Hanspeter Pfister, Jeroen Van Baar, and Markus Gross. Ewa volume splatting. In VIS, 2001.
[121] Jürgen Sturm, Nikolas Engelhard, Felix Endres, Wolfram Burgard, and Daniel Cremers. A benchmark for the evaluation RGB-D SLAM systems. In IROS, 2012. 7, 20, 21, 24, 26, 28
[122] Angela Dai. Angel X. Chang. Manolis Savva, Maciej Halber, Thomas A. Funkhouser, and Matthias Nießner. Scannet: Richly-annotated 3d reconstructions of indoor scenes. In CVPR, 2017. 20, 21, 22, 24, 26, 28
[123] Zhou Wang, Alan C. Bovik, Hamid R. Sheikh, and Eero P. Simoncelli. Image quality assessment: from error visibility to structural similarity. IEEE, 2004. 7
[124] Richard Zhang, Phillip Isola, Alexii A. Efros, Eli Shechtman, and Oliver Wang. The unreasonable effectiveness of deep features as a perceptual metric. CVPR, 2018.
[125] Nikhila Ravi, Valentin Gabeur, Yuan-Ting Ting Hu, Ronghang Hu, Chaitanya Ryali, Tengyu Ma, Haitham Khedr, Roman Rädle, Chloe Rolland, Laura Gustafson, al. Sam 2: Segment anything in images and videos. In ICLR, 2025.
[126] Quan Sun, Yuxin Fang, Ledell Wu, Xinlong Wang, and Yue Cao. Eva-clip: Improved training techniques for clip scale. CoRR, 2023.
[127] Haobo Yuan, Xiangtai Li, Chong Zhou, Yining Li, Kai Chen, and Chen Change Loy. Open vocabulayyy sam: Segment and recognize twenty-thousand classes interactively. In ECCV, 2024. 18
[128] Bangbang Yang, Yinda Zhang, Yinghao Xu, Yijin Li, Han Zhou, Hujun Bao, Guofeng Zhang, and Zhaopeng Cui. Learning object-compositional neural radiance field for editable scene rendering. In ICCV, 2021. 18
[129] Verica Lazova, Vladimir Guzov, Kyle Olszewski, Sergey Tulyakov, and Gerard Pons-Moll. Control-nerf: Editable feature volumes for scene rendering and manipulation. In WACV, 2023.
16

---

Published as a conference paper at ICLR 2026
[130] Edward Bartrum, Thu Nguyen-Phuoe, Chris Xie, Zhengqin Li, Numair Khan, Armen Avetisyan. Douglas Lanman, and Lei Xiao. Replaccanything3d: Text-guided 3d scene editing with compositional neural radiance fields. In NeurlPS, 2024.
[131] Qihang Zhang. Yinghao Yujun Shen, Bo Dai, Bolei Zhou, and Ceyuan Yang. Berfscene: Bev- conditioned equivariant radiance fields for infinite scene generation. In CVPR, 2024.
[132] Yinghao Xu, Menglei Chai, Zifan Shi, Sida Peng. Ivan Skorokhodov. Aliaksandr Siarohin, Ceyuan Yang. Yujun Shen, Hsin- Ying Lee, Bolci Zhou, and Sergey Tulyakov. Discoscene: Spatially disentangled generative radiance fields for controllable 3d-aware scene synthesis. In CVPR, 2023.
[133]
Mingqiao Ye, Martin Danclljan, Fisher Yu, and Lei Kc. Gaussian grouping: Segment and edit anything in 3d scenes, In ECCV, 2024. 18
[134] Jingyu Zhuang, Chen Wang. Liang Lin, Lingje Liu, and Guanbin Li. Dreameditor: Text-driven 3d scene editing with neural fields. In SIGGRAPH, 2023. 18
[135] Hyconscop Song, Scokhun Choi, Hoscok Do, Chul Lee, and Tachyeong Kim. Blending-nerft Text-driven localized editing in neural radiance fields. In ICCV, 2023,
[136] Ayaan Haque, Matthew Tancik. Alexei A. Efros, Aleksander Holynski, and Angjoo Kanazawa. Instruct- nerf2nerf: Editing 3d scenes with instructions. In ICCV, 2023.
[137] Yiwen Chen, Zilong Chen, Chi Zhang, Feng Wang, Xiaofeng Yang. Yikai Wang. Zhongang Cai, Lei Yang. Huaping Liu, and Guosheng Lin. Gaussianeditor: Swift and controllable 3d editing with gaussian splatting. In CVPR, 2024.
[138] Jiemin Fang, Junjie Wang, Xiaopeng Zhang, Lingxi Xie, and Qi Tian. Gaussianeditor: gaussians delicately with text instructions. In CVPR, 2024. 18
Editing 3d
[139] Tim Brooks, Aleksander Holynski. and Alexei A. Efros. Instructpix2pix: Learning to follow image editing instructions. In CVPR, 2023. 19
[140] Justin Johnson, Alexandre Alahi, and Li Fei-Fei. Perceptual losses for real-time style transfer and super-resolution. In ECCV, 2016. 19
[141] Yanmin Wu, Xinhua Cheng. Renrui Zhang. Zesen Cheng, and Jian Zhang, Eda: Explicit text-decoupling and dense alignment for 3d visual grounding. In CVPR, 2023. 22
[142] Zhiyuan Yu, Zheng Qin, Lintao Zheng, and Kai Xu. Learning instance- correspondences for robust multi instance point cloud registration in cluttered scenes. In CVPR, 2024. 22
17

---

Published as a conference paper at ICLR 2026
APPENDIX
This appendix contains additional details for the ICLR 2026 submission, titled Open-Set Semantic Gaussian Splatring SLAM with Expandable Representation. The appendix is organized follows:
SA provides more details on our method.
S presents more experiment results.
SC gives our pseudocode and limitations.
SD offers asset licenses and socictal impact.
A MORE DETAILS ON METHODS
A.1 PREPROCESS RGB FRAMES
We generate semantic maps SGT using RGB frames CGr. We first generate segment masks with an image segmentation model SAM [12]. We then extract pixel-aligned CLIP [11] features for each segmented object. Notably, our framework is not dependent on any specific semantic extraction method and can alternatively employ other segmentation approaches (e.g.. SAM [125]),
CLIP models (e.g., EVACLIP [126]), or unified semantic extraction methods (e,g. OVSAM [127]). Therefore, this preprocessing pipeline is an implementation choice for semantic initialization of our framework.
A.2 FRAMEWORK DETAILS
Auxiliary Semantic Head. Given a 2D semantic feature map S rendered from semantic features, we train semantic head to perform classification task. Specifically, for each pixel p in S, classify into one of the annotated categories, resulting in closed-set semantic map S(p):
S(p) classifier(S(p))
(16)
Supposed we have the input ground truth closed-set semantic map Ser for currrent time t, we can optimize the classfier with cross-entropy loss:
Lg = Lee L.e ($, Sar).
(17)
We implement the classification for the quantitative comparisons mentioned in $4.3.
Selected Mapping Frames and Keyframes. Instead of performing mapping at every frame, map only at selected frames. Optimization of Gaussians typically prioritizes recent frames, which can cause the system forget earlier optimization results. To mitigate this, trigger mapping only when significant number of new Gaussians need to be added to the scene in the current frame. Additionally, if no mapping occurs for several consecutive frames, we designate the current frame as a mapping frame to ensure full scene optimization.
We also maintain a of keyframes, storing every n-th frame along with the current frame. During mapping, we evaluate the overlap between the current frame and each keyframe using the current depth map, selecting the most relevant keyframes for optimization. One of them randomly chosen for each iteration of mapping.
A.3 DETAILS ON 3D SCENE EDITING
Previous research has given limited attention to selective editing individual objects within complex scenes. Most existing methods [128-133] for such scenes focus primarily object movement or removal. Although they are typically limited to small areas around the target object. These diffusion-based methods face challenges: on one hand, they require the target object to be clearly visible, which complicates editing in scenes with multiple objects, diffusion may struggle isolate the intended target (supp. 1A.3.1). On the other hand, diffusion models often unintentionally modify parts of non-target objects, resulting in unwanted changes larger, open scenes (supp. $A.3.2).
methods [134-138] can alter the appearance or style of objects,
A.3.1 CAMERA GENERATION
Camera Positions. Given center coordinate o, initial upper vector Vo. generate sphere with radius r in spherical coordinates (r, 8, sp). The angles are typically based on vo as the "north pole". However, for more flexible editing, we often need specify additional normal vector in
18

---

l.c Published as a conference paper at ICLR 2026
as the "north pole" to rotate the sphere. This enables alignment of the generated camera positions with the desired orientation, accommodating scenarios such as objects on walls, where the north pole should be perpendicular to the wall rather than the ground. To transform the sphere, we can obtain the rotation matrix between by and in with Rodrigues' rotation formula:
R R K²
(18)
5²
where
Uh U1, Up
UI,
|
02
4/y
(19)
K
11
12
Now we can transform any point to the place we need on the surface of the sphere with R. We divide the radian values of the spherical coordinate system 8, evenly to generate camera positions on the sphere, and transform them to Cartesian coordinate positions = (r, 2):
x r sin cos pp y = rsin 0 sin ip; r cos
(20)
Camera Pose Matrix. With the position e of a generated camera and the center coordinate o, we have the look direction vector / o which indicates the direction the camera is pointing. Given an upper vector (such as vo) u. we derive the right vector as r = u X L. To ensure the upper vector is completely orthogonal to both the look direction and right vector, we compute corrected upper vector I x r, which ensures r, w', I form an orthonormal basis, representing the x, y, 2 axes of the camera coordinate system respectively. Then we have the view matrix:
T=
Lr
c
V
Ty
u
ly
u c
(21)
P2s
l.
0
0
0
After generating a sequence of cameras positioned around the target object, we utilize the Gaussian differentiable renderer to render images centered on the object, meeting the requirements for diffusion- based editing.
A.3.2 PRECISE EDITING
Editing specific objects without affecting surrounding areas often relies on using 2D masks restrict loss computation. However, this approach faces key challenges. First, updating masks during editing incurs high computational costs and leads to progressively inaccurate mask generation. In contrast, static masks may fail to support precise edits, especially for modifications extending beyond the mask's boundaries. Second, during optimization, the impact is not confined solely to Gaussians belonging to the target object; parts of the scene occluded by the object but still within the mask region may also be unintentionally altered.
To address these challenges, our methodology leverages the embedded semantics in our representation. We employ the CLIP model [11] to extract features from the input text, evaluate their similarity to the semantics that Gaussians retrieve from the feature pool, and selectively refine the relevant Gaussians for the editing. This approach effectively improves editing precision by directly interacting with the scene's underlying representation.
During the editing process, we randomly select a view obtained from supp. $A.3.1 to render the original image Iori, and utilize InstructPix2PPxx [139] generate the edited image Tedst- We then compute the perceptual loss [140] between them to optimize the scene:
Learting = Eyeresptual (Lors, Ludit).
(22)
At each iteration, views are re-selected and editing is applied to the previously edited scene. This iter- ative approach minimizes inconsistencies that can arise from editing all view images simultaneously, allowing us to make further optimizations on the already edited scene.
A.3.3 REMOVAL AND MOVEMENT
Removal We remove object using similar approach to editing, We first extract the object's CLIP feature to identify its semantics and remove the corresponding Gaussians. Then, we render images from the generated camera poses and inpaint them using diffusion. During optimization, to avoid altering the surrounding environment, we only update Gaussians within the sphere centered at o with radius (supp. 1A...1). At each iteration, we re-select camera to continuously inpaint the scene.
19

---

Published as a conference paper at ICLR 2026
Poini SLAM [B] ICCV23 Our SplaTAM
Ground Truih
Point- SLAMI [18]cccv2) Ours SplaTAM
Ground Truth
Figure Qualitative Comparisons ($B.2.1) over Point-SLAM [18] on Replica [21].
Movement. We break the movement into two stages. First, similar to removal, we locate the target object, select its Gaussians, and perform inpainting at its original position. Second, move the selected Gaussians by applying the input translation and rotation to their coordinates.
B MORE EXPERIMENT RESULTS
B.1 EXPERIMENTAL SETUP
Datasets. The experiment results in this part are conducted on Replica [21], TUM-RGBD [121] and ScanNet [122], with evaluations on 8, and scenes.
Implementation Details are the same in $4.1. be more specific, during tracking, only camera parameters are optimized, with learning rates of 2e-3 for translations, 4e-4 for unnormalized rotations. During mapping, only Gaussian parameters and Kp, Fp are optimized, with learning rates 9e-5 for 3D positions, 2.5e-3 for colors, lc-1 for semantic-embede poot, le-3 for key pool, 8e-4 for rotations, 4e-2 for opacities, and 8e-4 for scales.
parameter, 1e-2 for semantic feature
B.2 MORE TRACKING AND APPEARANCE MAPPING RESULTS
Baselines. We apply our method on SplaTAM [20]. LoopSplat [19), and more recent works S3PO-GS [59] and SEGS SEGS-SLAM [60].
Metrics. We assess camera tracking and mapping with the same metrics as in $4.2.
B.2.1 MORE RESULTS ON REPLICA
More Quantitative Comparisons. We present more quantitative comparisons with more recent works in Table 8. Notably, S3PO-GS [59] is designed for open-world scenarios, and the Replica results shown here come from our reproduction.
More Qualitative Comparisons. We present more visual comparisons in
Method
PSNR
SSIM
LPIPS
ATE RMSE
S3PO-GS [59] JCce22))
32.71
0.943
0.138
2.26
Fig. 4. Compared to Point-SLAM [18], our method excels at capturing finer details, such as patterns, wrin- kles, plush textures, and even lighting variations.
S3PO-GS Ours
35.23
0.97
SEGS-SLAM (icez
39.42
0.975
0.021
0.43
SEGS-SLAM Ours
39.88
0.983
0.020
0.38
Table 8: More Quantitative Comparisons (supp. B.2. with more recent works Replica [21].
Other Visualizations. Fig. 13 shows more qualitative results on color, depth and semantic field reconstruction. Our method produces high-qualit color field reconstructions on Replica [21], which also reflected geometric details. We also reconstruct semantic fields with clear boundaries, enabling applications like 3D scene editing.
B.2.2 RESULTS ON SCANNET AND TUM
Camera Pose Estimation. As shown in Table 18, our method outperforms existing NeRF-based approaches, on the challenging TUM, where sparse depth information and severe motion blur are common. Despite these similar challenges, our approach also achieves competitive results on ScanNet when compared to the latest methods.
Quantitative Analysis of Scene Reconstruction. The rendering results on real-world data, shown in Table 19, highlight the superior performance of our method compared to previous approaches,
20

---

Published as a conference paper at ICLR 2026
Editing: "Transform the small round stool to appear as if is made of marble."
Removal: " "Remove the stool." "
Translation & Rotation
Figure 10: Qualitative Results of 3D Editing (supp. $B.5) on Replica [21].
underscoring its effectiveness in real-world settings. Due to the lack of semantic annotation in TUM, GS*LAM [67] cannot be on this dataset.
More Visualizations. We present more visual results on ScanNet [122] and TUM-RGBD [121] are shown Fig. 14. As seen, our method produces high-quality color and semantic field reconstructions on real-world data. Notably, while previous closed-set semantic methods struggle with datasets that lacks semantic annotations such as TUM, our approach successfully reconstructs semantic fields across any scene or dataset, enabling broader application possibilities.
B.3 MORE CLOSED-SER SEMANTIC MAPPING RESULTS
Datasets and Baselines. Following L.*M [67], we use real-world ScanNet [122] scene0059 00, which is also designated as an example their official GitHub repository.
Method
mloU
GS [67]
18.62
SplaTAM Ours
53.27
Table 9: Closed-set
Metrics. We employ mloU as our metric.
parisons (supp. B.3) with
Results. We use the same claasification head mentioned in supp. $A.2. Table demonstrates that our method outperforms previous
GS* [67] on ScanNet [122] scene0059 _00.
work on real-world data, which can attributed to the introduction of our consistent open semantics.
B.4 RUNTIME ANALYSIS
Table 10 presents the runtime of our framework.
Tracking /Frame (s)
Mapping /Frame (s)
Due to the introduction of semantic reconstruction, method incurs approximately 8% 21% addi- tional time overhead compared to SplaTAM [20], LoopSplat [19] and MonoGS [48]. For tracking. the additional computational overhead stems from semantic assignment from Fp; For mapping, in-
Method
SplaTAM
0.23
1.78
2.56
SplaTAM + Ours
0.16
2.33
3.82
LoopSplat
LoopSplat + Ours
0.37
0.97
1.72
MonoGS
MonoGS Ours
0.99
0.37
0.65
cludes semantic assignment, Intra-Inter Semantic Consistency Objective and Semantic Stability Guidance. Notably, the additional overhead introduced for MonoGS is significantly lower than that for SplaTAM and LoopSplat. This is because MonoGS requires far fewer Gaussians for reconstruction compared to them 10 v.s. x 10"), reducing the computational burden of semantic assignment and consistency optimization. This observation also suggests that as more compact 3DGS techniques emerge, our framework's efficiency can be further improved.
Table 10: Runtime (supp. B.4) on Replica Room 0.
B.5 MORE RESULTS 3D SCENE EDITING
Fig. 10 shows more results 3D scene editing. Beyond the editing mentioned in $4.4, our method also supports removal, translation and rotation, enabled by accurate semantic fields construct.
21

---

Published as a conference paper at ICLR 2026
"Shely"
"Musie"
"Mindspore"
Figure 11: Annotation examples (supp. $B.7) on three datasets.
B.6 DETAILS ON IN-THE- IN-THE-WILD DATA
We collected data using an iPad equipped with LiDAR. This dataset includes RGB-D images from 9 scenes, without any extrinsic camera parameters. These scenes are generally more complex than those used current works, such as scenes ScanNet [122]. In terms of scale, the number images scene ranges from 5,000 to 12,000 frames. Regarding complexity, the scenes encompass both indoor and outdoor environments and typically include intricate objects such as plants and flowers. In terms of challenges, many scenes were captured with loops, including large loops that end with the starting point and smaller loops within the capture process. This dataset is specifically designed for SLAM tasks and presents high level of complexity.
B.7 DETAILS OPEN-SET LOCALIZATION EXPERIMENTS
We perform 3D anmotations on Replica [21], ScanNet [122], and our in-the-wild data (Fig. 11). The 3D bounding boxes for Replica and ScanNet are obtained from the official Replica dataset and EDA [141], respectively, with some objects being re- re-annotated by us. For our data, all 3D bounding boxes are annotated by us. Each dataset is annotated with approximately 50 objects. To ensure the thoroughness and challenge of our experiments, we design complex textual descriptions for the objects, including common names, abstract uses, colors, and other visual attributes of the objects. Examples on three datasets are presented in Fig. 11.
We employ the following method for our experiments. For given text input, first convert into feature using CLIP and then compute the similarity between this feature and the Gaussians in the scene. We select a group of Gaussians with the highest similarity. Based on their coordinates, we can derive a predicted 3D bounding box. Subsequently, we use point cloud registration method [142] align this group of Gaussians with the 3D ground truth scene. Finally, we calculate the overlap between the predicted bounding box and the annotated object bounding box.
To evaluate the accuracy of our predictions, we use two metrics: acc @1 0.25 and acc a.c5 These metrics measure the percentage of predicted bounding boxes that have an IoU greater than or equal to 0.25 and 0.5, respectively, when compared to the ground truth bounding boxes.
B.8 MORE DIAGNOSTIC EXPERIMENTS
Experiments are also on Room 0 of Replica [21] with the baseline of SplaTAM [20], as in $4.5.
PSNR
ATE RMSE J
mloU
Wil MFS Ours
34.52
0.26
91.37
34,86
0.24
96.93
Mapping Frame Selection. We conduct an experiment
Table 11: Quantitative Results of Map- ping Frame Selection (MFS) (supp. $B.8).
without mapping frame selection, showing that improves scene reconstruction quality and camera pose estimation, as well as enhancing the accuracy of the semantic field.
MFS represents mapping frame selection.
More Details on Large-Scale Scenes Diagnostic Experiments. We select SGS-SIAM [64] and LAM [67] as baselines because they are both SplaTAM-basse methods, enabling more direct com- parisons. We generate labels using the methods mentioned on the Github page of LAM..
D, Storage (MB)
SplaTAM [20]
252,89
317,0%
w/o F
351.14
16
~u 650
w/ F,
318.60
321.95
Memory Footprint of Storing the Scene. Table gives the mem- ory requirements of Room with different D, w/ w/o the feature pool. When assigning a 16-dimensional feature to each Gaussian, the experiment fails due to the excessive GPU memory occupancy,
Table 12: Memory Footprint
(supp. $B.8) of storing Room of Replica [21].
22

---

Published as a conference paper at ICLR 2026
Original Views
w/o Semantics
w/ Semantics
Figure 12: Diagnostic Experiment on Semantic Features for 3D Editing (supp. $B.8). Prompt: Turn the cabinet red.
and the number estimated. Because of the additional attribute (k or the semantic feature for each Gaussian), our method requires more storage compared to SplaTAM [20]. Without Fp. storage demand increases greatly with higher Da, that using 5-dimensional feature requires more storage (30 MB) than 16-dimensional Fp. The results illustrate that our method can substantially reduce storage requirements.
Number of Nearest Neighbors (m) in Aggregation. In Table 13, we discuss the number of nearest neighbors to select during the aggregation. When m 1. the approach essentially stores semantics directly in the pool rather than aggregating them. The results show poor semantic field reconstruction, demonstrating the effectiveness of our aggregation strategy. As m increases, the performance improves significantly, indicating that aggregating more features is beneficial for semantic representation.
mloli
58,12
96.93
10
96.84
Table 13: m
Aggregation (supp. $B.8).
Novel Feature Determination Threshold During Pool Expansion. When a semantic feature is input, its similarity with features in F, is computed to determined whether it is novel feature with threshold ($3.2.1). In this section, we conduct an ablation study on it. As shown in Table 14, the mloU peaks when the threshold is set to 0.8. Based on this finding, we adopt this value the default setting for all other experiments.
Threshold
mloU
0.6
81.67
0.7
0.8
96.93
95.26
Table 14: Determination Threshold (supp. $B.8).
Determination of Empty Slots in Fg. To determine if there are available slots in Fp. we calculate the similarity between each semantic in current F, and its initial state. If the similarity of all slots in F, is below certain threshold, the current F, is considered full and will be expanded ($3.2. 1). In this section, results are shown in Table 15, including both mIoU performance and the final pool size. When the threshold is too low, some slots that may have already stored substantial semantic information are still judged as empty, causing excessive semantics be crammed into the pool and resulting in poor performance. When it is too high, the final F, becomes excessively large, leading a sharp increase in computational and storage costs. Notably, current experiments are conducted based SplaTAM. As mentioned in supp. $B.4, SplaTAM's reconstructed scenes contain an excessive number of Gaussians, which consumes substantial computational resources during semantic assignment. When more compact Gaussian representations are employed, the pool can be further expanded size, and can store more semantics in large-scale scenes.
Threshold
mloU
0.3
89.64
50
0.4
92.15
100
0.5
96.93
200
0.6
2-800
conduct ablation experiments on this threshold. The
Table 15: Determination
of Empty Slots in F, (supp. $B.8). When the threshold reaches 0.6, the final size ceeds 800, causing the periment fail.
Expansion Proportion (n)) of Semantic Pool. We further explore the impact of enlarging the pool size on performance in Table 16. Results show that the
mloU
1.50
91.88
greater the enlargement, the more pronounced the performance enhancement. This is attributed to the larger space offering more flexibility for the aggregation of new semantics, thereby storing new semantics in the extended part rather than continuing aggregate it within the original portion.
1.75
93.24
2,00
96.93
Table 16: v Sefmaatttic Pool (supp. $B.8).
23

---

Published as a conference paper at ICLR 2026
Semantic Features for 3D Editing. Fig. 12 shows that, without semantics, editing operations tend to affect the entire scene, which is particularly problematic in large-scale environments. With semantics, method enables selective refinement of targeted objects while preserving the surrounding scene. This highlights the importance of semantics in ensuring precise edits. The results further demonstrate the accuracy of the reconstructed semantic fields.
C
MORE DISCUSSION
C.I DETAILED FRAMEWORK ALGORITIIM
Algorithm provides the pseudocode for our framework. By leveraging the expandable semantic pool, the Intra-Inter Semantic Consistency Objective and Semantic Stability Guidance, our approach integrates expandable open-set semantic field reconstruction into SLAM, facilitating its application across various tasks.
C.2 MORE DISCUSSIONS ON PoOL EXPANSION IN LARGE SCENES
We acknowledge that for extremely large-scale scenes, the semantic pool may continue expand and could eventually approach practical limits imposed by hardware capacity and computational resources. Neverthe- less, we believe our method remains
Scenes
Total Frames
Expansion Frame lDx
Final Pool Size
Replica Room 0
2000
16, 22, 118
200
TUM Long Office
2585
2. 119. 347, 645, 4292
ScanNet 0000
5578
400
In-the-wild Data
8501
168, 1041, 4569
200
Table Statistics semantic pool expansion across relatively large-scale scenes. (supp. $C.2).
applicable beyond the scene scales demonstrated in this paper. As shown in Table 17, we observe self-limitin liffliiniing expansion behavior of the semantic pool: while semantic diversity continuously accumulated the scene grows, the frequency of pool growth decreases significantly over time. This suggests that as the semantic coverage of the scene becomes increasingly comprehensive, representation gradually stabilizes rather than expanding indefinitely, indicating strong potential for scaling to even larger environments.
D LEGAL AND ETHICAL CONSIDERATIONS
D.1 ASSET LICENSE
We conduct our method on three indoor datasets (e.g., Replica [21], TUM-RGBD [121] and ScanNet [122]), and two pretrained models (e.g., SAM [12] and CLIP [11]), which are all available for academic access. Replica (https://gith))
com/ htt.///bbboorreerccc///////o
Replica-Dataset) is released under this License. TUM-RGBD (https:/ https::/cvg..iii cit. tum tum.:ee/fatta/dttasstts//// http:///dtaaattttrrrrrtrdddaaaaa dataset)
released under this License. ScanNet org/) is released under this License. SAM (nttps:// segmentt-anything..omm com// is released under this License. CLIP (https: /tttep:://opeai.cco index/clip/) released under this MIT License.
ltttp://ww..canct
D.2 POTENTIAL SOCIETAL IMPACT
Enabling everyday devices to create interactive metric-semantic 3D) virtual worlds has long been a goal of AI research. We have made significant progress toward this by allowing SL.AM frameworks to reconstruct open-ended semantic fields in any scene. However, this raises privacy and legal concerns. We recommend that users and content publishers restrict the use of this system to public environments to protect privacy and ensure compliance with laws, such as portrait rights and trademark protections.
D.3 USAGE OF LLM
During paper writing, we use LLM polish our text and check for spelling and grammatical errors.
24

---

Published as a conference paper at ICLR 2026
Algorithm Framework Pscudocode, PyTorch-like style
Gaussians
scene
R: Gaussian renderer
M; number of tracking and mapping iterations
Li, SSIM: mean absclute ezror and structure similarity index measure
load every timestep frame from dataset
C, 6, s_p: current RGB, depta, semantic frames and previous semantic frames K_P, key pool feature pool
cam: antimmeed nameras
tl, t2, thresholds during expansion
R.p, P.F turch. rand Dk), tarch. Ds) init_F_p E_p.cione()
def EXPAND (s, F_P, init_F_p): unique_teatures torch.unique!t)
Compute number novel features -
novel for que, feature quattueu in uni unique_ festures:
www.fttteeemaeee novel_feature_
top_a_ wimmmmmiiiiiii
top_m(casine www.iiesssiiiiitttiieiaoaa (uriquuu F_p))
15 all wtwpopowsimmaarrst...e novel_
ti):
Check if there is enough erpty_slota = torch.sum sosimmmoooinieesilllrrtyyy
novel features
init_F_F) < t2)
novel_feature num empty_slota:
# Expand K_P and P_P with randomly initia" ized parts Expand_K_e expand_F_
curch.rand -J,, Dk), torch.rand((a 11,, (PLF, expard_P_pi
K_P, F_F tarch, cat cat P, expand [_p] init_ tarch. sat (init_F_p, =_pand_F_p)
return K_F, F_P, init_F_p
for c, d, in loader: Tracking
for in :T]:
# Assign semantic features to Gaussians (Eq.5) S.params.semanti FRATTOR__ODD..G.. key, K_P, P_P)
frames (, e_t, e_t, t_t (arpam,,, cam.params)
Compute tracking loss (Eqs.7-8) loss_t_c = L1 (c, d_t) less_t_d
ld, d_t)
loss_t_s - Li 1s, s_t) loss_t loss_t_e loss___d loss_t_s
# Update the cameras parametera bbackwarddd update (crmm))rams)
Mapping
for in (ange(M)
K_P, ELP, init_F_p EXPAND is, KLP, ELD. init_F_p)
G.params. semantic
(CTTORE_FOOOGGppaa.ri key, K_p, P_pl
S d_ P.F. (GG.pas cam.params)
# Compute RGB and depth loss 11-12) loss_MLe te, g_m) SSIKie, a_m) 10ss_m_d L1 ld, d_m)
Intra-Inter Semantic Consistency Object & Semantic Stabil Stability Guidance loss_m_eo CONSISTENCY D OBJECTIVEESS_M, a_p) m_sq STABILITY_GUIDA
a_P)
Semantic (Eq.13) loss_m_s m_5g * (s, s_m) + lass_n_co
Overall mapping loas (3q.14)
loss_m loss_m_e + loss_m_d + loss_m_s
# Update the Gaussian parametera boss_m.backward update (G.par)
25

---

Published as a conference paper at ICLR 2026
TUM-RGED [12]]
ScanNet [122]
Methods
Avg.
tx1/
desk
deak2
trl/
zoom
fx1/
Ex2/
xyz
Ex3/ off.
Arg.
0000
0059
0106
0169
0161
0207
Vox-Fusion [43]
11.31
3.52
6.00
19.53
1.49
26.01
26.90
68.84
24.18
8.41
27.28
23.30
9.41
NCE-SLAM. [41]
15.87
4.26
4.99
34.49
31.73
3.87
10.70
12.00
14.00
7.90
10.90
13.40
6.20
Point Min--SLun AMAMI [18]
8.92
4.34
4.54
30.92
1,31
3.48
12.19
10.24
7.81
8.65
22.16
14.77
9.54
GS [67]
15.56
12.98
7.56
SplaTAM [2]
5.48
3.35
6.34
11.13
1.24
5.16
1L.8
12.83
10.10
17.72
12.08
11.10
7.46
SplaTAM Ours
5.20 p.
3.18
6.48
10.95
1,29
4.12
10.27 1.61
14.56
9.41
7.82
11.59
11.11
7.12
SpopSplar
3.33
2.08
3.54
0.24
1.38
3.22
7.73
6.20
7.10
7.40
10.60
8.50
6.00
LoopSplat + Ours
3.13
1.94
3.32
6.05
1.37
2.99
7.54
jo.4
6.12
7.02
2.41
9.89
8.31
6.48
Table 18: Quantitative Results on Camera Pose Estimation (supp.
$B. TUM-RGBD [121] and
ScanNet [122] (ATE RMSE. [em]).
TUMERGBD [121]
Scan [122]
Methods
Metrics
Avg.
i:1/
Er2/
Er3/
Avg.
0000
0059
01.69
desk
desk2
roon
xya
Vax-Fusion [43]
PSNR1
SSIM
0.632
15.54
0.647
15.79
0.568
14.12
0.566
14.20
0.706
0.677
18.17
0673
19.06
0,062
0.655
0,753
0.620
0.666
0.696
LPIPS
0.502
0.523
0.541
0.359
01433
0.456
0.504
0.515
0.528
0.439
0.313
0,532
0,500
NICE-SLAM
PSNRT
13.59
13.83
12.00
11.39
17.87
12.89
17.51
18.71
16.35
17.29
18.75
15.56
18.35
0.629
0.562
LPIPS
0.494
0,482
0.520
0.629
0.344
0.498
0.548
0.561
0.534
0.510
0.534
0.602
0.552
[4L
PSKRT
13.42
11.29
12.30
9.06
17.46
17.02
15.29
15.70
14.48
15.44
14.56
14.23
17.32
SSIMT
LPIPS
0.358
0.421
0.192
0.69%
0.652
0.48%
0.449
0.529
0.486
0,483
0.534
POA [8]
PSNRT
SSIM
0.665
15.53
0.627
13.87
0.592
11.12
0.645
14.16
0.708
17.36
0.754
18.43
19.82
21.30
0.806
0.765
1978
0.676
16.80
0.686
18.33
22.27
0.823
0.750
20.36
0.471
GS [67]
PSNRT
22.86
23,02
20.96
22.57
25.85
20.58
24.39
SSIM
D.86%
0.352
858
0.890
11855
0.77
LPIPS
0.222
0.277
0.205
0.252
0.195
SplsTAM (20)
PSNRT
21.67
22,00
20.35
19.62
24.50
21.90
19.14
19.33
19.27
17.73
21.97
16.76
19.80
0.716
LPIPS
00208
0.222
0,270
0.235
0,100
0.202
0,358
0.488
0.289
0.376
0.281
0,420
0.341
FSNR
22.3.pm
23.68
21.55
20.89
24.77
22.28
22 mm
21.15
20.43
21.90
24.89
19.99
24.26
SplaTAM Ours
SSIM
0.895
0.920
0.873
0.950
0.836
0.764
1440
0.854
0.834
LPIPS
0.0
0.138
0.227
0.222
0.297
LoopSplat [19] 19302
PSKK
19700
24.92
26,80
24.81
26.31
SSIM
0.837
0.849
0.773
0.791
0.892
0.879
0.845
0.840
0.846
0.877
0.874
0.854
LPIPS
0.207
0. 1.9077
0.386
0.324
0.217
0.253
0.425
0.4%)
0.400
0.409
1.346
0.514
01440
PSKRT
22.25
22.99
20.37
20.15
23.61
24.13
25.53
25.30
24.11
23.98
27,41
25.34
27.06
LoopSplut +
0.892
0.896
0.918
0.885
LPIPS.
0.220 010F7
0.191
0.284
0.253
0.140
0.231
0.305 p0.
0.325
0.301
0.316
0.250
0.319
0.288
Table 19: Quantitative Comparison on Rendering Performance (supp. BB.2.)) with baselines on TUM- RGBD [121] and ScanNet [122].
26

---

Published as a conference paper at ICLR 2026
Rendered RGB
Rendered Depth
Rendered Semanties
Ground Truth
Office 0
Office I
Office 2
Office 3
Office 4
Room 0
Figure 13: More Qualitative Results (supp. $B.2.1) of RGB, depth and open-set semanties on Replica [21].
27

---

Published as a conference paper at ICLR 2026
Rendered RGB
Rendered Depth
Rendered Semantics
GT RGB
Depth
GT Semantics
scenee Ceneeetio
Scan [122]
TUM-RGBD [121]
fri Beek
ir From
IES office
Figure 14: Qualitative Results (supp. $B.2.2) of RGB, depth and open-set semantics on ScanNet [122] and TUM-RGBD [121].
28


Reviews:
[{"review_id": "HOSk5KuQas", "reviewer": "Reviewer_21bL", "summary": "The paper introduces an open-set semantic 3D Gaussian Splatting (3DGS) framework designed to enhance scene reconstruction and semantic segmentation under open-set and real-time conditions. The proposed method integrates semantic understanding into the Gaussian Splatting pipeline by associating per-Gaussian semantic embeddings and enabling online label adaptation. The system claims to handle unknown categories and domain shifts, and to improve both reconstruction quality and scene-level segmentation.\n\nExperiments are conducted mainly on the Replica dataset, showing improved reconstruction quality and semantic accuracy compared to several baseline methods. The results demonstrate promising performance in synthetic indoor environments, suggesting potential for robust open-world mapping.", "questions": "The questions are the same as weakness. This paper presents a solid and well-written approach that meaningfully extends semantic understanding into 3D Gaussian Splatting. However, the experimental evaluation is too narrow and overly dependent on synthetic data, which weakens claims of real-world robustness and open-set generalization. Including stronger baselines and more diverse datasets would substantially improve the paper’s credibility.\nTypos:\n1. In Figure 2, the label “Ground Truth RGB Frame” appears misplaced — please verify and correct this annotation.", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. Good presentation and writing quality.\nThe paper is clearly structured, logically consistent, and easy to follow. Figures are visually clean and the methodology is well explained.\n\n2. Strong results on benchmark datasets.\nThe proposed approach achieves good reconstruction and segmentation results on Replica and other small-scale datasets, showing the method’s effectiveness in controlled settings.\n\n3. Direct and effective idea.\nThe proposed framework is conceptually straightforward yet functional. It extends 3D Gaussian Splatting toward semantic understanding and contributes a practical perspective to open-set semantic reconstruction.", "weaknesses": "1. Overreliance on synthetic data.\nUsing Replica as the main evaluation dataset is limiting. Since Replica is synthetic and lacks real-world sensor noise, it is less meaningful for evaluating SLAM or localization robustness. The paper would be stronger with experiments on real-world semantic datasets such as ScanNet++, SemanticKITTI.\n\n2. Missing comparisons with recent baselines.\nSeveral recent 3DGS-based SLAM and semantic mapping methods (e.g., MonoGS, S3PO-GS, SEGS-SLAM) are not included in comparisons. Without these baselines, it is difficult to gauge the real advancement of the proposed approach relative to the current state of the art.\n\n3. Limited improvement from semantics to pose estimation.\nWhile the paper introduces semantic components, the pose estimation accuracy remains very close to previous methods, suggesting that semantics may not significantly contribute to the geometric optimization process.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "LQXMz3wumJ", "reviewer": "Reviewer_zwtS", "summary": "This paper proposes an open-set semantic SLAM system based on 3D Gaussian Splatting. The key innovation is the introduction of an expandable semantic feature pool, which decouples high-level semantics from individual Gaussians. This design significantly reduces memory usage and allows for dynamic integration of new semantic concepts during SLAM operation. To address the inconsistency issue from 2D foundation models, the authors also propose a consistency-aware optimization strategy and a semantic stability guidance mechanism. Extensive experiments on both synthetic and real-world datasets demonstrate the system's advantages in rendering quality, tracking accuracy, and open-set semantic understanding tasks, such as 3D localization and scene editing.", "questions": "1. The expansion factor n and the threshold for determining \"empty slots\" seem to be empirical. Could you discuss the sensitivity of the model's performance to these hyperparameters? Is there a risk of the pool expanding too aggressively in very large-scale scenes?\n﻿\n2. In the semantic stability guidance, how is the \"object that appears for the first time\" specifically identified? Is this based purely on the lack of correspondence in previous frames?\n﻿\n3. The paper demonstrates excellent results on in-the-wild data. Could you comment on the system's performance in highly dynamic environments where objects move frequently? Does the semantic pool and consistency mechanism handle such cases robustly?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "The core idea of a learnable and expandable semantic feature pool is novel and clever. It effectively separates the scene-level semantics from the geometric representation, enabling efficient and scalable open-set learning.  \n\nThe paper is structured logically, and the framework is explained step by step, making it understandable.   \n\nThis work pushes the boundary of semantic SLAM from closed-set to open-set, which is a crucial step towards general-purpose 3D scene understanding. The ability to run on commodity hardware and support applications like 3D editing greatly enhances its practical value.", "weaknesses": "While the comparison with existing SLAM methods is comprehensive, I feel the comparison with state-of-the-art open-set 3D understanding methods (e.g., OpenMask3D, OpenScene) in terms of pure semantic segmentation accuracy is somewhat lacking.   \n\nThe runtime analysis in the supplement shows a non-negligible overhead. Although the authors attribute this to the baseline, a more in-depth discussion on how to optimize the efficiency of the semantic module itself would be helpful for practical deployment.  \n\nThe threshold parameters for the pool expansion (e.g., the similarity thresholds in Algorithm 1) seem crucial but their selection process is not deeply analyzed.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "XcaoJObGd7", "reviewer": "Reviewer_r2wC", "summary": "The paper presents a Gaussian-based SLAM framework that integrates open-vocabulary semantic features derived from pretrained visual-language models. It introduces an updating mechanism for semantic features across frames, aiming to improve temporal consistency and robustness in semantic mapping. The method is evaluated on multiple RGB-D datasets, showing improvements in both geometric accuracy and semantic segmentation performance.", "questions": "Please refer to the weaknesses.", "limitations": "", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "- Leveraging pretrained models for enhancing scene-level semantic awareness is meaningful and practically relevant for scalable 3D mapping.\n\n- The integration of semantic information into a Gaussian-based SLAM framework is valuable and timely, addressing the growing need for open-vocabulary 3D understanding on real-time systems.\n\n- The experimental section covers multiple datasets and tasks， typically three in-the-wild scenes captured by everyday devices are interesting.\n\n- The visualization in the paper is clear and informative, effectively illustrating the semantic reconstruction results and helping understand the pipeline of the proposed method.", "weaknesses": "- The quantitative comparison in Table 1 lacks several recent SOTA RGB-D SLAM baselines such as MonoGS, Gaussian-SLAM, RTG-SLAM, and SplatSLAM. It is highly recommended to include these baselines to ensure a fair, comprehensive, and up-to-date evaluation of the proposed module's performance.\n\n- The organization of the paper needs substantial improvement. Essential information about how semantic features are extracted, processed, and integrated into the pipeline is relegated to Appendix A.1, while the main body spends excessive space on general preliminaries and related work (Eq. 1–5, 10–11). As a result, readers struggle to understand the core contribution without repeatedly consulting the supplementary material. The authors should move the key descriptions of the semantic feature pipeline, backbone choice, 2D-to-3D mapping, update rules, and training losses into the main Method section, and condense redundant preliminary equations.\n\n- The use of pretrained semantic features from CLIP and DINOv2 is no longer novel, as several recent works, such as SemGauss-SLAM and OVO-SLAM, have already demonstrated open-vocabulary mapping with similar embeddings. The only distinctive component here is the recurrent update mechanism applied to explicit per-map semantic features. However, the motivation for updating explicit feature representations across frames is not clearly justified. Updating/Memorization is intuitive for latent representations that capture temporal dependencies, but less so for fixed explicit map representations, as per pixel level, their cross-frame correspondence is not recurrently updated but shifted from adjacent pixels in a continuous trajectory.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "0jI8LPthrN", "reviewer": "Reviewer_fyiJ", "summary": "The paper, \"Open-Set Semantic Gaussian Splatting SLAM with Expandable Representation“, introduces a system designed to enable everyday devices like smartphones to capture and reconstruct 3D scenes with rich, open-set semantics. The core idea is to integrate an expandable semantic feature pool with a Gaussian Splatting SLAM (GS-SLAM) framework. This feature pool decouples scene-level semantics from individual 3D Gaussians, using lightweight indexing vectors to reduce memory overhead and support dynamic updates of semantics. The system also incorporates a consistency-aware optimization strategy and a Semantic Stability Guidance mechanism to improve cross-view semantic consistency.", "questions": "See the weaknesses.", "limitations": "", "rating": 8, "confidence": 5, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "The paper presents a compelling and timely contribution at the intersection of dense SLAM, 3D Gaussian splatting, and open-vocabulary semantic understanding. \n\nThe work demonstrates high originality through several key innovations: \n\n1. Expandable Semantic Feature Pool: Rather than embedding high-dimensional semantic features directly into each Gaussian (which is memory-prohibitive), the authors propose a shared, dynamic, and expandable semantic feature pool. Each Gaussian references semantics via a lightweight indexing vector. This design is both novel and pragmatic—it decouples scene-level semantics from per-point storage, enabling scalability and dynamic updates. \n\n2. Open-Set Semantic Integration in SLAM: While prior SLAM systems typically handle closed-set semantics (e.g., fixed object categories), this work enables open-set semantic understanding—supporting queries like “fruit in plastic bag” or “xbox”—by leveraging foundation models (e.g., CLIP) within a real-time SLAM pipeline. This bridges a critical gap between foundation model capabilities and real-world 3D reconstruction. \n\n3. Consistency-Aware Optimization: The paper introduces a dual mechanism—an Intra-Inter Semantic Consistency Objective (via contrastive learning) and Semantic Stability Guidance (via cosine-similarity-based reweighting)—to enforce cross-view and temporal semantic coherence. This addresses a fundamental limitation of naively lifting 2D foundation model outputs into 3D without considering geometric consistency.", "weaknesses": "1. While the paper claims to reduce memory overhead \"by orders of magnitude\" (line 037) through the expandable semantic feature pool, the memory footprint analysis in Table 5 and Table 10 still shows significant memory usage, particularly as the semantic feature dimension ($D_s$) or pool size ($L$) increases. For real-time applications on \"everyday devices (e.g., smartphones)\" (line 025), the current memory requirements might still be prohibitive.\n\n2. The runtime analysis in Table 8 shows that the proposed method incurs an \"approximately 8% – 21% additional time overhead\" compared to SplaTAM and LoopSplat. While this might be acceptable for some applications, achieving truly real-time performance on mobile devices for complex scenes with continuous semantic updates remains a challenge. The paper mentions that the \"additional computational overhead stems from semantic assignment from Fp\" and the consistency objectives.\n\n3. The paper states that the system \"demonstrates limited robustness in highly dynamic or large-scale (e.g., city-level) environments\" (lines 1150-1151). This is a significant limitation for a SLAM system designed for \"in-the-wild 3D scenes\" (line 026). While the Semantic Stability Guidance mechanism is introduced to mitigate semantic ambiguity, a more thorough analysis of its effectiveness in highly ambiguous or dynamic scenarios would be beneficial.\n\n4. The system relies on pre-trained 2D foundation models like CLIP [11] or SAM [12] for semantic distillation. While this is a common approach, the performance and generalizability of the system will be inherently tied to the capabilities and biases of these underlying models. The paper mentions that the framework is not dependent on \"any specific semantic extraction method\" (line 826), but the choice of these models can still impact the quality of the semantic representations.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)