SegMASt3R: Geometry Grounded Segment Matching
Authors: Rohit Jayanti
Swayam Agrawal
Vansh Garg
Siddharth Tourani
Muhammad Haris Khan
Sourav Garg
Madhava Krishna
Venue: NeurIPS 2025 spotlight ()
Keywords: Segment Matching
3D Instance Mapping
Object-Level Topological Navigation
Primary area: applications

Abstract: Segment matching is an important intermediate task in computer vision that establishes correspondences between semantically or geometrically coherent regions across images. Unlike keypoint matching, which focuses on localized features, segment matching captures structured regions, offering greater robustness to occlusions, lighting variations, and viewpoint changes. In this paper, we leverage the spatial understanding of 3D foundation models to tackle wide-baseline segment matching, a challenging setting involving extreme viewpoint shifts. We propose an architecture that uses the inductive bias of these 3D foundation models to match segments across image pairs with up to $180^\circ$ rotation. Extensive experiments show that our approach outperforms state-of-the-art methods, including the SAM2 video propagator and local feature matching methods, by up to 30\% on the AUPRC metric, on ScanNet++ and Replica datasets. We further demonstrate benefits of the proposed model on relevant downstream tasks, including 3D instance mapping and object-relative navigation.

SegMASt3R: Geometry Grounded Segment Matching
Rohit Jayanti 1+t
Swayam Agrawal
Vansh Garg"
Siddharth Tourani
Muhammad Haris Khan
Sourav Garg
Madhava Krishna'
"II' Hyderabad
University of Heidelberg
MBZUAI
'Independent
Project Page:
.g.istttrr.iiiiiii github. io/
Abstract
Segment matching is an important intermediate task in computer vision that estab- lishes correspondences between semantically geometrically coherent regions across images. Unlike keypoint matching, which focuses on localized features, segment matching captures structured regions, offering greater robustness clusions, lighting variations, and viewpoint changes. In this paper, we leverage the spatial understanding of 3D foundation models tackle wide-baseline seg- ment matching, challenging setting involving extreme viewpoint shifts. We pro- pose architecture that uses the inductive bias of these 3D) foundation models to match segments across image pairs with up to 180% rotation. Extensive experi- ments show that our approach outperforms state-of-the-art methods, including the SAM2 video propagator and local feature matching methods, by up to 30% on the AUPRC metric, on ScanNet++ and Replica datasets. We further demonstrate ben- efits of the proposed model on relevant downstream tasks, including 3D instance mapping and object-relative navigation.
Introduction
Segment matching establishes correspondences between coherent regions-objects parts, mantic segments-acros images. underpins video object tracking [7] scene-graph construc-
Because it matches tion [18 (23, robot navigation [16 15 32, and instance-level SLAM 26 44
extended structures rather than sparse, texture-sensitive points, it is more robust to noise, occlusion, and appearance change. Mapping structured regions instead of isolated pixels also boosts inter- pretability and allows geometric or semantic priors to be injected, enabling higher-level reasoning about scene content.
Segment matching degrades sharply under wide-baseline conditions, where images of the same scene taken from widely separated viewpoints that introduce severe perspective, scale, and up to 180 rotation changes [34]. Such cases arise in long-term video correspondence and robotic navi- gation, and demand models that reason over global 3D structure and enforce geometric consistency. Current approaches that depend on features from pre-trained encoders like DINOv2 30 or ViT [0]] often mismatch repetitive patterns fail to link drastically different views of the same object.
In this paper, we propose to leverage the strong spatial inductive bias of 3D foundation model (namely MASt3R (2]) to solve the problem of wide-baseline segment matching. 3D Foundation Models (3DFMs) are large-scale vision models trained to capture spatial and structural properties of scenes, such depth, shape, and pose. Unlike sopeaarrnne-foussee models. 3DFMs learn geometry- aware representations using 2D and 3D supervision, enabling strong generalization across tasks
Equal Contribution. Corresponding Author. Now at Google DeepMind.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

like reconstruction and pose estimation. Their inductive bias toward spatial reasoning makes them well-suited for applications requiring geometric consistency such our problem of wide-baseline segment matching.
We adapt MAS13R for segment matching by appending lightweight segment-feature head that transforms patch-level embeddings into segment-level descriptors. Given an image pair, these de- scriptors are matched establish segment correspondences. The head trained with a contrastive objective patterned after SuperGlue [35] Experiments show that our approach surpasses strong baselines, including SAM2's video propagator which is trained on far larger datasets and state-of- the-art local feature matching methods. Finally, we demonstrate its practical utility in two down- stream tasks: object-relative navigation and instance-leve mapping.
Contributions Our key contributions summarized below:
We introduce simple but effective approach for learning segment-aligned features by leveraging strong priors from a 3D foundation model (3DFM) MAS13R. A differentiable segment matching layer is employed align features across views, while segment-feaure head transforms dense pixel-level representations into robust segment-level descriptors.
To address the under-explored problem of wide-baseline segment matching, we construct a comprehensive benchmark comprising both direct segment association methods and those based on local feature matching. Our method demonstrates significant improvements over all baselines on challenging bide-baseline image pairs.
We validate the practical utility of our approach by applying it to the downstream ap- plications such as 3D instance mapping and object-relative topological navigation. Our method outperforms competitors by significant margins showcasing the efficacy of our de- sign choices.
2
Related Work
Segment Matching and Segmentation Foundation Models. Robust segment-level association has emerged crucial intermediate step for high-level vision tasks such as scene graph construc- tion, long-term object tracking, and multi-view instance association. While related sub-problems like video instance segmentation and object tracking have been studied extensively in
29 24 6). the broader challenge of matching segments across arbitrary viewpoints, modalities, and time remains comparatively under-explored. Large-scale class-agnostic segmentation models have begun to close this gap. The Segment Anything Model (SAM) [20] and its successor SAM2 3d deliver high-quality masks and include built-in propagator for associat- ing masks across video frames. However, this propagation module is optimized for short temporal windows and does not explicitly enforce geometric consistency under wide baselines or substantial appearance changes.
works [46 45
41
11 31
Learning to Match Segments and Overlap Prediction. Some recent methods address segment matching more directly. MASA (25 augments SAM's rich object proposals with synthetic geo- metric transformations to learn instance correspondences, and DMESA [49] extends these ideas to dense matching with improved efficiency. Despite such progress, these approaches are still limited by 2D supervision. An alternative line of work predicts the degree of visual overlap between im- ages [7] 2) By estimating shared content, these methods implicitly learn region correspondences, yet they also remain confined to 2D training signals.
Local Feature Matching. Sparse ers propagate pixel-level correspondences that can, in principle, transfer segment labels between views [15 [6]. Nonetheless, like the previous categories, they are trained exclusively on image data and struggle with extreme viewpoint changes.
40
36 and dense 12 4
local feature match-
Across segment matching, overlap prediction, and feature matching, reliance purely 2D super- vision leaves existing techniques brittle under wide-baseline conditions. Our method addresses this limitation by fine-tuning the 3D foundation model MASt3R (22), whose strong geometric priors enable reliable segment correspondence even when image pairs differ by nearly 180 in viewpoint.

---

Image
Shaned image Enooder
Fresen
2
Putsi Lavel Featanca
Masks
Bognent Fararr
Differentable Optimal Transport
Buchleene
Soore Noatk
Rowestie
A1
Transtor
Segmant
Decader
Fearure
Sinkhor
Croas-Atariee on
V
G2
(M,39
Seft Commepondencess
Figure 1: Pipeline Overview: An image pair is processed by frozen MASt3 backbone to extract patch-level features; segmentation masks are obtained either from a parallel segmentation module or ground truth annotation; the patch-level features are aggregated by the regment-featrr head to form segment-lewe descriptors; and these descriptors are then matched across images via differentiable optimal transport layer to produce segment-level correspondences.
Method
Figure T provides overview of method. builds upon the MAS(3R introducing a Feat2Seg Adapter that maps the patch-level features output by the MASt3R decoder get segment-level features, which are subsequently matched via differentiable optimal transport and row-wise argmax yielding the final segment-level correspondences.
architecture by 22
3.1 MASt3R Preliminaries
MASt3R is a 3D foundation model pre-trained on diverse collection 3D-vision datasets com- monly used for tasks such as metric depth estimation and camera-poss prediction. Given a pair of images, produces dense 3D point maps for each image and identifies correspondences between them. Thanks to training on data with wide-baseline pairs, MASt3R generalizes well to unseen im- age pairs [22 and consistently outperforms alternative 3D matching methods such MicKey ( which relies on DINOv2 backbone 3
Architecture: We summarize here the portions of the architecture we utilize in our pipeline. The two images 7 and l2 are processed in a Siamese manner by weight-sharing ViT encoder [T0) E, resulting in token sets H1. H2, i.e.
H1 E(I,,
H2 E(I2)
style intertwined transformer decoders jointly refines the two feature sets. By alternating self- and cross-attention, the decoders exchange information between inputs to capture both the relative viewpoints and the global 3-D structure of the scene. We show via an ablation study, these cross-view aware decoders provide significant boost to the segment matching in Section 5
Cross-view transformer decoder: Next, pair of CroCo 42 43
The resulting geometry aware representations denoted V1 and V2;
(V1, V/2 Decoder (H1, H²).
Assuming. H, W denote the height and width of the input images, the output geometry aware patch- level features are of size (H/16, W/16, 768). The original architecture subsequently uses two pre- diction heads output dense point-maps, as well as pixel-level features, they are not shown in Fig- ure they are not used in our pipeline. These portions of the pipeline remain frozen and their outputs are used as is, we instead introduce new segment-feature head.
3.2 Segment-Feature Head: Segment-Aligned Features
The MASt3R decoder produces patch-level embeddings V1, 1/2 shape (H/16, W/16, 768). In the original MASI3R [22). feature head upsamples these tensors to the resolution of the input image. For segment matching we introduce another head to transform the patch level features to M segment features. This introduced head is realized as an MLP that upsamples the patch-level

---

features (V1, V2) to image resolution, yielding feature maps of size (H, W, 24), 24 being the feature dimension. We denote this Feature-to-Segmett head the segment-feature head. In addition to the patch-features, the segment-feature takes as input M image resolution segment masks, obtained either from an external segmenter such as SAM2 [34] or from ground-truth annotations. Both the masks and feature maps are flattened along the spatial dimensions yielding flattened tensors.
Piat 22
Ma RM
To go from pixel-level descriptors segment descriptors, a single batched matrix multiplication aggregates the pixel descriptors inside each mask:
G Maat P Hltt E Mx24
The resulting segment embeddings for the images denoted G1 and G2 in Figure 1 and fed to the differentiable matching layer deseribed in Section 3.3 We set M 100 as an upper bound for batch processing. In practice, images typically contain 20-30 masks when training with ground truth annotations; we pad with zeros when fewer masks are present. At inference time, the number of masks be arbitrarily, independent of the training-time value of M.
3.3 Differentiable Segment Matching Layer
The goal of the differentiable segment matching layer is to establish permutation-styl correspon- dences between the M segments from each image, given segment descriptors (G1, G2) E (M,44)
similiysimilarity affinity. We first construct an affinity matrix product
M.M. with a simple dot
S (g. isis M, 1s S M2.
(2)
g and gÿ correspond to segment-level features from G₁ and G2 respectively. Ideally, segment features corresponding the same underlying 3D region should have a high similarity score and dis-similar regions a correspondingly low score.
Learnable dustbin. Following 35 incorporate dustbin row and column in the affinity ma- trix S to handle segments without correspondences in the other image, which is critical for wide- baseline matching. We augment (2) by concatenating an additional row and column initialized with learnable logit a E IR, yielding S IR (M+++))))+++)
a1
01
Soft Correspondences via Sinkhorn The similarity logits are transformed into soft assignment matrix P by T' iterations of the Sinkhorn normalisation [3
37 in log-space:
0 exp($/7).
(1)
1
(
1
E, p(o
Σ, PM
((+1) = ut (() (0)
0 S
where 7 is temperature hyper-parameter After convergence, P P-P is (approximately) doubly stochastic. Throughout all experiments conducted, we assume = 50. The output of the Sinkhorn algorithm (TT) is soft bi-stochastic matrix, which has be discretized obtain the final segment matches.
Discrete correspondences. To obtain the simple final segment matches, we perform a simple row- wise arg max over the non-dustbin columns:
mii)
argmax Puj- with assignment accepted if j * M 15/5Ma

---

3.4 Supervision
Training objective. We adopt the SuperGlue cross- cross-entropy loss Eso [3 extended with explicit terms for unmatched segments:
C
Σ (J))
log Pij
-Llog -loggP.M P.MM+ iEl4
Elog JEUlg
log PM+1J
(4)
where M is the set of ground-truth matches and U4 U2 are the unmatched indices in image I and 2, respectively. The dustbin parameter a is learned jointly with the rest of the network, enabling the layer to balance match confidence against the cost of declaring non-matches. This fully differen- tiable design allows the matching layer to be trained end-to-end together with the upstream segment encoders and downstream task losses.
3.4.1 Training Details
The model is trained using A optimizer with an initial learning rate of 1e-4, weight decay of 1e-4, and cosine annealing learning rate schedule without restarts, decaying up minimum learning rate of 1e-6 over the full training duration. We use a batch size of 36 and train the model for 20 epochs on single NVIDIA RTX A6000 GPU. The segment-feature heads are initialized with MASt3R's local feature head weights and finetuned further. For the differentiable segment matcher initialize the single learnable dustbin parameter 1.0. The number of Sinkhorn iterations set to 50. Training our model on ScanNet++ takes 22 hours, whereas a single forward pass during inference with batch size of takes 0.579 seconds.
4 Experiments
4.1 Datasets
Our network is trained on scenes from ScanNet++ 47 which contain diverse set of real-world scenes, primarily indoor settings. We test our model on novel scenes from ScanNet++ as well as perform cross-dataset generalization studies on Replica [39] and MapFree [2] datasets. The former contains high-quality photo-realistic indoor scenes while the latter a challenging outdoor visual- localization dataset, which is sufficiently out-of-distribution considering our training data.
ScanNet++ 47
ScanNet++ contains 006 indoor scenes captured with DSLR images and RGB- D iPhone streams, all co-registered to high-quality laser scans. The dataset supplies 3D semantic meshes, 2D instance masks, and accurate camera poses, which we utilize in our pipeline. We train on 860 k image pairs from 140 scenes and evaluate on 8 k pairs from 36 validation scenes, sampled with a fixed seed (42) and balanced across scenes and four pose bins: [0 [ [45 and [135- - 180%, defined by the rotational geodesic distance between camera orientations.
90% [90 133%),
The Replica dataset contain 18 high-quality, photo-realistic indoor room reconstruc- tions in the form dense meshes replete with per-primitive semantic class and instance information. In particular, we use pre-rendered version of this dataset from Semantic-NeRi (50), which directly provides poses, RGB-D sequences, and per-frame semantic masks. For evaluation, we employ the same pose-binning strategy as described above and randomly sample 3200 image pairs across scenes, again ensuring uniform sampling across both scenes and pose-bins.
Replica 33
MapFree Visual Re-localization (2)- MapFree is challenging benchmark for metric-relative pose estimation, featuring 655 diverse outdoor scenes (sculptures, fountains, murals) with extreme viewpoint changes, varying visual conditions, and geometric ambiguities. The training split contains 460 scenes with 0.5M images; we uniformly sample 50 scenes yielding 31K image pairs for training. Since test split ground truth is unavailable, we report results on the validation split using 7.8K pairs from 13 uniformly sampled scenes (out of 65 total).
4.2 Baselines
Local Feature Matching (LFM). Although few models target segment matching directly, wide range of local feature matchers (LFMs)) exists at different densities: sparse SuperPoint [9], semi- dense LoFTR [40], and dense RoMa and MASt3R [12 22) We harness these state-of-the-art LFMs
5

---

Type
Method
0' 0'-45"
45" -50
10'5
135 135"-180"
AUPRC
R@i
R
AUPRC
R@1
R485
AUPRC
R@
R65
AUPRC
RØ1
R@5
Local Feature
SP-LG
0 22
42.1
45.6
51.2
33.5
36.9
43.1
15.9
19.7
26.2
6.1
9.3
14.6
Matching
GiM-
59.1
64.9
69.7
54,9
60.2
66.1
39.6
44.5
51,8
21.3
25.9
32.7
(LFMM)
DKM
11
ReMA
123
61.6
68.7
73.5
58.9
66.4
73.0
47.4
56.1
65.5
30.0
39.5
49.7
MASCR 22
59.5
74,2
65.6
52.9
68.9
45.4
62.2
Sogmentt
Matching
SAM2 34
DINO2 24]
61.9
57.9
66.7
64.6
87.4
67.5
43.0
46.6
50.1
55.9
54.0
27.9
33.5
48.0
32.5
37.2
78.0
17.0
32.4
21.6
25.4
75.6
(SegMatch)
SegVLAD [4
44.2
58.6
81.4
32.1
49.5
76.5
23.2
42.2
70.5
20.0
39.6
66.8
MASER 22
51.7
54.6
69.9
45.6
49.8
68.5
41.4
47.9
69.2
39.5
48.7
72.6
Ours
SEOMAST3R
92.8
93.6
98.0
91.1
92:2
97.6
88.0
89.5
96.8
83.66
85.9
95.9
Table 1: Performance of selected methods across pose-hins ScanNet++
47
Blue cells mark
the best scores; Orange cells mark the second-best.
via the Matth toolkit [5] obtain segment correspondences. Each matched keypoint pair votes for the source and target masks that contain its coordinates, populating vote matrix of size M x N (with M and N segments in the two images). Correspondences are taken as the highest-scoring entries of this matrix; the full algorithm is provided in the supplementary.
Segment Matching (SegMatch). We also compute segment matches from dense features of two strong pre-trained vision encoders, DINOv2 30) and MASt3R (22), as well as SAM2's video prop- agator 34 to track masks across views. For feature based methods, masks are downsampled via neighor interpolation to match feature resolution, and segment descriptors are computed via masked average pooling. Cosine similarity between descriptors yields a match matrix, from which one-to-one links are selected via mutual-check. We further benchmark against SegVLAD (14), which aggregates features from neighbouring segments for segment retrieval based visual place recognition.
4.3 Evaluation Metrics
We report two complementary measures of segment correspondence quality AUPRC and Re- call. Area Under the Precision-Recall Curve (AUPRC) integrates precision over the entire recall axis, providing a threshold-independenn summary that is particularly informative under the high olass-imaalance characteristic of segment matching between image pairs. Recall@ k (R@k) de- notes the fraction of query segments whose ground-truth counterpart is found within the top ranked candidates, thus gauging how effectively the method surfaces correct matches among its highest-confidence predictions. Additional details for the dataset, baseline and experiments can be found in the supplementary, along with more qualitative results.
5 Results
Segment Matching In Table [ compare our proposed method SegMASt3R with the state- of-the-art methods using two categories of approaches in the literature: the well-established lo- cal feature matching (LFM) based on sparse or dense keypoint correspondences, and the recently emerging open-set instance association based on segment matching (Seg.aatch). It can be observed that SegMASt3R outperforms all the baselines for all the pose bins with huge margin. Amongst the LFM techniques, dense matchers (RoMa and MASt3R) outperform sparse matchers (SP-LG and GiM-DKM) on both the evaluation metrics for the task of segment matching. Notably, on the highly challenging wide-baseline settings, MASt3R outperforms other local matchers by a large margin. However, when the same backend features are aggregated at the segment-level, MASt3R's performance deteriorates significantly, e.g., AUPRC drops from 52.9 to 41.4 on the 90- 135 pose bin. This uncovers an inherent limitation: feature distinctiveness at the pixel level does not necessarily translate to the instance level. Considering the SegMatch techniques which are not trained for the LFM task, can be observed that DINOv2 and Seg VLAD perform particulary well on R@5 metric, which aligns with their typical use for coarse retrieval [19 14) On the other hand, SAM2's two- frame video propagation only works well for narrow-baseline matching, which can be expected as its training set is mostly comprised of dynamic object tracking. Overall, these results show that all

---

Type
Method
0'45"
45
90'-135"
135 135"-180"
AUPRC
R@
R@8
AUPRC
R@1
R@5
AUPRC
R@1
R-85
AUPRC
R@
R@5
LFM
MASER
78.2
86.5
89.4
69.5
77.6
81.0
48.0
60.4
64.6
32.5
49.0
54.1
SegMatch MASIER
51.0
23.6
45.9
Ours
SEGMAST3R
95.0
96.0
98.6
86.2
91.2
96.4
73.4
85.2
95.7
68.4
83.8
94.8
Table 2: Performance of selected methods across pose-bins on Replica [3 Blue cells mark the Orange cells mark the second-best.
best scores;
the baseline methods lack at least one of the fronts: pixel- vs segment-level distinctiveness, recall vs precision, and narrow- vs wide-baseline robustness. Our proposed method SegMAS13R achieves all these desirable properties with high performance across the board by learning segment-level rep- resentations with the training objective of instance association. In Section 5 we provide qualitative results which emphasize the ability of our method address the problem of perceptual instance aliasing and instance matching under extreme viewpoint shifts.
Generalization We assess our model's ability to generalize to new environments. First, in Ta- ble we present results on different indoor dataset, Replica, test the generalization ability of SegMASt3R, which is trained only on ScanNet++. We compare SegMASt3R against the LFM and SegMatch versions of MASt3R, these three methods closely resemble each other in terms of their network architecture (detailed comparisons are included in the supplementary). It can be observed that the performance patterns on Replica remain largely the same as that on ScanNet++, and SegMASt3R consistently outperforms the L.FM and SegMatch versions of MASt3R across the board.
Furthermore, we test generalization to challenging outdoor scenes from the MapFree dataset (2), with results shown in Table Since MapFree lacks instance-level ground truth, we use SAM2's video propagator on image sequences to generate a pseudo-ground truth for evaluation. Our indoor- trained model (SegMASt3R (SPP)) shows a regression in performance in comparison to DINOv2, highlighting the domain shift. However, this gap can be substantially closed by either re-training on MapFree data (SegMASt3R (MF)) or, even more simply by just recalibrating the single learn- able dustbin parameter using grid-search over small calibration set from the target domain (SegMASt3R (SPP, Dusthin MF)). This demonstrates the strong adaptability of our model's learned geometric features.
Method
Train Set
Overall IoU
0-45"
45-90'
90-135"
135-180'
DINOv2 30 MASI3R (Vanilla)
Multiple Multiple
84.4
85,4
85.2
83.5
83.8
69.2
73.4
69.8
70.0
66.1
SegMAScR (SPP)
ScanNet++
75.2
75.2
74.6
76.5
74.5
SegMASt3R (SPP, Dustbin MF)
ScunNet++
88.7
88.6
88.6
88.5
88.9
SegMASt3R (MF)
MapFree
93.7
93.3
93.7
93.9
93.9
Table Generalization performance on the outdoor MapFree dataset
Method
officel)
officel
office2
office3
office4
roomli
rooml
rooma2
AP/AP650
AP#50
AP/
AP#50
AP/AP620
AP#50
Concepirraphs (MobileSAM MARellSAMMMM Misks) [b
11.84/2843
20,31/43.79
22.82
8,07 /22.83
9:46 23
12.23
34.4
583/12.96
7.83/2382
ConceptGraphis Masks) [X]]
22.487 4071
60.69
32.06/55.44
39.63
44.89 16964
36.53
25.93/43.63
SegMAScRR(ars, GT Masks)
79.91/87.17
$4.89/6442
64.00/85.50
S8.02/79.0
67. 65.08/85.01
71.02/9122
64.99888
/8..0
56.35/7666
Table Class-Agnostic instance- mappnng performance (AP and AP@ 50) on Replica scenes, shown in percentage. The best value in each column is highlighted in blue.
3D Instance Mapping The goal of instance mapping to localize object instances in 3D so a robot can distinguish objects of the same class in both image and metric space [1]28] The main difficulty is preserving identities over long trajectories, especially when objects leave the camera's view and later reappear from different angles. Our pipeline employs SEGMAST3R to match object masks across image pairs. Given ground-truth masks and sampled pairs, we extract mask features, solve Sinkhorn assignment, and take the wwwe argmax obtain tentative correspondences.
7

---

Method
IoU 0.25
IoU 0.50
IoU > 0.75
AUPRC
R@1
R@5
AUPRC
R@1
R@5
AUPRC
R@1
R@5
SAM2 3
47.4
56.2
61.2
50.2
58.7
62.6
55.6
64.1
66.6
D 30
39.0
60.0
89.2
44.4
65.0
92.0
54.9
73.1
94.0
MASB3 (Vanilla) 22
50.7
58.0
76.2
52.4
59.5
77.0
49.6
57.4
76.3
SegMASI3R (Ours)
84.2
91.9
99.3
87.6
94.4
99.3
89.9
94.3
96.3
Table 5: Performance on ScanNet++
47
using noisy masks from FastSAM.
We then back-project each matched mask into 3D and drop links whose point-cloud IoU falls below 0.5, rejecting any match that fails this geometric check. Details are provided in the supplementary. Table[4 shows percentage AP versus ConceptGraphs [17]. We evaluate under two conditions: using masks generated by MobileSAM 48 in the original ConceptGraphs setup, and using ground- truth (GT) masks for a fairer comparison of the underlying matching capability. Our geometry- aware matching yields higher accuracy in both settings, particularly when objects exit and later re-enter the field of view, demonstrating robustness to both noisy masks and challenging viewpoints.
To assess the practical viability of our approach in real-world scenarios where ground-truth masks are unavailable, we evaluated all methods on Scan- Net++ using imperfect segmentations generated by FastSAM. A key challenge in this setting that evaluation can conflate the performance of the segment matcher with the quality of the upstream Automatic Mask Generator (AMG). A low score may not distinguish between a matching failure and an AMG failure where predicted masks do not align with any ground-tros truth segment.
Robustness to Noisy Segmentation Masks
To decouple these factors and isolate the core matching performance, we adopt an evaluation proto- col that does not penalize methods for AMG errors. Specifically, while all methods take noisy masks as input, the evaluation is performed only on the subset of ground-truth correspondences for which valid match between predicted segments was possible. As shown in Table SegMASt3R maintains substantial performance margin over all baselines across different evaluation thresholds. This re- sult confirms that the strong geometric priors learned by our method provide significant robustness, enabling superior matching even when conditioned on inconsistent and noisy segment inputs.
Object-level Topological Navigation Recent works [17 15 [6] have explored the of SAM2's open-set, aaaaanicalll--mannngnn
instance segmentation for topological mapping and navigation. These methods rely on accurate segment- association. To show the benefits of our proposed method on complex downstream tasks such as navigation, we considered an object topology-based mapping and navigation method, RoboHop [15), and swapped its SoperPPointLightGlue based seg- ment matching with our proposed segment matcher. This segment matching is used by RoboHop's localizer match each of the object instances in the query image with those in the sub-map images, which is then used to estimate the currently-vewwed object's path to the goal and correspondingly obtain a control signal. We used the val set of the ImageInstanceNav (21 dataset, which comprises real-world indoor scenes from HM3Dv0.2 [33]. We followed 33 for creating map trajectories and evaluating navigation performance.
In Table 6 we report navigation success for (vanilla) RoboHop and an enhanced version of it that uses MegMRS for segment association for localization and navigation. We use two metries: SPL (Success weighted by Path Length) [ and SSPL (soft SPL. g. Furthermore, we considered four different evaluation settings using two parameters that define the submap used for localizing the query image segments: Submap Span (S.). which defines the total number of images sampled from the map based on the distance from the robot's current position, and Submap Density (Sp), which defines a subsampling factor to uniformly skip map images. These parameter configurations aid in testing narrow- and bise-iae matching as well as the ability to avoid false positives. Table shows that SegMASt3R consistently outperforms vanilla RoboHop. In particular, we achieve an absolute improvement of 27% SPL on one of the hardest parameter settings: S, 16, S, 0.25, where only 4 submap images are sampled due to low density value. This shows that even with a very sparse submap, it is possible to maintain high navigation success rate, thus avoiding the typical trade-off between compute time and accuracy for the localizer. In Figure 2 we present qualitative comparison between vanilla RoboHop's 's segment matching and that based on SegMASt3R: (left) the mismatch between the wall (orange) and the vanity cabinet for the vanilla methods leads to an incor-
8

---

Method
5, 16, Sp - 0.25
$, =
16, Sp
0.5
5. = 32, S, - 0.25
= 32, Sp 0.5
SPL
SSPL
SPL
SSPL
SPL.
SSPL
SPL
SSPL
RoboHop 15
36.34
54.25
54.51
69.98
60.57
68.29
57.52
68.47
SegMASt3R (Ours)
63.60
78.84
63.60
78.33
66.62
75.20
63.56
73.89
Table 6: Navigation performance comparison. The best value in each column is highlighted in blue.
Query Image
Submap Ref Image
Query Image
Submap Ref Image
RoboHop w. DINOv2
w. SegMASt3R
Figure 2: Segment Matching-Guidee Navigation. (left) In vanilla RoboHop's segment matching, a wall segment (orange) gets mismatched with the vanity cabinet and misguides the agent to move towards its right, leading navigation failure. (right) SegMASt3R correctly recognizes the same cabinet as well as other segments and guides the robot into the bathroom, and eventually to the goal. manually probed the Note that the query and submap images vary across both the methods, point of failure for the baseline and the nearest agent state for ours.
rect rotation towards the right, consequently leading to navigation failure, whereas SegMASt3R's accurate segment matching (right) guides the agent into the bathroom and finally the goal object.
Impact of the Feature Encoder To isolate the contribution of the feature encoder in our pipeline, we replace MASt3R's cross- attention block with two alternative, purely 2D backbones: CroCo (shared by both MASt3R and SegMASt3R) and DINOv2, state-of-the-arr encoder often used for off-the- offtthe-shelf segment matching [5 14. As reported in Table neither backbone yields com- petitive eegment-matching accuracy. This result underscores that learning segment descriptors from a 2D foundation model alone inadequate; geometric context is essential. In contrast, MASt3R's cross-attention layers, 3D-aware training regimen, and explicit formulation of image matching in 3D jointly endow the model with the priors required for reliable instance association. The fact that CroCo is always second best suggests that cross- view-completion, by itself, can yield superior results for segment matching between image pairs.
Method
0'-45"
45"-90"
-135"
133"
ALPRC.
R@1
R@5
AUPRC
R@1
R@5
AUPRC.
R@1
R
AUPRC
R@1
R@5
DINOv2-SegFeat
64.7
71.5
89.3
55.7
65.9
87.3
45.4
59.1
84.4
36.8
53.4
81.3
CroCo-SegFeat
73.4
78.8
92.3
64.0
73.1
90.6
50.7
64.6
38.5
56.6
SogMASC3R (Ours)
92.8
93.6
98.0
91.1
92.2
97.6
88.0
89.5
96.8
83.6
85,9
95.9
Table 7: Proposed method and model ablations performance across pose-bins on ScanNet++ Blue cells mark the best scores; Orange cells mark the second-best.
47
Qualitative Results compares SegMASt3R with SAM2's two-frame video propagation- based segment matching under extreme viewpoint variations on the ScanNet++ dataset. Each row shows a reference image, SAM2's matches, SegMASt3R's matches, respectively. In the top row, wall (pink) and a door (blue) are mismatched by SAM2, whereas SegMASt3R correctly associates them, despite very limited visual overlap. In the bottom row, SAM2 gets confused between the two monitors, whereas SegMASt3R is able to correctly associate them despite the simultaneous effect of 180 viewpoint shift and percepiual instance aliasing (i.e., different instances of the same object category in an image potentially lead to mismatches). Figure presents qualitative results on the (outdoor) MapFree 2 dataset, where we compare our ScanNet-trained SegMASt3R with the off-the thel DINOv2
30 features. Each image triplet represents the query segment (left) and its

---

Query
SegMASt3R
DINOv2
Query
SegMASt3R
DINOv2
Figure 3: MapFree Outdoor Dataset Perceptual Instance Aliasing (left): the right leg of the signboard as a query segment (red) is correctly matched by our method but mismatched with its left leg by DINOv2. Sinkhorn Matches to Dusthin (right): the query segment (red) is not visible in the reference image and is correctly ignored by our method, whereas DINOv2 mismatches with vehicle segment.
Figure 4: ScanNel++ Dataset Wide-baseline Matching (top): The wall (pink) and the door (blue) in the query image (left) gets incorrectly associated by SAM2's video propagation (middle), whereas SegMASt3R (right) is able to correctly match them despite very limited visual overlap. Perceptual Instance Aliasing (bottom): unlike SAM2, SegMASt3R is able to correctly associate the pair of monitors under the simultaneous duress of an opposing viewpoint observation and perceptual in- stance aliasing.
retrieved matches using our method (middle) and DINOv2 (right). The query segment is displayed in red color, and its top three matches are respectively displayed in red, green, and blue colors. The left image triplet illustrates another case of perceptual instance aliasing. SegMASt3R is able to resolve this aliasing problem, whereas DINOv2 confuses the right and left legs of the signboard. The right image panel shows that Sinkhorn solver effectively learns dustbin allocations for explicitly rejecting negatives, that is, the segments which do not have any corresponding match, whereas DINOv2 leads to incorrect matches.
Conclusion
We proposed SegMASt3R, simple method to re-purpose an existing 3D foundation model MASt3R for image segment matching. Our proposed method achieves excellent results ScanNet+ and Replica with a simple pipeline and a minimal amount of training. It especially excels on wide- baseline segment matching between image pairs. addition, we show that SegMASt3R has prac- tical applicability by evaluating it's performance on the downstream tasks of 3D instance mapping and object-relative topological navigation, where we significantly outperform the corresponding baselines. Overall, this work attempts to establish segment matching as core computer vision ca- pability, which will enable even more downstream applications future alongside the advances in image segmentation and data association.
10

---

References
[1] Peter Anderson, Angel Chang, Devendra Singh Chaplot, Alexey Dosovitskiy, Saurabh Gupta, Vladlen Koltun, Jana Kosecka, Jitendra Malik, Roozbeh Mottaghi, Manolis Savva, al. On evaluation of embodied navigation agents. arXiv preprins arXiv: 10X.00/077 2018.
[2] Eduardo Arnold, Jamie Wynn, Sara Vicente, Guillermo Garcia-Hermando, Áron Monszpart, Victor Adrian Prisacariu, Daniyar Turmukhambetov, and Eric Brachmann. Map-free visual relocalization: Metric pose relative single image. In ECCV, 2022.
[3] Axel Barroso-Laguna, Sowmya Munukutla, Victor Adrian Prisacariu, and Eric Brachmann. Matching 2d images in 3d: Metric relative pose from metric correspondences. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 4852-4863, 2024.
[4] Gabriele Berton, Carlo Masone, Valerio Paolicelli, and Barbara Caputo. Viewpoint invariant dense matching for visual geolocalization. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 12169 12169-12178, 2021.
[5] Gabriele Berton, Gabriele Goletto, Gabriele Trivigno, Alex Stoken, Barbara Caputo, and Carlo Masone. Earthmatch: Iterative coregistration for fine-grained localization of astronaut photog- raphy. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recogni- tion (CVPR) Workshops, June 2024.
[6] Oreun Cetintas, Guillem Brasó, and Laura Leal-Taixé. Unifying short and long-term tracking with graph hierarchies. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 22877-22881, 2023.
[7] Ying Chen, Dihe Huang, Shang Xu, Jianlin Liu, and Yong Liu. Guide local feature match- ing by overlap estimation. In Proceedings of the AAAI conference on artificial intelligence, volume 36, pages 365-373. 2022.
[8] Samyak Datta, Oleksandr Maksymets, Judy Hoffman, Stefan Lee, Dhruv Batra, and Devi Parikh. Integrating egocentric localization for more realistic point-goal navigation agents. In Jens Kober, Fabio Ramos, and Claire Tomlin, editors, Proceedings of the 2020 Conference on Robot Learning. volume 155 of Proceedings of Machine Learning Research, pages 313- mmriinnns.........mmm..mm.m.mm.m mlr, presss/v15/datta2l1..
328. PMLR, 16-18 Nov 2021. URRLattss//pprceedinn
html
[9] Daniel DeTone, Tomasz Malisiewiez, and Andrew Rabinovich. Superpoint: Self-supervised interest point detection and description. In CVPR Deep Learning for Visual SLAM Workshop, 2018. http::/axxxiv org/ abs//712.... 07629
[10] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010.119999 2020.
[11] Johan Edstedt, Ioannis Athanasiadis, Mårten Wadenbäck, and Michael Felsberg. Dkm: Dense kernelized feature matching for geometry estimation. In Proceedings of the IEEE/CVF Con- ference on Computer Vision and Pattern Recognition, pages 17765- 17775, 2023.
[12] Johan Edstedt, Qiyu Sun, Georg Bökman, Mårten Wadenbäck, and Michael Felsberg. RoMa: Robust Dense Feature Matching. IEEE Conference on Computer Vision and Pattern Recogni- tion, 2024.
[13] Yujie Fu, Pengju Zhang, Bingxi Liu, Zheng Rong, and Yihong Wu. Learning to reduce scale differences for large-scale invariant image matching. IEEE Transactions on Circuits and Sys- tems for Video Technology, 13(5):1335-1344 2022.
[14] Kartik Garg, Sai Shubodh Puligilla, Shishir Kolathaya, Madhava Krishna, and Sourav Garg. Revisit anything: Visual place recognition via image segment retrieval. In European Confer- ence on Computer Vision (ECCV), 2024.

---

[15] Sourav Garg, Krishan Rana, Mehdi Hosseinzadeh, Lachlan Mares, Niko Suenderhauf, Feras Dayoub, and Ian Reid. Robohop: Segment-based topological map representation for open- world visual navigation. arXiv, 2023.
[16] Sourav Garg, Dustin Craggs, Vineeth Bhat, Lachlan Mares, Stefan Podgorski. Madhava Kr- ishna, Feras Dayoub, and lan Reid. Objectreact: Learning object-relative control for visual navigation. In Conference on Robot Learning. PMIR, 2025.
[17] Qiao Gu, Ali Kuwajerwala, Sacha Morin, Krishna Murthy Jatavallabhula, Bipasha Sen, Aditya Agarwal, Corban Rivera, William Paul, Kirsty Ellis, Rama Chellappa, et al. Conceptgraphs: Open-vocabulary 3d scene graphs for perception and planning. In 2024 IEEE International Conference on Robotics and Automation (ICRA), pages 5021-5028. IEEE, 2024.
(18) Nathan Hughes, Yun Chang, and Luca Carlone. Hydra: A real-time spatial perception system for 3d scene graph construction and optimization. arXiv preprint arXiv:2201. 133360 2022.
[19] Nikhil Keetha, Avneesh Mishra, Jay Karhade, Krishna Murthy Jatavallabhula, Sebastian Scherer, Madhava Krishna, and Sourav Garg. Anyloc: Towards universal visual place recog- nition. IEEE Robotics and Automation Letters, 9(2):1286-111 1293, 2023.
[20] Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander C Berg, Wan- Yen Lo, et al. Segment anything. In Proceedings of the IEEE/CVF international conference on computer vision, pages 4015-4026, 2023.
[21] Jacob Krantz, Stefan Lee, Jitendra Malik, Dhruv Batra, and Devendra Singh Chaplot. Instance- specific image goal navigation: Training embodied agents to find object instances. arXiv preprint arXiv:2211.15870 2022.
[22] Vincent Leroy, Yohann Cabon, and Jérôme Revaud. Grounding image matching in 3d with mast3r. In European Conference on Computer Vision, pages 71-91. Springer, 2024.
[23] Rongjie Li, Songyang Zhang, and Xuming He. Sgtr: End-to-end scene graph generation with transformer. In proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 19486-19496, 2022.
[24] Siyuan Li, Martin Danelljan, Henghui Ding, Thomas Huang, and Fisher Yu. Tracking every thing in the wild. In European conference on computer vision, pages 498-515. Springer, 2022.
[25] Siyuan Li, Lei Ke, Martin Danelljan, Luigi Piccinelli, Mattia Segu, Lue Van Gool, and Fisher Yu. Matching anything by segmenting anything. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 18963-18973, 2024.
[26] Rongguang Liang, Jie Yuan, Benfa Kuang, Qiang Liu, and Zhenyu Guo. Dig-slam: an accu- rate rgb-d slam based on instance segmentation and geometric clustering for dynamic indoor scenes. Measurement Science and Technology, 35(1):015401, 2023.
[27] Philipp Lindenberger, Paul-Edouard Sarlin, and Mare Pollefeys. LightGlue: Local Feature Matching at Light Speed. In ICCV, 2023.
[28] Shigemichi Matsuzaki, Takuma Sugino, Kazuhito Tanaka, Zijun Sha, Shintaro Nakaoka, Shin- taro Yoshizawa, and Kazuhiro Shintani. Clip-loc: Multi-modal landmark association for global localization in object-based maps. In Fortieth Inil Conference on Robotics and Automation, 2024. URL https:::s.. //tpx//avvvvv//////// org/ abs//402.aom 06092.
[29] Tim Meinhardt, Alexander Kirillov, Laura Laaxe and Christoph Feichtenhofer. Track- former: Multi-object tracking with transformers. In Proceedings of the IEEF/CVF conference on computer vision and pattern recognition, pages 8854, 2022.
[30] Maxime Oquab, Timothée Darcet, Théo Moutakanni, Huy Vo, Mare Szafraniec, Vasil Khali- dov, Pierre Fernandez, Daniel Haziza, Francisco Massa, Alaaeldin El-Nouby, et Dinov2: Learning robust visual features without supervision. arXiv preprint arXiv:2304.07199. 2023.
12

---

[31] Jiangmiao Pang, Linlu Qiu, Xia Li, Haofeng Chen, Qi Li. Trevor Darrell, and Fisher Yu. Quasi-dense similarity learning for multiple object tracking. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 164-173, 2021.
[32] Stefan Podgorski, Sourav Garg, Mehdi Hosseinzadeh, Lachlan Mares, Feras Dayoub, and Ian Reid. Tango: Traversablitty-awarr navigation with local metric control for topological goals. In 2025 IEEE International Conference on Robotics and Automation (ICRA). IEEE, 2025.
[33] Santhosh Kumar Ramakrishnan, Aaron Gokaslan, Erik Wijmans, Oleksandr Maksymets, Alexander Clegg, John M Turner, Eric Undersander, Wojeiech Galuba, Andrew Westhury, Angel X Chang, Manolis Savva, Yili Zhao, and Dhruy Batra. Habitat-matterpont 3d dataset (HM3d): 1000 large-scale 3d environments for embodied AI. In Thirty-fitth Conference on Neural Information Processing Systems Datasets and Benchmarks Track, 2021. URL https: . org/ abaaaba///10/ 08238
[34] Nikhila Ravi, Valentin Gabeur, Yuan-Ting Hu, Ronghang Hu, Chaitanya Ryali, Tengyu Ma, Haitham Khedr, Roman Rädle, Chloe Rolland, Laura Gustafson, al. Sam 2: Segment any- thing in images and videos. arXiv preprint arXiv:2408.0077 2024.
[35] Paul-Edouard Sarlin, Daniel DeTone, Tomasz Malisiewicz, and Andrew Rabinovich. Super- glue: Learning feature matching with graph neural networks. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 4938 4947, 2020.
[36] Xuclun Shen, Zhipeng Cai, Wei Yin, Matthias Müller, Zijun Li, Kaixuan Wang, Xiaozhi Chen, and Cheng Wang. Gim: Learning generalizable image matcher from internet videos. In The Twelfth International Conference on Learning Representations, 2024.
[37] Richard Sinkhorn. A relationship between arbitrary positive matrices and doubly stochastic matrices. The Annals of Mathematical Statistics, 35(2):876-879, 1964.
[38] Richard Sinkhorn and Paul Knopp. Concerning nonnegative matrices and doubly stochastic matrices. Pacific Journal of Mathematics, 21(2):343-344 1967.
[39]
Julian Straub, Thomas Whelan, Lingni Ma, Yufan Chen, Erik Wijmans, Simon Green, Jakob J. Engel. Raul Mur-Artal, Carl Ren, Shobhit Verma, Anton Clarkson, Mingfei Yan, Brian Budge, Yajie Yan, Xiaqing Pan, June Yon, Yuyang Zou, Kimberly Leon, Nigel Carter, Jesus Briales, Tyler Gillingham, Elias Mueggler, Luis Pesqueira, Manolis Savva, Dhruv Batra, Hauke M. Strasdat, Renzo De Nardi, Michael Goesele, Steven Lovegrove, and Richard Newcombe. The Replica dataset: A digital replica of indoor spaces. arXiv preprint 1wt.:v:9006007700 2019.
[40] Jiaming Sun, Zehong Shen, Yuang Wang, Hujun Bao, and Xiaowei Zhou. Loftr: Detector- free local feature matching with transformers. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 8922-8931, 2021.
[41] Zhongdao Wang, Liang Zheng, Yixuan Liu, Yali Li, and Shengjin Wang. Towards real-time multi-object tracking. In European conference on computer vision, pages 107-122. Springer, 2020.
[42] Philippe Weinzaepfel, Vincent Leroy, Thomas Lucas, Romain Brégier, Yohann Cabon, Vaib- hav Arora, Leonid Antsfeld, Boris Chidlovskii, Gabriela Csurka, and Jérôme Revaud. Croco: Self-supervised pre-training for 3d vision tasks by cross-viees view completion. Advances in Neural Information Processing Systems, 35:3502-355 -5166. 2022.
[43] Philippe Weinzaepfel, Thomas Lucas, Vincent Leroy, Yohann Cabon, Vaibhav Arora, Romain Brégier, Gabriela Csurka, Leonid Antsfeld, Boris Chidlovskii, and Jérôme Revaud. Croco v2: Improved cross-view completion pre- mee-trinnig for stereo matching and optical flow. In Proceedings of the IEEE/CVF International Conference Computer Vision, pages 17969- 17980, 2023.
[44] Binbin Xu, Wenbin Li, Dimos Tzoumanikas, Michael Bloesch, Andrew Davison, and Stefan Leutenegger. Mid-fusion: Octree-based object-level multi-instance dynamic slam. 2019 International Conference on Robotics and Automation (ICRA), pages 5231-5237. IEEE, 2019.
13

---

[45] Bin Yan, Yi Jiang, Peize Sun, Dong Wang, Zehuan Yuan, Ping Luo, and Huchuan Lu. Towards grand unification of object tracking. European conference on computer vision, pages 733- 751. Springer, 2022.
[46] Bin Yan, Yi Jiang, Jiannan Wu, Dong Wang, Ping Luo, Zehuan Yuan, and Huchuan Lu. Uni- versal instance perception as object discovery and retrieval. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 15325-15336, 2023.
[47] Chandan Yeshwanth, Yueh-Cheng Liu, Matthias Nießner, and Angela Dai. Scannet++: A high-fidelity dataset of 3d indoor scenes. In Proceedings of the International Conference on Computer Vision (ICCV), 2023.
[48] Chaoning Zhang, Dongshen Han, Yu Qiao, Jung Uk Kim, Sung-Ho Bac, Seungkyu Lee, and Choong Seon Hong. Faster segment anything: Towards lightweight sam for mobile applica- tions. arXiv preprint arXiv:2306. 14289, 2023.
[49] Yesheng Zhang and Xu Zhao. Dmesa: Densely matching everything by segmenting anything- arXiv preprint arXiv:2408.00077 2024.
[50] Shuaifeng Zhi, Tristan Laidlow, Stefan Leutenegger, and Andrew J. Davison. In-place scene labelling and understanding with implicit scene representation. In ICCV, 2021.
14

---

NeurIPS Paper Checklist
The checklist is designed encourage best practices for responsible machine learning research, addressing issues of reproducibility, transparency, research ethics, and societal impact. Do not re- move the checklist: The papers not including the checklist will be desk rejected. The checklist should follow the references and follow the (optional) supplemental material. The checklist does NOT count towards the page limit.
Please read the checklist guidelines carefully for information on how to answer these questions. For each question in the checklist:
You should answer [Yes]. [No]. or [NA]-
[NA] means either that the question Not Applicable for that particular paper or the relevant information is Not Available.
Please provide short (1 (1-2 sentence) justification right after your answer (even for NA).
The checklist answers are an integral part of your paper submission. They are visible to the reviewers, area chairs, senior area chairs, and ethics reviewers. You will be asked to also include it (after eventual revisions) with the final version of your paper, and its final version will be published with the paper.
The reviewers of your paper will be asked to the checklist ation. While "[Yes] generally preferable to "[No] is perfectly acceptable to answer "[No] provided proper justification is given (e.g., "error bars are not reported because it would be too computationally expensive" or "we were unable to find the license for the dataset we used"). In general, answering "[No]" "[NA] in binary way, we acknowledge that the true answer is often more nuanced, so please just use your best judgment and write a justification elaborate, All supporting evidence can appear either in the main paper or the supplemental material, provided in appendix. If you answer [Yes] to a question, the justification please point the section(s) where related material for the question can be found.
of the factors in their evalu-
grounds for rejection. While the questions are phrased
IMPORTANT, please:
Delete this instruction block, but keep the section heading "NeurIPS Paper Check- list",
Keep the checklist subsection headings, questions/answers and guidelines below.
Do not modify the questions and only the provided macros for your answers.
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: We re- p-rppspps 3d foundation model and show how it can be used for seg- ment matching. We show experiments showcasing the efficacy of our proposed method and down-stream applications justifying its use in real-world robotic settings.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
15

---

Answer: [Yes]
Justification:
The limitations are mentioned in the supplementary.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The au- thors should reflect on how these assumptions might be violated practice and what the implications would be.
The authors should reflect the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the ap- proach. For example, facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to- system might not be used reliably to provide closed captions for online lectures because fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to ad- dress problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't It acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proof's
Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?
Answer: [No]
Justification: Paper is purely algorithmic and application oriented and does not have any theory.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theo- rems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the of the paper should be comple- mented by formal proof's provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclu- sions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
16

---

Justification:
Training and dataset details are provided in Section 4.1 and Section 3.4.1
that allow for reproducibility.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is a dataset and/or model, the authors should describe the steps taken make their results reproducible verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or the contribution is a specific model and empirical evaluation, it may be necessary either make it possible for others replicate the model with the same dataset, or provide access to the model. In general. releasing code and data often one good way to accomplish this, but reproducibility detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of a large language model), relcasing of a model checkpoint, or other means that are appropriate to the research performed.
also be provided via
While NeurIPS does require releasing code, the conference does require all sub- missions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm.
(b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results way produce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which thors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path reproducing verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [No]
Justification:
We will release code and image pairs trained upon paper acceptance.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips cc/
for more details.
While we encourage the release of code and data, understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https: //nips cc/
wttp:/uwwmmiiiiooooooooooo.e for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
17

---

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g.. data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: The paper specifies dataset and splits used in Section 4.1 Implementation details are specified in Section 3.4.1
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropri- ate information about the statistical significance of the experiments?
Question:
Answer: [NA]
Justification:
No plots requiring error bars have been presented.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanicd by error bars, confi- dence intervals, statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK report I-sigma error bars, but should state it. The authors should prefer- ably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification: Mentioned in the supplementary.
18

---

Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of the individual experimental rums well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't 't make it into the paper).
9. Code ethies
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https: http:::ttttttuuuttt httpiipss.cpppulbe/tttteie
Answer:
[Yes]
Justification:
The code of ethics details are given in the supplementary.
Guidelines:
The answer NA means that the authors have not reviewed the NeurlPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there is a special consid- eration due to laws or regulations their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
These are mentioned in supplementary.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA No, they should explain why their work has societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact spe- cific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out, For example, it is legitimate to point that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, is not needed to point out that a generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitiga- tion strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Question: Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
19

---

Answer: [NA]
Justification:
Paper poses no such risks.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards allow for controlled use of the model, for example by re- quiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e-g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: Dataset details provided in Section 4.1 Their links are given in the sup- plementary. The licenses can be found by following the dataset links. The backbone models used are cited throughout the paper.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets released, the license, copyright information, and terms of use in the pack- age should be provided. For popular datasets, papppsswtthooee com/ datasets has curated licenses for some datasets. Their licensing guide help determine the li- cense of a dataset.
For existing datasets that are re-packaged, both the original license and the license of derived asset (if has changed) should be provided.
this information is not available online, the authors are encouraged to reach out to the asset's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documenta- tion provided alongside the assets?
Answer: [No]
Justification:
No new assets are introduced. Only existing datasets are utilized.
Guidelines:
The answer NA means that the paper does release new assets.
Researchers should communicate the details the dataset/codeemmodel. as part their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets applicable). You can either create anonymized URL or include an anonymized zip file.
20

---

14. Crowdsoureing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the pa- per include the full text of instructions given participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification:
We don't crowdsource data or use any human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Including this information in the supplemental material is fine, but if the main contri- bution of the paper involves human subjects, then much detail possible should be included the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, cura- tion, or other labor should paid at least the minimum wage the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects. and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: Not applicable our submission.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research conducted, IRB approval (or equiva- lent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Does the paper describe the usage of LLMs if it is important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, formatting purposes and does not impact the methodology. scientific rigorousness, or originality of the research, declaration is not required.
Question:
Answer: [NA]
Justification: LLMS were not used.
Guidelines:
The answer NA means that the method development in this research does not involve LLMs as any important, original, or non-standard components.
Please refer to our LLM policy (https://neurips.om http://errppscccc//oeeeeeeee//// for what should should not be described.
21


Reviews:
[{"review_id": "6fGtQzGxPj", "reviewer": "Reviewer_jith", "summary": "The paper introduces SegMASt3R, a segment matching framework that leverages a 3D foundation model (MASt3R) to establish wide-baseline segment correspondences. The key contribution lies in the segment-feature head, which aggregates patch-level features into segment-level descriptors, followed by a differentiable optimal transport layer for segment matching. The approach demonstrates strong performance improvements on ScanNet++ and Replica, supported by qualitative results on MapFree, and is further evaluated on downstream tasks such as 3D instance mapping and topological navigation.", "questions": "- The segmentation masks in this paper are from ground truth or SAM2. How does SegMASt3R perform when using imperfect or noisy segment masks?\n- Have you conducted ablations on fine-tuning the cross-attention decoder in MASt3R jointly with the segment-feature head? Could this further enhance segment matching?", "limitations": "Yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 3, "originality": 2, "strengths_and_weaknesses": "- Strengths\n\n  - Leverage 3D priors from pretrained model. SegMASt3R introduces MASt3R’s strong 3D inductive bias to address the gap in geometry-aware segment matching, enabling robust performance under wide viewpoint variations where 2D segment-based models may fail.\n\n  - Strong experimental performance. The proposed method significantly outperforms existing baselines across all pose bins (Tab. 1, Tab. 2), especially in the wide-baseline setting (135–180°) in the ScanNet++ and Replica datasets.\n\n  - Good downstream task application. The proposed method shows strong applicability to real-world tasks, including 3D instance mapping (Tab. 3) and object-based topological navigation (Tab. 4). These results highlight its ability to maintain object-level consistency across large viewpoint shifts.\n\n- Weaknesses\n\n    - Novelty concerns. While the paper leverage pretrained 3D foundation model as priors for segment matching is novel.  The core components are all adapted from prior work. \n        -  The corss-view patch features are from MASt3R prediction.\n        -  The core segment-feature head, is directly adpoted and initilized from MASt3R's descriptor head. \n        - The differentiable matching layer and sinknorn normalization closely follow the SuperGlue approach.\n\n     Hence, the contribution is largely engineering-driven.\n\n    - Writing and Presentation Clarity. \n        - Line 106: Patch-level features are of size (H/16, V/16, 768) → Should be (H/16, W/16, 768).\n        - Line 99: “Cross-view encoding Transformer”. Encoding transformer or decoder?\n        - Line 111: V1, V2 of shape (H/16, W/16, 24) → The patch-level embeddings V1, V2 should be (H/16, W/16, 768).\n        - The learnable logit \\alpha is not well defined and not used in the following equations. The description of $\\alpha$ in line 152 is even confusing.\n        - Lines 199–200: “A segment descriptor is the masked average of its feature map at a suitable resolution” — what is the suitable resolution? “Average of its feature map” — how is the average conducted?\n        - Table 1: Local Feature Matching (LFM). Are the results from local visual descriptor matching or from segment matching?\n        - Line 287: The ablation with CroCo encoder cannot demonstrate the effectiveness of the “explicit formulation of image matching”. To demonstrate it, it is better to use the DUSt3R backbone as it is not trained for the matching task.\n\n    - Generalization concerns to outdoor. The evaluation on the outdoor MapFree dataset is limited to qualitative results, without quantitative metrics or comparisons. Including quantitative analysis, additional outdoor datasets, or more in-the-wild examples would strengthen the claim of generalization capability.", "was_revised": true, "final_justification": "Thanks author for the detailed replies and additional experiments. My concerns are mostly solved, I'd like to raise my score to borderline accept.", "rebuttal": "**Typos:** We thank the reviewer for pointing out the typos.\n\n### Cross-view Encoding Transformer\n> Line 99: “Cross-view encoding Transformer”. Encoding transformer or decoder? \n\nThank you for pointing this out we will make the required change. It is indeed the decoder.\n\n### Description of Dustbin\n>$\\alpha$ Definition. Description of $\\alpha$ in L152 is confusing: \n\nThe learnable logit $\\alpha$ is a threshold below which any score from the score matrix will be classified as not-matched [34]. It is useful in situations when there are no common masks between an image pair, as it then maps such masks to the dustbin. We will clarify this in the paper.\n\n### Resolution and Averaging of Features\n> Lines 199–200: “A segment descriptor is the masked average of its feature map at a suitable resolution” - what is the suitable resolution? “Average of its feature map” - how is the average conducted?\n\nThis is a valid point since we haven't explicitly documented the formulation in our paper. For DINOv2 specifically, we work with patch level features at 1/14th the image resolution $(H/14, W/14, 1536)$. To assign segment level features, we choose an intermediate resolution (1/3rd image resolution, decided empirically), and upsample the patch-level features to $(H/3, W/3, D)$ via bilinear interpolation. Binary segemnt masks are downsampled to this same resolution $(H/3, W/3, M)$ via nearest-neighbour interpolation. Finally to obtain our mask level features, we perform a mask-average pooling operation, implemented as a mat-mul: $(M, H/3 \\times W/3) \\times (H/3 \\times W/3, D) = (M, D)$ to obtain a single D-dimensional vector per segment.\n\n\n### Local Feature Matching for Evaluation\n> Table 1: Local Feature Matching (LFM). Are the results from local visual descriptor matching or from segment matching?\n\n In this paper we believe we are the first to introduce wide-baseline segment matching. To develop a suitable baseline of methods, amongst others we chose to add LFM based baselines as they have been repurposed as segment matchers (for example in [15,31]) and there is a straightforward methodology to turn them into segment matchers. We will rectify this in the baselines subsection to clarify our motivation behind using them in the main paper.\n\n### DUSt3R evaluation and fine-tuning MASt3R\n> Line 287: The ablation with CroCo encoder cannot demonstrate the effectiveness of the “explicit formulation of image matching”. To demonstrate it, it is better to use the DUSt3R backbone as it is not trained for the matching task.\n> Have you conducted ablations on fine-tuning the cross-attention decoder in MASt3R jointly with the segment-feature head? Could this further enhance segment matching?\n\nIn Table 5 in the main paper and the Impact of the Feature Encoder (L281-L290) section, we were interested in performing an experiment to understand whether knowledge of the image pair and the subsequent cross-attention that features from both these images undergo is indeed a consequential part of these backbones for segment matching. As mentioned in the section, even though CroCo was not trained for the matching task and it was trained on much-less data than DINOv2, it still performs better than DINOv2 confirming our intuition. We explicitly did not choose DUSt3R because it is trained on a matching task and we imagined it would perform similar to MASt3R. We add the DUSt3R backbone comparison here for completion. As can be clearly seen SegMASt3R is much better than DUSt3R with the same training scheme as SegMASt3R on ScanNet++.  Additionally, we observe that fine-tuning the MASt3R decoder layers does not lead to further improvements in performance. Here we report by fine-tuning a total of 4 decoder layers (2 in each decoder).\n\n| Method                                             | 0–45 AUPRC | 0–45 R@1 | 0–45 R@5 | 45–90 AUPRC | 45–90 R@1 | 45–90 R@5 | 90–135 AUPRC | 90–135 R@1 | 90–135 R@5 | 135–180 AUPRC | 135–180 R@1 | 135–180 R@5 |\n|---------------------------------------------------|------------|----------|----------|-------------|-----------|-----------|----------------|-------------|-------------|------------------|---------------|---------------|\n| **SegMASt3R**                                     | **92.8**   | **93.6** | **98.0** | **91.1**    | **92.2**  | **97.6**  | **88.0**       | **89.5**    | **96.8**    | **83.6**         | **85.9**      | **95.9**      |\n| DUST3R-SegFeat                                    | 83.8       | 85.9     | 94.9     | 81.1        | 83.7      | 94.2      | 74.5           | 78.6        | 92.4        | 64.5             | 71.3          | 89.9          |\n| CroCo-SegFeat                                     | 73.4       | 78.8     | 92.3     | 64.0        | 73.1      | 90.6      | 50.7           | 64.6        | 87.5        | 38.5             | 56.6          | 84.1          |\n| SegMASt3R + Decoder Fine-Tuned                    | 90.3       | 91.5     | 96.7     | 88.6        | 90.1      | 96.3      | 84.9           | 87.1        | 95.2        | 78.8             | 82.5          | 93.7          |\n\n### Generalization / Outdoor Dataset\n>The evaluation on the outdoor MapFree dataset is limited to qualitative results, without quantitative metrics or comparisons.\n\nHere's a quantitative comparison on the outdoor MapFree (MF) dataset using SAM2 masks. SegMASt3R (MF) is the model which we have now trained on the MapFree dataset (in response to Reviewer **jith**, **mKs5** comments). SegMASt3R (SPP) is the same model as that used in the original manuscript. SegMASt3R (SPP, Dustbin MF) differs from the previous only in the choice of dustbin parameter $\\alpha$ which we borrowed from the SegMASt3R (MF) model to demonstrate that the indoor-to-outdoor generalization gap can be further reduced *easily* (e.g., this single parameter could simply be grid-searched on a small calibration set as an alternative to full training on MapFree). \n\n| **Method** | **Train Dataset** | **Eval Dataset** |  **Overall IoU** | IoU / **$0^\\circ-45^\\circ$** | IoU / **$45^\\circ-90^\\circ$** | IoU / **$90^\\circ-135^\\circ$** | IoU / **$135^\\circ-180^\\circ$** |\n| :--------: | :-------: | :--------: | :-------: | :-------: | :-------: | :-------: | :-------: |\n|DINOv2 | - | MapFree  | 84.4 | 85.4 | 85.2 | 83.5 | 83.8 |\n|MASt3R (Vanilla) | - | MapFree | 69.2 | 73.4 | 69.8 | 70.0 | 66.1 |\n|SegMASt3R (SPP)  | ScanNet++ | MapFree | 75.2 | 75.2 | 74.6 | 76.5 | 74.5 |\n|SegMASt3R (SPP, Dustbin MF)  | ScanNet++ | MapFree | 88.7 | 88.6 | 88.6 | 88.5 | 93.9 |\n|SegMASt3R (MF)  | MapFree | MapFree | 93.7 | 93.3 | 93.7 | 93.9 | 93.9 |\n\n**Ground Truth**: Most outdoor *real-world* datasets, including MapFree, don’t have ground truth instance masks, which is why we hadn't evaluated it quantitatively. The above evaluation uses a pseudo ground truth, obtained from SAM2 video propagation on the image sequence of a given scene. This works well in general because of the narrow baseline in adjacent video frames. We manually verified this ground truth on randomly selected scenes. All compared methods use the same pseudo ground truth.\n\n**Dataset/Training**: The official train split of the MapFree dataset comprises 460 unique scenes with 0.5 million images in total. To train SegMASt3R, we uniformly sampled 50 scenes to obtain 31K image pairs. Since the MapFree test split ground truth is not available publicly, we report results on their official val split using 7.8k image pairs from 13 scenes sampled uniformly from 65 total val scenes. Similar to our ScanNet++ evaluation, we use pose binning to present results across narrow and wide baselines.\n\n### Performance under noisy masks\n> How does SegMASt3R perform when using imperfect or noisy segment masks?\n\nOur method is robust to noisy masks, e.g., those obtained from SAM2/FastSAM. This is evident from the use of SAM2 masks in the above results on the MapFree dataset, and FastSAM masks in the navigation results on the HM3D dataset (Table 4). The following table presents results on the ScanNet dataset using FastSAM, which shows that the performance patterns are still similar to that observed in Table 1 with GT masks:\n\n| Method            | 0-45 AUPRC | 0-45 R@1 | 0-45 R@5 | 45-90 AUPRC | 45-90 R@1 | 45-90 R@5 | 90-135 AUPRC | 90-135 R@1 | 90-135 R@5 | 135-180 AUPRC | 135-180 R@1 | 135-180 R@5 | Overall AUPRC | Overall R@1 | Overall R@5 |\n|-------------------|------------|----------|----------|-------------|-----------|-----------|---------------|-------------|-------------|----------------|--------------|--------------|----------------|-------------|-------------|\n| SAM2 (Video Prop) | 0.6389     | **0.7516**   | **0.7681**   | 0.5029      | **0.5884**    | 0.615     | 0.3142        | 0.3823      | 0.4175      | 0.2019         | 0.2593       | 0.2875       | 0.4147         | 0.4956      | 0.5222      |\n| MASt3R            | 0.3437     | 0.4014   | 0.5203   | 0.2862      | 0.3443    | 0.4575    | 0.2472        | 0.3067      | 0.4242      | 0.2370         | 0.2974       | 0.4220       | 0.2786         | 0.3375      | 0.4560      |\n| DinoV2-g14-AVG    | 0.3920     | 0.4563   | 0.6109   | 0.2596      | 0.3296    | 0.4883    | 0.1991        | 0.2701      | 0.4173      | 0.1737         | 0.2455       | 0.3784       | 0.2562         | 0.3254      | 0.4738      |\n| SegMASt3R AMG (Automatic Mask Generator)    | **0.7089**     | 0.7443   | 0.7694   | **0.6825**      | 0.7205    | **0.7455**    | **0.6444**        | **0.6883**      | **0.7178**      | **0.6032**         | **0.6548**       | **0.6827**       | **0.6598**         | **0.7020**      | **0.7289**      |\n\n### Novelty\n> The core components are all adapted from prior work\n\nPlease refer to our response to Reviewer **mKs5**, titled Novelty."}, {"review_id": "DdblO8RlQu", "reviewer": "Reviewer_mKs5", "summary": "SegMASt3R is a model for segment matching, an under-explored problem. That is, the paper introduces the wide baseline segment matching problem between image pairs. An approach based on frozen MASt3R feature extractor, given objects masks, and the differentiable optimal transport framework of SuperGlue is provided. In addition, a benchmark for wide segment matching is proposed with\nquantitative evaluations on indoor datasets ScanNet++ and Replica and show visual results on MapFree Outdoor dataset. The proposed approach improves downstream tasks such as 3D instance mapping and object level topological navigation.", "questions": "Q1. Why downsample the masks for the DINOv2 and MASt3R baselines, rather than interpolating their features to match the\nmask resolution?\nQ2. The masks extraction is a crucial step in creating high quality segments. ScanNet++ and Replica dataset provide masks, however, in cases where masks are not provided annotating 100 masks (line\n123) for every input image seems unpractical. When ground truth\nmasks are not available, how do the authors propose to obtain it? What\nis the expected running time for this step?\nQ3. Have the authors considered comparing to \"Gaussian Grouping: Segment and Edit Anything in 3D Scenes\"?\n\n-Q4: Statistical Error Bars: Question 7 checklist: I don’t think it is correct – there is no\nstatistical error bars in any of the results (at least not mentioned) also in 3.4.1 the\ntime is not with standard deviation.", "limitations": "The paper overall limitation is that it requires a 3D ground truth annotation, which are quite\nrare. Furthermore, as a direct result of that, they cannot generalize well to other scenes and\ndatasets. Those issues were addressed by the authors in the Sup. Material.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 3, "originality": 2, "strengths_and_weaknesses": "Strengths:\n- An approach for the under explored problem of wide baseline segment matching is offered, showing improved results over existing segment matching methods on indoor datasets.\n- Using SegMASt3R improves performance on downstream tasks such as 3D instance mapping and Object level topological navigation\n-  The paper is, overall, well written, the math is clearly explained, and the description of the \nfigures is clear.\n\nWeaknesses:\n- All quantitative results are reported exclusively on indoor datasets.\n1. ScanNet++ , an indoor dataset, which the proposed method is\ntrained on although the competitor&#39;s methods aren&#39;t. 2. Replica, an\nindoor synthetic dataset.\n- The comparison with DINOv2 and MAStr3 methods in table 1 appears\nto favor the proposed approach. The proposed method first upsample\nthe features to the image resolution before the mask are applied (lines\n114-115), however, for extracting segment match from dense features\nencoders the authors downsample the masks to the features&#39;\ndegraded resolution (lines 199-202).\n-  The proposed approach mainly combines existing components without\nintroducing substantial methodological advances.\n\n\nMinor weaknesses: \n- Inaccuracy in the abstract. Lines 9-12 &quot;Extensive experiments show\nthat our approach outperforms state-of-the-art methods, including\nthe SAM2video propagator and local feature matching methods, by up\nto 30% on the AUPRC metric, on ScanNet++ and Replica datasets.&quot;;\nSince many of those methods were not inherently built for the task of\nsegment matching some required modifications by the authors to\nproduce segment correspondences. Therefore, it is not accurate to\nname those methods SOTA for this task or to claim the proposed\nmethod outperform SOTA methods.\n- The necessity for the learnable logit in the learnable dustbin is unclear.", "was_revised": true, "final_justification": "I thanks the authors regarding the clarification about the 3d annotation. While I'm still not convinced about the novelty, in light of this clarification I am raising the score from Borderline Reject to Borderline Accept.", "rebuttal": "### DinoV2 and MASt3R Feature Resolution\n> Q1. Why downsample the masks for the DINOv2 and MASt3R baselines, rather than interpolating their features to match the mask resolution?\n\nWe only downsample the masks for DINOv2, not for MASt3R (we will rectify this in L202). For DINOv2, the following table compares different feature/mask resolutions. It can be observed that performance does not change significantly. We had originally tested this on a subset and chose a lower resolution ([112, 171] - one-third of the original resolution) for efficiency.\n\n| Resolution ($[H,W]$)                                                       | 0–45 AUPRC | 0–45 R@1 | 0–45 R@5 | 45–90 AUPRC | 45–90 R@1 | 45–90 R@5 | 90–135 AUPRC | 90–135 R@1 | 90–135 R@5 | 135–180 AUPRC | 135–180 R@1 | 135–180 R@5 |\n|--------------------------------------------------------------|------------|----------|----------|-------------|-----------|-----------|----------------|-------------|-------------|------------------|---------------|---------------|\n| $[112,171]$       | **0.5794** | **0.6670** | **0.8742** | **0.4300**  | 0.5586    | 0.8319    | **0.3347**     | **0.4804**  | **0.7797**  | **0.3239**       | **0.4602**    | **0.7563**    |\n| $[168,256]$                 |   0.5784   | 0.6663    | 0.8740    |  0.4297     | 0.5584 | 0.8315 |  0.3323 |  0.4801 | 0.7794 | 0.3234 | 0.4597 | 0.7498 |\n| $[336,512]$                 | 0.5788     | 0.6657   | 0.8739   | 0.4297      | **0.5589**| **0.8330**| 0.3312         | 0.4803      | 0.7797      | 0.3203           | 0.4600        | 0.7563        |\n\n### Number of Masks, Non-GT Masks For Training, Running Time for Mask Generation At Inference\n\n> Q2. The masks extraction is a crucial step in creating high quality segments. ScanNet++ and Replica dataset provide masks, however, in cases where masks are not provided annotating 100 masks (line 123) for every input image seems unpractical. When ground truth masks are not available, how do the authors propose to obtain it? What is the expected running time for this step? \n\nWe respond to this in three parts: a) the choice of M=100, b) the need for ground truth (GT) masks during training, and c) expected runtime of non-GT masks during inference.\n\n### Number of Masks\na) **Number of Masks / M=100 masks**: This is simply an upper bound for convenience in batch processing. Most training/test pairs do not have 100 masks, regardless of using GT or SAM/FastSAM. The actual number of masks per image is between 20 to 30 when training using GT. For the missing masks, we just pad them with 0s. We will add this important detail to the paper (in Line 117). Additionally, during inference time, the number of masks can be set arbitrarily regardless of the set M during training.\n\n### Non-GT Masks For Training\nb) **Need for GT masks during *training***: It is indeed possible to train SegMASt3R with *weak supervision* based on mask annotations obtained from SAM2 video propagator. To show this, we have now trained a model on the MapFree dataset, which does not have GT instance masks but has image sequences per scene. We use this with SAM2's video propagator (which works well due to narrow baseline between adjacent frames) to generate segment correspondences for different image pairs within the image sequence. This pseudo ground truth is used for both training and evaluation. For results on MapFree which involves noisy (non-GT masks) please refer to the **Generalization / Outdoor Datasets** section of reviewer **jith**'s.\n\n### Running Time for Mask Generation At Inference\nc) Non-GT masks during *inference*:  When GT masks are unavailable, we can use SAM /FastSAM to obtain them. FastSAM works in real-time, so mask extraction should not be a bottleneck either in our training or inference and have real-time applicability.\n\n| **Method** |  **Overall IoU** |  **$0^\\circ-45^\\circ$** | **$45^\\circ-90^\\circ$** | **$90^\\circ-135^\\circ$** | **$135^\\circ-180^\\circ$** |\n| :--------: | :-------: | :-------: | :-------: | :-------: | :-------: |\n|DinoV2-SegFeat | 0.844 | 0.8543 | 0.8519 | 0.8351 | 0.8385 |\n|MASt3R (Vanilla) |  0.692 | 0.7336 | 0.6976 | 0.7001 | 0.6611 |\n|SegMASt3R (MF)  |  **0.937** | **0.9334** | **0.9374** | **0.9387** | **0.9398** |\n\n\n### Gaussian Grouping\n> Q3. Have the authors considered comparing to \"Gaussian Grouping: Segment and Edit Anything in 3D Scenes\"?\n\nWe thank the reviewer for pointing out Gaussian Grouping. We will cite the paper as it is related to this research, but can’t do a fair comparison, as our method is based on only *pairwise* image based segment matching. Gaussian Grouping on the other hand uses Gaussian splatting which requires *multiple overlapping views* of a scene to generate accurate segmentations.\n\n### Statistical Error Bars\n> Q4: Statistical Error Bars: Question 7 checklist: I don’t think it is correct – there is no statistical error bars in any of the results (at least not mentioned) also in 3.4.1 the time is not with standard deviation.\n\nThis was an oversight on our part. We meant to write no statistical error bars. \n\n\n### Generalization to outdoor datasets\n> All quantitative results are reported exclusively on indoor datasets.\n\nPlease refer to the *Generalization / Outdoor Dataset* section in reviewer **jith**'s rebuttal.\n\n>  comparison with DINOv2 and MAStr3 methods\n\nThis is covered above in Q1.\n\n### Novelty\n> The proposed approach mainly combines existing components without introducing substantial methodological advances.\n\nWhile some individual components used in our method are known within the community, we demonstrate that even these simple elements when integrated algorithmically can achieve outstanding results across a range of vision tasks, including wide-baseline segment matching, robotic navigation and instance segmentation. Crucially, we integrate these components in a novel and non-trivial configuration, which is key to the performance gains observed. Our approach also serves as an effective exploration of segment-based representations, which, despite the simplicity of our methodology, is shown to enhance performance in multiple robotic-vision settings showcasing the usefulness of segments as a useful intermediate-level scene representation. To our knowledge, this still remains under-explored. Furthermore, the introduction of a segment matching task and wide-baseline evaluation adds an additional layer of novelty, pushing the boundaries of current approaches in this domain.\n\n### Abstract\n> Inaccuracy in the abstract\n\nWe thank the reviewer for recognizing that we are introducing a new task (wide-baseline segment matching) and its evaluation methodology. To develop reasonable baselines, we chose to use SAM2 video propagator which is not inherently trained for this task, because it is the industry and academia standard for segment matching across multiple frames and is also trained on a large amount of data (more than MASt3R). \nWe chose LFM methods because they have been repurposed as segment matchers (for example in [15,31]) and there is a straightforward methodology to turn them into segment matchers. It was our oversight that we only mentioned the baselines but did not explain their motivations, and will add these motivations to the main paper.\n\n\n### Learnable Logit Interpretation and Learnability\n> The necessity for the learnable logit in the learnable dustbin is unclear.\n\nThe logit is basically a threshold below which any score from the score matrix will be classified as not-matched. It is useful in situations when there are masks not common to both images and allows for such masks to be mapped to the dustbin. While we could also have let it have been a tuneable hyper-parameter, we chose to make it learnable to allow for a more natural discovery of its appropriate value. We will clarify this in the paper.\n\n### Limitations\n> The paper overall limitation is that it requires a 3D ground truth annotation, which are quite rare. Furthermore, as a direct result of that, they cannot generalize well to other scenes and datasets. Those issues were addressed by the authors in the Sup. Material.\n\nWe have now presented results on the MapFree dataset where we use SAM2 video propagator instead of ground truth annotations for training/evaluation. While this strategy may not apply for all types of datasets, it shows that our method does not *strictly* depend on ground truth annotations."}, {"review_id": "Xs9HYJl6Bz", "reviewer": "Reviewer_vtN9", "summary": "This paper addresses the challenging problem of segment-level image correspondence under extreme viewpoint variations, leveraging the spatial reasoning capabilities of 3D foundation models. The proposed method, SegMASt3R, utilizes a frozen MASt3R backbone to extract geometry-aware patch-level features from an image pair. These features are upsampled and aggregated into segment-level descriptors via a lightweight MLP, using segmentation masks from external models (e.g., SAM2) or ground truth. Segment correspondences are then established through a differentiable optimal transport layer based on the Sinkhorn algorithm. The model is trained end-to-end on ScanNet++ using a SuperGlue-inspired contrastive loss that accounts for both matched and unmatched segments. SegMASt3R is evaluated on ScanNet++ and Replica, showing strong generalization across indoor scenes, and is qualitatively assessed on the challenging MapFree outdoor dataset. It is further validated on downstream tasks such as 3D instance mapping and object-goal navigation, where it delivers substantial improvements over baselines. The method achieves up to a 30% gain in AUPRC over both segment-matching and local-feature-based methods.", "questions": "- The rebuttal could elaborate on the robustness to segmentation noise\n- The rebuttal could elaborate on the dependence of performance on the number of segments\n- The final version of the paper could include already in the main paper an abridged version of the discussion on limitations and societal impact", "limitations": "Yes", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 4, "significance": 4, "originality": 2, "strengths_and_weaknesses": "The main strength of this paper is the strong performance on a widely relevant computer vision task, and strong results (>80) even for extreme viewpoint changes. This work undoubtedly has applications in practical tasks related to robotic navigation and instance mapping, as also show in the ablation and supplementary experiments. And the pipeline used is relatively simple, while very effective. Finally, the paper is very well structured and written, with a strong and exhaustive experimental section and an excellent methodology section, which discusses in clear details all aspects of the pipeline. \n\nAt the same time, the simplicity of the proposed architecture leads to limited novelty: it is a straightforward adaptation of MASt3R with an added head comprising an MLP and Sinkhorn-based matching. Additionally, as noted by the paper's limitation section (included in the supplementary material, not in the main paper), the experiments do not cover fully generalisation to outdoor settings, and training requires ground truth 3D instance segmentation, which is not always easy to obtain. An additional limitation not covered by the paper is how robust the method is to noisy segmentation masks. Finally, while the matching strategy is efficient enough (0.579s forward pass at inference) it is not discuss how scalable the approach is; the paper mentions using 100 segments per image to be 'sufficient to cover the entire image while keeping computation affordable', but this leaves space for a discussion on how performance would change adding more segments.\n\nDespite the weaknesses, this paper still provides a very strong contribution to the computer vision community, and the weaknesses are mitigated by the fact that in most practical applications 100 segments is a reasonable number (e.g., for SAM), and the strong performances with a simple framework are a positive.", "was_revised": true, "final_justification": "The authors have addressed all of my questions satisfactorily. After reviewing the other reports, I believe they engaged constructively in the review process and responded thoroughly to all concerns. I therefore confirm my recommendation to accept this paper.", "rebuttal": "### Novelty\n> it is a straightforward adaptation of MASt3R with an added head comprising an MLP and Sinkhorn-based matching. \n\nWhile some individual components used in our method are known within the community, we demonstrate that even these simple elements when integrated algorithmically can achieve outstanding results across a range of vision tasks, including wide-baseline segment matching, robotic navigation and instance segmentation. Crucially, we integrate these components in a novel and non-trivial configuration, which is key to the performance gains observed. Our approach also serves as an effective exploration of segment-based representations, which, despite the simplicity of our methodology, is shown to enhance performance in multiple robotic-vision settings showcasing the usefulness of segments as a useful intermediate-level scene representation. To our knowledge, this still remains under-explored. Furthermore, the introduction of a segment matching task and wide-baseline evaluation adds an additional layer of novelty, pushing the boundaries of current approaches in this domain.\n\n### 3D Instance Segmentation\n> Training requires ground truth 3D instance segmentation, which is not always easy to obtain:\n\nOur method relies more on 2D instance masks than 3D. The benefit with 3D instance masks is they are more boundary aware. 2D instance masks can be easily generated to both train the network as well as use the network to evaluate on novel scenes like shown in the quantitative MapFree results of **jith's** rebuttal. Our method, even when trained only on indoor datasets generalizes better than compared to baselines on the outdoor MapFree.\n\n\n\n### Noisy Masks\n> An additional limitation not covered by the paper is how robust the method is to noisy segmentation masks. \n\nIn reviewer **jith**’s rebuttal we show results for the MapFree dataset in the Generalization / Outdoor Dataset section which covers both outdoors and noisy masks as the masks generated are via SAM2. Additionally, we show results on noisy masks as generated by FastSAM in  **jith**'s  in the *Performance under noisy masks* section. The table shows that the performance patterns are still similar to that observed in Table 1 with GT masks.\n\n### Dependence of performance on the number of segments: \n> the paper mentions using 100 segments per image to be 'sufficient to cover the entire image while keeping computation affordable, but this leaves space for a discussion on how performance would change adding more segments.\n\nOur method is not reliant on a particular number of segments. The M (set to 100) parameter in L117 is an upper bound on the segments in the scene. It allows for more efficient batch processing. For the missing masks, we just pad them with 0s. We will add this important detail to the paper (in Line 117).  Additionally, during inference time, the number of masks can be set arbitrarily regardless of the set M during training. We presume beyond a certain point having really small segments would indeed degrade performance, but most segmenters (like SAM2, FastSAM) yield less than a 100 segments. Additionally, in the main paper, and in the rebuttal, we have explored several results with different segmentation algorithms. FastSAM is used in the robotic navigation experiment (Table 4 on the HM3D dataset), SAM2 AMG (Automatic Mask Generator) results are shown for the *Performance under noisy masks* section for reviewer **jith**. The various segmenters used also naturally cover a varying number of segments, showing that indeed our methods are robust to this quantity."}, {"review_id": "HQKc8cETXB", "reviewer": "Reviewer_nAMx", "summary": "This paper presents SegMASt3R, a novel framework that adapts the 3D foundation model MASt3R for wide-baseline segment matching. It introduces a lightweight segment-feature head to generate segment-level descriptors and applies a differentiable optimal transport solver with a learnable dustbin. Extensive experiments demonstrate that SegMASt3R significantly outperforms both local feature matchers and 2D segment-based methods. Its effectiveness is further validated in downstream tasks such as 3D instance mapping and topological navigation.", "questions": "My concerns mainly lie in the evaluation settings. Please refer to the weaknesses.", "limitations": "Yes, in the supplementary material.", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 4, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n1. Effective model design. The method builds upon a strong backbone (MASt3R). The design of the segment-level feature head and the differentiable matching mechanism with a learnable dustbin is reasonable.\n2. Strong performance and diverse downstream applications. The application of SegMASt3R in topological navigation is interesting and demonstrates its practical value.\n3. Clear writing. The paper is well-written and easy to follow.\n\nWeaknesses:\n1. Limited ablation:\n\nThe method uses a learnable dustbin to handle unmatched segments. It would strengthen the paper to include an ablation study evaluating its contribution.\n\n2. Evaluation settings.\n\n(1) It is unclear whether the baseline methods are re-trained or evaluated using their publicly released checkpoints. This is especially critical in wide-baseline settings, where fine-tuning or data augmentation strategies may significantly affect performance.\n\n(2) According to Supplementary Table 2, SegMASt3R is evaluated only using ground-truth masks, which are often unavailable during inference. It would be beneficial to also report results using masks from MobileSAM.\n\n3. Minor typo.\n\nIn Line 111, the channel dimension of V1, V2 should be 768, instead of 24.", "was_revised": true, "final_justification": "The authors have addressed my concerns. I think the proposed SegMASt3R provides an effective solution for segment matching, and its potential has been demonstrated through downstream applications. Therefore, I maintain my rating of accept.", "rebuttal": "### Limited ablation: \n>The method uses a learnable dustbin to handle unmatched segments. It would strengthen the paper to include an ablation study evaluating its contribution.\n\nThe table below shows the effect of removing the dustbin parameter from the SegMASt3R pipeline. The performance sharply degrades across all metrics without the dustbin parameter.\n\n### Ablation Study on Learnable Dustbin\n\n| Method                                             | 0–45 AUPRC | 0–45 R@1 | 0–45 R@5 | 45–90 AUPRC | 45–90 R@1 | 45–90 R@5 | 90–135 AUPRC | 90–135 R@1 | 90–135 R@5 | 135–180 AUPRC | 135–180 R@1 | 135–180 R@5 |\n|---------------------------------------------------|------------|----------|----------|-------------|-----------|-----------|----------------|-------------|-------------|------------------|---------------|---------------|\n| SegMASt3R (No Dustbin) | 65.0       | 73.8     | 95.9     | 61.8        | 71.5      | 95.4      | 58.7           | 69.1        | 94.6        | 55.1             | 66.4          | 93.6          |\n| **SegMASt3R**                                     | **92.8**   | **93.6** | **98.0** | **91.1**    | **92.2**  | **97.6**  | **88.0**       | **89.5**    | **96.8**    | **83.6**         | **85.9**      | **95.9**      |\n\n### MobileSAM Mask Evaluation\n> According to Supplementary Table 2, SegMASt3R is evaluated only using ground-truth masks, which are often unavailable during inference. It would be beneficial to also report results using masks from MobileSAM.\n\nDuring inference, we can rely on segment generators like MobileSAM. In Reviewer **jith**'s rebuttal, we show results on  evaluation using noisy masks as obtained from FastSAM in the \"*Performance under noisy masks*\" section. Additionally, in our robotic navigation task (L253-290) we use masks generated from FastSAM on the HM3D dataset (Table 4). This clearly shows that our method is not reliant on ground truth masks. We will mention this more explicitly in the paper.\n\n\n### Were Pre-trained Checkpoints Used\n>  It is unclear whether the baseline methods are re-trained or evaluated using their publicly released checkpoints. This is especially critical in wide-baseline settings, where fine-tuning or data augmentation strategies may significantly affect performance.\n\nYes, we use pre-trained checkpoints. This is indeed a missing detail on our part and we will add the pre-trained checkpoints used for evaluation in the paper. For DINOv2, we use the DINOv2 ViT-G pre-trained checkpoint. For CroCo, we use the ViT-L pre-trained checkpoint provided in their repo. For other models (SAM2, MASt3R, etc.), we use the official checkpoints available in their respective repos.\n\n\n### Typos\n> In Line 111, the channel dimension of V1, V2 should be 768, instead of 24.\n\n Thank you for pointing this out. We will fix the typo."}]

Author rebuttal:

Meta-review: This paper introduces SegMASt3R, a novel method for wide-baseline segment matching that effectively leverages MASt3R to establish robust correspondences between semantic segments across extreme viewpoint changes. The paper's key strength is its compelling scientific insight: while a 2D model (DINOv2) outperforms a geometric one (MASt3R) off-the-shelf for this task, fine-tuning the geometric backbone with a simple segment-matching head yields state-of-the-art performance, proving that explicit geometric reasoning is essential. The authors' rebuttal was exemplary, thoroughly addressing all reviewer concerns with substantial new quantitative evidence on outdoor datasets and under noisy mask conditions, significantly elevating the paper's impact and demonstrating strong generalization and robustness. The additional ablations, including the critical role of the learnable dustbin, further strengthened the paper. All reviewers acknowledged that their concerns were fully satisfied, confirming the paper's technical soundness, clarity, and substantial contribution to the field. The combination of defining a new task, delivering a counter-intuitive finding. Therefore, the AC recommends acceptance.
Decision: Accept (spotlight)