Title: SegMASt3R: Geometry Grounded Segment Matching
Venue: NeurIPS 2025
Accepted: yes
Keywords: Segment Matching

3D Instance Mapping

Object-Level Topological Navigation
Abstract: Segment matching is an important intermediate task in computer vision that establishes correspondences between semantically or geometrically coherent regions across images. Unlike keypoint matching, which focuses on localized features, segment matching captures structured regions, offering greater robustness to occlusions, lighting variations, and viewpoint changes. In this paper, we leverage the spatial understanding of 3D foundation models to tackle wide-baseline segment matching, a challenging setting involving extreme viewpoint shifts. We propose an architecture that uses the inductive bias of these 3D foundation models to match segments across image pairs with up to $180^\circ$ rotation. Extensive experiments show that our approach outperforms state-of-the-art methods, including the SAM2 video propagator and local feature matching methods, by up to 30\% on the AUPRC metric, on ScanNet++ and Replica datasets. We further demonstrate benefits of the proposed model on relevant downstream tasks, including 3D instance mapping and object-relative navigation.

SegMASt3R: Geometry Grounded Segment Matching

Abstract

Segment matching is an important intermediate task in computer vision that establishes correspondences between semantically or geometrically coherent regions across images. Unlike keypoint matching, which focuses on localized features, segment matching captures structured regions, offering greater robustness to occlusions, lighting variations, and viewpoint changes. In this paper, we leverage the spatial understanding of 3D foundation models to tackle wide-baseline segment matching, a challenging setting involving extreme viewpoint shifts. We propose an architecture that uses the inductive bias of these 3D foundation models to match segments across image pairs with up to 180 ◦ rotation. Extensive experiments show that our approach outperforms state-of-the-art methods, including the SAM2 video propagator and local feature matching methods, by up to 30% on the AUPRC metric, on ScanNet++ and Replica datasets. We further demonstrate benefits of the proposed model on relevant downstream tasks, including 3D instance mapping and object-relative navigation.

1 Introduction

Segment matching establishes correspondences between coherent regions-objects, parts, or semantic segments-across images. It underpins video object tracking [17], scene-graph construction [18, 23], robot navigation [16, 15, 32], and instance-level SLAM [26, 44]. Because it matches extended structures rather than sparse, texture-sensitive points, it is more robust to noise, occlusion, and appearance change. Mapping structured regions instead of isolated pixels also boosts interpretability and allows geometric or semantic priors to be injected, enabling higher-level reasoning about scene content.

Segment matching degrades sharply under wide-baseline conditions, where images of the same scene are taken from widely separated viewpoints that introduce severe perspective, scale, and up to 180 ◦ rotation changes [34]. Such cases arise in long-term video correspondence and robotic navigation, and demand models that reason over global 3D structure and enforce geometric consistency. Current approaches that depend on features from pre-trained encoders like DINOv2 [30] or ViT [10] often mismatch repetitive patterns or fail to link drastically different views of the same object.

In this paper, we propose to leverage the strong spatial inductive bias of a 3D foundation model (namely MASt3R [22]) to solve the problem of wide-baseline segment matching. 3D Foundation Models (3DFMs) are large-scale vision models trained to capture spatial and structural properties of scenes, such as depth, shape, and pose. Unlike appearance-focused models, 3DFMs learn geometryaware representations using 2D and 3D supervision, enabling strong generalization across tasks

like reconstruction and pose estimation. Their inductive bias toward spatial reasoning makes them well-suited for applications requiring geometric consistency such as our problem of wide-baseline segment matching.

We adapt MASt3R for segment matching by appending a lightweight segment-feature head that transforms patch-level embeddings into segment-level descriptors. Given an image pair, these descriptors are matched to establish segment correspondences. The head is trained with a contrastive objective patterned after SuperGlue [35]. Experiments show that our approach surpasses strong baselines, including SAM2's video propagator which is trained on far larger datasets and state-ofthe-art local feature matching methods. Finally, we demonstrate its practical utility in two downstream tasks: object-relative navigation and instance-level mapping.

Contributions Our key contributions are summarized below:

We introduce a simple but effective approach for learning segment-aligned features by leveraging strong priors from a 3D foundation model (3DFM) MASt3R. A differentiable segment matching layer is employed to align features across views, while a segment-feature head transforms dense pixel-level representations into robust segment-level descriptors.

To address the under-explored problem of wide-baseline segment matching, we construct a comprehensive benchmark comprising both direct segment association methods and those based on local feature matching. Our method demonstrates significant improvements over all baselines on challenging wide-baseline image pairs.

We validate the practical utility of our approach by applying it to the downstream applications such as 3D instance mapping and object-relative topological navigation. Our method outperforms competitors by significant margins showcasing the efficacy of our design choices.

2 Related Work

Segment Matching and Segmentation Foundation Models. Robust segment-level association has emerged as a crucial intermediate step for high-level vision tasks such as scene graph construction, long-term object tracking, and multi-view instance association. While related sub-problems like video instance segmentation and object tracking have been studied extensively in recent works [46, 45, 41, 11, 31, 29, 24, 6], the broader challenge of matching segments across arbitrary viewpoints, modalities, and time remains comparatively under-explored. Large-scale class-agnostic segmentation models have begun to close this gap. The Segment Anything Model (SAM) [20] and its successor SAM2 [34] deliver high-quality masks and include a built-in propagator for associating masks across video frames. However, this propagation module is optimized for short temporal windows and does not explicitly enforce geometric consistency under wide baselines or substantial appearance changes.

Learning to Match Segments and Overlap Prediction. Some recent methods address segment matching more directly. MASA [25] augments SAM's rich object proposals with synthetic geometric transformations to learn instance correspondences, and DMESA [49] extends these ideas to dense matching with improved efficiency. Despite such progress, these approaches are still limited by 2D supervision. An alternative line of work predicts the degree of visual overlap between images [7, 13, 2]. By estimating shared content, these methods implicitly learn region correspondences, yet they also remain confined to 2D training signals.

Local Feature Matching. Sparse [9, 35, 40, 27, 36] and dense [12, 4] local feature matchers propagate pixel-level correspondences that can, in principle, transfer segment labels between views [15, 16]. Nonetheless, like the previous categories, they are trained exclusively on image data and struggle with extreme viewpoint changes.

Across segment matching, overlap prediction, and feature matching, reliance on purely 2D supervision leaves existing techniques brittle under wide-baseline conditions. Our method addresses this limitation by fine-tuning the 3D foundation model MASt3R [22], whose strong geometric priors enable reliable segment correspondence even when image pairs differ by nearly 180 ◦ in viewpoint.

Image Pair

Shared Image

Encoder

Frozen

Transformer

Decoder

Cross-Attention

Transformer

Decoder

Patch Level

Features

Segment

Feature Head

Masks

Segment

Feature Head

Segment

Features

Dustbin

Dustbin

Score Matrix

Sinkhorn

Differentiable Optimal Transport

Row-wise argmax

Soft Correspondences

Segment

Matching

3 Method

Figure 1 provides an overview of our method. It builds upon the MASt3R [22] architecture by introducing a Feat2Seg Adapter that maps the patch-level features output by the MASt3R decoder to get segment-level features, which are subsequently matched via a differentiable optimal transport and a row-wise argmax yielding the final segment-level correspondences.

3.1 MASt3R Preliminaries

MASt3R is a 3D foundation model pre-trained on a diverse collection of 3D-vision datasets commonly used for tasks such as metric depth estimation and camera-pose prediction. Given a pair of images, it produces dense 3D point maps for each image and identifies correspondences between them. Thanks to training on data with wide-baseline pairs, MASt3R generalizes well to unseen image pairs [22] and consistently outperforms alternative 3D matching methods-such as MicKey [3], which relies on a DINOv2 backbone [3].

Architecture: We summarize here the portions of the architecture we utilize in our pipeline. The two images I 1 and I 2 are processed in a Siamese manner by a weight-sharing ViT encoder [10] E , resulting in token sets H 1 , H 2 , i.e.

Cross-view transformer decoder: Next, a pair of CroCo [42, 43] style intertwined transformer decoders jointly refines the two feature sets. By alternating self- and cross-attention, the decoders exchange information between inputs to capture both the relative viewpoints and the global 3-D structure of the scene. We show via an ablation study, these cross-view aware decoders provide a significant boost to the segment matching in Section 5

The resulting geometry-aware representations are denoted V 1 and V 2 :

Assuming, H,W denote the height and width of the input images, the output geometry-aware patchlevel features are of size ( H/ 16 , W/ 16 , 768) . The original architecture subsequently uses two prediction heads to output dense point-maps, as well as pixel-level features, they are not shown in Figure 1 as they are not used in our pipeline. These portions of the pipeline remain frozen and their outputs are used as is, we instead introduce a new segment-feature head.

3.2 Segment-Feature Head: Segment-Aligned Features

The MASt3R decoder produces patch-level embeddings V 1 , V 2 of shape ( H/ 16 , W/ 16 , 768) . In the original MASt3R [22], a feature head upsamples these tensors to the resolution of the input image. For segment matching we introduce another head to transform the patch level features to M segment features. This introduced head is realized as an MLP that upsamples the patch-level

features ( V 1 , V 2 ) to image resolution, yielding feature maps of size ( H,W, 24) , 24 being the feature dimension. We denote this Feature-to-Segment head as the segment-feature head. In addition to the patch-features, the segment-feature takes as input M image resolution segment masks, obtained either from an external segmenter such as SAM2 [34] or from ground-truth annotations. Both the masks and feature maps are flattened along the spatial dimensions yielding flattened tensors.

To go from pixel-level descriptors to segment descriptors, a single batched matrix multiplication aggregates the pixel descriptors inside each mask:

The resulting segment embeddings for the two images are denoted G 1 and G 2 in Figure 1 and are fed to the differentiable matching layer described in Section 3.3. We set M = 100 as an upper bound for batch processing. In practice, images typically contain 20-30 masks when training with ground truth annotations; we pad with zeros when fewer masks are present. At inference time, the number of masks can be set arbitrarily, independent of the training-time value of M .

3.3 Differentiable Segment Matching Layer

The goal of the differentiable segment matching layer is to establish permutation-style correspondences between the M segments from each image, given segment descriptors ( G 1 , G 2 ) ∈ R ( M, 24)

Cosine-similarity affinity. We first construct an affinity matrix S ∈ R M 1 × M 2 with a simple dot product

g 1 i and g 2 j correspond to segment-level features from G 1 and G 2 respectively. Ideally, segment features corresponding to the same underlying 3D region should have a high similarity score and dis-similar regions a correspondingly low score.

Learnable dustbin. Following [35], we incorporate a dustbin row and column in the affinity matrix S to handle segments without correspondences in the other image, which is critical for widebaseline matching. We augment (2) by concatenating an additional row and column initialized with a learnable logit α ∈ R , yielding ˜ S ∈ R ( M 1 +1) × ( M 2 +1) .

Soft Correspondences via Sinkhorn The similarity logits are transformed into a soft assignment matrix P by T iterations of the Sinkhorn normalisation [38, 37] in log-space:

where τ is a temperature hyper-parameter. After convergence, P = P ( T ) is (approximately) doubly stochastic. Throughout all experiments conducted, we assume T = 50 . The output of the Sinkhorn algorithm P ( T ) is a soft bi-stochastic matrix, which has to be discretized to obtain the final segment matches.

Discrete correspondences. To obtain the simple final segment matches, we perform a simple rowwise arg max over the non-dustbin columns:

̸

3.4 Supervision

Training objective. We adopt the SuperGlue cross-entropy loss L SG [35], extended with explicit terms for unmatched segments:

where M is the set of ground-truth matches and U 1 , U 2 are the unmatched indices in image 1 and 2, respectively. The dustbin parameter α is learned jointly with the rest of the network, enabling the layer to balance match confidence against the cost of declaring non-matches. This fully differentiable design allows the matching layer to be trained end-to-end together with the upstream segment encoders and downstream task losses.

3.4.1 Training Details

The model is trained using AdamW optimizer with an initial learning rate of 1e-4 , weight decay of 1e-4 , and a cosine annealing learning rate schedule without restarts, decaying up to a minimum learning rate of 1e-6 over the full training duration. We use a batch size of 36 and train the model for 20 epochs on a single NVIDIA RTX A6000 GPU. The segment-feature heads are initialized with MASt3R's local feature head weights and finetuned further. For the differentiable segment matcher we initialize the single learnable dustbin parameter to 1 . 0 . The number of Sinkhorn iterations is set to 50. Training our model on ScanNet++ takes 22 hours, whereas a single forward pass during inference with batch size of 1 takes 0.579 seconds.

4 Experiments

4.1 Datasets

Our network is trained on scenes from ScanNet++ [47] which contain a diverse set of real-world scenes, primarily in indoor settings. We test our model on novel scenes from ScanNet++ as well as perform cross-dataset generalization studies on Replica [39] and MapFree [2] datasets. The former contains high-quality photo-realistic indoor scenes while the latter is a challenging outdoor visuallocalization dataset, which is sufficiently out-of-distribution considering our training data.

ScanNet++ [47]. ScanNet++ contains 1 006 indoor scenes captured with DSLR images and RGBD iPhone streams, all co-registered to high-quality laser scans. The dataset supplies 3D semantic meshes, 2D instance masks, and accurate camera poses, which we utilize in our pipeline. We train on 860 k image pairs from 140 scenes and evaluate on 8 k pairs from 36 validation scenes, sampled with a fixed seed (42) and balanced across scenes and four pose bins: [0 ◦ -45 ◦ ] , [45 ◦ -90 ◦ ] , [90 ◦ -135 ◦ ] , and [135 ◦ -180 ◦ ] , defined by the rotational geodesic distance between camera orientations.

Replica [39]. The Replica dataset contain 18 high-quality, photo-realistic indoor room reconstructions in the form of dense meshes replete with per-primitive semantic class and instance information. In particular, we use a pre-rendered version of this dataset from Semantic-NeRF [50], which directly provides poses, RGB-D sequences, and per-frame semantic masks. For evaluation, we employ the same pose-binning strategy as described above and randomly sample 3200 image pairs across 8 scenes, again ensuring uniform sampling across both scenes and pose-bins.

MapFree Visual Re-localization [2]. MapFree is a challenging benchmark for metric-relative pose estimation, featuring 655 diverse outdoor scenes (sculptures, fountains, murals) with extreme viewpoint changes, varying visual conditions, and geometric ambiguities. The training split contains 460 scenes with 0.5M images; we uniformly sample 50 scenes yielding 31K image pairs for training. Since test split ground truth is unavailable, we report results on the validation split using 7.8K pairs from 13 uniformly sampled scenes (out of 65 total).

4.2 Baselines

Local Feature Matching (LFM). Although few models target segment matching directly, a wide range of local feature matchers (LFMs) exists at different densities: sparse SuperPoint [9], semidense LoFTR [40], and dense RoMa and MASt3R [12, 22]. We harness these state-of-the-art LFMs

via the EarthMatch toolkit [5] to obtain segment correspondences. Each matched keypoint pair votes for the source and target masks that contain its coordinates, populating a vote matrix of size M × N (with M and N segments in the two images). Correspondences are taken as the highest-scoring entries of this matrix; the full algorithm is provided in the supplementary.

Segment Matching (SegMatch). We also compute segment matches from dense features of two strong pre-trained vision encoders, DINOv2 [30] and MASt3R [22], as well as SAM2's video propagator [34] to track masks across views. For feature based methods, masks are downsampled via nearest-neighbor interpolation to match feature resolution, and segment descriptors are computed via masked average pooling. Cosine similarity between descriptors yields a match matrix, from which one-to-one links are selected via mutual-check. We further benchmark against SegVLAD [14], which aggregates features from neighbouring segments for segment retrieval based visual place recognition.

4.3 Evaluation Metrics

We report two complementary measures of segment correspondence quality AUPRC and Recall . Area Under the Precision-Recall Curve (AUPRC) integrates precision over the entire recall axis, providing a threshold-independent summary that is particularly informative under the high class-imbalance characteristic of segment matching between image pairs. Recall@ k ( R @ k ) denotes the fraction of query segments whose ground-truth counterpart is found within the top k ranked candidates, thus gauging how effectively the method surfaces correct matches among its highest-confidence predictions. Additional details for the dataset, baseline and experiments can be found in the supplementary, along with more qualitative results.

5 Results

Segment Matching In Table 1, we compare our proposed method SegMASt3R with the stateof-the-art methods using two categories of approaches in the literature: the well-established local feature matching (LFM) based on sparse or dense keypoint correspondences, and the recently emerging open-set instance association based on segment matching (SegMatch). It can be observed that SegMASt3R outperforms all the baselines for all the pose bins with a huge margin. Amongst the LFM techniques, dense matchers (RoMa and MASt3R) outperform sparse matchers (SP-LG and GiM-DKM) on both the evaluation metrics for the task of segment matching. Notably, on the highly challenging wide-baseline settings, MASt3R outperforms other local matchers by a large margin. However, when the same backend features are aggregated at the segment-level, MASt3R's performance deteriorates significantly, e.g., AUPRC drops from 52.9 to 41.4 on the 90-135 pose bin. This uncovers an inherent limitation: feature distinctiveness at the pixel level does not necessarily translate to the instance level . Considering the SegMatch techniques which are not trained for the LFM task, it can be observed that DINOv2 and SegVLAD perform particulary well on R@5 metric, which aligns with their typical use for coarse retrieval [19, 14]. On the other hand, SAM2's twoframe video propagation only works well for narrow-baseline matching, which can be expected as its training set is mostly comprised of dynamic object tracking. Overall, these results show that all

the baseline methods lack on at least one of the fronts: pixel- vs segment-level distinctiveness, recall vs precision, and narrow- vs wide-baseline robustness. Our proposed method SegMASt3R achieves all these desirable properties with high performance across the board by learning segment-level representations with the training objective of instance association. In Section 5, we provide qualitative results which emphasize the ability of our method to address the problem of perceptual instance aliasing and instance matching under extreme viewpoint shifts.

Generalization We assess our model's ability to generalize to new environments. First, in Table 2, we present results on a different indoor dataset, Replica, to test the generalization ability of SegMASt3R, which is trained only on ScanNet++. We compare SegMASt3R against the LFM and SegMatch versions of MASt3R, as these three methods closely resemble each other in terms of their network architecture (detailed comparisons are included in the supplementary). It can be observed that the performance patterns on Replica remain largely the same as that on ScanNet++, and SegMASt3R consistently outperforms the LFM and SegMatch versions of MASt3R across the board.

Furthermore, we test generalization to challenging outdoor scenes from the MapFree dataset [2], with results shown in Table 3. Since MapFree lacks instance-level ground truth, we use SAM2's video propagator on image sequences to generate a pseudo-ground truth for evaluation. Our indoortrained model (SegMASt3R (SPP)) shows a regression in performance in comparison to DINOv2, highlighting the domain shift. However, this gap can be substantially closed by either re-training on MapFree data (SegMASt3R (MF)) or, even more simply by just recalibrating the single learnable dustbin parameter α using a grid-search over a small calibration set from the target domain (SegMASt3R (SPP, Dustbin MF)). This demonstrates the strong adaptability of our model's learned geometric features.

3D Instance Mapping The goal of instance mapping is to localize object instances in 3D so a robot can distinguish objects of the same class in both image and metric space [17, 28, 44]. The main difficulty is preserving identities over long trajectories, especially when objects leave the camera's view and later reappear from different angles. Our pipeline employs SEGMAST3R to match object masks across image pairs. Given ground-truth masks and sampled pairs, we extract mask features, solve a Sinkhorn assignment, and take the row-wise argmax to obtain tentative correspondences.

We then back-project each matched mask into 3D and drop links whose point-cloud IoU falls below 0.5, rejecting any match that fails this geometric check. Details are provided in the supplementary. Table 4 shows percentage AP versus ConceptGraphs [17]. We evaluate under two conditions: using masks generated by MobileSAM [48] as in the original ConceptGraphs setup, and using groundtruth (GT) masks for a fairer comparison of the underlying matching capability. Our geometry-aware matching yields higher accuracy in both settings, particularly when objects exit and later re-enter the field of view, demonstrating robustness to both noisy masks and challenging viewpoints.

Robustness to Noisy Segmentation Masks To assess the practical viability of our approach in real-world scenarios where ground-truth masks are unavailable, we evaluated all methods on ScanNet++ using imperfect segmentations generated by FastSAM. A key challenge in this setting is that evaluation can conflate the performance of the segment matcher with the quality of the upstream Automatic Mask Generator (AMG). A low score may not distinguish between a matching failure and an AMG failure where predicted masks do not align with any ground-truth segment.

To decouple these factors and isolate the core matching performance, we adopt an evaluation protocol that does not penalize methods for AMG errors. Specifically, while all methods take noisy masks as input, the evaluation is performed only on the subset of ground-truth correspondences for which a valid match between predicted segments was possible. As shown in Table 5, SegMASt3R maintains a substantial performance margin over all baselines across different evaluation thresholds. This result confirms that the strong geometric priors learned by our method provide significant robustness, enabling superior matching even when conditioned on inconsistent and noisy segment inputs.

Object-level Topological Navigation Recent works [17, 15, 16] have explored the use of SAM2's open-set, semantically-meaningful instance segmentation for topological mapping and navigation. These methods rely on accurate segment-level association. To show the benefits of our proposed method on complex downstream tasks such as navigation, we considered an object topology-based mapping and navigation method, RoboHop [15], and swapped its SuperPoint+LightGlue based segment matching with our proposed segment matcher. This segment matching is used by RoboHop's localizer to match each of the object instances in the query image with those in the sub-map images, which is then used to estimate the currently-viewed object's path to the goal and correspondingly obtain a control signal. We used the val set of the ImageInstanceNav [21] dataset, which comprises real-world indoor scenes from HM3Dv0.2 [33]. We followed [32] for creating map trajectories and evaluating navigation performance.

In Table 6, we report navigation success for (vanilla) RoboHop and an enhanced version of it that uses SegMASt3R for segment association for localization and navigation. We use two metrics: SPL (Success weighted by Path Length) [1] and SSPL (soft SPL) [8]. Furthermore, we considered four different evaluation settings using two parameters that define the submap used for localizing the query image segments: Submap Span ( S s ), which defines the total number of images sampled from the map based on the distance from the robot's current position, and Submap Density ( S ρ ), which defines a subsampling factor to uniformly skip map images. These parameter configurations aid in testing narrow- and wide-baseline matching as well as the ability to avoid false positives. Table 6 shows that SegMASt3R consistently outperforms vanilla RoboHop. In particular, we achieve an absolute improvement of 27% SPL on one of the hardest parameter settings: S s = 16 , S ρ = 0 . 25 , where only 4 submap images are sampled due to a low density value. This shows that even with a very sparse submap, it is possible to maintain high navigation success rate, thus avoiding the typical trade-off between compute time and accuracy for the localizer. In Figure 2, we present a qualitative comparison between vanilla RoboHop's segment matching and that based on SegMASt3R: (left) the mismatch between the wall (orange) and the vanity cabinet for the vanilla methods leads to an incor-

QueryImage

SubmapRefImage

Querylmage

SubmapRef Image

rect rotation towards the right, consequently leading to navigation failure, whereas SegMASt3R's accurate segment matching (right) guides the agent into the bathroom and finally to the goal object.

Impact of the Feature Encoder To isolate the contribution of the feature encoder in our pipeline, we replace MASt3R's cross-attention block with two alternative, purely 2D backbones: CroCo (shared by both MASt3R and SegMASt3R) and DINOv2 , a state-of-the-art encoder often used for off-the-shelf segment matching [17, 15, 14]. As reported in Table 7, neither backbone yields competitive segment-matching accuracy. This result underscores that learning segment descriptors from a 2D foundation model alone is inadequate; geometric context is essential. In contrast, MASt3R's cross-attention layers, 3D-aware training regimen, and explicit formulation of image matching in 3D jointly endow the model with the priors required for reliable instance association. The fact that CroCo is always second best suggests that cross-view-completion, by itself, can yield superior results for segment matching between image pairs.

Qualitative Results Figure 4 compares SegMASt3R with SAM2's two-frame video propagationbased segment matching under extreme viewpoint variations on the ScanNet++ dataset. Each row shows a reference image, SAM2's matches, SegMASt3R's matches, respectively. In the top row, a wall (pink) and a door (blue) are mismatched by SAM2, whereas SegMASt3R correctly associates them, despite a very limited visual overlap. In the bottom row, SAM2 gets confused between the two monitors, whereas SegMASt3R is able to correctly associate them despite the simultaneous effect of 180 ◦ viewpoint shift and perceptual instance aliasing (i.e., different instances of the same object category in an image potentially lead to mismatches). Figure 3 presents qualitative results on the (outdoor) MapFree [2] dataset, where we compare our ScanNet-trained SegMASt3R with the off-the-shelf DINOv2 [30] features. Each image triplet represents the query segment (left) and its

Query

SegMASt3R

DINOv2

Move More.

4 laps = 1mile

Query

SegMASt3R

DINOv2

Image O with GT Masks

Image 1 - SAM2 Video Propagation

Image 1 with SegMast3R (Ours)

Image O with GT Masks

Image 1 - SAM2 Video Propagation

Image 1 with SegMast3R (Ours)

retrieved matches using our method (middle) and DINOv2 (right). The query segment is displayed in red color, and its top three matches are respectively displayed in red, green, and blue colors. The left image triplet illustrates another case of perceptual instance aliasing. SegMASt3R is able to resolve this aliasing problem, whereas DINOv2 confuses the right and left legs of the signboard. The right image panel shows that our Sinkhorn solver effectively learns dustbin allocations for explicitly rejecting negatives, that is, the segments which do not have any corresponding match, whereas DINOv2 leads to incorrect matches.

6 Conclusion

Weproposed SegMASt3R, a simple method to re-purpose an existing 3D foundation model MASt3R for image segment matching. Our proposed method achieves excellent results on ScanNet++ and Replica with a simple pipeline and a minimal amount of training. It especially excels on widebaseline segment matching between image pairs. In addition, we show that SegMASt3R has practical applicability by evaluating it's performance on the downstream tasks of 3D instance mapping and object-relative topological navigation, where we significantly outperform the corresponding baselines. Overall, this work attempts to establish segment matching as a core computer vision capability, which will enable even more downstream applications in future alongside the advances in image segmentation and data association.

References

Peter Anderson, Angel Chang, Devendra Singh Chaplot, Alexey Dosovitskiy, Saurabh Gupta, Vladlen Koltun, Jana Kosecka, Jitendra Malik, Roozbeh Mottaghi, Manolis Savva, et al. On evaluation of embodied navigation agents. arXiv preprint arXiv:1807.06757 , 2018.

Eduardo Arnold, Jamie Wynn, Sara Vicente, Guillermo Garcia-Hernando, Áron Monszpart, Victor Adrian Prisacariu, Daniyar Turmukhambetov, and Eric Brachmann. Map-free visual relocalization: Metric pose relative to a single image. In ECCV , 2022.

Axel Barroso-Laguna, Sowmya Munukutla, Victor Adrian Prisacariu, and Eric Brachmann. Matching 2d images in 3d: Metric relative pose from metric correspondences. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 4852-4863, 2024.

Gabriele Berton, Carlo Masone, Valerio Paolicelli, and Barbara Caputo. Viewpoint invariant dense matching for visual geolocalization. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 12169-12178, 2021.

Gabriele Berton, Gabriele Goletto, Gabriele Trivigno, Alex Stoken, Barbara Caputo, and Carlo Masone. Earthmatch: Iterative coregistration for fine-grained localization of astronaut photography. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) Workshops , June 2024.

Orcun Cetintas, Guillem Brasó, and Laura Leal-Taixé. Unifying short and long-term tracking with graph hierarchies. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 22877-22887, 2023.

Ying Chen, Dihe Huang, Shang Xu, Jianlin Liu, and Yong Liu. Guide local feature matching by overlap estimation. In Proceedings of the AAAI conference on artificial intelligence , volume 36, pages 365-373, 2022.

Samyak Datta, Oleksandr Maksymets, Judy Hoffman, Stefan Lee, Dhruv Batra, and Devi Parikh. Integrating egocentric localization for more realistic point-goal navigation agents. In Jens Kober, Fabio Ramos, and Claire Tomlin, editors, Proceedings of the 2020 Conference on Robot Learning , volume 155 of Proceedings of Machine Learning Research , pages 313328. PMLR, 16-18 Nov 2021. URL https://proceedings.mlr.press/v155/datta21a. html .

Daniel DeTone, Tomasz Malisiewicz, and Andrew Rabinovich. Superpoint: Self-supervised interest point detection and description. In CVPR Deep Learning for Visual SLAM Workshop , 2018. URL http://arxiv.org/abs/1712.07629 .

Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010.11929 , 2020.

Johan Edstedt, Ioannis Athanasiadis, Mårten Wadenbäck, and Michael Felsberg. Dkm: Dense kernelized feature matching for geometry estimation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 17765-17775, 2023.

Johan Edstedt, Qiyu Sun, Georg Bökman, Mårten Wadenbäck, and Michael Felsberg. RoMa: Robust Dense Feature Matching. IEEE Conference on Computer Vision and Pattern Recognition , 2024.

Yujie Fu, Pengju Zhang, Bingxi Liu, Zheng Rong, and Yihong Wu. Learning to reduce scale differences for large-scale invariant image matching. IEEE Transactions on Circuits and Systems for Video Technology , 33(3):1335-1348, 2022.

Kartik Garg, Sai Shubodh Puligilla, Shishir Kolathaya, Madhava Krishna, and Sourav Garg. Revisit anything: Visual place recognition via image segment retrieval. In European Conference on Computer Vision (ECCV) , 2024.

Sourav Garg, Krishan Rana, Mehdi Hosseinzadeh, Lachlan Mares, Niko Suenderhauf, Feras Dayoub, and Ian Reid. Robohop: Segment-based topological map representation for openworld visual navigation. arXiv , 2023.

Sourav Garg, Dustin Craggs, Vineeth Bhat, Lachlan Mares, Stefan Podgorski, Madhava Krishna, Feras Dayoub, and Ian Reid. Objectreact: Learning object-relative control for visual navigation. In Conference on Robot Learning . PMLR, 2025.

Qiao Gu, Ali Kuwajerwala, Sacha Morin, Krishna Murthy Jatavallabhula, Bipasha Sen, Aditya Agarwal, Corban Rivera, William Paul, Kirsty Ellis, Rama Chellappa, et al. Conceptgraphs: Open-vocabulary 3d scene graphs for perception and planning. In 2024 IEEE International Conference on Robotics and Automation (ICRA) , pages 5021-5028. IEEE, 2024.

Nathan Hughes, Yun Chang, and Luca Carlone. Hydra: A real-time spatial perception system for 3d scene graph construction and optimization. arXiv preprint arXiv:2201.13360 , 2022.

Nikhil Keetha, Avneesh Mishra, Jay Karhade, Krishna Murthy Jatavallabhula, Sebastian Scherer, Madhava Krishna, and Sourav Garg. Anyloc: Towards universal visual place recognition. IEEE Robotics and Automation Letters , 9(2):1286-1293, 2023.

Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander C Berg, Wan-Yen Lo, et al. Segment anything. In Proceedings of the IEEE/CVF international conference on computer vision , pages 4015-4026, 2023.

Jacob Krantz, Stefan Lee, Jitendra Malik, Dhruv Batra, and Devendra Singh Chaplot. Instancespecific image goal navigation: Training embodied agents to find object instances. arXiv preprint arXiv:2211.15876 , 2022.

Vincent Leroy, Yohann Cabon, and Jérôme Revaud. Grounding image matching in 3d with mast3r. In European Conference on Computer Vision , pages 71-91. Springer, 2024.

Rongjie Li, Songyang Zhang, and Xuming He. Sgtr: End-to-end scene graph generation with transformer. In proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 19486-19496, 2022.

Siyuan Li, Martin Danelljan, Henghui Ding, Thomas E Huang, and Fisher Yu. Tracking every thing in the wild. In European conference on computer vision , pages 498-515. Springer, 2022.

Siyuan Li, Lei Ke, Martin Danelljan, Luigi Piccinelli, Mattia Segu, Luc Van Gool, and Fisher Yu. Matching anything by segmenting anything. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 18963-18973, 2024.

Rongguang Liang, Jie Yuan, Benfa Kuang, Qiang Liu, and Zhenyu Guo. Dig-slam: an accurate rgb-d slam based on instance segmentation and geometric clustering for dynamic indoor scenes. Measurement Science and Technology , 35(1):015401, 2023.

Philipp Lindenberger, Paul-Edouard Sarlin, and Marc Pollefeys. LightGlue: Local Feature Matching at Light Speed. In ICCV , 2023.

Shigemichi Matsuzaki, Takuma Sugino, Kazuhito Tanaka, Zijun Sha, Shintaro Nakaoka, Shintaro Yoshizawa, and Kazuhiro Shintani. Clip-loc: Multi-modal landmark association for global localization in object-based maps. In Fortieth Intl Conference on Robotics and Automation , 2024. URL https://arxiv.org/abs/2402.06092 .

Tim Meinhardt, Alexander Kirillov, Laura Leal-Taixe, and Christoph Feichtenhofer. Trackformer: Multi-object tracking with transformers. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 8844-8854, 2022.

Maxime Oquab, Timothée Darcet, Théo Moutakanni, Huy Vo, Marc Szafraniec, Vasil Khalidov, Pierre Fernandez, Daniel Haziza, Francisco Massa, Alaaeldin El-Nouby, et al. Dinov2: Learning robust visual features without supervision. arXiv preprint arXiv:2304.07193 , 2023.

Jiangmiao Pang, Linlu Qiu, Xia Li, Haofeng Chen, Qi Li, Trevor Darrell, and Fisher Yu. Quasi-dense similarity learning for multiple object tracking. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 164-173, 2021.

Stefan Podgorski, Sourav Garg, Mehdi Hosseinzadeh, Lachlan Mares, Feras Dayoub, and Ian Reid. Tango: Traversablility-aware navigation with local metric control for topological goals. In 2025 IEEE International Conference on Robotics and Automation (ICRA) . IEEE, 2025.

Santhosh Kumar Ramakrishnan, Aaron Gokaslan, Erik Wijmans, Oleksandr Maksymets, Alexander Clegg, John M Turner, Eric Undersander, Wojciech Galuba, Andrew Westbury, Angel X Chang, Manolis Savva, Yili Zhao, and Dhruv Batra. Habitat-matterport 3d dataset (HM3d): 1000 large-scale 3d environments for embodied AI. In Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track , 2021. URL https://arxiv.org/abs/2109.08238 .

Nikhila Ravi, Valentin Gabeur, Yuan-Ting Hu, Ronghang Hu, Chaitanya Ryali, Tengyu Ma, Haitham Khedr, Roman Rädle, Chloe Rolland, Laura Gustafson, et al. Sam 2: Segment anything in images and videos. arXiv preprint arXiv:2408.00714 , 2024.

Paul-Edouard Sarlin, Daniel DeTone, Tomasz Malisiewicz, and Andrew Rabinovich. Superglue: Learning feature matching with graph neural networks. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 4938-4947, 2020.

Xuelun Shen, Zhipeng Cai, Wei Yin, Matthias Müller, Zijun Li, Kaixuan Wang, Xiaozhi Chen, and Cheng Wang. Gim: Learning generalizable image matcher from internet videos. In The Twelfth International Conference on Learning Representations , 2024.

Richard Sinkhorn. A relationship between arbitrary positive matrices and doubly stochastic matrices. The Annals of Mathematical Statistics , 35(2):876-879, 1964.

Richard Sinkhorn and Paul Knopp. Concerning nonnegative matrices and doubly stochastic matrices. Pacific Journal of Mathematics , 21(2):343-348, 1967.

Julian Straub, Thomas Whelan, Lingni Ma, Yufan Chen, Erik Wijmans, Simon Green, Jakob J. Engel, Raul Mur-Artal, Carl Ren, Shobhit Verma, Anton Clarkson, Mingfei Yan, Brian Budge, Yajie Yan, Xiaqing Pan, June Yon, Yuyang Zou, Kimberly Leon, Nigel Carter, Jesus Briales, Tyler Gillingham, Elias Mueggler, Luis Pesqueira, Manolis Savva, Dhruv Batra, Hauke M. Strasdat, Renzo De Nardi, Michael Goesele, Steven Lovegrove, and Richard Newcombe. The Replica dataset: A digital replica of indoor spaces. arXiv preprint arXiv:1906.05797 , 2019.

Jiaming Sun, Zehong Shen, Yuang Wang, Hujun Bao, and Xiaowei Zhou. Loftr: Detectorfree local feature matching with transformers. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 8922-8931, 2021.

Zhongdao Wang, Liang Zheng, Yixuan Liu, Yali Li, and Shengjin Wang. Towards real-time multi-object tracking. In European conference on computer vision , pages 107-122. Springer, 2020.

Philippe Weinzaepfel, Vincent Leroy, Thomas Lucas, Romain Brégier, Yohann Cabon, Vaibhav Arora, Leonid Antsfeld, Boris Chidlovskii, Gabriela Csurka, and Jérôme Revaud. Croco: Self-supervised pre-training for 3d vision tasks by cross-view completion. Advances in Neural Information Processing Systems , 35:3502-3516, 2022.

Philippe Weinzaepfel, Thomas Lucas, Vincent Leroy, Yohann Cabon, Vaibhav Arora, Romain Brégier, Gabriela Csurka, Leonid Antsfeld, Boris Chidlovskii, and Jérôme Revaud. Croco v2: Improved cross-view completion pre-training for stereo matching and optical flow. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 1796917980, 2023.

Binbin Xu, Wenbin Li, Dimos Tzoumanikas, Michael Bloesch, Andrew Davison, and Stefan Leutenegger. Mid-fusion: Octree-based object-level multi-instance dynamic slam. In 2019 International Conference on Robotics and Automation (ICRA) , pages 5231-5237. IEEE, 2019.

Bin Yan, Yi Jiang, Peize Sun, Dong Wang, Zehuan Yuan, Ping Luo, and Huchuan Lu. Towards grand unification of object tracking. In European conference on computer vision , pages 733751. Springer, 2022.

Bin Yan, Yi Jiang, Jiannan Wu, Dong Wang, Ping Luo, Zehuan Yuan, and Huchuan Lu. Universal instance perception as object discovery and retrieval. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 15325-15336, 2023.

Chandan Yeshwanth, Yueh-Cheng Liu, Matthias Nießner, and Angela Dai. Scannet++: A high-fidelity dataset of 3d indoor scenes. In Proceedings of the International Conference on Computer Vision (ICCV) , 2023.

Chaoning Zhang, Dongshen Han, Yu Qiao, Jung Uk Kim, Sung-Ho Bae, Seungkyu Lee, and Choong Seon Hong. Faster segment anything: Towards lightweight sam for mobile applications. arXiv preprint arXiv:2306.14289 , 2023.

Yesheng Zhang and Xu Zhao. Dmesa: Densely matching everything by segmenting anything. arXiv preprint arXiv:2408.00279 , 2024.

Shuaifeng Zhi, Tristan Laidlow, Stefan Leutenegger, and Andrew J. Davison. In-place scene labelling and understanding with implicit scene representation. In ICCV , 2021.

NeurIPS Paper Checklist

The checklist is designed to encourage best practices for responsible machine learning research, addressing issues of reproducibility, transparency, research ethics, and societal impact. Do not remove the checklist: The papers not including the checklist will be desk rejected. The checklist should follow the references and follow the (optional) supplemental material. The checklist does NOT count towards the page limit.

Please read the checklist guidelines carefully for information on how to answer these questions. For each question in the checklist:

You should answer [Yes] , [No] , or [NA] .

[NA] means either that the question is Not Applicable for that particular paper or the relevant information is Not Available.

Please provide a short (1-2 sentence) justification right after your answer (even for NA).

The checklist answers are an integral part of your paper submission. They are visible to the reviewers, area chairs, senior area chairs, and ethics reviewers. You will be asked to also include it (after eventual revisions) with the final version of your paper, and its final version will be published with the paper.

The reviewers of your paper will be asked to use the checklist as one of the factors in their evaluation. While "[Yes] " is generally preferable to "[No] ", it is perfectly acceptable to answer "[No] " provided a proper justification is given (e.g., "error bars are not reported because it would be too computationally expensive" or "we were unable to find the license for the dataset we used"). In general, answering "[No] " or "[NA] " is not grounds for rejection. While the questions are phrased in a binary way, we acknowledge that the true answer is often more nuanced, so please just use your best judgment and write a justification to elaborate. All supporting evidence can appear either in the main paper or the supplemental material, provided in appendix. If you answer [Yes] to a question, in the justification please point to the section(s) where related material for the question can be found.

IMPORTANT, please:

Delete this instruction block, but keep the section heading 'NeurIPS Paper Checklist" ,

Keep the checklist subsection headings, questions/answers and guidelines below.

Do not modify the questions and only use the provided macros for your answers .

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: We re-purpose a 3d foundation model and show how it can be used for segment matching. We show experiments showcasing the efficacy of our proposed method and down-stream applications justifying its use in real-world robotic settings.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: The limitations are mentioned in the supplementary.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory assumptions and proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [No]

Justification: Paper is purely algorithmic and application oriented and does not have any theory.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental result reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: Training and dataset details are provided in Section 4.1 and Section 3.4.1 that allow for reproducibility.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [No]

Justification: We will release code and image pairs trained on upon paper acceptance.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental setting/details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: The paper specifies dataset and splits used in Section 4.1. Implementation details are specified in Section 3.4.1.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment statistical significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [NA]

Justification: No plots requiring error bars have been presented.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments compute resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: Mentioned in the supplementary.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code of ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: The code of ethics details are given in the supplementary.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [Yes]

Justification: These are mentioned in supplementary.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: Paper poses no such risks.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: Dataset details are provided in Section 4.1. Their links are given in the supplementary. The licenses can be found by following the dataset links. The backbone models used are cited throughout the paper.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [No]

Justification: No new assets are introduced. Only existing datasets are utilized.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and research with human subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: We don't crowdsource data or use any human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional review board (IRB) approvals or equivalent for research with human subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: Not applicable to our submission.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

16. Declaration of LLM usage

Question: Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.

Answer: [NA]

Justification: LLMS were not used.

Guidelines:

The answer NA means that the core method development in this research does not involve LLMs as any important, original, or non-standard components.

Please refer to our LLM policy ( https://neurips.cc/Conferences/2025/LLM ) for what should or should not be described.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents SegMASt3R, a novel framework that adapts the 3D foundation model MASt3R for wide-baseline segment matching. It introduces a lightweight segment-feature head to generate segment-level descriptors and applies a differentiable optimal transport solver with a learnable dustbin. Extensive experiments demonstrate that SegMASt3R significantly outperforms both local feature matchers and 2D segment-based methods. Its effectiveness is further validated in downstream tasks such as 3D instance mapping and topological navigation.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n1. Effective model design. The method builds upon a strong backbone (MASt3R). The design of the segment-level feature head and the differentiable matching mechanism with a learnable dustbin is reasonable.\n2. Strong performance and diverse downstream applications. The application of SegMASt3R in topological navigation is interesting and demonstrates its practical value.\n3. Clear writing. The paper is well-written and easy to follow.\n\nWeaknesses:\n1. Limited ablation:\n\nThe method uses a learnable dustbin to handle unmatched segments. It would strengthen the paper to include an ablation study evaluating its contribution.\n\n2. Evaluation settings.\n\n(1) It is unclear whether the baseline methods are re-trained or evaluated using their publicly released checkpoints. This is especially critical in wide-baseline settings, where fine-tuning or data augmentation strategies may significantly affect performance.\n\n(2) According to Supplementary Table 2, SegMASt3R is evaluated only using ground-truth masks, which are often unavailable during inference. It would be beneficial to also report results using masks from MobileSAM.\n\n3. Minor typo.\n\nIn Line 111, the channel dimension of V1, V2 should be 768, instead of 24.\n\nQUESTIONS:\nMy concerns mainly lie in the evaluation settings. Please refer to the weaknesses.\n\nLIMITATIONS:\nYes, in the supplementary material.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 4, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-14"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper addresses the challenging problem of segment-level image correspondence under extreme viewpoint variations, leveraging the spatial reasoning capabilities of 3D foundation models. The proposed method, SegMASt3R, utilizes a frozen MASt3R backbone to extract geometry-aware patch-level features from an image pair. These features are upsampled and aggregated into segment-level descriptors via a lightweight MLP, using segmentation masks from external models (e.g., SAM2) or ground truth. Segment correspondences are then established through a differentiable optimal transport layer based on the Sinkhorn algorithm. The model is trained end-to-end on ScanNet++ using a SuperGlue-inspired contrastive loss that accounts for both matched and unmatched segments. SegMASt3R is evaluated on ScanNet++ and Replica, showing strong generalization across indoor scenes, and is qualitatively assessed on the challenging MapFree outdoor dataset. It is further validated on downstream tasks such as 3D instance mapping and object-goal navigation, where it delivers substantial improvements over baselines. The method achieves up to a 30% gain in AUPRC over both segment-matching and local-feature-based methods.\n\nSTRENGTHS AND WEAKNESSES:\nThe main strength of this paper is the strong performance on a widely relevant computer vision task, and strong results (>80) even for extreme viewpoint changes. This work undoubtedly has applications in practical tasks related to robotic navigation and instance mapping, as also show in the ablation and supplementary experiments. And the pipeline used is relatively simple, while very effective. Finally, the paper is very well structured and written, with a strong and exhaustive experimental section and an excellent methodology section, which discusses in clear details all aspects of the pipeline. \n\nAt the same time, the simplicity of the proposed architecture leads to limited novelty: it is a straightforward adaptation of MASt3R with an added head comprising an MLP and Sinkhorn-based matching. Additionally, as noted by the paper's limitation section (included in the supplementary material, not in the main paper), the experiments do not cover fully generalisation to outdoor settings, and training requires ground truth 3D instance segmentation, which is not always easy to obtain. An additional limitation not covered by the paper is how robust the method is to noisy segmentation masks. Finally, while the matching strategy is efficient enough (0.579s forward pass at inference) it is not discuss how scalable the approach is; the paper mentions using 100 segments per image to be 'sufficient to cover the entire image while keeping computation affordable', but this leaves space for a discussion on how performance would change adding more segments.\n\nDespite the weaknesses, this paper still provides a very strong contribution to the computer vision community, and the weaknesses are mitigated by the fact that in most practical applications 100 segments is a reasonable number (e.g., for SAM), and the strong performances with a simple framework are a positive.\n\nQUESTIONS:\n- The rebuttal could elaborate on the robustness to segmentation noise\n- The rebuttal could elaborate on the dependence of performance on the number of segments\n- The final version of the paper could include already in the main paper an abridged version of the discussion on limitations and societal impact\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "CLARITY": 4, "IMPACT": 4, "ORIGINALITY": 2, "DATE": "2025-06-20"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nSegMASt3R is a model for segment matching, an under-explored problem. That is, the paper introduces the wide baseline segment matching problem between image pairs. An approach based on frozen MASt3R feature extractor, given objects masks, and the differentiable optimal transport framework of SuperGlue is provided. In addition, a benchmark for wide segment matching is proposed with\nquantitative evaluations on indoor datasets ScanNet++ and Replica and show visual results on MapFree Outdoor dataset. The proposed approach improves downstream tasks such as 3D instance mapping and object level topological navigation.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n- An approach for the under explored problem of wide baseline segment matching is offered, showing improved results over existing segment matching methods on indoor datasets.\n- Using SegMASt3R improves performance on downstream tasks such as 3D instance mapping and Object level topological navigation\n-  The paper is, overall, well written, the math is clearly explained, and the description of the \nfigures is clear.\n\nWeaknesses:\n- All quantitative results are reported exclusively on indoor datasets.\n1. ScanNet++ , an indoor dataset, which the proposed method is\ntrained on although the competitor&#39;s methods aren&#39;t. 2. Replica, an\nindoor synthetic dataset.\n- The comparison with DINOv2 and MAStr3 methods in table 1 appears\nto favor the proposed approach. The proposed method first upsample\nthe features to the image resolution before the mask are applied (lines\n114-115), however, for extracting segment match from dense features\nencoders the authors downsample the masks to the features&#39;\ndegraded resolution (lines 199-202).\n-  The proposed approach mainly combines existing components without\nintroducing substantial methodological advances.\n\n\nMinor weaknesses: \n- Inaccuracy in the abstract. Lines 9-12 &quot;Extensive experiments show\nthat our approach outperforms state-of-the-art methods, including\nthe SAM2video propagator and local feature matching methods, by up\nto 30% on the AUPRC metric, on ScanNet++ and Replica datasets.&quot;;\nSince many of those methods were not inherently built for the task of\nsegment matching some required modifications by the authors to\nproduce segment correspondences. Therefore, it is not accurate to\nname those methods SOTA for this task or to claim the proposed\nmethod outperform SOTA methods.\n- The necessity for the learnable logit in the learnable dustbin is unclear.\n\nQUESTIONS:\nQ1. Why downsample the masks for the DINOv2 and MASt3R baselines, rather than interpolating their features to match the\nmask resolution?\nQ2. The masks extraction is a crucial step in creating high quality segments. ScanNet++ and Replica dataset provide masks, however, in cases where masks are not provided annotating 100 masks (line\n123) for every input image seems unpractical. When ground truth\nmasks are not available, how do the authors propose to obtain it? What\nis the expected running time for this step?\nQ3. Have the authors considered comparing to \"Gaussian Grouping: Segment and Edit Anything in 3D Scenes\"?\n\n-Q4: Statistical Error Bars: Question 7 checklist: I don’t think it is correct – there is no\nstatistical error bars in any of the results (at least not mentioned) also in 3.4.1 the\ntime is not with standard deviation.\n\nLIMITATIONS:\nThe paper overall limitation is that it requires a 3D ground truth annotation, which are quite\nrare. Furthermore, as a direct result of that, they cannot generalize well to other scenes and\ndatasets. Those issues were addressed by the authors in the Sup. Material.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 2, "DATE": "2025-07-01"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper introduces SegMASt3R, a segment matching framework that leverages a 3D foundation model (MASt3R) to establish wide-baseline segment correspondences. The key contribution lies in the segment-feature head, which aggregates patch-level features into segment-level descriptors, followed by a differentiable optimal transport layer for segment matching. The approach demonstrates strong performance improvements on ScanNet++ and Replica, supported by qualitative results on MapFree, and is further evaluated on downstream tasks such as 3D instance mapping and topological navigation.\n\nSTRENGTHS AND WEAKNESSES:\n- Strengths\n\n  - Leverage 3D priors from pretrained model. SegMASt3R introduces MASt3R’s strong 3D inductive bias to address the gap in geometry-aware segment matching, enabling robust performance under wide viewpoint variations where 2D segment-based models may fail.\n\n  - Strong experimental performance. The proposed method significantly outperforms existing baselines across all pose bins (Tab. 1, Tab. 2), especially in the wide-baseline setting (135–180°) in the ScanNet++ and Replica datasets.\n\n  - Good downstream task application. The proposed method shows strong applicability to real-world tasks, including 3D instance mapping (Tab. 3) and object-based topological navigation (Tab. 4). These results highlight its ability to maintain object-level consistency across large viewpoint shifts.\n\n- Weaknesses\n\n    - Novelty concerns. While the paper leverage pretrained 3D foundation model as priors for segment matching is novel.  The core components are all adapted from prior work. \n        -  The corss-view patch features are from MASt3R prediction.\n        -  The core segment-feature head, is directly adpoted and initilized from MASt3R's descriptor head. \n        - The differentiable matching layer and sinknorn normalization closely follow the SuperGlue approach.\n\n     Hence, the contribution is largely engineering-driven.\n\n    - Writing and Presentation Clarity. \n        - Line 106: Patch-level features are of size (H/16, V/16, 768) → Should be (H/16, W/16, 768).\n        - Line 99: “Cross-view encoding Transformer”. Encoding transformer or decoder?\n        - Line 111: V1, V2 of shape (H/16, W/16, 24) → The patch-level embeddings V1, V2 should be (H/16, W/16, 768).\n        - The learnable logit \\alpha is not well defined and not used in the following equations. The description of $\\alpha$ in line 152 is even confusing.\n        - Lines 199–200: “A segment descriptor is the masked average of its feature map at a suitable resolution” — what is the suitable resolution? “Average of its feature map” — how is the average conducted?\n        - Table 1: Local Feature Matching (LFM). Are the results from local visual descriptor matching or from segment matching?\n        - Line 287: The ablation with CroCo encoder cannot demonstrate the effectiveness of the “explicit formulation of image matching”. To demonstrate it, it is better to use the DUSt3R backbone as it is not trained for the matching task.\n\n    - Generalization concerns to outdoor. The evaluation on the outdoor MapFree dataset is limited to qualitative results, without quantitative metrics or comparisons. Including quantitative analysis, additional outdoor datasets, or more in-the-wild examples would strengthen the claim of generalization capability.\n\nQUESTIONS:\n- The segmentation masks in this paper are from ground truth or SAM2. How does SegMASt3R perform when using imperfect or noisy segment masks?\n- Have you conducted ablations on fine-tuning the cross-attention decoder in MASt3R jointly with the segment-feature head? Could this further enhance segment matching?\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 2, "IMPACT": 3, "ORIGINALITY": 2, "DATE": "2025-07-01"}