LoCo: Learning 3D Location-Consistent Image Features with a Memory-Efficient Ranking Loss
Authors: Dominik Kloepfer
Joao F. Henriques
Dylan Campbell
Venue: NeurIPS 2024 poster (2024)
Keywords: Feature Learning
Self-Supervised Learning
3D Vision
Primary area: machine_vision
TL;DR: A loss function for training feature extractors to extract location-consistent features in a self-supervised manner.

Abstract: Image feature extractors are rendered substantially more useful if different views of the same 3D location yield similar features while still being distinct from other locations. A feature extractor that achieves this goal even under significant viewpoint changes must recognise not just semantic categories in a scene, but also understand how different objects relate to each other in three dimensions. Existing work addresses this task by posing it as a patch retrieval problem, training the extracted features to facilitate retrieval of all image patches that project from the same 3D location. However, this approach uses a loss formulation that requires substantial memory and computation resources, limiting its applicability for large-scale training. We present a method for memory-efficient learning of location-consistent features that reformulates and approximates the smooth average precision objective. This novel loss function enables improvements in memory efficiency by three orders of magnitude, mitigating a key bottleneck of previous methods and allowing much larger models to be trained with the same computational resources. We showcase the improved location consistency of our trained feature extractor directly on a multi-view consistency task, as well as the downstream task of scene-stable panoptic segmentation, significantly outperforming previous state-of-the-art.

LoCo: Learning 3D Location-Consistent Image Features with a Memory-Efficient Ranking Loss
Dominik A. Kloepfer
João Henriques
Dylan Camphell School of Computing
Visual Geometry Group University of Oxford dominikGrobots. ox ac uk
Visual Geometry Group University of Oxford joao@robots. ox ac uk
Australian National University
dylan campbel @anu, edu.
Abstract
Image feature extractors are rendered substantially more useful if different views of the same 3D location yield similar features while still being distinct from other locations. A feature extractor that achieves this goal even under significant viewpoint changes must recognise not just semantic categories in scene, but also understand how different objects relate to each other in three dimensions. Existing work addresses this task by posing as patch retrieval problem, training the extracted features to facilitate retrieval all image patches that project from the same 3D location. However, this approach uses loss formulation that requires substantial memory and computation resources, limiting its applicability for large- scale training. We present method for memory-efficien learning of location- consistent features that reformulates and approximates the smooth average precision objective. This novel loss function enables improvements in memory efficiency by three orders of magnitude, mitigating a key bottleneck of previous methods and allowing much larger models be trained with the same computational resources. We showcase the improved location consistency of our trained feature extractor directly on multi-vier -iet consistency task, as well as the downstream task of scene- stable panoptic segmentation, significantly outperforming previous state-of-the-art
Our approach-Loor features. That is, features that backproject to nearby 3D locations are encouraged to have similar image patch features (illustrated here by the pair of blue stacked cubes and the pair of red stacked cubes), while those that backproject to well-separated points are trained to have more dissimilar features (here, the blue red cube stacks). This is achieved via a novel ranking loss that reformulates
Figure 1:
memory-efficien learning of location-consisentt (LoCo)
and corrects the smooth average precision loss proposed in previous work [4, (4)26)) This facilitates the derivation of close approximation the loss that significantly more efficient compute, allowing the method to scale to much larger models with the same computational resources.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

Introduction
Reasoning in 3D is critical for developing useful visual understanding of an environment. However, image-centric approaches, including patch feature extractors like DINO [1]. are not 3D-consistent, as recent paper by El Banani et al. [12] demonstrates. That is, the same 3D location may yield significantly different features from different viewpoints in space and time, due to occlusions, self- occlusion, reflections, lighting variations, and motion. When this occurs, it is challenging maintain a spatially and temporally consistent model of the world. Thus, it is useful to first convert visual observations into a form that is stable across different spatio-temporal viewpoints.
40 aggregate or distil visual features in 3D, ensuring view- consistency at the cost of requiring a full 3D reconstruction pipeline. In contrast, recent work [26] explores more flexible image-centered representation that encourages similarity between image patch features that backproject to the same region of 3D space, within a spatial tolerance. It employs a ranking- based loss function, smooth Average Precision (sAP)), to encourage all spatially co-located features to be similar and all non-colocated features to be dissimilar, as illustrated in Fig.
Existing approaches 3 18
29
Like this approach, we formulate the learning problem as one of patch retrieval: given one image patch, retrieve with high precision and recall all patches in other views that project from the same 3D region. However, the smooth Average Precision (sAP) loss function requires substantial memory and computation, precluding its deployment large-scale training. By rewriting the loss function in terms of pairs of image patches rather than individual patches, we derive a more general form of the sAP objective that lends itself to approximation. This novel formulation enables improvements in memory efficiency by three orders of magnitude, mitigating a key bottleneck and allowing larger models to be trained with the same computational resources. By applying this novel loss function within new training strategy, we obtain method for memory-efficien learning of pocationcconistten (LoCo) features that are semantically-nn maaaiiinllyymmminnfllllll and stable across viewpoints. Our contributions are:
novel reformulation and approximation of the smooth average precision loss function that can be computed significantly more efficiently than the original; training strategy for scalable and memory-efficient learning of location-consistent image fea- tures; and
Applications to pixel correspondence estimation and scene-stable panoptic segmentation.
The approach is evaluated on a recently proposed multi-view consistency task [12] and is tested on two real-world indoor datasets, significantly outperforming state-of-the-art feature extractors.
2 Related Work
The topics of panoptic image segmentation 25
visual place recognition 23).23], image retrieval 4 B 32), and visual feature learning 5 16 are well-studied. Here, we focus on the most recent
and related work.
Image segmentation. Visual features from pre-trained models were shown to be very useful for Interestingly, location-consistee visual features unlock the possibility of "scene-stable panoptic segmentation", where instance IDs are consistent across multiple views of the same scene [26]. This is related to identity-preserving video segmentation (24) where object identities are tracked, leveraging temporal smoothness.
panoptic (instance and semantic) image segmentation [5] 25 43
Image retrieval. Learning representations that facilitate the ranking of images according their relevance to a query has been studied extensively ( 32). One class of approaches, metric learning, uses contrastive losses [8 42) to encourage positive instances be close and negative instances to be further apart, while others optimise ranking-based metrics like Average Precision (AP) directly 4 34). For example, Smooth-AP 4 recommends the use of an approximated AP ranking function, which targets the correct ranking without being concerned with the absolute feature distances. We pose our
feature learning problem as a patch retrieval problem, allowing us to adapt strategies from the image retrieval literature.
Self-supervised visual feature learning. Self-supervision has emerged as a dominant strategy for training foundation models in computer vision on large-scale image datasets. Notably, DINO framework to learn to extract semantically meaningful feature
leveraged a mnowledge-distillan maps. Building upon DINO's foundation, DINOv2 [ refines the training algorithm and scales up model sizes on larger datasets, resulting in enhanced performance. Another noteworthy approach

---

is the Masked Autoencoder (MAE) [7], which employs an autoencoder architecture to reconstruct masked-out patches, and demonstrated the scalability of autoencoders large datasets.
In a similar vein, CroCo 4 and its successor, CroCo-v2 [44, also adopt an reconstructs architecture for reconstructing masked-out patches. However, in these works the decoder the patches based on feature map extracted from a non-masked image of the same scene but observed from a different viewpoint. While we also use different viewpoints as the source of the supervision signal like CroCo, the learning process quite different and makes more explicit use of geometric constraints in forming positive and negative pairs of patches (ef'. Section 3.2). Most closely related to our work LoCUS [26). which uses a similar problem set-up and loss function. Where they approach the task from the perspective of extracting distinctive landmarks (individual patches or points in the scene), our areas of interest are pairs of patches. Crucially, this allows us to massively decrease the algorithm's 's memory consumption and unlock significant performance gains.
We provide further detail in Section 3
Other works [27] 29 40) distil visual features in 3D in order to ensure craattoncoonsistencyy the same goal our approach. For example, N3F [40] distilled DINO image features into 3D feature field using the same rendering loss as NeRF [30]. While the resulting features are 3D-consistent by design, this comes at the cost of requiring full 3D reconstruction pipeline. In contrast, our approach image-centric and lightweight, predicting location-consistent features from one image at a time without requiring input poses re-training for new scenes.
Efficiently Learning LoCo Image Features
In this section, we outline our method for learning 3D location-consisennt (LoCo) image features in a scalable, memory-efficiem way. We first formalise the problem definition and define the positive and negative sets, then we reformulate the retrieval objective function and show how this facilitates very significant reductions in the memory requirements. This allows method to overcome critical bottleneck in the learning process for these features, permitting us to scale up the model size. Sec. shows that this has large impact on performance, justifying the need for the reformulation.
3.1 Problem Definition
Our goal is to extract feature maps so that the extracted features are similar for sets of image patches that depict the same region in 3D space. Like Kloepfer et al. (26), make this task tractable by recognising that feature maps solving this task can also be used for patch retrieval: given an image patch and associated feature vector, retrieve all patches that project from the same region in 3D space. This allows us to adapt methods from the extensive literature on image retrieval.
Formally, given a set I of training images I, E R Hx x3 drawn from an environment e, we aim to train feature extractor & I > R hx Here, h < H and w S W the height and width of the extracted feature map, and d is the dimension of the extracted feature vectors, so each feature vector is associated with image patch HĄ C I, consisting of 4 W. pixels.
We denote image patch as Ik. the associated feature vector 6,, patch pair as C. (z,, and the associated cosine similarity score between the feature vectors of patch pair as sa = 0. ||) We use Greek subscripts index pairs of patches, rather than individual patches.
Each patch zi E I; is also associated with point Pk R in the environment, obtained by backprojecting the centre coordinates of the patch into the environment using estimated or provided camera intrinsic and extrinsic parameters and depth [15]. This auxiliary information only required at training time; at test time the trained feature extractor & only requires raw RGB image.
3.2 Positive and Negative Sets
we define two patches 2; and Zy. not necessarily from the same image, as a 'positive' pair if and only if they are drawn from images in the same environment and the distance between their associated 3D points p: and p5 is below a threshold p. Therefore, given the index of the environment associated to each patch 2;, the set of all positive pairs is
As illustrated Fig. 2a
IP {x,, .; llp py & p A e; = e,
Pairs of patches that not observe the same 3D region form 'negative' pair. As a form of hard negative mining, restrict our attention to those negative pairs whose patches depict locations
3

---

P- ((1 (2). N (sa
0:66--52
TayBy
Re.FA
(a) Defining the positive and negative sets.
(b) Memory-effiiiett strategy for smooth AP.
Figure 2: (a) The distance between 3D points p; associated with patch pairs (I1, 25z in the positive set is less than p. denoted by the green arrows connecting them. The distance between those in the negative set is in (p. ki], denoted by the red arrows connecting them. All unconnected 3D point pairs are in neither the positive nor negative set, since they are separated by a distance greater than k. (b) When the absolute similarity difference [sg sal between image pair and pair 3 is large, the sigmoid in the loss function (blue curve) becomes saturated and does not impact learning. We can avoid the memory cost of pack-propagation in these cases by separating the positive P and negative pairs into subsets: saturated below, unsaturated, and saturated above. We choose the saturation threshold A such that the sigmoid gradient there is 0.2% of its maximum value.
within a distance of of each other, and define the set of negative pairs as N = {z r2): llp- - 225l
e; = =.}. For convenience we also define the set of training pairs 9 P N. For given patch 2,, the thresholds p and define a positive and a negative region around the associated 3D point ps. All other patches that observe points inside the radius-p sphere centred at P: will form a positive patch pair with T;, Likewise, all other patches that observe points outside this sphere but inside the radius-k sphere centred at p; will form negative patch pair with The features of the positive pairs are encouraged to be more similar than those of the negative pairs.
should also be noted that the finite size of these regions means that the training robust noise in the depth maps and camera poses used to compute the patch locations in 3D. Noisy patch locations effectively only slightly change the size of these regions and so we expect them not to fundamentally alter the learning algorithm. This could open the door using less accurate estimates for depth and camera poses in future, which may be easier to obtain, particular for large-scale datasets.
3.3 A Ranking Loss Function for Patch Retrieval
The smooth average precision (AP) loss function, originally introduced in Brown al. 4. was adapted to the setting of patch retrieval around "tentative 3D landmarks" by Kloepfer el al. (26). resulting vectorised form of the loss function. We streamline this setting by eliminating the need for these landmarks, and instead focus exclusively on retrieving positive pairs of patches.
The aforementioned vectorised smooth AP loss function can be rewritten terms of patch pairs as
1 EeseMics 0r(83 sa)
[(2;T,P,1))
1 E
ZegeRi (co) 0r(8n sa)
Ca
E EeseM T- (59 - sa)
CP (pEP)(cw) (3 - sa) + + Ec,EN o (s, - sa)
where (I) = (1 + exp(-x/7))-1 is the sigmoid function with temperature
This loss function computes a differentiable approximation to the average precision of binary classifier that classifies pairs of patches as positive or negative based on the similarity of each pair. This approximation becomes exact as 7 0 and the sigmoid approaches the indicator function. For each positive pair C., Eq. (2) calculates the ratio of the rank of c, among all positive pairs and its rank among all pairs (positive and negative) when ranking the pairs by decreasing similarity. More details can be found in Brown et al. [4].
Compared with standard contrastive losses, like triplet loss 19 or SimCLR 77. the gradient ac/Os, of this ranking loss with respect to the similarity of positive pair ca will disappear as

---

soon as Sa is higher than the similarities of all negative pairs. That is, it does not force + +1 for positive pairs and Sn + -1 for negative pairs, merely encourages some boundary to exist somewhere between positive and negative pair similarities. This "gentler" contrastive characteristic greatly aids training convergence [26].
3.4 Correction Terms for the Batched Loss Function
The sums in Eq. (2 run over all pairs of (positive) patches the training set. Clearly, computing the exact loss function is completely infeasible for large datasets. However, when sampling batches of positive and negative pairs, correction terms are needed to make the expectation value of the batched loss equal the exact (unbatched) loss. In partieular, some care needs to be taken since each term in the loss function depends on multiple different samples within the batch. Using the subscript B refer to the batched versions of the sets I' C I and Ng C of positive and negative pairs, the batched version of our loss function becomes
DegEFal(oa) (r sa)
Ln
Σ
PB
1+
EeseP 0-(53 (-(38-8a)) sa)+ Nal
[8|
Ec,ENn (r sa)
c. C.E
PPl
The correction factors
NN Ps 18 necessary ensure that the expectation value of the batched loss is as close as possible to the loss computed across the entire dataset. This automatically the case for standard loss functions that average over per-sample loss, due to the linearity of the expected value. However, since the ranking loss computes the (non-linear) ratio of expectations over samples, this linearity is lost. Each loss term depends on multiple pairs, and the +1 terms in numerator and denominator introduce additional complications for finding an unbiased estimator. For detailed derivation of Eq. we refer the reader to Appendix B where show that it is ratio estimator 377 that is simple to calculate and consistent, but has bias of order (11[[Q
Pl
and
We note here that neither Brown el which cause the losses in those works to deviate even further from the desired average precision approximation. We quantify this deviation in the appendix.
nor Kloepfer al. (26] include these correction factors,
3.5 Improving the Memory Efficiency
A key bottleneck of the loss in Eq. (3) is the memory consumption of the |PA| x (IPnl + ())) matrix containing all the values of s- S., and s.y Sax and the associated computation graph. Since each occurrence of the similarity of particular patch pair in the loss function only provides a supervision signal for the feature vectors of two individual patches, the batches of positive and negative patch pairs need to be quite large (Kloepfer et al. (26) use |P R 13, 000 and N z 100, 000).
To alleviate this problem, we design two ways to significantly reduce the memory consumption of this matrix. First, we observe that the positive pairs c, in Eq. (3) do not need be drawn from the same subset of all positive pairs as the ones in cg. As long as both are sampled uniformly from the set of all positive pairs, the expected value of the batched loss continues to equal the non-batched loss of Eq. (2). Sampling ca from a small set of positive pairs P'B C P and cg from a large set FB C P. with IP B < ||. reduces the size of the matrix of similarity differences to IP's x (IIP nl + [N])): only the second dimension is large, while previously both dimensions were large and of comparable size. At the same time, this still retains supervision signal for a large number of feature vectors, namely those used to construct the larger sets IPB and Ng-
Second, we observe that, in practice, a large number of the computed similarity differences saturate the sigmoid function. That is, when [8,3 sal >> O, the gradient of the sigmoid Var(ss - 8.) vanishes: these terms make no material difference to the loss gradient. We can use this fact to significantly reduce the number of similarity differences the computation graph. To do so, we set a threshold A > 0 and use the approximation, visualised in Fig. 2b
if sp 8y > A
0r(sp Sv) 2 o (sp sy)
if p sy < A
(4)
sp Sy -A
5

---

to divide the uniformly sampled sets of patch pairs Pg and Ng into three subsets for each Co E P
Ng N UNB UN
(5)
N B {cg E NB: 53 $a > 4},
(6)
B {cB NB [8 5- 2 A},
(7)
NB {co E NB: 53 sa < AAA,
(8)
and similarly for Pg- We now significantly reduce the number of similarity differences in the computation graph by only including patch pairs from Ng and P in the loss computation, to
(P [ 1) in total. Computing these subsets for each C still requires us to calculate Ex.er,
all [I x (IPs) + (Nal) similarity differences, but because most pairs' gradients are close to they not used to compute the loss any gradients, so these parts of the computational graph can be deleted, leading to substantial memory savings.
To compensate for the non-uniform sampling of F and 2 we also need to add additional correction terms to our loss function. Our final memorry-efficieee batched loss function is given by
Σ
Eg 0r - sa) 5%
LB
Σ EF
FB
P'B
Σ ogET
o (sg sa)
+
Σ 0r (1.,-5a) + + da
Pnl
Nol
Cy C
with the correction terms given by
N
and 8-- = I
(10)
B
For detailed derivation of these correction terms, well as derivation of an upper bound on the error due to the approximation in Eq. (4), we refer the reader to Appendix C
Restricting our sampling of patch pairs to those whose similarities fall within a certain range is reminiscent of hard negative mining, as previously employed in the context of contrastive learning by, egg,, Robinson et al. [33.. However, standard approaches are not easily applicable to our loss function since it operates on pairs rather than individual samples. Furthermore, both motivation and effect of our approach differs from hard negative mining methods. We discuss this further in Appendix EE] where we also compare empirically with standard contrastive learning and hard negative mining.
4 Experiments
In this section, present experiments, where evaluate the performance of LoCo features at multi-view consistency task and at scene-stable panoptic segmentation.
4.1 Experimental Setup
The training dataset images to 256 x 320 pixels. The Matterport3D dataset is particularly suitable for our task of enforcing multi-view consistency due to its diversity and the way it captures varied viewpoints of the same scene through panorama cropping. Datasets such as ScanNet [T0] provide less viewpoint variation per scene due their trajectory-basee data collection. Contrary to Kloepfer et al. [26), we use all available images in the Matterport3D training scenes instead of restricting to images taken in the horizontal plane.
comprises 59 environments of the Matterport3D dataset, resizing the
Due limited computational resources, we were unable to use our loss function to train a full foundation model from scratch. Instead, we adapt the architecture used by DINO-Tracker [41]. keeping pre-trained DINO [3] features frozen and training a convolutional neural network to learn additive residuals to those features. We use values of p 0.5m for the positive radius, к 5.0m for the negative radius, 7 0.01 for the sigmoid temperature, and A 0.076 for the saturation threshold. With these values, o, ()) 0.9995 and the gradient is 0.2% of the maximum gradient of the sigmoid function, making this conservative choice with little impact on the training dynamics. The threshold A is a hyperparameter that can be decreased to obtain further memory savings, at the expense of some performance decrease. We provide further implementation details regarding the efficient sampling of patch pairs in Appendix D and will publicly release our training code.

---

Table 1: Results on the pixel correspondence task on the Paired split [36 of ScanNet [[0], as
introduced by El Banani et al. [1]. We report the recall of accurate pixel correspondences at
reprojection error threshold of 10 pixels, for image pairs with the respective viewpoint changes. We
also report the GPU Memory required for training LoCUS-based and LoCo models (for LoCUS we use the authors' values). uses 64-dimensional feature vectors.
Model
GPU
-15
-5--30-
30--60"
600-1800
Memory
LoCUS!
11GB
23.5
18.8
DINO 5
45.0
34.3
22.6
10.7
DINOv2 3
37.0
27.5
19.7
11.2
DINOv2
Blocks 1-6
47.1
36.4
22.4
8.4
CroCo-v2
4
16.8
12.4
7.4
3.7
LoCo (A 0.076) (Ours)
48GB
61.8
52.7
31.8
10.3
LoCo 0.053)
42GB
59.9
49.8
29.1
LoCo (A = 0.029)
40GB
$7.9
47.5
28.1
9.6
LoCo (w/ DINOv2 backbone)
38GB
58.2
49.2
30,0
10.5
LoCo (w/ LoCUS architecture)"
6GB
27.8
18.0
12.1
6.9
4.2
Baselines
We compare our LoCo features those from several state-of-the-arr feature extractors, as well as models specialised to the downstream segmentation task.
DINO [5). We use the pre-trained ViT-base model and extract f68-dimensional features for each patch 8 pixels by discarding the class token and reshaping the output of the final Transformer- Block into feature map.
DINOv2 31 We use the pre-trained ViT-base model and extract 768-dimensional features for each the output of the final Tranformer-Block We generally use higher-resolution images as inputs for this model to extract feature maps of the same shape as for LoCo and DINO features.
patch of 14 14 pixels by again discarding class token and reshaping
CroCo-v2 [44]. We use the ViT-base encoder that was pre-trained with the Case-Decoder and extract 768 768-dimensional features for each patch of 16 16 pixels, again using higher-resolution inputs.
LoCUS [26). We use the pre-trained LoCUS weights that publicly available for landmark radius of 0.2m. We use it to extract 64-diimessional feature vectors for each patch of 8 x 8 pixels.
MaskDINO 28 For the scene-stable object segmentation task (ef- Section 4.4 we also compare with MaskDINO,, state-of-the-arr specialised panoptic segmentation algorithm. however is not designed to recognise the same object in different images. We therefore match the per-image object indices produced by the algorithm the ground-truth per-scene object index whose mask has the highest IoU with the object mask in question.
4.3 Multi- View Consistency
We first directly evaluate the location consistency of the features extracted by different models across different views. To do so, we follow the protocol introduced by El Banani al. [I2 test the multi-view consistency of feature extractors on pixel correspondence estimation task. Brieffy, given a pair of images, we extract a fixed number pixel matches by filtering the nearest neighbour matches using ratio test. For more details, the reader is referred to the original paper [12].
Like El Banani et al., we evaluate on the Paired ScanNet [10] split proposed by Sarlin et al. [36). reporting the recall a reprojection error threshold 10 pixels for different viewpoint changes in Table I As we can see, our model outperforms the baselines by significant margin. We also see that DINOv2 [31] performs worse than the original DINO [5] This situation is somewhat reversed when sourcing the features from the first six Transformer blocks (instead of the final feature map). This suggests that while earlier layers of DINOv2 are still reasonably location-consistent, later layers create more semantically meaningful features that accordingly do not vary much by the patches' location in 3D space, explaining why they perform worse in this setting. CroCo-v2 44 performs relatively poorly on this task. despite its training objective being explicitly designed for multi-view

---

Table Scene-stable panoptic segmentation results on unseen Matterport3D 6 and ScanNet [T0] environments. Except for MaskDINO, each method extracts d-dimensional feature vectors for 30 x 40 patches that are then classified into a scene-wide object index using a linear probe. The feature dimension is d = 768, except for LoCUS (d 64) due to its high memory consumption. *Per-image instance indices are matched the ground truth per-scene indices based on mask IoU.
Matterport3D loU
ScanNet IoU
Model
Jac
AP
Jac
AP
LoCUS 26
64
28.6
29.6
40.5
68.9
59.2
68.7
DINO 5
768
65.2
65.5
81.6
81.8
73.7
84.9
DINOv2 31
768
65.9
60.0
80.5
79.7
71.1
82.
CroCo-v2 49
768
65.5
67.0
87.0
81.5
73.8
89.4
MaskDINO
28
768
54.8
38.3
35.0
58.7
39.7
36.5
LoCo (Ours)
768
66.5
67.5
84.5
83.5
76.3
88,8
LoCo (w/ DINOv2 backbone)
768
61.1
58.5
80,4
80.4
71.0
83.8
LoCo (w/ LoCUS architecture)
10.5
7.3
11.2
58.9
47.3
58.1
tasks. However, when training CroCo, the encoder features are first used by a transformer-bas decoder module before loss function is applied. This means that there is incentive for CroCo features to location-consistee under a simple and interpretable cosine-similarity operation, and would require complex adapter to support similarity-basee operations.
4.4 Scene-Stable Panoptic Segmentation
The task of scene-stable panoptic segmentation was originally introduced by Kloepfer et al. [26]. Given a of images single scene, the goal create segmentation mask for the objects in each of these images, where, crucially, different views of the same individual object are labelled consistently with the same identity.
Formally, for set I {4: of images of the same scene, set {0, (o,...,, L semantic classes that is split into 'stuff" (amorphous classes such as floor, walls, etc) and 'things' (clearly distinct objects) subsets Ca and Cth respectively. The latter are also split into set O (O...., N - 1 of object instance IDs within the scene. The goal is to map each pixel Pj to its semantic class if e, E C*, and to its scene-wide object instance o, if ey E Chh This is similar to the standard panoptic segmentation task [25] with the crucial difference that the object instance indices are consistent across different images of the same scene.
Ca U of
This task requires the algorithm to not just differentiate between multiple object instances of the same object class in the conventional instance segmentation task, but also recognise when different images show the same object, which requires a broader understanding of the scene geometry. The need retain consistent object identities across different images is reminiscent of panoptic video segmentation [24]. However, in our task the images are unordered and have much larger viewpoint changes, so methods cannot rely on pixel tracking or optical flow keep object identities consistent.
As standard in self-supervised learning from the feature vectors for every image patch.
16 26, 31,
we train a linear probe predict class labels
4.4.1 Datasets.
Both the Matterport3D 6 and ScanNet [10] datasets provide 3D-mesh reconstructions of their constituent environments, segmented into individual objects. This allows to generate ground-truth segmentation masks for consssconiittet object segmentation by finding the individual objects in each scene that ray through given pixel intersects. Scripts generate this data will be included in our code release. For the Matterport31) dataset, we evaluate on 18 unseen scenes, and for ScanNet evaluate 21 unseen scenes selected to show a range of different types of environments.
4.4.2 Results.
We report our results on the Matterport3 dataset and on the ScanNet dataset in Section 4.4. We measure the scene-stable panoptic segmentation performance using three metrics, which we caleulate for cach object instance individually (for each object instance treating the segmentation masks as

---

Figure 3: Scene-stable object segmentations for three images drawn from the Matterport3D [6] dataset. Ground-truth segmentations in the top predicted segmentations in the bottom row. The object identities and segmentation masks remain stable across significant viewpoint changes.
binary) and then average first across object instances in each scene and then across scenes. The Jaccard index (Jac) for each object instance is calculated by TP/(PP + FN), given the counts for True Positive (TP), False Positive (FP), and False Negative (FN) predictions. The intersection-ove--Unee with the ground-truth masks. The Average Precision (AP) that
(loU) is the intersection-over-uiiie of the linear classifier in a one-vs-all mode, taking all other pixels as negative labels.
Our LoCo-trained features perform better than all baselines, and have comparable performance to CroCo-v2 44], which also makes use of multi-view supervision. However, ours has far fewer trainable parameters (only 28.9 million), since only train comparatively small CNN. In contrast, CroCo trains the entire network (85 million parameters) with a multi-view loss and on significantly larger datasets with greater computational resources (8 A100 GPUs vs. RTX8000 GPU). We note also the strong performance of the original DINO compared to the newer DINOv2 method. As in the pixel correspondence task, this might be due the final DINOv2 features focusing semantic meaning, and so struggling to differentiate between, e.g., different chairs.
4.5 Ablations
DINOv2 Backbone. We also train our method using frozen DINOv2 [I] backbone, again training a convolutional neural network to learn additive residuals and keeping other hyperparameters the same.
The resulting features significantly outperform the original pre- preitrained DINOv2 features for finding accurate pixel correspondences (Table 1.. showing the advantage of LoCo-training in tasks that require location-consistent features. However, slightly underperforms the LoCo model trained with the KINO-ViT-BaL backbone. We hypothesize that this arises from the coarser feature map of the DINOv2- S-D02--VT-Baaae 14 backbone (with patch size of 14 instead of 8).
On the panoptic scene-stable segmentation task (Section 4.4.1) the LoCo model trained with the DINOv2 backbone only outperforms the original DINOv2 feature extractor on some of the metrics. This is likely attributable the coarser feature map of this backbone, which leads less fine-grained patch-level supervision during training.
LoCUS Architecture. To further investigate the impact of our alterations to the loss function and training algorithm compared to Kloepfer al. (26), train the original LoCUS architecture with the LoCo loss function and algorithm.
On the multi-view consistency task (Table M]. this model outperforms the original LoCUS model for small viewpoint changes, but underperforms for image pairs with larger viewpoint changes.
In fact, the LoCUS architecture trained with the LoCo-algorithm performs worse than the original LoCUS model on the panoptic scene-ste stable segmentation task (Section 4.4.I).
For this ablation, we trained for the same number of epochs as our other LoCo models, so possible that the vision transformer blocks in the LoCUS architecture require longer training times than the

---

convolutional layers of the LoCo models. In any case, this ablation illustrates that the improvements memory efficiency do not by themselves lead to improvements in performance. Their advantage is that they allow for the training of larger models and dingher-dmmesiioall feature vectors with the same computational budget, the effect of which far outweighs any performance decrease due to our loss function and training algorithm changes.
Effect of A. We analyse the impact of restricting further the range of similarities from which N and P are sampled by decreasing the saturation threshold A. The results of are shown in the last two rows of Table [ We that while there is small decrease in performance as the threshold decreases, overall, the training is remarkably robust to more aggressive filtering of patch pairs. This confirms the intuition that only small number of patch pairs contribute meaningfully the gradient and that most patch pairs can be discarded without significantly impacting the training behaviour.
4.6 Memory Efficiency Analysis
Using their training code, find that the hyperparameters used in Kloepfer al. (26 result in positive and negative pair set sizes of |Pal = 13, 000 and [Ngl = 98, 000. The resulting matrix size, using single- pessee--eeiioonn floating point numbers, of |P x (P + nl) 51GB exceeds most computational limits, so the authors subsample 10% of the negative pairs to reduce the matrix size to 5.7GB for a matrix with 1.4 billion entries. We instead use value of P 32 and find empirically that even with the conservative value A, roughly 80% of the pair differences are vell-approxim as having zero-gen grro-gradient. This means that [P = (L - 0.8) B and [ = (1 ) [ Bl. Assuming the same computational budget of 5.7GB. our method can therefore use batches I' B and Ng that are larger by factor of ~ 2000. In our experiments, we limit [ < 800 and [ 3, 000, resulting in the matrix occupying only 500KB of memory, thereby freeing up GPU memory to train models with more parameters and with substantially larger feature vectors.
Conclusion
In this paper, have proposed method for the memory-effiieent learning of location-consisten features. In particular, we present a reformulation of the smooth average precision ranking loss that corrects for biases induced by batching, and introduce an approximation that facilitates significant memory reductions without distorting the training signal. This mitigated a key memory bottleneck, allowing larger models to be trained with the same computational resources. Equipped with this novel retrieval-based objective function, we are able to efficiently learn to modulate DINO 5 31
ViT features towards
Our feature extractor demonstrates compelling performance on the downstream tasks of scene-stable panoptic segmentation and visual place recognition, outperforming previous state-of-the-art feature extractors. This work goes some way towards scaling up the training pipeline; however, there is significant scope for applying these techniques on truly large scale image or video data in an entirely self-supervised manner by estimating depth maps and camera poses using off-the-shelff methods.
Acknowledgements. The authors acknowledge the generous support of the Royal Academy of Engineering
and EPSRC (VisualAI, EP/T028572/1.
References
[1] Amar Ali-bey, Brahim Chaib-draa, and Philippe Giguère. MixVPR: feature mixing for visual place recognition. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pages 2998-3007, 2023.
[2] Relja Aranjjelovic, Petr Gronat, Akihiko Torii, Tomas Pajdla, and Josef Sivic. NetVLAD: CNN architecture for weakly supervised place recognition. In CVPR, pages 5297-5307, 2016.
[3] Samarth Brahmbhatt, Jinwei Gu, Kihwan Kim, James Hays, and Jan Kautz. Geometry-aware learning of maps for camera localization. In CVPR, pages 2616-2625, 2018
[4] Andrew Brown, Weidi Xie, Vicky Kalogeiton, and Andrew Zisserman, Smooth-ap: Smoothing the path towards large-scale image retrieval. In ECCV, 2020.
10

---

[5] Mathilde Caron, Hugo Touvron. Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties in self sef--uuerviied vision transformers. In Proceedings of the IEEF/CVF international conference on computer vision, pages 9650-9660, 2021.
[6] Angel Chang, Angela Dai, Thomas Funkhouser, Maciej Halber, Matthias Niessner, Manolis Savva, Shuran Song, Andy Zeng, and Yinda Zhang. Matterport3d: Learning from rgb-d data in indoor environments. International Conference on 3D Vision (3DV), 2017.
[7] Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. International conference on machine learning, pages 1597-1607. PMLR, 2020.
[8] Sumit Chopra, Raia Hadsell, and Yann LeCun. Learning a similarity metric discriminativell, with application to face verification. In CVPR, volume 1, pages 539-546. IEEE, 2005,
[9] William Gemmell Cochran. Sampling techniques. john wiley sons, 1977.
[10] Angela Dai, Angel X Chang, Manolis Savva, Maciej Halber, Thomas Funkhouser, and Matthias Nießner. Scannet: Richly-annotated 3d reconstructions of indoor scenes. Proceedings of the IEEE conference computer vision and pattern recognition, pages 5828-5839 2017.
[11] Matthijs Douze, Alexandr Guzhva, Chengqi Deng, Jeff Johnson, Gergely Szilvasy, Pierre- Emmanuel Mazaré, Maria Lomeli, Lucas Hosseini, and Hervé Jégou. The faiss library. 2024.
[12] Mohamed EI Banani, Amit Raj. Kevis-Kokitsi Maninis, Abhishek Kar, Yuanzhen Li, Michael Rubinstein, Deqing Sun, Leonidas Guibas, Justin Johnson, and Varun Jampani. Probing the 3D Awareness of Visual Foundation Models. In CVPR, 2024.
[13] Albert Gordo, Jon Almazán, Jerome Revaud, and Diane Larlus. Deep image retrieval: Learning global representations for image search. In ECCV, pages 241-257. Springer, 2016.
[14] Michael Gutmann and Aapo Hyvärinen. Noise-contrastive estimation: A new estimation principle for unnormalized statistical models. In Proceedings of the thirteenth international con- ference artificial intelligence and statistics, pages 297-304. JMLR Workshop and Conference Proceedings, 2010.
[15] Richard I. Hartley and Andrew Zisserman. Multiple View Geometry in Computer Vision. Cambridge University Press, ISBN: 0521540518, second edition, 2004.
[16] Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. Masked autoencoders are scalable vision learners. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 16000-16009, 2022.
[17] Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. Masked autoencoders are scalable vision learners. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 16000-16009, 2022.
[18] Joao Henriques and Andrea Vedaldi. Mapnet: An allocentric spatial memory for mapping environments. In CVPR, 2018
[19] Elad Hoffer and Nir Ailon. Deep metric learning using triplet network. In Similarity- -assed Pattern Recognition: Third International Workshop, SIMBAD 2015, Copenhagen, Denmark, October 12- 14, 2015. Proceedings 3, pages 84 92. Springer, 2015.
[20] Hervé Jégou, Matthijs Douze, Cordelia Schmid, and Patrick Pérez. Aggregating local descriptors into a compact image representation. 2010 IEEE computer society conference on computer vision and patlern recognition, pages 3304 3311 IEEE, 2010.
[21] Sangryul Jeon, Dongbo Min, Seungryong Kim, and Kwanghoon Sohn. Mining better samples for contrastive learning of temporal correspondence. In 202/ IEFF/CVF Conference on Computer Vision and Puttern Recognition (CVPR), pages 1034-1044, 2021. doi: 10. 10//CV44444.
000190
11

---

[22] Yannis Kalantidis, Mert Bulent Sariyildiz, Noe Pion, Philippe Weinzaepfel, and Diane Larlus. Hard negative mixing for contrastive learning. In Neural Information Processing Systems (NeurIPS), 2020.
[23] Nikhil Keetha, Avneesh Mishra, Jay Karhade, Krishna Murthy Jatavallabhula, Sebastian Scherer, Madhava Krishna, and Sourav Garg. Anyloc: Towards universal visual place recognition. IEEE Robotics and Automation Letters, 2023.
[24] Dahun Kim, Sanghyun Woo, Joon-Young Young Lee, and In So Kweon. Video panoptic segmentation. In Proceedings of the IEEE//CV Conference on Computer Vision and Pattern Recognition, pages 9859-9868, 2020.
[25] Alexander Kirillov, Kaiming He, Ross Girshick, Carsten Rother, and Piotr Dollár, Panoptic segmentation. In Proceedings of the IEEE/CVE conference on computer vision and pattern recognition, pages
2019.
[26] Dominik Kloepfer, Dylan Camphell, and Joao F. Henriques. Locus: Learning multiscale 3d-consistent features from posed images. In ICCV, pages 16634-16644, October 2023.
[27] Sosuke Kobayashi, Eiichi Matsumoto, and Vincent Sitzmann. Decomposing NeRF for editing via feature field distillation. NeurlPS, volume 35, 2022.
[28] Feng Li, Hao Zhang, Huaizhe Xu, Shilong Liu, Lei Zhang, Lionel M Ni, and Heung- Yeung Shum. Mask dino: Towards a unified transformer-bawee framework for object detection and segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 3041-3050, 2023.
[29] Kirill Mazur, Edgar Sucar, and Andrew Davison. Feature-realistic neural fusion for real-time, open set scene understanding. International Conference on Robotics and Automation (ICRA), 2023.
[30] Ben Mildenhall, Pratul Srinivasan, Matthew Taneik, Jonathan T Barron, Ravi Ramamoorthi, and Ren Ng. NeRF: representing scenes as neural radiance fields for view synthesis. In ECCV, pages 405-421. Springer, 2020.
[31] Maxime Oquab, Timothée Darcet, Théo Moutakanni, Huy V. Vo, Marc Szafraniec, Vasil Khalidov, Pierre Fernandez, Daniel Haziza, Francisco Massa, Alaaeldin El-Nouby, Mido Assran, Nicolas Ballas, Wojciech Galuba, Russell Howes, Po-Yao Huang, Shang-Wen Li, Ishan Misra, Michael Rabbat, Vasu Sharma, Gabriel Synnaeve, Hu Xu, Herve Jegou, Julien Mairal, Patrick Labatut, Armand Joulin, and Piotr Bojanowski. DINOv2: Learning robust visual features without supervision. Transactions on Machine Learning Research, 2024. ISSN 2835-8856. URL https:///peerrrviee http://pwwrreeeeeeeeeeeerrrrr/aa wtt../ooooor////d--66
[32] Filip Radenović, Giorgos Tolias, and Ondřej Chum. Cnn image retrieval learns from bow: Unsupervised fine-tuning with hard examples. In ECCV, pages 3-20. Springer, 2016.
[33] Joshua Robinson, Ching-Yas Yao Chuang, Suvrit Sra, and Stefanie Jegelka. Contrastive learning with hard negative samples. International Conference on Learning Representattions 2021.
[34] Michal Rolínek, Vít Musil, Anselm Paulus, Marin Vlastelica, Claudio Michaelis, and Georg Martius. Optimizing rank-based metrics with blackbox differentiation. In CVPR, pages 7620- 7630, 2020.
[35] Raghavender Sahdev and John K Tsotsos. Indoor place recognition system for localization of mobile robots. In 2016 13th Conference on computer and robot vision (CRV), pages 53-60. IEEE, 2016.
[36] Haul-Edouard Sarlin, Daniel DeTone, Tomasz Malisiewicz, and Andrew Rabinovich. SuperCilue: Learning feature matching with graph neural networks. In CVPR, 2020.
[37] Alastair Scott and Chien-Fu Wu. On the asymptotic distribution of ratio and regression estimators. Journal of the American Statistical Association, 76(373);98-102. 1981.
12

---

[38] Xun Sun, Yuanfan Xie, Pei Luo, and Liang Wang. A dataset for benchmarking image-based lo- calization. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 7436-7444, 2017.
[39] Niko Sünderhauf, Sareh Shirazi, Feras Dayoub, Ben Upcroft, and Michael Milford. On the performance of convnet features for place recognition. In 2015 JEEE/RSJ international conference intelligent robots and systems (IROS), pages 4297-4304. IEEE, 2015.
[40] Vadim Tschemezki, Iro Laina, Diane Larlus, and Andrea Vedaldi. Neural feature fusion fields:
3D distillation of self-supervised 2D image representations. In Proceedings of the International Conference 3D Vision (3DV), 2022.
[41] Narek Tumanyan, Assaf Singer, Shai Bagon, and Tali Dekel. Dino-tracker: Taming dino for self-supervised point tracking in single video, March 2024.
[42] Kilian Q Weinberger, John Blitzer, and Lawrence Saul. Distance metric learning for large margin nearest neighbor classification. NeurIPS, volume 18, 2006.
[43] Philippe Weinzaepfel. Vincent Leroy, Thomas Lucas, Romain Brégier, Yohann Cabon, Vaibhav Arora, Leonid Antsfeld, Boris Chidlovskii, Gabriela Csurka, and Jérôme Revaud. Croco: Self-supervised pre-training for 3d vision tasks by cross-view view completion. Advances Neural Information Processing Systems, 35:3502-3516, 2022.
[44] Philippe Weinzaepfel, Thomas Lucas, Vincent Leroy, Yohann Cabon, Vaibhav Arora, Romain Brégier, Gabriela Csurka, Leonid Antsfeld, Boris Chidlovskii, and Jérôme Revaud. Croco v2: Improved eross-view completion pre-training for stereo matching and optical flow. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 17969- 17980, 2023.
13

---

Table 3: Hyperparameters of the convolutional layers of the residual network used for the pixel-
correspondence task in Section 4.3
Layer
Kerne Size
Output Dimension
Dilation
Downsample Factor
1
5
64
2
5
128
3
5
256
4
5
512
5
5
768
6
5
768
A
Architecture of Residual Network
In Table 3 describe in more detail the architecture of our fully convolutional network that use to compute residuals to frozen DINO [5] features. For the respective layers we downsample the resolution by the given factor using BlurPool layer.
B Correction Factors for Batched Loss
B.1 Derivation of Correction Factors
In the following we will derive the batch correction factors PI/[F/] [| and |N/[N from Sec. 3.4 in the main paper. We will also retain all notation conventions used in the main paper.
We can re-write the Eq. (2) as
L
ELa La
(11)
PP| OoEP
1 Σ LegEPlieo 0r(33 - sa)
with La
1+ Σ
(oo) 0r(33 - sa) + Ec, 0- - 8a)
and similarly Eq. (3) as
PB Σ LBa
LB
(12)
1+ EnsePs 07(38 sa)
with LBa 1+ fe Ecge (5 sa) fa EcyENo o 0r(8 Sa)
where and fs the of yet unknown batch correction factors.
We want choose fo and such that
f = E[Cn)
(13)
From the linearity of the expected value have
E[Cal
Σ C
B[Lgal
(14)
P
Computing the expectation value of LB exactly is challenging, as numerator and denominator are not independent. We approximate IE. using the ratio estimator (37), which for two random variables X and Y approximates
X
E
E
(15)
Y
This estimator can be shown to consistent, though biased. the expectation values E[X] and [E[] estimated using samples, the bias can be shown to decrease O(n [9].
14

---

Applying this estimator to Eq. 14 with
x=1+ fe. Σ
0r(83 Sa)
(16)
CsEPn((co)
and
==1+
E
0r(83 80) + fN Σ
0rl8 8a):
(17)
<pEPa\(eu
c,ERg
we get
1 fr. E EcgePni(cn 0r 8a)
E [CB]
(18)
fz
E
EcgePn (588 sa)
+ fN.B. E EEgeBhn 0z 80))
Since for uniform sampling of pairs from
E fo- sa)l
Σ
(r sa).
(19)
IP| csEM
we obtain
P
E
Σ
(r(33 -a) =
0r -sa)
(20)
PI
CtEPnilen
caEPMe.)
and similarly for E. E results in
EC (1(83- sa)] Plugging this into Eq.
11 and Eq.
13
immediately
P
fe
(21)
IP B
[
fN
(22)
We now investigate the bias due to the approximation Eq. (5). By Taylor expanding Eq. around px and Y = [E[]] = jay to second order we get
X
E[X]
Px
1 (Y uv)
+ (X - -x)
(23)
Y
JYY
NrY
+ マタ (X - xxx) (Y - )x
pxx
My
1
(X (x)(V - py))
higher order terms.
n
Applying the expectation value operator results in
fx
+ E (Y - px) 1x nhz
E
=
(24)
Y
Ny
E[(x - -Y - -uy)]
+ higher order terms.
Using our definitions of X and Y from Eq. 16) and Eq.
we get
P
(X (x)
Σ
0r(30
sa) PP E (EoePP((o)) (o-))
ss)]
(25)
PB
csEFni(es)
(26)
|PB
15

---

because the standard error of the sample mean decreases with the inverse square root of the number of samples. Analogously, we have
(x-
+0
0
(27)
v
N
v122Bl
which then results in
E
1
(28)
Y
12B
The bias therefore decreases with the inverse of the batch size.
B.2 Error of Prior Loss Functions
Previous uses of the Smooth Average Precision loss function in Brown et al. 4 and Kloepfer et al. (26) did not batch correction factors.
The additional bias that results from this grows the batch size decreases, because the batch correction factors decrease for larger batch sizes, and also because for larger batch sizes the sums in numerator and denominator grow and the additional +-1 term that is unaffected by the batch correction factors becomes less and less relevant.
In our experiments, the sums would generally have magnitudes of at least O(10)), so the main function of the batch correction factors is correctly balance the effect of positive and negative pairs in the denominator if fp * fv. This is the case in our experiments, where our hyperparameter choices mean that we generally sample a larger fraction of positive than of negative pairs so that fr/ SN sJ 0.1. Without the batch correction factors, the loss function would place excessive weight on increasing the similarity of positive pairs.
Since these correction factors are simple calculate, is therefore valuable to use the loss function that includes them and that is built on a stronger theoretical foundation.
C Correction Terms for Memory Efficiency Improvements
C.1 Derivation
We derive here the exact form of the correction term 6, in Eq. (9) of the main paper. The derivation for 5tt proceeds entirely analogously.
Using the notation from the main paper, we can write
N
Σ cp
05
sa)
Σ
05 - -sa)
(29)
N
N
N
0r - sa)
Ng
ENA
N
0-(88 - sa).
N
E
Using the approximation from Eq. (4) in the main paper, we have
Veg E No 0-(88 sa) R
(30)
and
Veg E No or (83 - sa) R 1.
(31)
Plugging into Eq. yields
N
N
N
Σ
sa)
o-(53
sa)
(32)
N
N
E
CaENN
with the second term exactly the correction term 8,-
No
N
B
N
16

---

C.2
Approximation Error
Using the sum expansion of Eq. 29
Eq.
12
results in
La
1
Σ
(5g sa)
or
(33)
PB
Cy
P
(r sa)
FB
B
c (c,)
IP
0- 8a)
P
E
+
Σ
o ($g - sa)
PB
Cp ELD
Σ
0-(33 sa)
FB
C {ca)
P
Σ
o (5 -80)
PB
c B
N
Σ
0,33 sa)
N
ENN
N
Σ
0- - 8a)
N
C
N
Σ
0 sal)
B
In the worst case, the inequalities defining No. No. Fg, P are tight, so
Vcg No : (53 sa) = AA
(34)
Ncs E No P or (8g 5,)= = A.
(35)
17

---

Plugging in and using o, (4(4) = = =1-0,(-A) yields
PPP
La
1 +
PB or (-4)
(36)
PP
|P
Σ
(5 80)
[Pa
+ 5, - or (-1))
[P
+
PB or (-4)
(37)
B
|P]
Σ
0 0- 8a)
PB
(ea)
+ 6 0r (-A))
N
1 (-A)
N
N
05 sa)
N
Gp 2
0 - (1-or( (-A))
Taking Taylor expansion with respect to (-(-A) ~ around to first order gives
1
PB
PI
Σ
EPg
EBg Afeo) OT 8a) + 5
BB
(38)
LeaEPg (ea 0r (8g 8a)
+ A4 EcgeNg 0r(0g - sa) + dx + da
Fgl
|P
o (8g 8a)
N B
PP
B
Cy E
IP
+ 8
1 +
P
07 sa)
PB B
PB
esEPo Hea)
N
P
No
1+ +
0r600 8a))
PA
CfEPg Mea
[IP|
2
Σ
T (8g 8n) +
0. - 5.) + + + 5á²
PA
N
cpcPa 3 (ra)
c
+ 0x23)
We see that the loss function used in the main paper (Eq. (9)) re-appears as the zeroth order term of this expansion, while the higher order term(s) provide upper bound for the error due approximating the sigmoids in the worst case.
D Implementation Details for Efficient Patch-Pair Sampling
We pre-compute the 3D world coordinates of each patch in the dataset by backprojecting the centre coordinate into the scene using information about camera pose and depth values at that patch.
During training, we sample the positive patch-pairs ca P by uniformly randomly sampling a patch from a scene and then using the highly efficient FAISS [TT library find and sample a patch within the positive region of the original patch. This approach does not uniformly sample from all positive
18

---

patch- pairs since patches that are part of a larger number of positive pairs are not more likely to be chosen, but computing the number of positive pairs that include given patch to correct for this would be computationally unfeasible. This also does not change our derivations, since all pairs are still sampled from the same (even though non-uniform) distribution.
In each training step, we extract feature maps for only those images that contain the patches for the ca pairs. This is to use our limited computational budget in a way that maximises (P'pl. The pairs Cg and e, E N are then sampled from among the patches in the images for which feature maps have been extracted. This samples using the same distribution as the Co pairs, since for each co the other images in the batch represent merely uniformly random restriction of the patches that can be sampled.
E
Comparison with Hard Negative Mining
E.1
Discussion
One of the contributions of this paper, selecting pairs of patches that do not saturate a sigmoid function in the loss function (cf. sec. 3.5), is reminiscent of the better-known technique of hard negative mining in self-supervised learning 22 33
However, there are three important differences:
Firstly, our method selects pairs of samples rather than individual hard (negative) samples, a result of general reformulation of the problem of learning recatio-coonsssent features. In hard negative mining, given a single anchor sample, other samples are selected based on their features, whereas in our case, given pair of samples (patches), other pairs of patches are selected based on their similarity.
Secondly, while hard negative mining is a one-sided filter, our method is two-sided. We do not just filter out 'too easy" pairs like hard negative mining, i.e. pairs that are easily and correctly classified into positive or negative pairs, but also 'too difficult' pairs, i.e. pairs that are confidently but incorrectly classified. Both kinds of sample pairs saturate the sigmoid function and do not contribute to the gradient. Most standard contrastive learning loss functions do not contain a sigmoid function, so such two-sided filter is not appropriate there.
Thirdly, hard negative mining approaches aim to improve downstream model performance by placing more weight on difficult samples. The hard negative sampling changes the loss function to learn more useful features. In contrast, method aims exclusively to reduce the training algorithm's 's memory consumption without impacting the loss landscape.
E.2 Performance of NCE with Hard Sample Mining
We also trained a model using a more standard contrastive learning loss together with hard mining. We again use modified version of the model architecture of DINO-Tracker [41, training a fully convolutional neural network to learn residual features to frozen DINO 5 features.
14 From batch of 24 images, we sample 256 query patches, for each of which we sample 128 positive and 384 negative patches.
Instead of our loss, we train using the noise contrastive estimation (NCE) loss function
We use hard sample mining for both positive and negative samples. Inspired by Robinson et al. 33 we sample the negative samples randomly from among the patches the negative region with a probability x cxp(3s;j). where Sij is the similarity between the query patch and the potential negative patch. The coefficient 3 is a hyperparameter that we set to 3 = 0. Similarly, we sample the positive patches from among the patches in the positive region with a probability exp(-8s.j This means that we over-sample negative patches with high similarities and positive patches with low similarities, as these are the worst-performing samples.
The results of this model on the pixel correspondence estimation task are shown in Table 4 Though outperforms the baseline models and does comparatively well at large viewpoint changes, our loss is more suitable for learning multi-view consistent features.
19

---

Table 4: Results on the pixel correspondence task on the Paired split [66 of ScanNet [0], as introduced by El Banani et al. [1]. We report the recall of accurate pixel correspondences at reprojection error threshold of 10 pixels, for image pairs with the respective viewpoint changes.
Model
-5-15
-
30--60 60%
60--1800
LoCUST 26
23.5
18.8
13.5
7.5
DINO 5
45.0
34.3
22.6
10.7
DINOv2
31
37.0
27.5
19.7
11.2
DINOv2
31
Blocks 1-6
47.1
36.4
22.4
8.4
CroCo-v2
44
16.8
12.4
7.4
3.7
LoCo (Ours)
61.8
52.7
31.8
10.3
NCE Hard Mining
50.6
40.8
26.7
11.4
F
Visual Place Recognition
We also demonstrate the utility
features on the task Visual Place Recognition (VPR). Given a R of reference images and set Q of query images with QnR 0, ground truth function Q + 2 maps query image to the subset of reference images that show the same 'place' as the query image. Since the reference descriptor methods, which compute global descriptor for each image and for each query image retrieve the most similar reference images.
is typically large, here focus global
Recently, Keetha al. 23 demonstrated that excellent place recognition performance be achieved using a simple recipe: extract patch-level feature maps using a pre-trained feature extractor, then aggregate these into a global image descriptor using the VLAD [20] algorithm. We follow this approach and report the results for different feature extractors, We use 128 clusters for VLAD and construct the VLAD vocabulary only from the respective of reference images.
F11 Datasets.
The indoor datasets that are typically used to evaluate VPR methods (e.g. Baidu Mall 38 Gardens Point [39], and 17Places (35)) tend suffer from one of two defects. For some, like Gardens Point, the development of large-scale pre-trained image models means that VPR performance is strong enough that these datasets can no longer resolve small performance differences, with Keetha al. [23] reporting 99.5% Recall@5 for three variants of their method. Others, like Baidu Mall and 17Places, suffer from poor ground truth. Reference images labelled as 'ground truth' frequently have no image overlap with the query image or do not include images with significant overlap as ground truth. We therefore use the pose and depth information available with the Matterport3I) and ScanNet datasets to construct VPR datasets for which the loU between query and ground-truth images falls between 0.2 and 0.4. This ensures that the place recognition task challenging while remaining tractable for purely vision-based systems. The code to deterministically generate the datasets will be released publicly.
The images that comprise these datasets are drawn from environments that were unseen during training. For Matterport3D, using images from 18 unseen scenes results in 8637 reference and 2194 query images, and for ScanNet, using images from 21 unseen scenes results 4295 reference and 239 query images.
F.2 Results.
We give Visual Place Recognition results Table quoting the Recall@) and Recall@5, While the performance of our LoCo features remains respectable, it does not outperform the baseline feature extractors by as much as scene-stable panoptic segmentation task. This is potentially due the VLAD feature aggregation discarding information about the relative arrangement of different feature vectors the image, thereby removing a significant advantage of location-stable features compared to feature extractors focused more semantic information like DINO.
20

---

Table 5: Visual Place Recognition Results on VPR datasets constructed from unseen images in Matterport3D [6] and ScanNet [TO] MixVPR produces global image descriptors directly, all other methods extract d-dimensional feature vectors for 30 x 40 patches that are then aggregated into global descriptors using VLAD. LoCUST (26 uses d = 64, rather than d = 768.
Matterport3D
ScanNet
Model
@@1
R@5
R@1
R@5
LoCUS 26
20.8
38.4
61.1
79.9
DINO 5
45.7
77.8
81.0
92.9
DINOv2 61
25.0
52.0
59.4
78.2
CroCo-v2
44
26.6
51.6
63.6
81.1
MixVPR 1
26.9
51.4
72.4
84.5
LoCo (Ours)
46.8
79.1
81.2
92.5
G Qualitative Visualisations
In Fig. 4, we show point-cloud reconstructions obtained by projecting the pixel patches of a set of images showing a living room into 3D, and colouring the each point by the similarity of its patch's feature vector with a query feature vector taken from a query patch. The query patch looks at a point on the wall to the top left of the picture hanging on the wall. The upper image shows the similarities as reconstructed using our LoCo method, the lower as reconstructed using DINO (5) features.
The region of space where feature vectors have a high similarity with the query feature is much more localised for method than for the DINO features.
H Limitations
The training algorithm described in the paper assumes the availability of camera poses and depth maps for each RGB image. While number of datasets cxist for whom this information is available, this may not always be the case. There exist number of comparatively accurate algorithms to estimate this information, but the paper does not investigate whether such methods are sufficiently accurate reproduce the stated performance.
The paper also limits itself to training comparatively small convolutional networks, finetuning only a small number of layers of a pre-trained foundation model rather than training a similarly sized model from scratch. There is therefore the possibility that the training algorithm described requires an initialisation with relatively high-quality features and not entirely appropriate for training visual foundation models from scratch.
Thirdly, the paper only runs experiments training comparatively small dataset only containing few dozen different environments. While all testing is performed on unseen environments, the paper does not investigate the existence of scaling laws available training data increases.
I Broader impacts
This work, memory-efficient learning of 3D-consistent image features, has potential positive and negative social impacts. On the positive side, reducing the memory and compute requirements of a common type of self-supervised loss (a ranking loss) permits wider pool of researchers and interest groups to investigate and train models that use such losses, on restricted compute budgets. In addition, self-supervised learning reduces the need for often exploitative human labour practices associated with obtaining cheap labelled data.
On the negative side, training large self self-supervised models has considerable environmental impacts due carbon emissions that lead to significant societal impacts, including direct consequences (displacement, damage from extreme weather events, etc.) and indirect consequences. Moreover, location-consistent image features are likely be useful for tracking, which can be invasive of privacy and can be used for malicious purposes.
21

---

(a) Similarities of the features of patches taken from set of images with query feature vector. using LoCo
features, projected into 3D. The query feature vector's location is to the top left of the picture on the wall.
(b) Similarities of the features of patches taken from a set of images with query feature vector, using DINO S features, projected into 3D. The query feature vector's location is to the top left of the picture the wall.
Figure 4: 3D visualisation coloured by the similarity with respect to a query feature vector.
22

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: Yes, the claim of novel memory-efficien loss function and training strategy accurately reflect the contributions and scope.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: Limitations of the work are briefly discussed in the conclusion and in greater detail in the appendix.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (c.g.. independence assumptions, noiseless settings, model sell-speciiiiatinn asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-tex, system might not be used reliably to provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimesss.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
23

---

Answer: [Yes]
Justification:
Every theoretical result has an associated proof in the appendix.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification: While some details regarding the training setup will only be released to- gether with the documented code-base, the main claims of memory efficiency and general performance can be reproduced with the information provided in the paper.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, other means that appropriate to the research performed.
also be provided via detailed
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reprodueibility may be tricky in some cases, in which authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path reproducing or verifying the results.
5. Open access to data and code
24

---

Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [No]
Justification: At time of submission, have not released the code data, however will do so before publication for reproducibility.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://niis.cc
aublic/guidde//ooeeeuuiiiis ott.eeeuuwuiiiiiooooooooioooog for more details.
While encourage the release of code and data, understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https
/ wttps/cccppplccguua.eeaeaecccuea
witeeee//ooSSSeeeS...ee.a www.ssssiicggcooo for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: We provide this information either in the Experiments section or in the appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented the core the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, appendix, as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [No]
Justification: Error bars are not provided in the paper, due to the significant time cost associated with re-training from different initialisations.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by emoo bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
25

---

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CL, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that out of range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification:
The required GPU resources are outlined in the experiment section.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary failed experiments that didn't 't make into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics
Answer:
[Yes]
Justification:
We conform the NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No. they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
The societal impacts are discussed in the appendix.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
26

---

Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), faireess considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: This paper,
technique for self-supervised learning of 3D-consistent image features, poses no such risks.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification:
We credit the original creators of other's assets and respect their licenses.
Guidelines:
The answer NA means that the paper does use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
27

---

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com/dataseta
latasets.papersstt
has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
this information is not available online, the authors are encouraged to reach out to the asset's s creators.
13. New Assets
Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Question:
Answer: [NA]
The assets introduced in this paper (code and model weights) will be released before publication with full documentation. Since they are not released at the time of submission, this question is answered as NA.
Justification:
Guidelines:
The answer NA means that the paper does release assets.
Researchers should communicate the details the dataset/code/mooell as part their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsoureing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
The paper does not involve crowdsoureing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
28

---

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
29


Reviews:
[{"review_id": "Q8e4NWnwsh", "reviewer": "Reviewer_jhZZ", "summary": "This paper presents a new training scheme for vision foundation models. The key goal of the paper is to enhance the multi-view consistency of vision foundation models. To this end, the paper revisits the idea from soft average precision, and applies the idea for training vision foundation models. Specifically, the loss loosely enforces similar score values for positive patch pairs, unlike explicitly enforcing positive / negative affinities to reach a pre-designated value as in conventional contrastive learning approaches. The authors further propose pruning positive and negative samples during backpropagation to ensure memory efficiency. Experiments demonstrate that the proposed method can outperform tested baselines in local feature matching, panoptic segmentation, and visual place recognition.", "questions": "Please refer to the weaknesses section above. Below are a few additional questions:\n\n1. What are the exact computational requirements of the paper? The only place that I could find hardware requirements was L302. Is a single RTX8000 GPU all we need?\n2. What is the reason for training the model on Matterport3D? I feel datasets such as ScanNet will contain a much more diverse range of scenes. Since the only model being trained is the feature refinement CNN, training on ScanNet will not be prohibitively expensive.", "limitations": "Yes the limitations are stated in the supplementary material.", "rating": 3, "confidence": 5, "soundness": 2, "presentation": 2, "contribution": 1, "strengths": "1. The paper tackles an important task in training vision foundation models, namely enforcing multi-view consistency.\n2. The presentation is clear and straightforward to follow.\n3. The proposed pruning scheme enables training multi-view consistent vision foundation models at a much smaller computational cost than existing approaches.", "weaknesses": "1. My major concern is with the experiments. First, it is unfair to compare existing vision foundation models directly against the proposed method, since the method is additionally trained using Matterport3D data. The baselines in Table 1 should also have been additionally fine-tuned on Matterport3D. Further, since the paper is proposing a generic loss function applicable to any pre-trained vision foundation model, the technical contributions would have been better elucidated if the method improved multi-view consistency for other vision foundation models as well, for example pre-trained DINOv2 features.\n\n2. The motivation for setting the saturation threshold to 0.076 in L220 is unclear. Why is the gradient being 0.2% of the \"maximum gradient of the sigmoid function\" important for training models with the proposed loss?\n\n3. The technical contributions of the paper is unclear. From a methodological point of view, the paper adapts soft average precision loss from prior literature with a threshold-based pruning to ensure memory efficiency, which is not strongly novel. Further, the experimental results are largely limited to fine-tuning a small CNN operating on top of DINO features. Therefore the scalability of the proposed training scheme is unclear.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your thorough review and for highlighting both the strengths and weaknesses of our work. We appreciate your feedback, and we would like to address your concerns in detail below.\n\n---\n\n**Weakness 1: Fairness of comparisons and generalization of the proposed method**\n\nWe understand your concern regarding the comparison with existing vision foundation models. However, we would like to clarify that our training and evaluation strategy is designed to test generalization across diverse scenes and datasets, not just on Matterport3D. Specifically, we evaluate on different environments than those used in training, including environments from a different dataset (ScanNet), ensuring that our network is not simply overfitting on the training set, but instead learning transferable and generalizable features. \n\nRegarding the inclusion of a DINOv2 backbone, we have extended our experimental analysis to include results with a DINOv2 ViT-Base backbone (as described in the global author response). The results show clear performance improvements over the original DINOv2 features, which supports the general applicability of our method across different backbone architectures. This new evidence strengthens the claim that our method improves multi-view consistency in a broader context.\n\n---\n\n**Weakness 2: Motivation for the saturation threshold $\\Delta$**\n\nThe threshold $\\Delta$ is indeed a key hyper-parameter in our method, and its selection is crucial for balancing memory efficiency with the accuracy of the loss function approximation. The chosen value for $\\Delta$ represents a trade-off: a smaller $\\Delta$ leads to greater memory savings but can also introduce distortions to the loss function, affecting the training dynamics.\n\nWe selected $\\Delta$ such that the gradient at $\\sigma(\\Delta)$ is 0.2% of the maximum gradient of the sigmoid function. This choice allows us to achieve significant memory savings (by a factor of approximately 5) while minimizing the distortion of the loss gradients. In this way, we ensure that the memory efficiency gains do not come at the expense of training stability or performance.\n\nThe ablations provided in Table 1 of the paper offer insight into these trade-offs, showing how different values of $\\Delta$ impact both memory usage and model performance.\nWe agree that the explanation for our specific choice could have been more explicit in the paper, and we appreciate your feedback on this. We believe that our current justification is grounded in practical considerations of training efficiency and accuracy, and we have provided empirical evidence to support it.\n\n---\n\n**Weakness 3: Technical contributions and scalability concerns**\n\nWe would like to emphasize that our approach is more than just an adaptation of the LoCUS architecture with threshold-based pruning. The pruning mechanism we propose is based on a detailed analysis of the gradient behavior, and it includes correction terms to ensure that the loss remains *unbiased*. We also address the bias in existing batch implementations of the general Smooth-AP loss, deriving the compensating correction terms necessary if using this loss for mini-batches, as is always the case in practice. This addresses a fundamental flaw in this loss function as used in the literature. Overall, our careful loss design addresses the challenges associated with unbiasing the loss signal, improving the memory efficiency and stabilising the gradients, making our method a significant contribution beyond a simple combination of existing techniques.\n\nRegarding scalability, we agree that larger-scale experiments would be beneficial to fully explore the potential of our approach. However, even within the computational constraints we had, we observed consistent performance improvements with increased model size, as evidenced by our experiments with the LoCUS architecture. This suggests that our method scales well, at least within the range we were able to test. We hope that future work, possibly with greater computational resources, will further validate the scalability of our approach on larger foundation models.\n\n---\n\n**Questions and Additional Clarifications**\n\n1. **Computational Requirements**: As noted in our paper, our experiments were conducted using a single RTX8000 GPU, with each training run taking approximately 1.5 days. This setup reflects the memory efficiency of our method, making it accessible even for researchers with limited computational resources.\n\n2. **Choice of Matterport3D Dataset**: The method most similar to ours, LoCUS, trained its models on the Matterport3D dataset only, so we followed their choice to avoid additional confounding factors in comparing the two methods. The Matterport3D dataset is particularly suitable for our task of enforcing multi-view consistency due to its diversity and the way it captures varied viewpoints of the same scene through panorama cropping. We acknowledge that ScanNet is another valuable dataset; however, due to its trajectory-based data collection, there is less viewpoint variation per scene compared to Matterport3D, which influenced our dataset selection.\n\n---\n\nWe hope these clarifications address your concerns and provide a better understanding of the contributions and significance of our work. We appreciate your thoughtful feedback and believe that our additional explanations and results strengthen the case for our contributions. We respectfully invite you to reconsider your assessment of our work in light of the additional evidence and explanations provided.\n\nThank you once again for your time and effort in reviewing our submission.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "4PEQ899FLN", "reviewer": "Reviewer_YTBi", "summary": "The paper introduces a memory efficient loss for location consistent image features. The paper addresses the problem of high memory footprint of previous work, by significantly reducing the memory footprint, by 3 orders of magnitude. Memory efficiency is achieved by sampling the positive pairs from a smaller subset and using a threshold on similarity differences between pairs, and discarding the pairs with gradients close to 0, which do not contribute to the loss. To compensate for the non-uniform sampling, some correction terms are added to the loss.", "questions": "How does LoCo perform with features coming from different models (e.g. DINO vs DINO v2)?", "limitations": "- limited to datasets with 3D reconstructions available.", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 4, "strengths": "* Significant memory footprint reduction (3 orders of magnitude), allowing for larger batch sizes\n* Identifying that the gradients from most pairs are close to 0 and don't contribute to the losses.\n* Analysis of correction to the batched loss function.", "weaknesses": "* relies on existing 3D mesh reconstruction of the environment (for segmentation masks of individual objects in the scenes) -- this limits evaluation to datasets with existing mesh reconstructions.\n* while direct comparison with LoCUS on the same feature dimension is not possible due to high memory consumption, it would be interesting to see a comparion with LoCo on a feature dimension d=64 (Tab 2) for a fair comparison.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your thoughtful and encouraging review. We appreciate your recognition of the strengths of our work, particularly in terms of the memory efficiency gains and the analysis of the correction to the loss function. Below, we address your concerns and questions in more detail.\n\n---\n\n**Weakness 1: Reliance on existing 3D mesh reconstruction for evaluation**\n\nWe understand your concern regarding the reliance on 3D mesh reconstructions for the Scene-Stable Panoptic Segmentation task. However, we would like to clarify that this requirement is only necessary for *evaluation* purposes, not for training or general application of the LoCo features. The primary goal of this task is to highlight the improved location-consistency of our trained LoCo features compared to existing feature extractors.\n\nWhile it is true that this evaluation necessitates datasets with 3D reconstructions, we believe that this is not a significant limitation. Several existing datasets provide this data, and our method's ability to perform well in this setting demonstrates its robustness and applicability in scenarios requiring high-precision spatial understanding. The improvements in location-consistency, as demonstrated in this task, are transferable to other tasks that do not necessarily require 3D meshes.\n\n---\n\n**Weakness 2: Comparison with LoCo at a feature dimension of d=64**\n\nWe appreciate your suggestion to include a comparison between LoCo and LoCUS at a feature dimension of d=64 for a more direct comparison. In the global author response, we provided additional results for fine-tuning the LoCUS architecture using our memory-efficient loss. While it is true that a direct comparison at a high feature dimension is challenging due to memory constraints, our additional experiments demonstrate that even at a low feature dimension our method offers significant improvements in memory efficiency while maintaining strong performance.\n\nThe improvements seen with larger models trained using our method further validate the value of our modifications to the loss function and training algorithm in overcoming memory limitations. This supports the scalability and robustness of our approach across different architectures and feature dimensions.\n\n---\n\n**Question: Performance of LoCo with features from different models (e.g., DINO vs. DINOv2)**\n\nIn response to your question about the performance of LoCo with features from different models, we have provided additional results using a DINOv2 ViT-Base backbone in the global author response. These results show that our method significantly outperforms the original DINOv2 feature extractor in tasks requiring accurate pixel correspondences. \n\nWhile the performance with the DINOv2 backbone is slightly lower than with the DINOv1 backbone, this likely arises from the differences in the patch size of the backbone, which affects the granularity of the feature map. Nevertheless, the fact that training with our method improves multi-view consistency with different backbone architectures demonstrates its flexibility and effectiveness across various feature extraction models.\n\n---\n\n**Limitations: Dataset requirements**\n\nWe acknowledge the limitation regarding the availability of 3D reconstructions in certain datasets. As discussed, this requirement is specific to the evaluation of certain tasks and not an inherent limitation of our method. The technique we propose is broadly applicable and can be adapted to other datasets and tasks that do not require 3D reconstruction data.\n\n---\n\nWe hope these clarifications address your concerns and provide a deeper understanding of the contributions and versatility of our work. We appreciate your constructive feedback and believe that our paper makes a valuable contribution to the field of vision foundation models, particularly in improving memory efficiency and multi-view consistency. \n\nThank you again for your careful review and for the opportunity to improve our work through your insights.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "xjybvLpWxg", "reviewer": "Reviewer_aW6W", "summary": "This work focuses on the best strategy to pre-train a feature extractor network optimized to be invariant wrt to the viewpoint in the image. To do so the authors use a dataset of paired views from which they extract pairs of positive patches (same 3D point in two different views) and negatives (different 3D points). Given these sets a feature extractor model is trained for patch retrieval using a loss proposed by the authors. In particular the paper starts from the existing Average PRecision Loss and adapts it to make it significantly more memory efficient. Provided with the data and the loss the authors train a tiny CNN adapter on top of Dino features to make them more robust wrt to the viewpoint. The performance of the trained model is evaluated on few ad-hoc benchmarks (multi view feature consistency, consistency of panoptic segmentation and visual place retrieval) where the proposal achieves improvements wrt to the competitors.", "questions": "**Questions**\n\n\n1. Can you comment on weakness [b] and whether generalization  is to be expected or very limited?\n\n2. Can you comment on weakness [c] \n\n**Typos**\n\n* L 166: Eq. 12 → Eq. 3?", "limitations": "Limitations have been adequatelly discussed in the manuscript.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "+ Presentation: the paper is very well written and presented. The paper has a straightforward core idea and contribution and the author make a good job at motivating and explaining it.\n\n+ Reusable: the proposed improved ranking loss can likelly be re-used for other tasks that require some form of metric learning and it’s not specifically tailored for the multi-view consistency use case. As such the contribution of this work can probably extend to nearby fields.\n\n+ Efficiency: the proposal seems to be quite effective even when training somewhat shallow models on somewhat small datasets (although what the author train is a residual model on top of a pre-trained Dino one). This is quite interesting as it shows that a relatively small learning budget is enough for learning multi-view consistency.", "weaknesses": "a. Limited experimental validation: while the author tested their model against competitors across 3 tasks, these are mostly internal comparisons on tasks that would privilege the proposed model. In particular, Table 1 where the method has the biggest gains, is basically measuring the performance on the task LoCo was trained on vs models that were not trained for the task (except for LoCUS). Regarding LoCUS in table 1 it is shown with a † that has no match in the caption of the table, following the rest of the paper I’m assuming it means that the method only has dimensionality 64 for features.  It would have been interesting to include in the evaluation some of the correspondence based tasks that CroCo-v2 has been exploring like stereo depth estimation, optical flow or relative pose estimation.\n\nb. Generalization: the model is trained on samples from matterport3D and evaluated on unseen scenes from Matterport and on scenes from ScanNet. This constrains quite a lot the scenarios being tested and does not give insight on the generalization capabilities of the proposed method. For example how would it perform in very different scenarios like an outdoor scene? Note that there are plenty of  dataset for evaluation of correspondences method that could be used for this task like [MegaDepth](https://www.cs.cornell.edu/projects/megadepth/)\n\nc. Potential unfair comparison to competitors: the closest competitor that has been trained with a comparable objective to the one proposed by the authors is LoCUS that according to the evaluation in Tab.1 and Tab. 2 achieves consistently worse results. However LoCUS uses only 2 FC layers on top of the features f a frozen DINO (500K parameters) while the proposed methods use a full CNN (~20M parameters), as such the comparison is not particularly fair. It would have been interesting to have a version of the LoCo method trained in the exact same settings as LoCUS (models and data) to verify what is the gain or losses of the proposed new loss function compared to the one introduced in LoCUS.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your detailed and constructive review. We appreciate your positive remarks regarding the presentation, efficiency, and potential reusability of our proposed method. We also acknowledge the valuable concerns you’ve raised and address them below.\n\n---\n\n**Weakness (a): Limited experimental validation**\n\nWe understand your concern regarding the focus of our experimental validation on tasks where our model excels. These tasks were chosen to highlight the core strengths of our approach in a controlled setting.\n\nFor tasks like surface normal prediction and monocular depth estimation (as reported in, e.g., El Banani et al. [12]), the correct prediction depends not just on the scene geometry, but also on the camera pose. Our work however explicitly aims to make the features location-consistent and thereby invariant to changes in camera pose. The LoCo-trained features therefore lack crucial information for such tasks.\nIn their experiments on optical flow and stereo depth estimation, CroCov2 [44] fine-tune their entire (pre-trained) network rather than training a comparatively small probe. Experiments of that scale were beyond our computational resources. \n\nIn the appendix, we provide an evaluation on Visual Place Recognition, which further demonstrates the versatility and utility of the features learned by our method across different applications.\n\n---\n\n**Weakness (b): Generalization capabilities**\n\nWe appreciate the concern regarding generalization and agree that this is an important aspect of any method’s robustness. Due to computational limitations, we were constrained to training relatively small models on indoor scene datasets. In performing well on many unseen diverse indoor environments, the models do demonstrate the capability to generalise to unfamiliar environments. Due to the circumscribed training domain, we do not expect these models to perform well on environments with a larger domain shift (such as outdoor environments), where performance would mostly depend on the degree of their similarity with the training domain. \n\nHowever, the training method we propose is designed to be fully general and adaptable to a variety of settings. While our current experiments focus on indoor scenes, the methodology itself should be applicable to other scenarios and domains without significant adjustments. \n\n---\n\n**Weakness (c): Potential unfair comparison to competitors**\n\nWe appreciate your concern about the comparison between our method and LoCUS, given the differences in model architecture and parameter count. To address this, we conducted additional experiments, as detailed in our global author response, where we fine-tuned the LoCUS architecture using our memory-efficient loss.\n\nThese experiments reveal that while our method does not outperform LoCUS in all tasks in terms of accuracy, it significantly enhances memory efficiency. This efficiency is what unlocks the training of larger models that produce higher-dimensional feature vectors within the same computational budget, leading to substantial overall performance improvements. The ability to scale up model capacity while maintaining computational feasibility is a key advantage of our approach, illustrating its potential beyond the specific settings and model scales we tested. We do however appreciate the importance of having this apples-to-apples comparison in the paper.\n\n---\n\n**Typos**\n\nWe will correct the typo on Line 166 (Eq. 12 → Eq. 3) and address the missing match for the dagger symbol in Table 1 before publication. We appreciate your attention to detail in pointing these out.\n\n---\n\nWe hope these clarifications address your concerns and provide additional context for evaluating our work. We are confident that our paper offers valuable contributions, particularly in advancing methods for training location-consistent feature extractors in a memory-efficient manner. \n\nWe appreciate your careful consideration of our work and look forward to any further feedback.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We thank all reviewers for their careful reading and thoughtful comments. In this section, we outline the additional ablations and evaluations requested by the reviewers, which are presented in the PDF attached to this response.

## Pixel Correspondence Estimation (Table 1)
### LoCo with DINOv2 Backbone
As requested by reviewers **jhZZ** and **YTBi**, we provide results where the DINO backbone used in the original submission is replaced by the DINOv2 ViT-Base backbone ("LoCo w/ DINOv2 backbone"). The resulting features significantly outperform the original pre-trained DINOv2 features (also reported in Table 1) for finding accurate pixel correspondences, showing its advantage in tasks that require location-consistent features. However, it slightly underperforms the LoCo model trained with the DINO-ViT-Base8 backbone used in the original submission. We hypothesize that this arises from the coarser feature map of the DINOv2-ViT-Base14 backbone compared to the DINO-ViT-Base8 backbone.

### LoCo with LoCUS architecture
As requested by reviewers **YTBi** and **aW6W**, we also provide results where the network architecture used in the original submission is replaced by that used in LoCUS ("LoCo w/ LoCUS architecture"). This model outperforms the original LoCUS model for small viewpoint changes, but underperforms for image pairs with larger viewpoint changes. This illustrates that the improvements in memory efficiency do not by themselves lead to improvements in performance, but that they allow for the training of larger models and higher-dimensional feature vectors with the same computational budget, the effect of which far outweighs any performance decrease due to our loss function and training algorithm changes.

## Panoptic Scene-Stable Segmentation (Table 2)
### LoCo with DINOv2 Backbone
For this task, the LoCo model trained with the DINOv2 backbone only outperforms the original DINOv2 feature extractor on some of the metrics. This is likely attributable to the coarser feature map of this backbone, which leads to less fine-grained patch-level supervision during training.

### LoCo with LoCUS architecture
For this task, the LoCUS architecture trained with the LoCo algorithm performs worse than the original LoCUS model. For this ablation, we trained for the same number of epochs as our other LoCo models, and so cannot rule out that the vision transformer blocks in the LoCUS architecture require longer training times than the convolutional layers of the LoCo model. As before, this result illustrates the value in our efficiency improvements as they ultimately unlock the training of larger models with larger feature dimensions.

Meta-review: This paper received mixed reviews. While reviewers praise the interesting ideas, efficiency and good presentation, there were concerns around potentially unfair comparisons and unjustified hyperparameters. The rebuttal clarified several concerns, and there was a good discussion between the reviewers to weigh the pros and cons. Even though there was no final consensus on the ratings, the AC recognizes that the work can be accepted into NeurIPS, given all its positive aspects and the additional information provided in the rebuttal. We require that the clarifications and all relevant information provided in the rebuttal be incorporated in the camera-ready version.
Decision: Accept (poster)