LoCo: Learning 3D Location-Consistent Image Features with a Memory-Efficient Ranking Loss

Authors: dominik a kloepfer
joão  henriques
dylan  campbell
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-15 12:29:21

Abstract: Image feature extractors are rendered substantially more useful if different views of the same 3D location yield similar features while still being distinct from other locations. A feature extractor that achieves this goal even under significant viewpoint changes must recognise not just semantic categories in a scene, but also understand how different objects relate to each other in three dimensions. Existing work addresses this task by posing it as a patch retrieval problem, training the extracted features to facilitate retrieval of all image patches that project from the same 3D location. However, this approach uses a loss formulation that requires substantial memory and computation resources, limiting its applicability for large-scale training. We present a method for memory-efficient learning of location-consistent features that reformulates and approximates the smooth average precision objective. This novel loss function enables improvements in memory efficiency by three orders of magnitude, mitigating a key bottleneck of previous methods and allowing much larger models to be trained with the same computational resources. We showcase the improved location consistency of our trained feature extractor directly on a multi-view consistency task, as well as the downstream task of scene-stable panoptic segmentation, significantly outperforming previous state-of-the-art.

Main paper: 
 introduction: 
 Reasoning in 3D is critical for developing a useful visual understanding of an environment. However, image-centric approaches, including patch feature extractors like DINO [5, 31] , are not 3D-consistent, as a recent paper by El Banani et al. [12] demonstrates. That is, the same 3D location may yield significantly different features from different viewpoints in space and time, due to occlusions, selfocclusion, reflections, lighting variations, and motion. When this occurs, it is challenging to maintain a spatially and temporally consistent model of the world. Thus, it is useful to first convert visual observations into a form that is stable across different spatio-temporal viewpoints.Existing approaches [3, 18, 27, 29, 40] aggregate or distil visual features in 3D, ensuring viewconsistency at the cost of requiring a full 3D reconstruction pipeline. In contrast, recent work [26] explores a more flexible image-centered representation that encourages similarity between image patch features that backproject to the same region of 3D space, within a spatial tolerance. It employs a ranking-based loss function, smooth Average Precision (sAP), to encourage all spatially co-located features to be similar and all non-colocated features to be dissimilar, as illustrated in Fig. 1 .Like this approach, we formulate the learning problem as one of patch retrieval: given one image patch, retrieve with high precision and recall all patches in other views that project from the same 3D region. However, the smooth Average Precision (sAP) loss function requires substantial memory and computation, precluding its deployment in large-scale training. By rewriting the loss function in terms of pairs of image patches rather than individual patches, we derive a more general form of the sAP objective that lends itself to approximation. This novel formulation enables improvements in memory efficiency by three orders of magnitude, mitigating a key bottleneck and allowing larger models to be trained with the same computational resources. By applying this novel loss function within a new training strategy, we obtain a method for memory-efficient learning of location-consistent (LoCo) features that are semantically-meaningful and stable across viewpoints. Our contributions are:1. A novel reformulation and approximation of the smooth average precision loss function that can be computed significantly more efficiently than the original; 2. A training strategy for scalable and memory-efficient learning of location-consistent image features; and 3. Applications to pixel correspondence estimation and scene-stable panoptic segmentation.The approach is evaluated on a recently proposed multi-view consistency task [12] and is tested on two real-world indoor datasets, significantly outperforming state-of-the-art feature extractors. 
related work: 
 The topics of panoptic image segmentation [25] , visual place recognition [20, 23] , image retrieval [2, 4, 13, 32] , and visual feature learning [5, 16] are well-studied. Here, we focus on the most recent and related work.Image segmentation. Visual features from pre-trained models were shown to be very useful for panoptic (instance and semantic) image segmentation [5, 25, 43] . Interestingly, location-consistent visual features unlock the possibility of "scene-stable panoptic segmentation", where instance IDs are consistent across multiple views of the same scene [26] . This is related to identity-preserving video segmentation [24] , where object identities are tracked, leveraging temporal smoothness.Image retrieval. Learning representations that facilitate the ranking of images according to their relevance to a query has been studied extensively [2, 13, 32] . One class of approaches, metric learning, uses contrastive losses [8, 42] to encourage positive instances to be close and negative instances to be further apart, while others optimise ranking-based metrics like Average Precision (AP) directly [4, 34] . For example, Smooth-AP [4] recommends the use of an approximated AP ranking function, which targets the correct ranking without being concerned with the absolute feature distances. We pose our location-consistent feature learning problem as a patch retrieval problem, allowing us to adapt strategies from the image retrieval literature.Self-supervised visual feature learning. Self-supervision has emerged as a dominant strategy for training foundation models in computer vision on large-scale image datasets. Notably, DINO [5] leveraged a knowledge-distillation framework to learn to extract semantically meaningful feature maps. Building upon DINO's foundation, DINOv2 [31] refines the training algorithm and scales up model sizes on larger datasets, resulting in enhanced performance. Another noteworthy approach is the Masked Autoencoder (MAE) [17] , which employs an autoencoder architecture to reconstruct masked-out patches, and demonstrated the scalability of autoencoders to large datasets.In a similar vein, CroCo [43] and its successor, CroCo-v2 [44] , also adopt an encoder-decoder architecture for reconstructing masked-out patches. However, in these works the decoder reconstructs the patches based on a feature map extracted from a non-masked image of the same scene but observed from a different viewpoint. While we also use different viewpoints as the source of the supervision signal like CroCo, the learning process is quite different and makes more explicit use of geometric constraints in forming positive and negative pairs of patches (cf . Section 3.2). Most closely related to our work is LoCUS [26] , which uses a similar problem set-up and loss function.Where they approach the task from the perspective of extracting distinctive landmarks (individual patches or points in the scene), our areas of interest are pairs of patches. Crucially, this allows us to massively decrease the algorithm's memory consumption and unlock significant performance gains. We provide further detail in Section 3.Other works [27, 29, 40] distil visual features in 3D in order to ensure location-consistency, the same goal as our approach. For example, N3F [40] distilled DINO image features into a 3D feature field using the same rendering loss as NeRF [30] . While the resulting features are 3D-consistent by design, this comes at the cost of requiring a full 3D reconstruction pipeline. In contrast, our approach is image-centric and lightweight, predicting location-consistent features from one image at a time without requiring input poses or re-training for new scenes.Question: Does the paper discuss the limitations of the work performed by the authors?Answer: [Yes] Justification: Limitations of the work are briefly discussed in the conclusion and in greater detail in the appendix.Guidelines:• The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper. • The authors are encouraged to create a separate "Limitations" section in their paper.• The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be. • The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated. • The authors should reflect on the factors that influence the performance of the approach.For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon. • The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size. • If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness. • While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations. 
efficiently learning loco image features: 
 In this section, we outline our method for learning 3D location-consistent (LoCo) image features in a scalable, memory-efficient way. We first formalise the problem definition and define the positive and negative sets, then we reformulate the retrieval objective function and show how this facilitates very significant reductions in the memory requirements. This allows our method to overcome a critical bottleneck in the learning process for these features, permitting us to scale up the model size. Sec. 4 shows that this has a large impact on performance, justifying the need for the reformulation.Our goal is to extract feature maps so that the extracted features are similar for sets of image patches that depict the same region in 3D space. Like Kloepfer et al. [26] , we make this task tractable by recognising that feature maps solving this task can also be used for patch retrieval: given an image patch and associated feature vector, retrieve all patches that project from the same region in 3D space. This allows us to adapt methods from the extensive literature on image retrieval. Formally, given a set I of training images I i ∈ R H×W ×3 drawn from an environment e, we aim to train a feature extractor ϕ : I → R h×w×d . Here, h ≤ H and w ≤ W are the height and width of the extracted feature map, and d is the dimension of the extracted feature vectors, so each feature vector is associated with an image patch x k ⊆ I i consisting of H h × W w pixels. We denote an image patch as x k , the associated feature vector as θ k , a patch pair as c α = (x i , x j ), and the associated cosine similarity score between the feature vectors of a patch pair as s α = θ ⊤ i θ j /(∥θ i ∥∥θ j ∥). We use Greek subscripts to index pairs of patches, rather than individual patches. Each patch x k ∈ I i is also associated with a point p k ∈ R 3 in the environment, obtained by backprojecting the centre coordinates of the patch into the environment using estimated or provided camera intrinsic and extrinsic parameters and depth [15] . This auxiliary information is only required at training time; at test time the trained feature extractor ϕ only requires a raw RGB image.As illustrated in Fig. 2a , we define two patches x i and x j , not necessarily from the same image, as a 'positive' pair if and only if they are drawn from images in the same environment and the distance between their associated 3D points p i and p j is below a threshold ρ. Therefore, given the index of the environment e i associated to each patch x i , the set of all positive pairs isP = {(x i , x j ) : ∥p i -p j ∥ ⩽ ρ ∧ e i = e j }.(1) Pairs of patches that do not observe the same 3D region form a 'negative' pair. As a form of hard negative mining, we restrict our attention to those negative pairs whose patches depict locations The distance between 3D points p i associated with patch pairs (x i , x j ) in the positive set is less than ρ, denoted by the green arrows connecting them. The distance between those in the negative set is in (ρ, κ], denoted by the red arrows connecting them. All unconnected 3D point pairs are in neither the positive nor negative set, since they are separated by a distance greater than κ.(b) When the absolute similarity difference |s β -s α | between image pair α and pair β is large, the sigmoid in the loss function (blue curve) becomes saturated and does not impact learning. We can avoid the memory cost of back-propagation in these cases by separating the positive P and negative N pairs into 3 subsets: saturated below, unsaturated, and saturated above. We choose the saturation threshold ∆ such that the sigmoid gradient there is 0.2% of its maximum value.within a distance of κ > ρ of each other, and define the set of negative pairs asN = {(x i , x j ) : ρ < ∥p i -p j ∥ ⩽ κ ∧ e i = e j }.For convenience we also define the set of all training pairs Ω = P ∪ N.For a given patch x i , the thresholds ρ and κ define a positive and a negative region around the associated 3D point p i . All other patches that observe points inside the radius-ρ sphere centred at p i will form a positive patch pair with x i . Likewise, all other patches that observe points outside this sphere but inside the radius-κ sphere centred at p i will form a negative patch pair with x i . The features of the positive pairs are encouraged to be more similar than those of the negative pairs.It should also be noted that the finite size of these regions means that the training is robust to noise in the depth maps and camera poses used to compute the patch locations in 3D. Noisy patch locations effectively only slightly change the size of these regions and so we expect them not to fundamentally alter the learning algorithm. This could open the door to using less accurate estimates for depth and camera poses in future, which may be easier to obtain, in particular for large-scale datasets.The smooth average precision (AP) loss function, originally introduced in Brown et al. [4] , was adapted to the setting of patch retrieval around "tentative 3D landmarks" by Kloepfer et al. [26] , resulting in a vectorised form of the loss function. We streamline this setting by eliminating the need for these landmarks, and instead focus exclusively on retrieving positive pairs of patches.The aforementioned vectorised smooth AP loss function can be rewritten in terms of patch pairs asEQUATION= -1 |P| cα∈P 1 + c β ∈P\{cα} σ τ (s β -s α ) 1 + c β ∈P\{cα} σ τ (s β -s α ) + cγ ∈N σ τ (s γ -s α ) ,where σ τ (x) = (1 + exp(-x/τ )) -1 is the sigmoid function with temperature τ .This loss function computes a differentiable approximation to the average precision of a binary classifier that classifies pairs of patches as positive or negative based on the similarity of each pair. This approximation becomes exact as τ → 0 and the sigmoid approaches the indicator function. For each positive pair c α , Eq. ( 2) calculates the ratio of the rank of c α among all positive pairs and its rank among all pairs (positive and negative) when ranking the pairs by decreasing similarity. More details can be found in Brown et al. [4] .Compared with standard contrastive losses, like a triplet loss [19] or SimCLR [7] , the gradient ∂L/∂s α of this ranking loss with respect to the similarity of a positive pair c α will disappear as soon as s α is higher than the similarities of all negative pairs. That is, it does not force s α → +1 for positive pairs and s α → -1 for negative pairs, it merely encourages some boundary to exist somewhere between positive and negative pair similarities. This "gentler" contrastive characteristic greatly aids training convergence [26] .The sums in Eq. ( 2) run over all pairs of (positive) patches in the training set. Clearly, computing the exact loss function is completely infeasible for large datasets. However, when sampling batches of positive and negative pairs, correction terms are needed to make the expectation value of the batched loss equal to the exact (unbatched) loss. In particular, some care needs to be taken since each term in the loss function depends on multiple different samples within the batch. Using the subscript B to refer to the batched versions of the sets P B ⊂ P and N B ⊂ N of positive and negative pairs, the batched version of our loss function becomesEQUATION)The correction factors |P| |P B | and |N| |N B | are necessary to ensure that the expectation value of the batched loss is as close as possible to the loss computed across the entire dataset. This is automatically the case for standard loss functions that average over a per-sample loss, due to the linearity of the expected value. However, since the ranking loss computes the (non-linear) ratio of expectations over samples, this linearity is lost. Each loss term depends on multiple pairs, and the +1 terms in numerator and denominator introduce additional complications for finding an unbiased estimator. For a detailed derivation of Eq. ( 3) we refer the reader to Appendix B, where we show that it is a ratio estimator [37] that is simple to calculate and consistent, but has a bias of order O(1/|Ω B |).We note here that neither Brown et al. [4] nor Kloepfer et al. [26] include these correction factors, which cause the losses in those works to deviate even further from the desired average precision approximation. We quantify this deviation in the appendix.A key bottleneck of the loss in Eq. ( 3) is the memory consumption of the |P B | × (|P B | + |N B |) matrix containing all the values of s β -s α and s γ -s α , and the associated computation graph. Since each occurrence of the similarity of a particular patch pair in the loss function only provides a supervision signal for the feature vectors of two individual patches, the batches of positive and negative patch pairs need to be quite large (Kloepfer et al. [26] use |P B | ≈ 13, 000 and |N B | ≈ 100, 000).To alleviate this problem, we design two ways to significantly reduce the memory consumption of this matrix. First, we observe that the positive pairs c α in Eq. (3) do not need to be drawn from the same subset of all positive pairs as the ones in c β . As long as both are sampled uniformly from the set of all positive pairs, the expected value of the batched loss continues to equal the non-batched loss of Eq. ( 2). Sampling c α from a small set of positive pairs P ′ B ⊂ P and c β from a large set P B ⊂ P, with |P ′ B | ≪ |P B |, reduces the size of the matrix of similarity differences to|P ′ B | × (|P B | + |N B |):only the second dimension is large, while previously both dimensions were large and of comparable size. At the same time, this still retains a supervision signal for a large number of feature vectors, namely those used to construct the larger sets P B and N B .Second, we observe that, in practice, a large number of the computed similarity differences saturate the sigmoid function. That is, when |s β -s α | ≫ 0, the gradient of the sigmoid ∇σ τ (s β -s α ) vanishes: these terms make no material difference to the loss gradient. We can use this fact to significantly reduce the number of similarity differences in the computation graph. To do so, we set a threshold ∆ > 0 and use the approximation, visualised in Fig. 2b ,EQUATIONto divide the uniformly sampled sets of patch pairs P B and N B into three subsets for each c α ∈ P ′ B ,EQUATIONEQUATIONEQUATIONEQUATIONand similarly for P B . We can now significantly reduce the number of similarity differences in the computation graph by only including patch pairs from N α B and P α B in the loss computation, tocα∈|P ′ B | (|P α B | + |N α B |) in total.Computing these subsets for each c α still requires us to calculate all |P ′ B | × (|P B | + |N B |) similarity differences, but because most pairs' gradients are close to 0 they are not used to compute the loss or any gradients, so these parts of the computational graph can be deleted, leading to substantial memory savings.To compensate for the non-uniform sampling of P α B and N α B , we also need to add additional correction terms to our loss function. Our final memory-efficient, batched loss function is given byEQUATIONwith the correction terms given byEQUATION)For a detailed derivation of these correction terms, as well as a derivation of an upper bound on the error due to the approximation in Eq. ( 4), we refer the reader to Appendix C.Restricting our sampling of patch pairs to those whose similarities fall within a certain range is reminiscent of hard negative mining, as previously employed in the context of contrastive learning by, e.g., Robinson et al. [33] . However, standard approaches are not easily applicable to our loss function since it operates on pairs rather than individual samples. Furthermore, both motivation and effect of our approach differs from hard negative mining methods. We discuss this further in Appendix E, where we also compare empirically with standard contrastive learning and hard negative mining.Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?Answer: [No] Justification: At time of submission, we have not released the code or data, however we will do so before publication for reproducibility.Guidelines:• The answer NA means that paper does not include experiments requiring code. • Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted. 
experiments: 
 In this section, we present our experiments, where we evaluate the performance of LoCo features at a multi-view consistency task and at scene-stable panoptic segmentation.The training dataset we use comprises 59 environments of the Matterport3D dataset, resizing the images to 256 × 320 pixels. The Matterport3D dataset is particularly suitable for our task of enforcing multi-view consistency due to its diversity and the way it captures varied viewpoints of the same scene through panorama cropping. Datasets such as ScanNet [10] provide less viewpoint variation per scene due to their trajectory-based data collection. Contrary to Kloepfer et al. [26] , we use all available images in the Matterport3D training scenes instead of restricting to images taken in the horizontal plane.Due to limited computational resources, we were unable to use our loss function to train a full foundation model from scratch. Instead, we adapt the architecture used by DINO-Tracker [41] , keeping pre-trained DINO [5] features frozen and training a convolutional neural network to learn additive residuals to those features. We use values of ρ = 0.5m for the positive radius, κ = 5.0m for the negative radius, τ = 0.01 for the sigmoid temperature, and ∆ = 0.076 for the saturation threshold. With these values, σ τ (∆) = 0.9995 and the gradient is 0.2% of the maximum gradient of the sigmoid function, making this a conservative choice with little impact on the training dynamics. The threshold ∆ is a hyperparameter that can be decreased to obtain further memory savings, at the expense of some performance decrease. We provide further implementation details regarding the efficient sampling of patch pairs in Appendix D, and will publicly release our training code.Table 1 : Results on the pixel correspondence task on the Paired split [36] of ScanNet [10] , as introduced by El Banani et al. [12] . We report the recall of accurate pixel correspondences at a reprojection error threshold of 10 pixels, for image pairs with the respective viewpoint changes. We also report the GPU Memory required for training LoCUS-based and LoCo models (for LoCUS we use the authors' values). † uses 64-dimensional feature vectors.Model GPU Memory EQUATIONWe compare our LoCo features to those from several state-of-the-art feature extractors, as well as models specialised to the downstream segmentation task.DINO [5] . We use the pre-trained ViT-base model and extract 768-dimensional features for each patch of 8 × 8 pixels by discarding the class token and reshaping the output of the final Transformer-Block into a feature map.DINOv2 [31] . We use the pre-trained ViT-base model and extract 768-dimensional features for each patch of 14 × 14 pixels by again discarding the class token and reshaping the output of the final Transformer-Block. We generally use higher-resolution images as inputs for this model to extract feature maps of the same shape as for LoCo and DINO features.CroCo-v2 [44] . We use the ViT-base encoder that was pre-trained with the Base-Decoder and extract 768-dimensional features for each patch of 16 × 16 pixels, again using higher-resolution as inputs.LoCUS [26] . We use the pre-trained LoCUS weights that are publicly available for a landmark-radius of 0.2m. We use it to extract 64-dimensional feature vectors for each patch of 8 × 8 pixels.MaskDINO [28] . For the scene-stable object segmentation task (cf . Section 4.4) we also compare with MaskDINO, a state-of-the-art specialised panoptic segmentation algorithm. It however is not designed to recognise the same object in different images. We therefore match the per-image object indices produced by the algorithm to the ground-truth per-scene object index whose mask has the highest IoU with the object mask in question.We first directly evaluate the location consistency of the features extracted by different models across different views. To do so, we follow the protocol introduced by El Banani et al. [12] to test the multi-view consistency of feature extractors on a pixel correspondence estimation task. Briefly, given a pair of images, we extract a fixed number of pixel matches by filtering the nearest neighbour matches using a ratio test. For more details, the reader is referred to the original paper [12] .Like El Banani et al., we evaluate on the Paired ScanNet [10] split proposed by Sarlin et al. [36] ,reporting the recall at a reprojection error threshold of 10 pixels for different viewpoint changes in Table 1 . As we can see, our model outperforms the baselines by a significant margin. We also see that DINOv2 [31] performs worse than the original DINO [5] . This situation is somewhat reversed when sourcing the features from the first six Transformer blocks (instead of the final feature map). This suggests that while earlier layers of DINOv2 are still reasonably location-consistent, later layers create more semantically meaningful features that accordingly do not vary much by the patches' location in 3D space, explaining why they perform worse in this setting. CroCo-v2 [44] performs relatively poorly on this task, despite its training objective being explicitly designed for multi-view tasks. However, when training CroCo, the encoder features are first used by a transformer-based decoder module before a loss function is applied. This means that there is no incentive for CroCo features to be location-consistent under a simple and interpretable cosine-similarity operation, and would require a more complex adapter to support similarity-based operations.The task of scene-stable panoptic segmentation was originally introduced by Kloepfer et al. [26] . Given a set of images of a single scene, the goal is to create a segmentation mask for the objects in each of these images, where, crucially, different views of the same individual object are labelled consistently with the same identity.Formally, for a set I = {I i } of images of the same scene, a set C = {0, . . . , L -1} = C st ∪ C th of semantic classes that is split into 'stuff' (amorphous classes such as floor, walls, etc) and 'things' (clearly distinct objects) subsets C st and C th respectively. The latter are also split into a set O = {0, . . . , N -1} of object instance IDs within the scene. The goal is to map each pixel p j to its semantic class c j if c j ∈ C st , and to its scene-wide object instance o j if c j ∈ C th . This is similar to the standard panoptic segmentation task [25] with the crucial difference that the object instance indices are consistent across different images of the same scene.This task requires the algorithm to not just differentiate between multiple object instances of the same object class as in the conventional instance segmentation task, but also to recognise when different images show the same object, which requires a broader understanding of the scene geometry. The need to retain consistent object identities across different images is reminiscent of panoptic video segmentation [24] . However, in our task the images are unordered and have much larger viewpoint changes, so methods cannot rely on pixel tracking or optical flow to keep object identities consistent.As is standard in self-supervised learning [5, 16, 26, 31] , we train a linear probe to predict class labels from the feature vectors for every image patch.Both the Matterport3D [6] and ScanNet [10] datasets provide 3D-mesh reconstructions of their constituent environments, segmented into individual objects. This allows us to generate ground-truth segmentation masks for scene-consistent object segmentation by finding the individual objects in each scene that a ray through a given pixel intersects. Scripts to generate this data will be included in our code release. For the Matterport3D dataset, we evaluate on 18 unseen scenes, and for ScanNet we evaluate on 21 unseen scenes selected to show a range of different types of environments.We report our results on the Matterport3D dataset and on the ScanNet dataset in Section 4.4.1. We measure the scene-stable panoptic segmentation performance using three metrics, which we calculate for each object instance individually (for each object instance treating the segmentation masks as Our LoCo-trained features perform better than all baselines, and have comparable performance to CroCo-v2 [44] , which also makes use of multi-view supervision. However, ours has far fewer trainable parameters (only 28.9 million), since we only train a comparatively small CNN. In contrast, CroCo trains the entire network (85 million parameters) with a multi-view loss and on significantly larger datasets with greater computational resources (8 A100 GPUs vs. 1 RTX8000 GPU). We note also the strong performance of the original DINO compared to the newer DINOv2 method. As in the pixel correspondence task, this might be due to the final DINOv2 features focusing on semantic meaning, and so struggling to differentiate between, e.g., different chairs.DINOv2 Backbone. We also train our method using a frozen DINOv2 [31] backbone, again training a convolutional neural network to learn additive residuals and keeping other hyperparameters the same.The resulting features significantly outperform the original pre-trained DINOv2 features for finding accurate pixel correspondences (Table 1 ), showing the advantage of LoCo-training in tasks that require location-consistent features. However, it slightly underperforms the LoCo model trained with the DINO-ViT-Base8 backbone. We hypothesize that this arises from the coarser feature map of the DINOv2-ViT-Base14 backbone (with a patch size of 14 instead of 8).On the panoptic scene-stable segmentation task (Section 4.4.1), the LoCo model trained with the DINOv2 backbone only outperforms the original DINOv2 feature extractor on some of the metrics. This is likely attributable to the coarser feature map of this backbone, which leads to less fine-grained patch-level supervision during training. LoCUS Architecture. To further investigate the impact of our alterations to the loss function and training algorithm compared to Kloepfer et al. [26] , we train the original LoCUS architecture with the LoCo loss function and algorithm.On the multi-view consistency task (Table 1 ), this model outperforms the original LoCUS model for small viewpoint changes, but underperforms for image pairs with larger viewpoint changes.In fact, the LoCUS architecture trained with the LoCo-algorithm performs worse than the original LoCUS model on the panoptic scene-stable segmentation task (Section 4.4.1).For this ablation, we trained for the same number of epochs as our other LoCo models, so it is possible that the vision transformer blocks in the LoCUS architecture require longer training times than the convolutional layers of the LoCo models. In any case, this ablation illustrates that the improvements in memory efficiency do not by themselves lead to improvements in performance. Their advantage is that they allow for the training of larger models and higher-dimensional feature vectors with the same computational budget, the effect of which far outweighs any performance decrease due to our loss function and training algorithm changes.Effect of ∆. We analyse the impact of restricting further the range of similarities from which N α B and P α B are sampled by decreasing the saturation threshold ∆. The results of are shown in the last two rows of Table 1 . We see that while there is a small decrease in performance as the threshold decreases, overall, the training is remarkably robust to more aggressive filtering of patch pairs. This confirms the intuition that only a small number of patch pairs contribute meaningfully to the gradient and that most patch pairs can be discarded without significantly impacting the training behaviour.Using their training code, we find that the hyperparameters used in Kloepfer et al. [26] result in positive and negative pair set sizes of |P B | ≈ 13, 000 and |N B | ≈ 98, 000. The resulting matrix size, using single-precision floating point numbers, of|P B | × (|P B | + |N B |) ≈ 51GBexceeds most computational limits, so the authors subsample 10% of the negative pairs to reduce the matrix size to 5.7GB for a matrix with 1.4 billion entries. We instead use a value of |P ′ B | = 32 and find empirically that even with the conservative value ∆, roughly 80% of the pair differences are well-approximated as having zero-gradient. This means that|P α B | ≈ (1 -0.8)|P B | and |N α B | ≈ (1 -0.8)|N B |.Assuming the same computational budget of 5.7GB, our method can therefore use batches P B and N B that are larger by a factor of ∼ 2000. In our experiments, we limit |P α B | < 800 and |N α B | < 3, 000, resulting in the matrix occupying only 500KB of memory, thereby freeing up GPU memory to train models with more parameters and with substantially larger feature vectors. 
conclusion: 
 In this paper, we have proposed a method for the memory-efficient learning of location-consistent features. In particular, we present a reformulation of the smooth average precision ranking loss that corrects for biases induced by batching, and introduce an approximation that facilitates significant memory reductions without distorting the training signal. This mitigated a key memory bottleneck, allowing larger models to be trained with the same computational resources. Equipped with this novel retrieval-based objective function, we are able to efficiently learn to modulate DINO [5, 31] ViT features towards location-consistency.Our feature extractor demonstrates compelling performance on the downstream tasks of scene-stable panoptic segmentation and visual place recognition, outperforming previous state-of-the-art feature extractors. This work goes some way towards scaling up the training pipeline; however, there is significant scope for applying these techniques on truly large scale image or video data in an entirely self-supervised manner by estimating depth maps and camera poses using off-the-shelf methods.  
experimental setting/details: 
 Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?Answer: [Yes] Justification: We provide this information either in the Experiments section or in the appendix.Guidelines:• The answer NA means that the paper does not include experiments.• The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them. • The full details can be provided either with the code, in appendix, or as supplemental material. 
experiment statistical significance: 
 Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?Answer: [No] Justification: Error bars are not provided in the paper, due to the significant time cost associated with re-training from different initialisations.Guidelines:• The answer NA means that the paper does not include experiments.• The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.• The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions). • The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.) • The assumptions made should be given (e.g., Normally distributed errors). • It should be clear whether the error bar is the standard deviation or the standard error of the mean. Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.• Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations. • The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?Answer: [NA] Justification: This paper, on a new technique for self-supervised learning of 3D-consistent image features, poses no such risks.Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?Answer: [Yes] Justification: We credit the original creators of other's assets and respect their licenses.Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL. • The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.• If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. • For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. • If this information is not available online, the authors are encouraged to reach out to the asset's creators. 
new assets: 
 Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?Answer: [NA] Justification: The assets introduced in this paper (code and model weights) will be released before publication with full documentation. Since they are not released at the time of submission, this question is answered as NA.Guidelines:• The answer NA means that the paper does not release new assets.• Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc. • The paper should discuss whether and how consent was obtained from people whose asset is used. • At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file. 
crowdsourcing and research with human subjects: 
 Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?Answer: [NA]Justification: The paper does not involve crowdsourcing nor research with human subjects.Guidelines:• The answer NA means that the paper does not involve crowdsourcing nor research with human subjects. • Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper. • According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector. 
institutional review board (irb) approvals or equivalent for research with human subjects: 
 Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?Answer: [NA]Justification: The paper does not involve crowdsourcing nor research with human subjects.Guidelines:• The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.• Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper. • We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution. • For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review. 
: 
 Figure 1 : Our approach-LoCo-offers memory-efficient learning of location-consistent (LoCo) features. That is, features that backproject to nearby 3D locations are encouraged to have similar image patch features (illustrated here by the pair of blue stacked cubes and the pair of red stacked cubes), while those that backproject to well-separated points are trained to have more dissimilar features (here, the blue vs. red cube stacks). This is achieved via a novel ranking loss that reformulates and corrects the smooth average precision loss proposed in previous work [4, 26] . This facilitates the derivation of a close approximation to the loss that is significantly more efficient to compute, allowing the method to scale to much larger models with the same computational resources. 
 
 Appendix: In Table 3 we describe in more detail the architecture of our fully convolutional network that we use to compute residuals to frozen DINO [5] features. For the respective layers we downsample the resolution by the given factor using a BlurPool layer.In the following we will derive the batch correction factors |P|/|P B | and |N|/|N B | from Sec. 3.4 in the main paper. We will also retain all notation conventions used in the main paper.We can re-write the Eq. ( 2) asEQUATIONwithL α = 1 + c β ∈P\{cα} σ τ (s β -s α ) 1 + c β ∈P\{cα} σ τ (s β -s α ) + cγ ∈N σ τ (s γ -s α ) .and similarly Eq. (3) asEQUATIONwithL Bα = 1 + f P c β ∈P B \{cα} σ τ (s β -s α ) 1 + f P c β ∈P B \{cα} σ τ (s β -s α ) + f N cγ ∈N B σ τ (s γ -s α ) ,where f P and f N are the as of yet unknown batch correction factors.We want to choose f P and f N such thatEQUATIONFrom the linearity of the expected value we haveEQUATIONComputing the expectation value of L B exactly is challenging, as numerator and denominator are not independent. We can approximate EL Bα using the ratio estimator [37] , which for two random variables X and Y approximatesEQUATIONThis estimator can be shown to be consistent, though biased. If the expectation values E[X] and E[Y ] are estimated using n samples, the bias can be shown to decrease as O(n -1 ) [9] .Applying this estimator to Eq. ( 14) withEQUATIONandEQUATIONwe getEQUATION)Since for uniform sampling of pairs from PEQUATIONwe obtainEQUATIONand similarly forE[ c β ∈N B σ τ (s β -s α )].Plugging this into Eq. ( 14) and Eq. ( 13), immediately results inEQUATIONEQUATION)We now investigate the bias due to the approximation Eq. (15) . By Taylor expanding Eq. ( 15) aroundX = E[X] = µ X and Y = E[Y ] = µ Y to second order we get X Y = µ X µ Y + (X -µ X ) • 1 µ Y -(Y -µ Y ) • µ x µ 2 y (23) + 1 2 (X -µ X ) 2 • 0 + (Y -µ Y ) 2 • µ X µ 3 Y -(X -µ X )(Y -µ Y ) • 1 µ 2 Y+ higher order terms.Applying the expectation value operator results inE X Y = µ X µ Y + E (Y -µ Y ) 2 • µ X µ 3 Y (24) -E [(X -µ X )(Y -µ Y )] • 1 µ 2 Y+ higher order terms.Using our definitions of X and Y from Eq. ( 16) and Eq. ( 17) we getEQUATION= O 1EQUATIONbecause the standard error of the sample mean decreases with the inverse square root of the number of samples.Analogously, we haveEQUATIONwhich then results inEQUATION)The bias therefore decreases with the inverse of the batch size.Previous uses of the Smooth Average Precision loss function in Brown et al. [4] and Kloepfer et al. [26] did not use batch correction factors.The additional bias that results from this grows as the batch size decreases, because the batch correction factors decrease for larger batch sizes, and also because for larger batch sizes the sums in numerator and denominator grow and the additional +1 term that is unaffected by the batch correction factors becomes less and less relevant.In our experiments, the sums would generally have magnitudes of at least O(10 4 ), so the main function of the batch correction factors is to correctly balance the effect of positive and negative pairs in the denominator if f P ̸ = f N . This is the case in our experiments, where our hyperparameter choices mean that we generally sample a larger fraction of positive than of negative pairs so that f P /f N ≈ 0.1. Without the batch correction factors, the loss function would place excessive weight on increasing the similarity of positive pairs.Since these correction factors are simple to calculate, it is therefore valuable to use the loss function that includes them and that is built on a stronger theoretical foundation.We derive here the exact form of the correction term δ - α in Eq. ( 9) of the main paper. The derivation for δ + α proceeds entirely analogously. Using the notation from the main paper, we can writeEQUATION+ |N| |N B | c β ∈N α B σ τ (s β -s α ) + |N| |N B | c β ∈N α B σ τ (s β -s α ).Using the approximation from Eq. ( 4) in the main paper, we haveEQUATION) Plugging into Eq. ( 29) yieldsEQUATIONwith the second term exactly the correction term δ- α = |N α B | |N| |N B | .Using the sum expansion of Eq. ( 29) in Eq. ( 12) results inEQUATION+ |P| |P B | c β ∈P α B \{cα} σ τ (s β -s α ) + |P| |P B | c β ∈P α B σ τ (s β -s α ) / 1 + |P| |P B | c β ∈P α B σ τ (s β -s α ) + |P| |P B | c β ∈P α B \{cα} σ τ (s β -s α ) + |P| |P B | c β ∈P α B σ τ (s β -s α ) + |N| |N B | c β ∈N α B σ τ (s β -s α ) + |N| |N B | c β ∈N α B σ τ (s β -s α ) + |N| |N B | c β ∈N α B σ τ (s β -s α ) .In the worst case, the inequalities definingEQUATION)EQUATION)Plugging in and using σ τ (∆) = 1 -σ τ (-∆) yieldsEQUATIONEQUATION+ |P| |P B | c β ∈P α B \{cα} σ τ (s β -s α ) + δ + α (1 -σ τ (-∆)) + |N| |N B | |N α B |σ τ (-∆) + |N| |N B | c β ∈N α B σ τ (s β -s α ) + δ - α (1 -σ τ (-∆)) .Taking a Taylor expansion with respect to z ≡ σ τ (-∆) ≈ 0 around z = 0 to first order givesEQUATION)+ z • |N||P| |N B ||P B | (|P α B | -|P α B |) c β ∈N α B σ τ (s β -s α ) + δ - α   1 + |P| |P B | |P α B | + |P| |P B | c β ∈P α B \{cα} σ τ (s β -s α )   - |N| |N B | |N α B |   1 + δ + α + |P| |P B | c β ∈P α B \{cα} σ τ (s β -s α )   / 1 + |P| |P B | c β ∈P α B \{cα} σ τ (s β -s α ) + |N| |N B | c β ∈N α B σ τ (s β -s α ) + δ + α + δ - α 2 + O(z 2 )We see that the loss function used in the main paper (Eq. ( 9)) re-appears as the zeroth order term of this expansion, while the higher order term(s) provide an upper bound for the error due to approximating the sigmoids in the worst case.We pre-compute the 3D world coordinates of each patch in the dataset by backprojecting the centre coordinate into the scene using information about camera pose and depth values at that patch.During training, we sample the positive patch-pairs c α ∈ P by uniformly randomly sampling a patch from a scene and then using the highly efficient FAISS [11] library to find and sample a patch within the positive region of the original patch. This approach does not uniformly sample from all positive patch-pairs since patches that are part of a larger number of positive pairs are not more likely to be chosen, but computing the number of positive pairs that include a given patch to correct for this would be computationally unfeasible. This also does not change our derivations, since all pairs are still sampled from the same (even though non-uniform) distribution.In each training step, we extract feature maps for only those images that contain the patches for the c α pairs. This is to use our limited computational budget in a way that maximises |P ′ B |. The pairs c β ∈ P and c γ ∈ N are then sampled from among the patches in the images for which feature maps have been extracted. This samples using the same distribution as the c α pairs, since for each c α the other images in the batch represent merely a uniformly random restriction of the patches that can be sampled.One of the contributions of this paper, selecting pairs of patches that do not saturate a sigmoid function in the loss function (cf . sec. 3.5), is reminiscent of the better-known technique of hard negative mining in self-supervised learning [21, 22, 33] . However, there are three important differences: Firstly, our method selects pairs of samples rather than individual hard (negative) samples, a result of our general reformulation of the problem of learning location-consistent features. In hard negative mining, given a single anchor sample, other samples are selected based on their features, whereas in our case, given a pair of samples (patches), other pairs of patches are selected based on their similarity.Secondly, while hard negative mining is a one-sided filter, our method is two-sided. We do not just filter out 'too easy' pairs like hard negative mining, i.e. pairs that are easily and correctly classified into positive or negative pairs, but also 'too difficult' pairs, i.e. pairs that are confidently but incorrectly classified. Both kinds of sample pairs saturate the sigmoid function and do not contribute to the gradient. Most standard contrastive learning loss functions do not contain a sigmoid function, so such a two-sided filter is not appropriate there.Thirdly, hard negative mining approaches aim to improve downstream model performance by placing more weight on difficult samples. The hard negative sampling changes the loss function to learn more useful features. In contrast, our method aims exclusively to reduce the training algorithm's memory consumption without impacting the loss landscape.We also trained a model using a more standard contrastive learning loss together with hard mining. We again use a modified version of the model architecture of DINO-Tracker [41] , training a fully convolutional neural network to learn residual features to frozen DINO [5] features.Instead of our loss, we train using the noise contrastive estimation (NCE) loss function [14] . From a batch of 24 images, we sample 256 query patches, for each of which we sample 128 positive and 384 negative patches.We use hard sample mining for both positive and negative samples. Inspired by Robinson et al. [33] , we sample the negative samples randomly from among the patches in the negative region with a probability ∝ exp(βs ij ), where s ij is the similarity between the query patch and the potential negative patch. The coefficient β is a hyperparameter that we set to β = 0.1. Similarly, we sample the positive patches from among the patches in the positive region with a probability ∝ exp(-βs ij . This means that we over-sample negative patches with high similarities and positive patches with low similarities, as these are the worst-performing samples.The results of this model on the pixel correspondence estimation task are shown in Table 4 . Though it outperforms the baseline models and does comparatively well at large viewpoint changes, our loss is more suitable for learning multi-view consistent features. Table 4 : Results on the pixel correspondence task on the Paired split [36] of ScanNet [10] , as introduced by El Banani et al. [12] . We report the recall of accurate pixel correspondences at a reprojection error threshold of 10 pixels, for image pairs with the respective viewpoint changes. Model 0 • -15 We also demonstrate the utility of location-consistent features on the task of Visual Place Recognition (VPR). Given a set R of reference images and a set Q of query images with Q ∩ R = ∅, a ground truth function g : Q → 2 R maps a query image to the subset of reference images that show the same 'place' as the query image. Since the reference set is typically large, here we focus on global descriptor methods, which compute a global descriptor for each image and for each query image retrieve the most similar reference images.Recently, Keetha et al. [23] demonstrated that excellent place recognition performance can be achieved using a simple recipe: extract patch-level feature maps using a pre-trained feature extractor, then aggregate these into a global image descriptor using the VLAD [20] algorithm. We follow this approach and report the results for different feature extractors. We use 128 clusters for VLAD and construct the VLAD vocabulary only from the respective set of reference images.F.1 Datasets.The indoor datasets that are typically used to evaluate VPR methods (e.g. Baidu Mall [38] , Gardens Point [39] , and 17Places [35] ) tend to suffer from one of two defects. For some, like Gardens Point, the development of large-scale pre-trained image models means that VPR performance is strong enough that these datasets can no longer resolve small performance differences, with Keetha et al. [23] reporting 99.5% Recall@5 for three variants of their method. Others, like Baidu Mall and 17Places, suffer from poor ground truth. Reference images labelled as 'ground truth' frequently have no image overlap with the query image or do not include images with significant overlap as ground truth. We therefore use the pose and depth information available with the Matterport3D and ScanNet datasets to construct VPR datasets for which the IoU between query and ground-truth images falls between 0.2 and 0.4. This ensures that the place recognition task is challenging while remaining tractable for purely vision-based systems. The code to deterministically generate the datasets will be released publicly.The images that comprise these datasets are drawn from environments that were unseen during training. For Matterport3D, using images from 18 unseen scenes results in 8637 reference and 2194 query images, and for ScanNet, using images from 21 unseen scenes results in 4295 reference and 239 query images.We give Visual Place Recognition results in Table 5 , quoting the Recall@1 and Recall@5. While the performance of our LoCo features remains respectable, it does not outperform the baseline feature extractors by as much as in the scene-stable panoptic segmentation task. This is potentially due to the VLAD feature aggregation discarding information about the relative arrangement of different feature vectors in the image, thereby removing a significant advantage of location-stable features compared to feature extractors focused more on semantic information like DINO. In Fig. 4 , we show point-cloud reconstructions obtained by projecting the pixel patches of a set of images showing a living room into 3D, and colouring the each point by the similarity of its patch's feature vector with a query feature vector taken from a query patch. The query patch looks at a point on the wall to the top left of the picture hanging on the wall. The upper image shows the similarities as reconstructed using our LoCo method, the lower as reconstructed using DINO [5] features.The region of space where feature vectors have a high similarity with the query feature is much more localised for our method than for the DINO features.The training algorithm described in the paper assumes the availability of camera poses and depth maps for each RGB image. While a number of datasets exist for whom this information is available, this may not always be the case. There exist a number of comparatively accurate algorithms to estimate this information, but the paper does not investigate whether such methods are sufficiently accurate to reproduce the stated performance.The paper also limits itself to training comparatively small convolutional networks, or finetuning only a small number of layers of a pre-trained foundation model rather than training a similarly sized model from scratch. There is therefore the possibility that the training algorithm described requires an initialisation with relatively high-quality features and is not entirely appropriate for training visual foundation models from scratch.Thirdly, the paper only runs experiments training on a comparatively small dataset only containing a few dozen different environments. While all testing is performed on unseen environments, the paper does not investigate the existence of scaling laws as available training data increases.This work, on memory-efficient learning of 3D-consistent image features, has potential positive and negative social impacts. On the positive side, reducing the memory and compute requirements of a common type of self-supervised loss (a ranking loss) permits a wider pool of researchers and interest groups to investigate and train models that use such losses, on restricted compute budgets. In addition, self-supervised learning reduces the need for often exploitative human labour practices associated with obtaining cheap labelled data.On the negative side, training large self-supervised models has considerable environmental impacts due to carbon emissions that lead to significant societal impacts, including direct consequences (displacement, damage from extreme weather events, etc.) and indirect consequences. Moreover, location-consistent image features are likely to be useful for tracking, which can be invasive of privacy and can be used for malicious purposes.(a) Similarities of the features of patches taken from a set of images with a query feature vector, using our LoCo features, projected into 3D. The query feature vector's location is to the top left of the picture on the wall.(b) Similarities of the features of patches taken from a set of images with a query feature vector, using DINO [5] features, projected into 3D. The query feature vector's location is to the top left of the picture on the wall. Guidelines:• The answer NA means that the abstract and introduction do not include the claims made in the paper. • The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers. • The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings. • It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper. 


Decision: This paper received mixed reviews. While reviewers praise the interesting ideas, efficiency and good presentation, there were concerns around potentially unfair comparisons and unjustified hyperparameters. The rebuttal clarified several concerns, and there was a good discussion between the reviewers to weigh the pros and cons. Even though there was no final consensus on the ratings, the AC recognizes that the work can be accepted into NeurIPS, given all its positive aspects and the additional information provided in the rebuttal. We require that the clarifications and all relevant information provided in the rebuttal be incorporated in the camera-ready version.

Reviews: review: paper_summary: paper_summary: This paper presents a new training scheme for vision foundation models. The key goal of the paper is to enhance the multi-view consistency of vision foundation models. To this end, the paper revisits the idea from soft average precision, and applies the idea for training vision foundation models. Specifically, the loss loosely enforces similar score values for positive patch pairs, unlike explicitly enforcing positive / negative affinities to reach a pre-designated value as in conventional contrastive learning approaches. The authors further propose pruning positive and negative samples during backpropagation to ensure memory efficiency. Experiments demonstrate that the proposed method can outperform tested baselines in local feature matching, panoptic segmentation, and visual place recognition.
strength_weakness: strength_weakness: 1. My major concern is with the experiments. First, it is unfair to compare existing vision foundation models directly against the proposed method, since the method is additionally trained using Matterport3D data. The baselines in Table 1 should also have been additionally fine-tuned on Matterport3D. Further, since the paper is proposing a generic loss function applicable to any pre-trained vision foundation model, the technical contributions would have been better elucidated if the method improved multi-view consistency for other vision foundation models as well, for example pre-trained DINOv2 features.

2. The motivation for setting the saturation threshold to 0.076 in L220 is unclear. Why is the gradient being 0.2% of the "maximum gradient of the sigmoid function" important for training models with the proposed loss?

3. The technical contributions of the paper is unclear. From a methodological point of view, the paper adapts soft average precision loss from prior literature with a threshold-based pruning to ensure memory efficiency, which is not strongly novel. Further, the experimental results are largely limited to fine-tuning a small CNN operating on top of DINO features. Therefore the scalability of the proposed training scheme is unclear.
questions: questions: Please refer to the weaknesses section above. Below are a few additional questions:

1. What are the exact computational requirements of the paper? The only place that I could find hardware requirements was L302. Is a single RTX8000 GPU all we need?
2. What is the reason for training the model on Matterport3D? I feel datasets such as ScanNet will contain a much more diverse range of scenes. Since the only model being trained is the feature refinement CNN, training on ScanNet will not be prohibitively expensive.
limitations: limitations: Yes the limitations are stated in the supplementary material.
paper_summary: paper_summary: The paper introduces a memory efficient loss for location consistent image features. The paper addresses the problem of high memory footprint of previous work, by significantly reducing the memory footprint, by 3 orders of magnitude. Memory efficiency is achieved by sampling the positive pairs from a smaller subset and using a threshold on similarity differences between pairs, and discarding the pairs with gradients close to 0, which do not contribute to the loss. To compensate for the non-uniform sampling, some correction terms are added to the loss.
strength_weakness: strength_weakness: * relies on existing 3D mesh reconstruction of the environment (for segmentation masks of individual objects in the scenes) -- this limits evaluation to datasets with existing mesh reconstructions.
* while direct comparison with LoCUS on the same feature dimension is not possible due to high memory consumption, it would be interesting to see a comparion with LoCo on a feature dimension d=64 (Tab 2) for a fair comparison.
questions: questions: How does LoCo perform with features coming from different models (e.g. DINO vs DINO v2)?
limitations: limitations: - limited to datasets with 3D reconstructions available.
paper_summary: paper_summary: This work focuses on the best strategy to pre-train a feature extractor network optimized to be invariant wrt to the viewpoint in the image. To do so the authors use a dataset of paired views from which they extract pairs of positive patches (same 3D point in two different views) and negatives (different 3D points). Given these sets a feature extractor model is trained for patch retrieval using a loss proposed by the authors. In particular the paper starts from the existing Average PRecision Loss and adapts it to make it significantly more memory efficient. Provided with the data and the loss the authors train a tiny CNN adapter on top of Dino features to make them more robust wrt to the viewpoint. The performance of the trained model is evaluated on few ad-hoc benchmarks (multi view feature consistency, consistency of panoptic segmentation and visual place retrieval) where the proposal achieves improvements wrt to the competitors.
strength_weakness: strength_weakness: a. Limited experimental validation: while the author tested their model against competitors across 3 tasks, these are mostly internal comparisons on tasks that would privilege the proposed model. In particular, Table 1 where the method has the biggest gains, is basically measuring the performance on the task LoCo was trained on vs models that were not trained for the task (except for LoCUS). Regarding LoCUS in table 1 it is shown with a † that has no match in the caption of the table, following the rest of the paper I’m assuming it means that the method only has dimensionality 64 for features.  It would have been interesting to include in the evaluation some of the correspondence based tasks that CroCo-v2 has been exploring like stereo depth estimation, optical flow or relative pose estimation.

b. Generalization: the model is trained on samples from matterport3D and evaluated on unseen scenes from Matterport and on scenes from ScanNet. This constrains quite a lot the scenarios being tested and does not give insight on the generalization capabilities of the proposed method. For example how would it perform in very different scenarios like an outdoor scene? Note that there are plenty of  dataset for evaluation of correspondences method that could be used for this task like [MegaDepth](https://www.cs.cornell.edu/projects/megadepth/)

c. Potential unfair comparison to competitors: the closest competitor that has been trained with a comparable objective to the one proposed by the authors is LoCUS that according to the evaluation in Tab.1 and Tab. 2 achieves consistently worse results. However LoCUS uses only 2 FC layers on top of the features f a frozen DINO (500K parameters) while the proposed methods use a full CNN (~20M parameters), as such the comparison is not particularly fair. It would have been interesting to have a version of the LoCo method trained in the exact same settings as LoCUS (models and data) to verify what is the gain or losses of the proposed new loss function compared to the one introduced in LoCUS.
questions: questions: **Questions**


1. Can you comment on weakness [b] and whether generalization  is to be expected or very limited?

2. Can you comment on weakness [c] 

**Typos**

* L 166: Eq. 12 → Eq. 3?
limitations: limitations: Limitations have been adequatelly discussed in the manuscript.

Comments: title: Acknowledgement
comment: Thank you for considering our response and for clearly outlining your main concerns. We appreciate the opportunity to address your remaining concerns and provide further clarification below:

1. We acknowledge the concern and will include this fine-tuning experiment in the final revision; we need more time to complete this for all compared methods. However, we would like to emphasise that one of the compared methods (CroCo-v2) is already trained with a multi-view loss on a very much larger multi-view dataset (a combination of simulated Habitat images and real images from ARKitScenes, MegaDepth, 3D Street view, IndoorVL; a total of 5.3 million image pairs) than Matterport3D and so has a significant advantage over LoCo, especially on the ScanNet experiments. LoCo’s better performance on the validation tasks highlights the benefits of explicitly including location-consistency as a goal of the loss function.

2. The 0.2% threshold is arbitrary, albeit selected empirically (see Table 1 of the paper). It represents a trade-off between performance and memory-efficiency. Setting it to 0.2% of the maximum gradient leads to minimal impact on the gradient signal (and minimal impact on performance) while achieving a 5x memory reduction. In this respect it resembles other hyperparameters used in training algorithms, e.g. learning rates, where the practitioner’s choice is guided by empirical performance rather than theoretical guarantees of optimality.

We hope these additional clarifications address your concerns more comprehensively. We are committed to improving our work and appreciate the constructive nature of your feedback. Thank you once again for your valuable insights.
Thank you, we will include these in the main paper to better reflect the setting and contributions of our proposed approach.
Thank you - we will definitely include this result in our revision and agree that it helps more concretely pinpoint the advantages of the proposed approach.
Thank you for the response. While the rebuttal has allayed some of my concerns, I am willing to keep my initial rating. Here is why:

1. I acknowledge that the authors have conducted experiments regarding generalization on other datasets such as ScanNet. However, my major concern was that all the tested baselines should have been fine-tuned on the training dataset for LoCo (i.e., Matterport3D), and then be evaluated on the test set. Otherwise, the comparisons will not be fair, since only LoCo had access to the multi-view information from Matterport3D.

2. Regarding the motivation of the saturation threshold, I am still not clear why "0.2% of the maximum gradient of the sigmoid function" is a key desiderata for choosing its value. Where does the number "0.2%" come from? The authors' explanation is not sufficient to clarify their decision on the saturation threshold value.
I would like to thank the authors for their responses and the other reviewers for their thoughtful feedback and questions.

I think the motivation of choice of Matterport (raised by Reviewer jhZZ ) and apples to apples comparison with LocUS (raised by Reviewer aW6W) should make it to the main paper, emphasizing the reduced memory footprint of the proposed method.
Thank you authors for submitting a detailed rebuttal.
I would suggest to incorporate the result of LocUS (w Loco architecture) in the future versions of the work because they help quantifying the effect on performance of the tradeoff introduced by the memory optimized loss function used.