Exploiting Distribution Constraints for Scalable and Efficient Image Retrieval
Authors: Mohammad Omama
Po-han Li
Sandeep P. Chinchali
Venue: ICLR 2025 Poster (2025)
Keywords: image retrieval
efficiency
foundation models
Primary area: unsupervised, self-supervised, semi-supervised, and supervised representation learning
TL;DR: AE-SVC improves image retrieval by refining foundation model embeddings, boosting performance by up to 15.5%. (SS)2D enhances efficiency by learning adaptive-sized embeddings, adding a 10% performance gain for smaller sizes.

Abstract: Image retrieval is crucial in robotics and computer vision, with downstream applications in robot place recognition and vision-based product recommendations. Modern retrieval systems face two key challenges: scalability and efficiency.
State-of-the-art image retrieval systems train specific neural networks for each dataset, an approach that lacks scalability. Furthermore, since retrieval speed is directly proportional to embedding size, existing systems that use large embeddings lack efficiency. To tackle scalability, recent works propose using off-the-shelf foundation models. However, these models, though applicable across datasets, fall short in achieving performance comparable to that of dataset-specific models. Our key observation is that, while foundation models capture necessary subtleties for effective retrieval, the underlying distribution of their embedding space can negatively impact cosine similarity searches. We introduce Autoencoders with Strong Variance Constraints (AE-SVC), which, when used for projection, significantly improves the performance of foundation models. We provide an in-depth theoretical analysis of AE-SVC. Addressing efficiency, we introduce Single-Shot Similarity Space Distillation ((SS)2D), a novel approach to learn embeddings with adaptive sizes that offers a better trade-off between size and performance. We conducted extensive experiments on four retrieval datasets, including Stan-
ford Online Products (SoP) and Pittsburgh30k, using four different off-the-shelf foundation models, including DinoV2 and CLIP. AE-SVC demonstrates up to a 16% improvement in retrieval performance, while (SS)2D shows a further 10% improvement for smaller embedding sizes.

Published as a conference paper at ICLR 2025
EXPLOITING DISTRIBUTION CONSTRAINTS SCALABLE AND EFFICIENT IMAGE RETRIEVAL
FOR
Mohammad Omama, Po-han Li, Sandeep Chinchali
The University of Texas at Austin
{mohd, omama, pohan1 1 i, sandeepc duteeeeepppeuttxxxeeeeeeee
ABSTRACT
Image retrieval is a crucial problem in robotics and computer vision, with down- stream applications robot place recognition and vision-based product recom- mendations. Modern retrieval systems face two key challenges: scalability and efficiency. State-of-the-ar image retrieval systems train specific neural networks for each dataset, an approach that lacks scalability. Furthermore, since retrieval speed is directly proportional to embedding size, existing systems that use large embeddings lack efficiency. To tackle scalability, recent works propose using off- the-shelf foundation models. However, these models, though applicable across datasets, fall short in achieving performance comparable to that of -ataset-speccfic models. Our key observation is that, while foundation models capture necessary subtleties for effective retrieval, the underlying distribution of their embedding space can negatively impact cosine similarity searches. We introduce Autoen- coders with Strong Variance Constraints (AE-SVC), which, when used for projec- tion, significantly improves the performance of foundation models. We provide an in-depth theoretical analysis of SVC. Addressing efficiency, we introduce Single-shot Similarity Space Distillation ( ((SS) 2D), a novel approach to learn em- beddings with adaptive sizes that offers a better trade-off between size and perfor- mance. We conducted extensive experiments Stanford Online Products (SoP) and Pittsburgh30k, using four different off-the- shelf foundation models, including D2 and CLIP. AE-SVC demonstrates up to a 16% improvement in retrieval performance, while (S5) shows a further 10% improvement for smaller embedding sizes.
four retrieval datasets, including
INTRODUCTION
Image retrieval involves finding the closest match of given image (query) in vast database of images (often called the reference set). It has numerous applications, from product recommendations in e-commerce to place recognition in robotics. In general, state-of-the-as (SOTA) image retrieval systems are dataset-specific. They are trained in a contrastive manner on a training split of the data, typically hand-labeled for positive and negative pairs for each query. Imagine scenario where a server has a vast reference database of fashion clothing images and receives query images from users find the closest matches in the reference set. Creating a separate split and hand-labeling positive and negative pairs to train a dataset-specific model is infeasible. A more scalable solution use off-the-shelf feature extractors like Dino (Caron et al., 202T), DinoV2 (Oquab al. 2024), CLIP (Radford et al., (021). etc. However, the performance of off-the-shelf feature extractors is generally lower compared to dataset-speeific models. Q1 (Scalability): How can we enhance the performance of these off-the-shele models in completely unsupervised way?
Another key problem in retrieval that the retrieval speed directly proportional to the size of the embeddings. For embedding of size d and reference set with N image vectors, the retrieval speed for a single query is O(d .). This retrieval time becomes significant as the size the refer- ence set (N) increases, with modern retrieval systems containing millions of images. Additionally, the embedding size directly affects the storage space required for the reference set and the com- munication bandwidth needed to send the query vector to the server in distributed image retrieval systems, which are common in product recommendations and robotic place recognition. Standard dimensionality reduction techniques, such Principal Component Analysis (PCA), aim preserve

---

Published as a conference paper at ICLR 2025
(A)
Training Step
(C)
Inference
(SS)_D
Foundation Model
Adaptive Embedding
Foundation Model
Constraints
Training Step
Results
(SS).D.
Make Similarity Spaces of Student
0.5
and Teacher Similar (KL Divergence)
m4P04
101
0.1
NxN
NxN
(oppp (Cums)
Student Simiarity Spaces
Simlarit
Space
GFLOPs
Figure Two-step pipeline for the proposed approach. (A) AE-SVC (discussed in Sec. 3.1 trains an autocncoder with our constraints to improve foundation model embeddings. (B) (SS) 2D (discussed in Sec. 3.2) uses the improved embeddings from AE-SVC to learn adaptive embeddings for improved retrieval any embedding size. (C) Once trained, (SS) 2D be directly applied foundation model embeddings to generate adaptive embeddings for improved retrieval. (D) AE-SVC (orange) boosts performance significantly, while (SS) 2D (green) further enhances results with smaller embeddings. Dino (blue) achieves optimal performance at 9 GLOPs, whereas (SS) 2D on top of AE-SVC achieves similar performance only 2.5 GLOPs.
information, while Autoencoders (AEs) (Hinton & Salakhutdinov 2006) and Variational Autoen- coders (VAEs) (Kingma & Welling 2013 focus reconstruction quality. However, these methods approaches like AES and VAEs re-
are not optimized for retrieval tasks. Additionally, learning- quire separate training for each embedding size, which is impractical given varying compute, space, and communication constraints. Q2 (Efficiency): Is there an effective unsupervised dimension- ality reduction method that strongly preserves the similarity structure the full embeddings, and is adaptive, i.e., does not need to be trained for each dimension separately?
address Q1 (Scalability), we propose Autoencoders with Strong Variance Con- straints (AE-SVC). Our primary idea is that, while foundation models capture the necessary details for effective retrieval, the underlying distribution of their embeddings can negatively impact cosine similarity searches. AE-SVC trains an autoencoder while enforcing three constraints on the latent space: an orthogonality constraint, a mean centering constraint, and a unit variance constraint. empirically show and mathematically prove that these constraints cause shift in the cosine simi- larity distribution, making it more discriminative. These constraints not only lead to more effective dimensionality reduction but also outperform the complete embeddings of foundation models. We discuss the motivation and technical details of AE-SVC in section 3. We also provide an in- depth mathematical analysis of SVC in section To address Q2 (Efficiency), we propose Single Shot Similarity Space Distillation ( ((SS) 2D). (SS) 2D aims at reducing embeddings to smaller ones while preserving their similarity relationships. The embedding learned with (SS) adaptive, i.e,, smaller segments of the output embedding also perform well in retrieval tasks. In summary, our approach consists of two steps (as shown in Fig. []] first, we use AE-SVC enhance the baseline performance of foundation models. Then, we leverage the improved embeddings teacher for efficient dimensionality reduction through (S.)
Contributions:
Fig. (D) also shows the retrieval performance versus retrieval speed in giga-FLOPs (floating oper- ations) on the Pittsburgh30K dataset (Tori et al. 2013)) using the Dino model (Caron et al. 202 Note that retrieval speed is function of the dimension d and the reference set size N, O(d x N), with N = 17, 000 for Pittsburgh30K. The performance of PCA (blue) represents the default off- the-shelf Dino model. We of the off-the-shelf Dino model (blue). Notice that outperforms the complete Dino embedding. Applying (5S) 2D top of AE-SVC further improves retrieval performance at smaller embedding
that AE-SVC (orange) significantly enhances the performance
2

---

Published as a conference paper at ICLR 2025
sizes (green). Section experimentally shows the advantages of the proposed approaches in detail on four different datasets using different foundation models.
2 RELATED WORK
Deep Metric Learning: Image retrieval is modeled as a Deep Metric Learning (DML) problem (Wang et al. 2017). DML focuses on learning latent representations where similar items are close and dissimilar items are far apart. Various techniques have been proposed (Oh Song et al. 2016
Schroff et al. 2015 Ustinova & Lempitsky 2016; Sohn 2016; Wang et al., 2019) to learn mcan-
ingful latent representations for effective retrieval, including contrastive loss (Hadsell et al.. 2006)
and triplet loss (Schroff et al. 2015). More recent works use advanced N-pair loss (Sohn (016) or multi-similarity loss (Wang et al. (019). Analogous to image retrieval, Visual Place Recognition (VPR) is also modeled
DML problem (Garg al. (02T). Various training objectives have been proposed for VPR (Ge et al. 2020 Xiao et al. 2023 -eyva-Valling et al., [2023] Berton et al. 2022]
Shubodh et al.] 2024) learn latent representations for effective retrieval. However, all of these approaches are dataset-specific, meaning they train specific neural networks for each dataset.
Foundation Models in Image Retrieval: The rise of foundation models, such as DINOv2 (Oquab) itt al. 2024). CLIP (Radford et al., (02I). ViT (Dosovitskiy et al. 202T). has allowed recent works to explore the applicability of these off-the-shelf models image retrieval. As shown in (Keetha et al. 2023 Lu et al. 2024), off-the-shelf models, in some cases, can have comparable retrieval per- formance to -ataset-specific models. However, these approaches either use the off-the-shelf models as-is (Keetha et al. 2023 Omama et al. 2023 han Li et al., 2024) or adapt them using labeled held-out dataset (Lu et al., 2024). In stark contrast, our approach, AE-SVC, exploits the inher- ent properties of the distribution improve the performance of these off-the-shelf models without requiring any labeled data.
Distribution Constraints in Semi-supervised Representation Learning: Semi-supervised learn- ing has seen significant advances with methods that leverage self-supervised objectives to learn robust representations. A notable approach is Barlow Twins, which employs additional cross- correlation loss to align representations while minimizing redundancy across dimensions, promoting feature decorrelation (Zbontar al. (021). Extending this idea, VICReg introduces an additional variance constraint that ensures sufficient spread in the learned features, preventing collapse and further enhancing representation quality (Bardes al.) [022].. However, date, no work has specif- ically explored the impact of such constraints in the context of autoencoders for retrieval tasks. In this paper, we are the first to address this gap, providing a comprehensive theoretical analysis of these constraints and their impact on retrieval.
Similarity Preserving Knowledge Distillation: Similarity preserving knowledge distillation (SPKD) focuses transferring knowledge from a teacher model to student model, similar to standard distillation (Hinton et al. 2013 Buciluà et al. 2006 Ba Caruana 2014 Zagoruyko
& Komodakis 2017 Zhang al. 2019 Passalis & Tefas (2018). while maintaining the structure of similarity of the data in the feature space. One of the pioncering works in this area is the Fit- Nets approach, which uses intermediate representations guide the student model (Romero al., 2014). The attention transfer method extends this by aligning the attention maps of the student and teacher models (Zagoruyko Komodakis] 2017). More recent advancements include (Tung & Mori 2019)), which explicitly preserves pairwise similarities between data points in the feature space during distillation. Contrastive Representation Distillation (CRD) further improves performance by using contrastive loss to maximize mutual information between teacher and student representations (Tian et al., (2220). Additionally, the Relational Knowledge Distillation (RKD) approach emphasizes the importance of preserving the relational knowledge among data points (Park et al., 2019). All of these works focus on training a lightweight student network to mimic heavy teacher network. We are the first to explore this idea in a dimensionality reduction setting for image retrieval.
Adaptive Feature Embeddings: A core requirement for dimensionality reduction in retrieval sys- tems is adaptive embeddings due to varying compute constraints. Previous works on adaptive feature embeddings have created efficient and adaptive embedding spaces for compression (Li et al.) 2023) and image classification (Kusupati et al., 2022), while ours focuses image retrieval. The key idea is to use only one neural network model to output a feature embedding ensuring that smaller sub-chunks of the embedding also perform well in retrieval tasks.
3

---

Published as a conference paper at ICLR 2025
3 METHODOLOGY
We now formally define the image retrieval problem:
Suppose we have a query set Q of M query image feature vectors and a reference set R of N reference image feature vectors, where each vector is d-dimensional. Given qj IR randomly drawn from Q, the task to identify the feature vector r-- R that has the smallest distance or highest similarity with qj- This be mathematically expressed as:
Ti- arg min dis(g) Ta); reff
with the distance dis generally based on cosine similarity, i.., dis(gj, ri) = - cos(g),7) ri), where cos denotes the cosine similarity.
We will now describe our approach in detail. First, we will explain the step-by-step procedure for AE-SVC. Next, we will present an in-depth explanation of 2D. Note that both AV-SVC and (D utilize only the reference set R during the training time. The query set Q is assumed to be unavailable to the user during training. A standard assumption when doing dimensionality reduction in image retrieval settings (Keetha et al. 2023).
3.1 AUTOENCODERS WITH STRONG VARIANCE CONSTRAINTS (AE-SVC)
Our primary motivation for AE-SVC that, while foundation models capture necessary subtleties for effective retrieval, the underlying dis- tribution of their embedding space can negatively impact cosine similarity searches. In Sec. we discuss in mathematical detail that the discrimina- tive power of the retrieval task is maximized when the variance of the cosine similarity distribution minimized. AE-SVC introduces three constraints on the latent space: an orthogonality constraint, a mean centering constraint, and unit variance constraint. In Sec. 4 we prove that these constraints on the la- tent space minimize the variance of the cosine simi- larity distribution. We now provide a more intuitive explanation of why these constraints are helpful, Consider the example illustrated in Fig. 2 We have a reference set comprising four categories of clothing: men's tank-tops, men's 's half-shirts, women's 's tank- tops, and women's 's half-shirts. The task is to match the query with the correct clothing type. After applying dimensionality reduction (like standard PCA) to the reference set, we obtain two orthogonal dimensions: a person dimension (with high variance) and shirt shape dimension (with low variance). If similarity is overly dependent on the high-variance dimension, as shown mathematically in Sec. 4 retrieval performance suffers. For in- stance, query for white man in tank-top might incorrectly match with a white man in half-shirt due to the dominant person dimension. Ideally, we need a space where both the person and shirt di- mensions equally influence the distinction between men and women, well between half-shirts and tank-tops. The goal of AE-SVC is to learn projection from the reference set R such that the latent dimensions are orthogonal and have equal variance.
Motivation:
Query
Reference
AE-SVC
standard dimensionality reduc- tion (say PCA), cosine similarity is dispropor- tionately influenced by high-variance dimen- sions, leading to poor retrieval. Given task to match the query with the correct clothing type. query image a white man a tank top may be incorrectly matched to white man in half-shirt due to the dominant person dimension. Ideally, both orthogonal dimensions should have an equal influence on cosine similarity.
Figure 2:
AE-SVC (as shown in Fig. [A)) takes an input embedding I E Rd coming from the foundation model, encodes it to a latent representation Rd. We have three additional losses to enforce the constraints mentioned above: constraint, a mean centering constraint, and a unit variance constraint. Note that we can choose any size for the latent representation We show the results for multiple sizes of 2. However, since AE-SVC is used to guide (SS) 2D later, we will use the best performing embedding, i.e., z E Rd for this guidance.
Approach:
IRd, and produces a reconstruction I' e
orthogonality
4

---

Published as a conference paper at ICLR 2025
Reconstruction Loss: We minimize the Mean Squared Error (MSE) between the imput embedding I and the reconstruction l':
Lue
14 1.1.2.
where n is the number of samples.
Covariance Loss: To promote orthogonality, we penalize off-diagonal terms of the covariance ma- trix:
1 (z (Z 11)
Len
where Z E IRad is the matrix of latent representations, / IR is the mean of Z, and II is the identity matrix.
Variance Loss: We constrain the variance of each latent dimension to be one:
Lowr Var(zz)) - -1)²
where 2" is the i-th latent dimension of and Var(=*) is the variance of the i-th latent dimension across all n samples.
Mean-Centering Loss: To ensure the latent space is centered around zero, we minimize the mean of the latent dimensions:
Lmcan
(a))
(4)
where n² is the mean of the i-th latent dimension across all n samples.
Total Loss: The final objective combines all four losses:
LARSVVC Aue + dea Aun + Amean Creen
(5)
where Arec Aaw; Avar and Amean hyperparameters diseussed in A.6 By optimizing this loss, AE-SVC learns a latent space that improves retrieval performance through accurate reconstruction, decorrelated and normalized feature dimensions, and zero-centered mean. The latent representa- tion, = E Rd, is then used to guide the training of 2D.
3.2 SINGLE SHOT SIMILARITY SPACE DISTILLATION ((SS)
Motivation: As previously mentioned, embedding size significantly impacts retrieval speed. For an embedding of size d and reference set with N image vectors, the retrieval speed for a single query O(d x N). Therefore, is crucial to reduce the embedding size while maintaining performance. Standard dimensionality reduction techniques, such as PCA, Autoencoders (AEs), and Variational Autoencoders (VAEs), are not optimized for retrieval. Additionally, AEs and VAES require separate training for each embedding size, which is impractical given varying computational constraints. Therefore, there is a need for adaptive embeddings that maintain retricval performance and are flexible, ensuring that smaller sub-chunks of the embedding also perform well in retrieval tasks.
Previous works have employed Sinierrtty-rreeerinng Knowledge Distillation (SPKD) both in hidden layer activations (Smith al. 2023) and output layers (Wu al.) 2022). SPKD trains a student network to ensure that input pairs producing similar (or dissimilar) activations in the teacher network yield similar (or dissimilar) activations in the student network. However, these stud- ies have not explored cenilarity-preeerviin approach is conceptually similar SPKD, but rather than focusing on network distillation (teacher to student), we perform umbedding-leve distillation, reducing a large embedding to smaller one while preserving similarity relationships. Using SPKD directly for dimensionality reduction requires training separate networks for each dimension size. Previous works on adaptive embeddings have utilized random projections in distributed compression setting (Li et al., 2023) or optimized multi- class classification loss for each nested dimension (Kusupati et al. 2022). Following the approach in (Kusupati et al. 2022), we optimize a similarity preserrin loss for each nested dimension. By combining similarity preseeripge loss with adaptive embedding training pipeline, introduce our approach, Single Shot Similarity Space Distillation, or (SS) 2D. Fig. T(B) illustrates the central idea of (5S) 2D which will now describe in detail.
Approach:
distillation for efficient dimensionality reduction. Our
5

---

Published as a conference paper at ICLR 2025
Let Rdd be the latent representation of the reference set obtained via AE-SVC. Our goal is to learn a smaller embedding such that the performance of the smaller embedding approximates that of the full embedding. In essence, the complete embedding serves emheddings. Although (SS) 2D could, in theory, directly use the foundation model embedding for guidance, the latent representation is shown to perform better in retrieval tasks, thus providing stronger guidance.
guide for creating smaller
Consider set M C [d] of embedding sizes and a neural network F(; 0) parameterized by We want to learn a projection å F(1;0) 0) such that each of the first m dimensions, for m E MM, of the learned embedding vector (2 1 R" independently preserves the retrieval performance of z. To learn the function F, we first compute a cosine similarity matrix C E R NNN by calculating the cosine similarities of every with every other in the reference set R. We refer this matrix as the teacher's cosine similarity space. Let C; denote the 7th row of C.
Next, for each m E .,, we define a student similarity space Č"M R N.N calculated using 1mm We then introduce a Kullback-Leible (KL) divergence loss 1n between Č and C for each m as follows:
1 Σ
DKL DKL(CTT (C,).
(6)
The overall loss L is then expressed as:
L (S8)
Σmm
1T Σ> DKL (CP
D
(7)
4 THEORETICAL ANALYSIS OF AE-SVC
In Section 3.1
we stated that the discriminative power of the retrieval task is maximized when the variance of the cosine similarity distribution is minimized. Given background distribution B (in our case, the distribution the reference set) and a distribution S representing vectors of a specific class, define the discriminative power using statistical threshold raa), where is the quantile of B at confidence level a. Specifically, the discriminative power is the probability that the cosine similarity between vectors S,, Ss E S exceeds this threshold:
P (cos(Se, So) T(ai) erf (Ejcus(S... S.)] T(())
(8)
Varlloos Sa. St)
where erf denotes the error function. Eq. can be reformulated in terms of the variance of the background cosine similarity distribution, demonstrating that the discriminative power is maximized when this variance is minimized. For a detailed proof of this result, along with alternative approaches to modeling discriminative power, refer the reader Smith et al. (2023).
We now verify the results of Gaussian assumptions also hold for real datasets. Fig. shows the distribution of cosine similarities of the reference set feature vectors on the Pittsburgh30k dataset Torii et al. 2013). We compute cosine similarities between all pairs of feature vectors in the refer- ence set and plot the probability mass function (PMF) with and without AE-SVC for two models: the off-the--helf foundation model Dino (Fig. 3ab and the SOTA dataset-specific model Cosplace (Berton al. 2022) (Fig. 3b). Cosplace, which excels retrieval, exhibits a PMF with signifi-
cantly less variance compared to Dino, seen in the blue curves in Fig. 3a and 3b The lower variance indicates more discriminative feature vectors, which aligns with the findings from Eq. that smaller variance enhances discriminative power and retrieval performance. Applying AV-SVC reduces the variance in both models (orange curves), with a more pronounced shift in Dino than in Cosplace. This suggests that AE-SVC benefits the off-the-shelf foundation model more (10%) than the dateset- model (2%), as reflected in the retrieval performance plot in Fig. 3c remainder of this work focuses off-the-shell foundation models.
Thus, the
We have mathematically and empirically demonstrated that the discriminative power of the retrieval task is maximized when the variance of the cosine similarity distribution is minimized. Next, we will

---

Published as a conference paper at ICLR 2025
DINO-S8
Cosplace
Default
Default
Frequency
10
AE-SVC (Ours)
Frequency
30
AE SVC (Ours)
mApP
10
Dis will PCA
Drieat Dre
Dia with AE-UIC (Can)
Coplace with PCA
X Difult Caplace
Coplace with 42 SVC (Qan)
0.0
0.5
1.0
0
0.0
0.2
0.4
0.6
0.8
1.0
60
Similarity
Similarity
Embedding Dimension
(a)
(b)
(c)
Figure 3: AE-SVC reduces the variance of cosine similarity distributions in both foundation (a) and datasct-speciffc models (b), with a more significant shift in foundation models (a). This results in greater improvement in retrieval performance for the foundation model (Dino) compared to the dataset-specific model (Cosplace), shown in (c).
prove that the constraints introduced on the latent space in Sec. 3.1 indeed minimize this variance.
Let p(z) be the distribution of AE-SVE SVC, i.c., ~ p(z). By applying constraints in Eq. 2 and Eq. the p(z) has a zero mean, μ = and a diagonal covariance matrix E Rdxd with eigenvalues
X of where of is the variance of the individual dimensions of 2. Given two vectors X and Y,
X, Y p(z), the cosine similarity between X and Y is:
Σ E X.Y. VV,Y ((XTX)(YTY)
X
X X
cOs( (Y
99)
where subscript denotes the i-th element a vector and ||||| and |Y|l denote the Euclidean norms of vectors X and Y respectively. Each term in the numerator of Eq. i just product of two random variables, but the denominator makes things complicated. A closed-form solution of a ratio of two random variables is diffieult calculate.
Let us introduce a relaxation that can be used to approximate the denominator and simplify the calculation. Using cosine's scale invariance (shown in A.2). we define a relaxation r(z) as:
r(2)
(10)
where m o Relaxation r has two properties. First, the expected value of the squared norm after relaxation, denoted as [=]lr(=)| 2J, ), equal 1. Second, the ratio of variance to square of the mean is given by Σg 2n4 and it approaches 0 as the dimensionality d tends to infinity under 22
the condition that the contribution of any single covariance eigenvalue is sufficiently small. Smith al. (2033) show that these properties then be used approximate the cosine similarity as:
cos cos(XY)) = cos(r(X), Σ El r())))) [x())II]+((]
Σ (r)))
(11)
With this approximation, the moments of the cosine similarity can be easily calculated:
Elcos( X, Y)] = 0,
(12)
d Σ 1=1
Var(cos( (X,Y))
(13)
Σ o3)²
We from Eq. 13 the cosine similarity distribution is highly dependent the dimensions with high variance. Taking the gradient of Eq. 13 with respect to o; we get:
Var (X, Y)) = do?
(14)
We can see that the global minimum the approximation for Var X, Y)) ) is achieved when all individual variances of the original distribution equal, i.e., of 09 V i,j. This is enforced

---

Published as a conference paper at ICLR 2025
in AE-SVC by the variance constraint (Eq. 3). Therefore, theoretically and empirically justi- fied that AE-SVC minimizes the variance of the cosine similarity distribution, improving retrieval performance.
EXPERIMENTS
5.1 EXPERIMENTAL SETUP
Datasets: We evaluate our approach on four distinct image retrieval datasets: InShop (Liu et al, 2016), Stanford Online Products (SOP) (Song al. 2016), Pittsburgh30K (Tori al. (003)), and Tokyo (Tori et al. 2013). InShop and SOP are sfate-of-the-ar (SOTA) image retrieval datasets commonly used in metric learning research, while Pittsurgh30K and Val recognized as SOTA place recognition datasets, frequently used in robotics research.
To demonstrate the broad applicability of approach, ducted experiments using four different foundational models trained with a variety of techniques and pre-training datasets: CLIP (Radford al. 2021), DINO (Caron al. 202I). DINOv2 (Oquab)
Foundation Models Evaluated:
et al., 2024)), and VIT (Dosovitskiy et al. (021). The CLIP model was pre treitrd on the LAION- 2B dataset (Community (023) using a contrastive learning approach. Both DINO and ViT models were pre-trained in a sele-suue manner on the ImageNet dataset (Russakovsky et al.) 2015). The DINOv2 model was pre-trained in a self-supervised manner on the LVDD-42M dataset (Quua et al. 2023). For the DINO and DINOv2 models, we evaluated different model sizes: specifically, DINO-S8 with 22.7 million parameters and DINO-B16 with 300 million parameters, as well as
DINav2-Small with 22.1 million parameters and DINOv2-Large with 300 million parameters.
Settings: Our work presents two contributions: the introduction
novel space, AA-SVV and a new method for learning adaptive embedding, (SS) yD, on top of AE-SVC. Consequently, experiments are divided into two parts. First, we demonstrate that SV- significantly enhances the retrieval performance of foundation models we show that (SS) 2D facilitates more effective dimensionality reduction on top of AE-SVC, further improving performance at lower dimensions. We compare (SS) 2D with the following baselines: Variational Auto Encoder (VAE) and PCA. Additionally, we compare (SS) 2D with using Similarity Space Distillation applied each dimension separately, referred to as SSD. Note that SSD serves as a theoretical upper bound for (SS) 2D. We use mean Average Precision at k (mAP@ko @k) a standard metric for evaluating retrieval performance. We show additional results on the Recall k metric in
various datasets and model choices. Second,
Appendix A.5
5.2
RESULTS
InShop
SOP
Pittsburgh30k
TokyoVal
0.8
0.6
mAP@4
mAP84
mAP34
0.6
0.4
0.4
0.4
6.2
0.2
200
400
500
800
200
400
600
800
200
400
600
300
0
200
400
600
800
Embedding Dimension
Embedding Dimension
Embedding Dimension
Embedding Dimension
with
VIT with AE-SVC (Ours)
Default
DiNO-V2-Large with DINO-V2-Large AE-SVC (Ours)
VIT Dafault
DINO 80---1660 with PCA
DINO 916 with AE-SVC (Ours)
DINO Lange Default
Figure 4: AV- significantly improves the retrieval performance of foundation models. AE-SVC (solid lines) consistently outperforms the off-the-shelf foundation models, i.e., PCA (dashed lines), four datasets, achieving a 15.5% average improvement retrieval performance.
Fig. 4 shows the performance comparison between BVC and PCA across 4 datasets, InShop, Stanford Online Products (SOP), Pittsburgh30K, and TokyoVal, using foundation models (ViT, DINOv2-Large, and DINO-B16). The performance of PCA (dotted line) represents the default off-the-shelf performance of the foundation models, while the performance of AE-SVO is shown with solid lines. AE-SVC consistently outperforms PCA across all datasets and embeddings. For
8

---

Published as a conference paper at ICLR 2025
example, the Largee embedding (yellow) shows 24% improvement InShop, 10% on SoP, 2% on Pittsburgh30K, and 22% on Tokyo at full embedding size. Overall, AE-SVC achieves average improvement of 15.5% across all datasets and embeddings at full size. Also, AE-SVC consistently surpasses PCA at smaller embedding sizes, providing a hetter size-performance trade- off. For additional results with more foundation models, Appendix A.3]
Fig. S illustraate the advantages of (SS) 2D on two datasets: InShop and Tokyo Val, utilizing two foundation models, DINO-B16 and VIT. We compare (8S) 2D with PCA (blue dashes) and non- linear dimensionality reduction technique, Variational Auto Encoder (VAE) (black crosses), Addi- tionally, we compare it with Similarity Space Distillation (SSD), which represents the theoretical upper bound of (SS) 2D. Note that unlike VAE (black crosses) and SSD (red crosses), which are trained for each dimension separately, (SS) 2D learns an adaptive embedding in one shot. From the plot, we observe that AE-SVC (orange) initially enhances the performance of the original embed- ding (blue), and (SS) 2D subsequently provides additional improvements of up 10% at smaller embedding sizes. Also note that since SSD is trained for each dimension separately, it serves as the theoretical upper bound 2D. The results indicate that (SS) 2D closely approaches this upper bound. For additional results involving more datasets and foundation models, please refer to Appendix [A.4] As discussed in Sec. | a smaller embedding size directly translates to faster retrieval speeds. Therefore, the effective dimensionality reduction achieved with (SS) 2D results in improved retrieval speeds, as demonstrated in Fig. TD).
InShop with DINO-B16
InShop with VIT
TokyoVal with DINO-816
TokyoVal with ViT
0.6
mAP84
MAPG4
MAPGR
50
150
200
100
200
250
100
150
250
200 Embedding Dimension
50
100
130
Embedding Dimension
Embedding Dimension
Embudding Dimension
PCA
AE-SVC (Ours)
(5512D (Ours)
Default
Teacher for SSD and (SS)2D
SSD (Upper Bound)
Figure 5: Applying (SS) DD AE- svC leads further performance boost at lower emhedding sizes. Compared VAE and SSD, (SS) 2D offers superior single-shol dimensionality reduction, achieving up to a 10% enhancement at smaller embedding sizes, closely approaching SSD's 's theoretical upper bound.
5.3 LIMITATIONS
While AE-SVC demonstrates significant improvements when applied to foundation models, its abil- ity to enhance dataset-specific models is limited. The pre-trained models tailored for specific dataset already capture the nuances required for effective retrieval, leaving less room for further op- timization via AE-SVC. Additionally, (SS) 2D introduces computational overhead during training due to the need to compute the loss functions for various embedding sizes. This overhead increases training time, though it is still substantially lower than the cost of retraining neural networks for different embedding sizes from scratch.
CONCLUSION AND FUTURE WORK
In this paper, we proposed AE-SVC and (S5) 2D, which improve retrieval scalability and efficiency. Our results show significant improvement on 4 datasets, from image retrieval to place recognition tasks. AE-SVC demonstrates up to a 15.5% improvement shows improvement of 10% for smaller embedding sizes, which further boosts retrieval efficiency. Future work includes new retrieval models that orchestrate the variance-aware property in training loss of foundation models. Although observe the resulting difference in the cosine similarity distributions of the foundation models and the data set-specific models (Figure the fundamental discrepancy between the embedding spaces remains unknown to the community.
retrieval performance, and (SS) 2D

---

Published as a conference paper at ICLR 2025
REPRODUCIBILITY STATEMENT
In the theoretical analysis section (Sec. 4 of this work, we clearly state all the assumptions as and when necessary. Appendix A.I and A.2 provides additional details necessary understand the proofs in Sec. 4 We also discussed all the hyperparameters in Appendix A.6
ACKNOWLEDGEMENT
This work was supported in part by National Science Foundation grants No. 2133481 and No. 2148186, and by the Office of Naval Research (ONR) under Grant No. N00014-22-1-2254. Any opinions, findings, conclusions, or recommendations expressed in this material are those of the authors and do not necessarily reflect the views of these agencies.
10

---

Published as a conference paper at ICLR 2025
REFERENCES
Jimmy Ba and Rich Caruana. Do deep nets really need to be deep? In Advances in neural informa- tion processing systems, pp. 2654-2662, 2014.
Adrien Bardes, Jean Ponce, and Yann LeCun. VICReg: Varrine--imarrinnc-ccaaaaane regulariza- tion for self-superrvieed learning. In International Conference on Learning Representations, 2022. URL httttsss///peeeeeieeett net. wwt.rriiiiieeeeenmmeeem id- 10ormmmid-x66022211111
Gabriele Berton, Carlo Masone, and Barbara Caputo. Rethinking visual geo-locaiization for large- scale applications. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 4878-4888, 2022.
Cristian Buciluă, Rich Caruana, and Alexandru Niculescu-Mizil. Model compression. In Pro- ceedings of the 12th ACM SIGKDD international conference on Knowledge discovery and data mining, pp. 335-541,77 2006.
Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties in self-supervised vision transformers, 2021.
Ting Chen, Simon Korbblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In International conference machine learning, pp. 1597-1607. PMLR, 2020.
LAION Community. Laion-2b: A large-scale dataset for multimodal learning. https:// suuggangfaee.coo//aeeettts/aaaaa
2023. Accessed: 2024-05-16.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszko- reit, and Neil Houlsby, An image worth 16x16 words: Transformers for image recognition at scale, 2021.
Sourav Garg, Tobias Fischer, and Michael Milford. Where is your place, visual place recognition? In ICCII volume 8, pp. 4416-4425, 2021.
Yixiao Ge, Haibo Wang, Feng Zhu, Rui Zhao, and Hongsheng Li. Supe-supe fin-graiied region similarities for large-scale image localization. In Computer Asion-ECCY 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part IV 16, pp. 369-386. Springer, 2020.
Raia Hadsell, Sumit Chopra, and Yann LeCun. Dimensionality reduction by learning an invariant mapping. In 2006 IEEE computer society conference computer vision and pattern recognition (CVPR'06) volume 2, pp. 1735-1742. IEEE, 2006.
Po han Li, Yunhao Yang, Mohammad Omama, Sandeep Chinchali, and Ufuk Topcu. Any2any: Incomplete multimodal retrieval with conformal prediction, 2024. URL httts::/arxxt
10513
Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge in neural network. arXiv preprint arXiv:1503.022. 02531, 2015.
Geoffrey E Hinton and Ruslan R Salakhutdinov. Reducing the dimensionality of data with neural networks. Science, 313(5786):504 507, 2006.
Aapo Hyvärinen, Jarmo Hurri, and Patrik O. Hoyer. Principal Components and Whitening, pp. 93-130. Springer London, London, 2009. ISBN 978-1-84882-491-11 978 1-84888---44--1155 URL http://.......00000000060666-660
doi: 10010
Yannis Kalantidis, Carlos Lassance, Jon Almazan, and Diane Larlus. Tidr: Twin learning for di- mensionality reduction. Transactions on Machine Learning Research, 2021.
Nikhil Keetha, Avneesh Mishra, Jay Karhade, Krishna Murthy Jatavallabhula, Sebastian Scherer, Madhava Krishna, and Sourav Garg. Anyloc: Towards universal visual place recognition. arXiv preprint Xrvi 00688. 2023.
11

---

2023. Utilized in the pretraining Published as a conference paper at ICLR 2025
Diederik P Kingma and Max Welling. arXiv:13/2.611,, 2013.
Auto-encoding variational
bayes.
arXiv preprint
Aditya Kusupati, Gantavya Bhatt, Aniket Rege, Matthew Wallingford, Aditya Sinha, Vivek Ra- manujan, William Howard-Snyder, Kaifeng Chen, Sham Kakade, Prateek Jain, et al. Matryoshka representation learning. Advances in Neural Information Processing Systems. 35:30233-30244 2022.
María Leyva- Nicola Strisciuglio, and Nicolai Petkov. Data-efficient large scale place recog- nition with graded similarity supervision. In Proceedings of the IEEE/CVF Conference on Com- puter Vision and Pattern Recognition, pp. 23487-23496, 2023.
Po-han Li, Sravan Kumar Ankireddy, Ruihan Philip Zhao, Hossein Nourkhiz Mahjoub, Ehsan Moradi Pari, Ufuk Topeu, Sandeep Chinchali, and Hyeji Kim. Task-aware distributed source coding under dynamic bandwidth. Advances in Neural Information Processing Systems, 36, 2023.
Ziwei Liu, Ping Luo, Shi Qiu, Xiaogang Wang, and Xiaoou Tang. Deepfashion: Powering robust clothes recognition and retrieval with rich annotations. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 1096-1104, 2016.
Feng Lu, Lijun Zhang, Xiangyuan Lan, Shuting Dong, Yaowei Wang, and Chun Yuan. To- wards seamless adaptation of pre-trained models for visual place recognition. arXiv preprint arXiv:2402. 14505, 2024.
Hyun Oh Song. Yu Xiang, Stefanie Jegelka, and Silvio Savarese. Deep metric learning via lifted structured feature embedding. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 4004-4012, 2016.
Mohammad Omama, Pranav Inani, Pranjal Paul, Sarat Chandra Yellapragada, Krishna Murthy Jatavallabhula. Sandeep Chinchali, and Madhava Krishna. Alt-pilot: Autonomous navigation with language augmented topometric maps, 2023.
Maxime Oquab, Timothée Darcet, Theo Moutakanni, Huy V. Vo, Mare Szafraniec, Vasil Khalidov, Pierre Fernandez, Daniel Haziza, Francisco Massa, Alaaeldin El-Nouby, Russell Howes, Yao Huang. Hu Xu, Vasu Sharma, Shang-War Wen Li, Wojciech Galuba, Mike Rabbat, Mido Assran, Nicolas Ballas, Gabriel Synnaeve, Ishan Misra, Herve Jegou, Julien Mairal, Patrick Labatut, Ar- mand Joulin, and Piotr Bojanowski. Lvd-142m: A curated dataset for self-supervised learning. ht https: /tt/tgtttuui com/ http://cbooooeeearrchdiiiieeeeee of DINOv2 models.
Maxime Oquab, Timothée Darcet, Théo Moutakanni, Huy Vo, Mare Szafraniee, Vasil Khalidov, Pierre Fernandez. Daniel Haziza, Francisco Massa, Alaacldin El-Nouby, Mahmoud Assran, Nico- Ballas, Wojciech Galuba, Russell Howes, Po- Yao Huang, Shang- Wen Li, Ishan Misra, Michael Rabbat, Vasu Sharma, Gabriel Synnaeve, Hu Xu, Hervé Jegou, Julien Mairal, Patrick Labatut, Ar- mand Joulin, and Piotr Bojanowski. Dinov2: Learning robust visual features without supervision, 2024.
Wonpyo Park, Dongju Kim, Yan Lu, and Minsu Cho. Relational knowledge distillation. In Pro- ceedings of the IEEE Conference Computer Vision and Pattern Recognition (CVPR), 2019.
Nikolaos Passalis and Anastasios Tefas. Learning deep representations with probabilistic knowledge transfer. In Proceedings of the European Conference on Computer Vision (ECCV), pp. 268-284, 2018.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pp- PMLR. 2021.
Adriana Romero, Nicolas Ballas, Samira Ebrahimi Kahou, Antoine Chassang, Carlo Gatta, and Yoshua Bengio. Fitnets: Hints for thin deep nets. arXiv preprint arXiv: 1412.6550 2014.
12

---

Published as a conference paper at ICLR 2025
Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, et al. Imagenet large scale visual recognition challenge. International journal of computer vision. 115:211-2555 2015.
Florian Schroff, Dmitry Kalenichenko, and James Philbin. Facenet: A unified embedding for face the IEEE conference on computer vision and pattern
recognition and clustering. In Proceedings the recognition, pp. 815 823,823,2222 2015.
Sai Shubodh, Mohammad Omama, Husain Zaidi, Udit Singh Parihar, and Madhava Krishna. Lip- loc: Lidar image pretraining for cross-modal localization. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision (WACV) Workshops, pp. 948-957, January 2024.
Ian Smith, Janosch Ortmann, Farnoosh Abbas-Aghabale Aaaabaaghaab,,,,,,, Petr Smirnov, and Benjamin Haibe- Kains. On the distribution of cosine similarity with application to biology. arXiv preprint 1rXiv:2310.13994, 2023.
Kihyuk Sohn. Improved deep metric learning with multi-class n-pair loss objective. Advances in neural information processing systems, 29, 2016.
Hyun Oh Song, Yu Xiang, Stefanie Jegelka, and Silvio Savarese. Deep metric learning via lifted structured feature embedding. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2016.
Yonglong Tian, Dilip Krishnan, and Phillip Isola. Contrastive representation distillation. In Inter- national Conference on Learning Representations (ICLR), 2020.
Akihiko Torii, Josef Sivic, Tomas Pajdla, and Masatoshi Okutomi. Visual place recognition with repetitive structures. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 883-890, 2013.
Akihiko Torii, Relja Arandjelovic, Josef Sivic, Masatoshi Okutomi, and Tomas Pajdla. 24/7 place recognition by view synthesis. In Proceedings of the IEEE conference on computer vision and paltern recognition, pp. 1808-1817, 2015.
Frederick Tung and Greg Mori. Similarity-prseerinng knowledge distillation. In Proceedings of the IEEE International Conference on Computer Vision (ICCV), 2019.
Evgeniya Ustinova and Victor Lempitsky. Learning deep embeddings with histogram loss. Advances in neural information processing systems, 29, 2016.
Jiang Wang, Yang Song, Thomas Leung, Chuck Rosenberg, Jingbin Wang, James Philbin, Bo Chen, and Ying Wu. Deep metric learning: A survey, arXiv preprint 1rXiv:1706.097200 2017.
Xun Wang, Xintong Han, Weilin Huang, Dengke Dong, and Matthew R Scott. Multi-similarity loss with general pair weighting for deep metric learning. In Proceedings of the IEEE/CVE conference on computer vision and puttern recognition, pp. 5022-5030, 2019.
Hui Wu, Min Wang, Wengang Zhou, Houqiang Li, and Qi Tian. Contextual similarity distillation for asymmetric image retrieval. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 94899949999 2022.
Jiuhong Xiao, Gao Zhu, and Giuseppe Loianno. Visual geo-localization with self-supervised repre- sentation learning. arXiv preprint arXiv:2308.00000 2023.
Sergey Zagoruyko and Nikos Komodakis. Paying more attention to attention: Improving the per- formance of convolutional neural networks via attention transfer. In International Conference on Learning Representations (ICLR), 2017.
Jure Zhontar, Li Jing, Ishan Misra, Yann LeCun, and Stéphane Deny. Barlow twins: Self-supervised learning via redundancy reduction. In International conference on machine learning, pp. 12310- 12320. PMLR, 2021.
Chiyuan Zhang, Samy Bengio, Moritz Hardt, Benjamin Recht, and Oriol Vinyals. Your classifier is secretly an energy based model and you should treat it like one. In International Conference on Learning Representations (ICL.), 2019.
13

---

Published as a conference paper at ICLR 2025
A APPENDIX
A.1 INVARTANCE OF ORTHOGONAL. TRANSFORMATION
We show that cosine similarity is invariant to orthogonal transformation. Define Torth E {T we see that
T T = I] as an orthogonal transformation. Following equation
cOs X, To (Torth X) Torth Y
1ooth XI|
| Torth YI
X I-IY
VXTLXYTTI
(15)
cos(X, Y) 0.
A.2 COSINE SCALE INVARIANCE
Similar to the derivation in A. define constant and follow equation g
we now show that cosine similarity is invariant to scaling. Again,
X
Y
XT
c'c
IXIIII///2
X
(16)
XX
cX,s(X,))
A.3 AE-SVC ADDITIONAL RESULTS WITH MORE FOUNDATION MODELS
In Fig.
we showed the advantage of using AE-SVC for embeddings of three models. Here, in show additional results with three more embeddings, namely: CLIP, DINO-V2-SmalI, and DINO-S8. We observe similar trends as previously noted.
Fig. 6
InShop
SOP
Pittsburg130K
TokyoVal
1.0
mAPDD
Errialding minssion
Enroalding
with CLP AE-SVC (Gum)
DINO-S0 Debalt
DING-V2-Smal with
DINO SVC (Oun)
CUI Difult
DINO-SE 58 with PCA
DINO 58 with AV- (0an)
DINO-V2.Smul Defactt
Figure 6: AE-SVC significantly improves the retrieval performance of foundation models. Here, we see similar trend as discussed in the main paper but with three additional foundation models.
A.4 (SS) 2D ADDITIONAL RESULTS WITH MORE DATASETS AND FOUNDATION MODELS
In Fig. we showed the advantage of using (SS) 2D top of AE-SVC for additional improvement in retrieval performance at smaller dimensions. Here, Fig. 7 we show additional results for (SS) on more datasets and embedding combinations. We observe similar trends previously noted.
A.5 SVC ADDITIONAL RESULTS ON THE RECALL METRIC
In Fig. show the advantage of using AE-SVC for embeddings of three models on the mean Av- erage Precision at k (mAP@) k) metric. Here, in Fig. g we show additional results on the Recall@ metric. We observe very similar trends the Recall @ metric did the mAP@ k metric.
14

---

Published as a conference paper at ICLR 2025
InShop with DINO-V2-Large
TokyoVal with DINO-V2-Large
SOP with DINO-B16
SOP with VIT
0.5
MAPS4
mAPg4
mAP
MAPS
0.3
50
100
150
200
250
30
100
150
200
230
50
100
150
200
250
50
100
130
200 250
Embedding Dimension
Embedding Dimension
Embedding Dimension
Embedding Dimension
PCA
AE-SVC (Ours)
(S5)20 (Ours)
VAE
(Upper
Figure 7: Applying (SS) 2D AE-SVC leads to further performance boost at lower embedding sizes. Here, we similar trend discussed the main paper but with additional foundation models and datascts.
InShop
SOP
P P1ttsburg333K
TokyoVai
Roal
Roci
Embbnddng Dimession
Embedding Dimemion
Embedding Dimension
Embedding Dinemios
(Ous)
DIVO-V2-Lage
DINO-V2-Lage (Ous))
WIT Defast
DING-816 B16 with PCA
DINO.BUS with AE-SVE (Oun)
DIVO-V2-Lage Defash
Figure 8: SV- significantly improves the retrieval performance of foundation models on the recall metric. AE-SVC (solid lines) consistently outperforms the off-the-shelff foundation models, i.e., PCA (dashed lines), four datasets.
A.6 HYPERPARAMETERS
We discussed four hyperparameters in Sec. 3.1 Arees Aems Avars and Amean We empirically deter- mined the values of these hyperparameters. ensuring optimal performance across all datasets. The fixed values are: Avee 25, Acov = 1. Avar = 15, and Amean = 1.
A.7 NETWORK SIZE ABLATIONS
Table I: Retrieval performance (map@4) with different sizes of AE-SVC models at varying em- bedding dimensions. As the network size increases, retrieval performance scales up. However, the results show diminishing returns and possible overfitting with deeper networks.
Model
8
32
64
128
256
384
AE AV-SVC
0.04
0.25
0.332
0.388
0.403
0.413
AV-SVC2
0.093
0.340
0.450
0.510
0.520
0.550
AE-SVC3
0.093
0.342
0.461
0.538
0.556
0.568
AE AE-SVCs
0.093
0.340
0.400
0.450
0.480
0.432
AE IV-SVC10
0.01
0.06
0.09
0.10
0.17
0.20
PCA
0.04
0.250
0.324
0.370
0.403
0.411
Table shows the retrieval performance (map@4) of AE-SVC models with varying network sizes (indicated by the subscript in AE-SVC,,, where is the number of encoder/decoder layers) at different embedding dimensions (8, 32, 64, 128, 256, and 384). These experiments are done on the InShop dataset using the DINO-S8 feature extractor.
Larger map@ @4 4 values (f) indicate better retrieval performance. The performance of linear AE-SVC (AE-SVC,) comparable to PCA. As the network size increases, retrieval performance scales up, with AE-SVC, achieving the best results across all dimensions. We have used AE-SVC, in our experiments across datasets and embedding types.
15

---

Published as a conference paper at ICLR 2025
However, performance begins to drop AE-SVCs and significantly deteriorates at AE-SVC1o- indi- cating diminishing returns and possible overfitting with deeper networks. This behavior is consistent across all embedding dimensions.
A.8
CONSTRAINT LOSSES APPLIED AT DIFFERENT LAYERS
Table 2 presents the retrieval performance @ of various AE-SVC configurations across different embedding dimensions (8, 32, 64, 128, 256, and 384). This evaluation encoder-decoder architecture. The constraint losses, which consist of variance, covariance, and mean-centering losses, are applied at different stages of the network. AE-SVC applies these losses immediately after the encoder at the representation level. AE-SVC!, AE-SVC?, and AE-SVÇ² ap- ply these losses after the first, second, and third decoder layers, respectively. Since the network has 3-layer encoder-decoder architecture, AE- EVC- corresponds to applying these losses after the final decoder layer which aligns with existing representation learning methods such as Zbontar et al. 2021. Chen al. (2020), and Kalantidis et al. (202I). In all configurations, reconstruction loss is applied after the final decoder layer.
3-layer
The proposed configuration (AE-SVC) consistently achieves the best retrieval performance across all embedding dimensions, highlighting the effectiveness of applying the loss at the representation level rather than later in the decoder network (like in previous approaches). Performance gener- ally deteriorates when the loss is applied at deeper decoder layers, as observed in the results for AE SE-sVC!, AE-SVC², and AE -vc..
Table 2: Retrieval performance (map@) @4) with constraints applied at different layer of the AE-SVC decoder. Consistent with our theoretical analysis the main paper, the constraints should be applied directly after the encoder and we see a performance drop as they are applied later in the decoder.
Model
8
32
64
128
256
384
AE-SVC
0.093
0.342
0.461
0.538
0.556
0.568
AE AE-SVC!
0.090
0.342
0,444
0,518
0.538
0.550
2
0.010
0.340
0.436
0.473
0.480
0.484
AE sVC
0.090
0.330
0.404
0.443
0.463
0.471
A.9 ADDITIONAL RESULTS WITH MORE BASELINES aND HELD-OUT TRAIN SET
InShop
SOP
Pittsburgh30K
TokyoVal
00
mAP04
2.4
0.5
x
00 XEX
36
200
20
100
20
301
Embadding Dimersion
Embedding Dimeesion
Embedding Dimension
Embedding Dimenson
PCA
Defull
AE-SVC (Ours)
PCA Whitering
VAE
TLOR
Figure 9: We show the retrieval performance of AE-SVC (orange) compared against PCA (blue), PCA Whitening (purple), TLDR (magenta) and Variational Autoencoder (black). For this experi- ment, use the Dino-S8 feature extractor. While PCA Whitening and TLDR improve the perfor- mance of the default feature extractor, AE AV-SVC achieves a significantly greater improvement.
We have incorporated PCA Whitening (Hyvärinen et al.) 2009) a baseline for our approach (VC- We have also incorporated TLDR (Kalantidis et al. [021], a recent dimensionality re- duction work focussed image retrieval. TLDR also focuses dimensionality reduction for trieval. However, TLDR uses only covariance loss. In stark contrast, we use a combination of three losses, including variance loss, which, as per theoretical analysis. is the most important. Fur- thermore, following Barlow Twins, TLDR applies the covariance loss at the output the decoder, whereas apply it at the representation level.
16


Reviews:
[{"review_id": "Gh9x5JA8Kr", "reviewer": "Reviewer_1v5t", "summary": "This paper proposes a method for facilitating the image retrieval task. It focuses on solving two challenges, scalability and efficiency, existed in the image retrieval. To tackle scalability, strong variance constraints are introduced to restrict the latent features. The variance of feature in different dimensions could be similar, so that improving the discriminative ability of the latent features. For the efficiency, a Single-shot Similarity Space Distillation method is introduced to generate adaptive size features, and keep the cosine similarity of sample features. The experiments show that the AE-SVC improve the retrieval performance, especially for the foundation models. The (SS)_2D can improve the performance for smaller embedding sizes.", "questions": "1. During training, does the back propagation happen after traversing the whole dataset or a batch of the images? Would the choice of two ways influence the performance?\n2. Does the embedding size reduction influence the cosine similarity variance or other constraints for AE-SVC of the output features? \n3. Does the performance be changed if the order of SVC and (SS)_2D switch? Or would the performance be improved if append SVC on the resized embeddings?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "1. The paper presents Autoencoders with Strong Variance Constraints (AE-SVC), an unsupervised method. By enforcing orthogonality, mean centering, and unit variance constraints in the latent space, AE-SVC shifts the cosine similarity distribution to be more discriminative. This advancement addresses the scalability issue in current SOTA methods.\n2. The authors introduce Single Shot Similarity Space Distillation ((SS)_2D), that preserves the similarity structure of embeddings without requiring retraining for different dimensions. (SS)_2D produces adaptive embeddings where smaller segments maintain high retrieval performance, addressing the impracticality of separate training for varying embedding sizes found in methods like PCA or traditional autoencoders.\n3. The paper's two-step approach, first enhancing embeddings with AE-SVC and then applying (SS)_2D, yields significant improvements in retrieval performance across multiple datasets and foundation models. Experimental results show that this method not only accelerates retrieval speed by reducing embedding dimensions but also surpasses the accuracy of existing unsupervised techniques and even the original full-sized embeddings.", "weaknesses": "1. The detail of the projection function F in L280 may need more explanations. The size of the embedding \\hat{z} is adaptive. The change of the embedding size whether influences the projection F. In other words, there are matched functions for different size m.\n2. The figures in Fig. (3,4,5) are small. It is hard to distinguish data points of lower embedding dimensions. It could be better that the quantity results are listed in a table.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "NssF4ytuQh", "reviewer": "Reviewer_mXfW", "summary": "This paper discusses two major concerns regarding image retrieval: scalability and efficiency. The concerns are more meaningful when existing retrieval methods use foundation models as feature extractors. To improve the foundation model's performance on retrieval tasks, the authors proposed Autoencoders with Strong Variance Constraints, which enforced three constraints on the learned latent space. Meanwhile, Single-shot Similarity Space Distillation outputs more compact embeddings with flexible dimensions for the model efficiency.", "questions": "1. My major concern is whether the learned normal distribution is still effective in the biased target domain, which is common in dataset-specific tasks. Without the prior information of the target domain, it is ambiguous for neural networks to distinguish instances by discriminative features or global features ( discriminate shirt shape or visually more similar image in the example in Fig.2, the former would be right in commerce recommendation system and the latter is more likely to be the output of the common search engine)\n2. The settings of Fig.4 and 5 are not consistent, and it is hard to discriminate results in different styles (color, line and dot). It would be better to list the results in tables for better comparison.\n3. It is contrary to common sense that the performance of ViT > DINO-Base > DINO-Large in InShop results in Fig.4, but the results of Pittsburgh30K and TokyoVal are normal.\n4. Without additional data, student models learned from teachers would get inferior results. Based on AE-SVC (orange) in Fig.5, it seems that applying feature distillation (green) further improves the retrieval performance. It would be better to add an analysis of this point in the experiment.", "limitations": "", "rating": 5, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "The authors focus on major concerns about using the foundation model in image retrieval or even more CV tasks: how to promote foundation model features and balance model performance and efficiency. By aligning teacher and student similarity spaces, the proposed distillation with flexible output dimensions indeed reduces retrieval compute resources.", "weaknesses": "1. The motivation of AE-SVC is worth discussing: it encourages the network to learn the normal embedding distribution (mean to 0, std to 1), but gt labels of dataset-specific tasks may biased.\n2. The authors claim that they are the first to explore representation learning ideas in image retrieval, considering retrieval is essentially a feature learning task, the soundness is not as remarkable as claimed.\n3. AE-SVC is only compared with the standard PCA in Sec.5.2, more competitors (e.g. PCA variants or AnyLoc ) are needed to demonstrate the effectiveness of AE-SVC, if possible.\n4. Experiment settings in Fig.4 and Fig.5 are not consistent. Compared with AE-SVC, the results of (SS)2D seems incomplete.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "dUVbULCYdW", "reviewer": "Reviewer_f9Kc", "summary": "This paper addresses two key challenges in large-scale image retrieval systems: scalability and efficiency. To enhance scalability, the authors focus on improving the performance of existing foundation models. For efficiency, they propose an effective unsupervised dimension reduction technique.", "questions": "See weaknesses.", "limitations": "", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- Overall, the paper is well-motivated and well-written, aiming to solve an important problem in large-scale image retrieval.\n- The theoretical aspects are well-developed, though it is unclear if the approximation in Eq. 11 is sufficiently supported. It may benefit from references to statistics or mathematical literature rather than an Arxiv paper in biology.\n- Experiments demonstrate the method's effectiveness, showing significant improvement.", "weaknesses": "- I am somewhat skeptical about the scenario described in L39-L42; in fact, self- or semi-supervised learning might address this issue (see related works [1, 2]).\n- I have some doubts about the novelty of AE-SVC, as it combines several common losses. Additionally, there are four losses in Eq. 5, but no explanation is provided on how they are balanced. Experiments or theoretical support in this regard would be beneficial.\n- The experimental results are somewhat limited. Is there additional related work beyond PCA for comparison? Also, I noticed a lack of ablation studies, as only two experiments, shown in Fig.4 and Fig.5, are provided.\n\n[1] eProduct: A Million-Scale Visual Search Benchmark to Address Product Recognition Challenges\n\n[2] V2L: Leveraging Vision and Vision-language Models into Large-scale Product Retrieval", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "72EbVoablP", "reviewer": "Reviewer_vMAk", "summary": "This paper aims to improve image retrieval under the context of using pretrained image foundation models. Two concrete techniques are designed: (1) Autoencoders with Strong Variance Constraints (AE-SVC) regularizes the embeddings from foundation models to improve the the retrieval performance, (2) Single-shot Similarity Space Distillation ((SS)_2D) compresses the embedding in an adaptive way to accelerate the retrieval. The paper provides theoretical and empirical analyses to justify the effectiveness of AE-SVC. The experimental results show that the proposed approaches can effectively improve the retrieval performance/efficiency of embedding from four different visual foundation models across four datasets.", "questions": "The paper focuses on improving the embedding from image foundation models by unsupervised training without any labels. However, given the recent advancements in multimodal LLMs, it seems possible to leverage these models (like GPT-4o) to automatically produce reasonable labels for fine-tuning image embedding models. If this is feasible, would the proposed approaches still be practically useful, given the fact mentioned in the paper that dataset-specific models are better than foundation models?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. The proposed approaches are quite simple yet effective. AE-SVC is backed up by some reasonable empirical analyses in Sec.4. The adaptive dimension reduction brought by (SS)$_2$D is interesting and can be practically useful to avoid training multimodal models for different embedding dimensions.\n\n2. The authors conduct thorough experiments to demonstrate the effectiveness of the proposed approaches: 4 image retrieval datasets and 4 visual foundation models are considered, and AE-SVC + (SS)$_2$D consistently show positive results for efficient and accurate image retrieval.\n\n3. A codebase is provided in the supplementary material. Roughly looking through the code, all necessary model/data components are included, ensuring the reproducibility of the paper.", "weaknesses": "1. The theoretical analysis in Sec.4 is only conducted based on the Gaussian distribution assumption, which does not necessarily hold for the embeddings from the visual foundation models. Based on the PDF visualization in Figure 3, the true distribution of cosine similarity between image embeddings is far from Gaussian.\n\n2. In large-scale retrieval, approximate nearest neighbour (ANN) is commonly used to accelerate the retrieval process, so the complexity analysis of image retrieval in the paper is not necessarily right in real-world applications. \n\n3. The figure placement can be optimized. The method details should not be in the first figure, which increases the burden of understanding the introduction, and forces people to switch back and forth when reading the method section.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "TSAQCo92KP", "reviewer": "Reviewer_Mq3y", "summary": "This work uses a foundation model for image retrieval, making two contributions: (1) AE-SVC trains an autoencoder on top of the foundation model such that the latent vectors have statistical (mean and covariance) properties inspired or equivalent to centering, PCA and whitening. (2) SS2D performs dimensionality reduction jointly for multiple target dimensions by distilling a similarity matrix from the AE-SVC encoder to another student encoder. Both are learned on the test (reference) set, while the foundation model is pre-trained and frozen.", "questions": "1. The mean and covariance quantities in (4) and (2) (also variance in (3), but this is redundant as discussed above), are global in the sense that they are measured on all $n$ samples of the test set. What happens when the test set is large and how are they treated when learning with a stochastic optimizer in mini-batches? Are mini-batches used in this work?\n\n2. Can the authors confirm or refute my understanding that the variance of similarities is a concave function of the mean and is maximized when the mean is zero? In the latter case, how would they update the paper?", "limitations": "", "rating": 6, "confidence": 5, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "The idea is well motivated. The methodology is sound. The writing is of high quality and mostly clear. The experiments support the claims on a number of datasets and models. The two contributions are shown to be effective comparing with simple baselines.", "weaknesses": "The novelty is incremental, while important related work and baselines are missing. The setting of learning on the test set is questionable. Comparisons with baselines are lacking, not fair and not informative enough. Section 4 fails to properly separate existing work by Smith et al. (2023) from this work and is partly unclear. More details follow.\n\n1. The paper addresses image retrieval and performs evaluation on standard deep metric learning and place recognition datasets. It misses revelant foundational work on image retrieval, including methodologies, benchmarks and protocols:\n\n    > [*1] Gordo et al. 2016, Deep Image Retrieval: Learning Global Representations for Image Search\n    >\n    > [*2] Radenovic et al. 2017, Fine-Tuning CNN Image Retrieval with No Human Annotation\n    >\n    > [*3] Radenovic et al. 2018, Revisiting Oxford and Paris: Large-Scale Image Retrieval Benchmarking\n    >\n    > [*4] Weyand et al. 2020, Google Landmarks Dataset v2: A Large-Scale Benchmark for Instance-Level Recognition and Retrieval\n\n    This work should be repositioned against this work in terms of novelty and protocols used. The related work and experiments sections should be extended accordingly, considering additional training and test datasets and evaluation protocols from these works.\n\n2. The motivation for AE-SVC is scalability. That is, it is challenging to have a labeled dataset-specific training set, hence it is suggested to learn in an unsupervised on the test (reference) set. On one hand, this setting is extremely problematic because it transfers the burden from training to testing, by having to learn something on the test set. The latter is in many practical applications constantly changing, hence the model would need to be constantly updated. It is a kind of overfitting to the test set that has never been used in standard work like the references above. To my knowledge, it has only been used in\n\n    > [*5] Liu et al. 2019, Guided Similarity Separation for Image Retrieval\n\n    Hence, this work should be a baseline for comparison if the authors insist in their setting of learning on the test set. On the other hand, if the motivation to learn in an unsupervised way, this can still be done on a separate training set. For example:\n\n    > [*6] Kim et al. 2022, Self-Taught Metric Learning without Labels\n\n    Besides, learning in [2*] may be supervised, but with labels obtained automatically without human annotation. Self-supervised learning should not be a reason to learn on the test set.\n\n3. In AE-SVC, I believe variance loss (3) is redundant. It is merely the diagonal part of covariance loss (2). Other than that, the covariance loss (2) and centering loss (4) are inspired by centering, PCA and whitening. Hence, AE-SVC should be compared to a baseline where centering, PCA and whitening are performed as post-processing on features extracted by the foundation model, without learning. This baseline is missing. Whitening has a long history in image retrieval but ignored here. Also very successful is supervised discriminative whitening [*2], which could be adapted for self-supervision to be used as a baseline.\n\n4. In SS2D, I believe the loss is the same as\n\n    > [*7] Park et al. 2019, Relational Knowledge Distillation\n\n   The authors cite this paper but do not discuss the actual relation with their work. I believe the only new component here is applying it for dimensionality reduction and learning for multiple target dimensions at once, which are pretty incremental contributions. Also very relevant is\n\n    > [*8] Kalantidis et al. 2022, TLDR: Twin Learning for Dimensionality Reduction\n\n    in the sense that it is also learning a kind of autoencoder on top of frozen feature for dimensionality reduction. Instead of a reconstruction loss, it uses the Barlow Twins loss on the output vectors, which is the same as the covariance loss here, but uses cross-correlation instead. Both the architecture and cross-correlation matrix should be compared to as baselines.\n\n5. Section 4 is useful and interesting, but it is presented as new to a large extent while it actually all comes from Smith et al. (2023). The only new part is the experiment of Figure 3. In addition, parts are copied from Smith et al. (2023) without carrying over all relevant information, such that the resulting text is unclear. For example, (8) refers to \"discriminative power\" but there is nothing to discriminate here, whereas Smith et al. (2023) compare the distribution of target similarities to a \"null\" or \"background\" distribution where the term \"discriminative\" makes sense. As another example, (14) results in a \"global minimum\" without saying anything about the function being convex, which is shown indeed by Smith et al. (2023).\n\n    In the analysis of Section 4, the data is taken to be centered and nothing is said about the effect of this choice to the distribution of similarities, either here or in Smith et al. (2023). To my understanding, the variance of similarities is a **concave** function of the mean and is **maximized** when the mean is zero. This is the opposite of what happens with whitening and raises questions on the motivation of minimizing the variance. To the very least, the experiment of Figure 3 should be extended to also show the distribution of similarities with centered data but without PCA and whitening. It should also include centering, PCA and whitening as post-processing steps rather than being learned with (2), (3) and (4).\n\n6. In the experiments of Figures 4 and 5, the current work (AE-SVC, SS2D or SSD) uses an additional encoder on top of the foundation model, while the \"default\" and \"PCA\" are using the foundation model alone, hence are not comparable. These results tell us nothing about whether the benefit is coming from the properties encouraged by the loss functions, the additional learned parameters or the adaptation to the test set. These are three factors that should be evaluated independently in some way, for example by fine-tuning the last layers of the foundation model rather than adding an extra encoder.\n\n    Whitening by post-processing after PCA should be definitely added as a baseline. TLDR or other state of the art dimensionality reduction methods should also be added, because centering, PCA, whitening and VAE are very basic baselines.\n\n## Post-rebuttal\n\nThe following is an assessment after rebuttal and considering the other reviews.\n\nThe contribution is incremental, which is not necessarily a problem if there is a clear and fair positioning with respect to previous work, a clear message and good results. The situation has clearly improved during rebuttal and discussion, but I am afraid that the paper is still weak in these three aspects. In particular:\n\n- *Losses*. The authors state in the dicussion that there is \"stark contrast\" with TLDR in that they use three losses rather than one. The covariance loss is very close to Barlow Twins loss used in TLDR, which is cross-correlation. An experimental comparison between the two is not there. The variance loss only refers to the diagonal of the covariance loss and treating it separately makes sense only because of using a different loss weight, which is not mentioned anywhere in the main text. The appendix A6 says $\\lambda_{\\text{cov}}=1$, $\\lambda_{\\text{var}}=15$, but the effect of each $\\lambda$ is not shown. What if $\\lambda_{\\text{cov}}=15$ for example? A contribution is meaningful when effective, especially if incremental. The mean-centering loss is something ubiquitous. The relative benefit of each loss component is not shown anywhere. Under these conditions, the contribution claim of three losses is still not adequately supported by evidence.\n\n- *Theoretical support*. The authors claim in the paper and discussion that providing theoretical analysis in section 4 is a contribution but this study is carried out by Smith at al. (2023) and section 4 does not clarify if there is anything new here except the experiment of Figure 3. This is kind of over-claiming contribution. Minimizing the variance of cosine similarity distribution is still not a clear narrative as motivation since, as a function of the mean, it is maximized. Using the term \"discriminative\" when there is nothing to discriminate from is still confusing.\n\n- *Distillation*. The authors state in the paper that their approach \"is conceptually similar to SPKD\", but this is different from saying \"we use RKD (Park et al., 2019)\", for example. Again, this is kind of over-claiming contribution.\n\n- *Supervision*. On one hand, there is a supervised PCA whitening approach that is common in image retrieval, as performed by Radenovic et al. (2017) mentioned in my review. This would be another baseline, plus the authors could use this kind of supervision on their own approach. On the other hand, there are more unsupervised metric learning methods like Kim et al. (2022) that I have mentioned in my review, which could be used for distillation or self-supervision, as competitors or as components of this method.\n\n- *Two-step*. The authors state in the discussion that their approach is two-step in \"stark contrast\" with previous work, but they do not experiment with an one-step variant of their approach to show the benefit and they do not explain why two-step is better.\n\nThe most important additions during the rebuttal are the experiment of training on a held-out training set, the comparison with TLDR and applying the losses at different layers (Table 2 of rebuttal document). The last one possibly accounts partially for the performance improvement over TLDR but the effect of the losses is not shown. However, as discussed above, several things are still missing in terms of fair positioning with previous work, some rewriting to make the message clearer and more experiments. I believe all of these issues could be resolved in theory, in which case the paper would be interesting enough to publish despite the incremental contribution. In case the paper is accepted, I strongly recommend the authors to revise accordingly for the camera-ready.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)