EXPLOITING DISTRIBUTION CONSTRAINTS FOR SCALABLE AND EFFICIENT IMAGE RETRIEVAL

## Abstract

Abstract Image retrieval is crucial in robotics and computer vision, with downstream applications in robot place recognition and vision-based product recommendations. Modern retrieval systems face two key challenges: scalability and efficiency.
State-of-the-art image retrieval systems train specific neural networks for each dataset, an approach that lacks scalability.
Furthermore, since retrieval speed is directly proportional to embedding size, existing systems that use large embeddings lack efficiency.
To tackle scalability, recent works propose using off-the-shelf foundation models. However, these models, though applicable across datasets, fall short in achieving performance comparable to that of dataset-specific models.
Our key observation is that, while foundation models capture necessary subtleties for effective retrieval, the underlying distribution of their embedding space can negatively impact cosine similarity searches.
We introduce Autoencoders with Strong Variance Constraints ( AE-SVC ), which, when used for projection, significantly improves the performance of foundation models. We provide an in-depth theoretical analysis of AE-SVC .
Addressing efficiency, we introduce Single-shot Similarity Space Distillation ( (SS) 2 D ), a novel approach to learn embeddings with adaptive sizes that offers a better trade-off between size and performance.
We conducted extensive experiments on four retrieval datasets, including Stanford Online Products (SoP) and Pittsburgh30k, using four different off-the-shelf foundation models, including DinoV2 and CLIP. AE-SVC demonstrates up to a 16 % percent 16 16\% 16 % improvement in retrieval performance, while (SS) 2 D shows a further 10 % percent 10 10\% 10 % improvement for smaller embedding sizes.

## 1 Introduction

Image retrieval involves finding the closest match of a given image (query) in a vast database of images (often called the reference set). It has numerous applications, from product recommendations in e-commerce to place recognition in robotics. In general, state-of-the-art (SOTA) image retrieval systems are dataset-specific. They are trained in a contrastive manner on a training split of the data, typically hand-labeled for positive and negative pairs for each query. Imagine a scenario where a server has a vast reference database of fashion clothing images and receives query images from users to find the closest matches in the reference set. Creating a separate split and hand-labeling positive and negative pairs to train a dataset-specific model is infeasible. A more scalable solution is to use off-the-shelf feature extractors like Dino (Caron et al., ), DinoV2 (Oquab et al., ), CLIP (Radford et al., ), and ViT (Dosovitskiy et al., ).
However, the performance of off-the-shelf feature extractors is generally lower compared to dataset-specific models.
Q1 (Scalability): Is there a way to enhance the performance of these off-the-shelf models in a completely unsupervised way by using just the reference set?

Figure: Figure 1: Two-step pipeline for the proposed approach. (A) AE-SVC (discussed in Sec. 3.1) trains an autoencoder with our constraints to improve foundation model embeddings. (B) (SS)2D (discussed in Sec. 3.2) uses the improved embeddings from AE-SVC to learn adaptive embeddings for improved retrieval at any embedding size. (C) Once trained, (SS)2D can be directly applied to foundation model embeddings to generate adaptive embeddings for improved retrieval. (D) AE-SVC (orange) boosts performance significantly, while (SS)2D (green) further enhances results with smaller embeddings. Dino (blue) achieves optimal performance at 9 GLOPs, whereas (SS)2D on top of AE-SVC achieves similar performance at only 2.5 GLOPs.
Refer to caption: extracted/5913932/figs/pipeline.png

Another key problem in retrieval is that the retrieval speed is directly proportional to the size of the embeddings. For an embedding of size $d$ and a reference set with $N$ image vectors, the retrieval speed for a single query is $O(d\times N)$. This retrieval time becomes significant as the size of the reference set ($N$) increases, with modern retrieval systems containing millions of images. Additionally, the embedding size directly affects the storage space required for the reference set and the communication bandwidth needed to send the query vector to the server in distributed image retrieval systems, which are common in product recommendations and robotic place recognition. Standard dimensionality reduction techniques, such as Principal Component Analysis (PCA), aim to preserve information, while Autoencoders (AEs) (Hinton & Salakhutdinov, ) and Variational Autoencoders (VAEs) (Kingma & Welling, ) focus on reconstruction quality. However, these methods are not optimized for retrieval tasks. Additionally, learning-based approaches like AEs and VAEs require separate training for each embedding size, which is impractical given varying compute, space, and communication constraints. Q2 (Efficiency): Is there an effective unsupervised dimensionality reduction method that strongly preserves the similarity structure of the full embeddings, and is adaptive, i.e., does not need to be trained for each dimension separately?

Contributions: To address Q1 (Scalability), we propose Autoencoders with Strong Variance Constraints (AE-SVC). Our primary idea is that, while foundation models capture the necessary details for effective retrieval, the underlying distribution of their embeddings can negatively impact cosine similarity searches. AE-SVC trains an autoencoder while enforcing three constraints on the latent space: an orthogonality constraint, a mean centering constraint, and a unit variance constraint. We empirically show and mathematically prove that these constraints cause a shift in the cosine similarity distribution, making it more discriminative. These constraints not only lead to more effective dimensionality reduction but also outperform the complete embeddings of foundation models. We discuss the motivation and technical details of AE-SVC in section 3.1. We also provide an in-depth mathematical analysis of AE-SVC in section 4. To address Q2 (Efficiency), we propose Single Shot Similarity Space Distillation ((SS)2D). (SS)2D aims at reducing embeddings to smaller ones while preserving their similarity relationships. The embedding learned with (SS)2D is adaptive, i.e., smaller segments of the output embedding also perform well in retrieval tasks. In summary, our approach consists of two steps (as shown in Fig. 1): first, we use AE-SVC to enhance the baseline performance of foundation models. Then, we leverage the improved embeddings as a teacher for efficient dimensionality reduction through (SS)2D.

Fig. 1(D) also shows the retrieval performance versus retrieval speed in giga-FLOPs (floating operations) on the Pittsburgh30K dataset (Torii et al., ) using the Dino model (Caron et al., ). Note that retrieval speed is a function of the dimension $d$ and the reference set size $N$, $O(d\times N)$, with $N=17,000$ for Pittsburgh30K. The performance of PCA (blue) represents the default off-the-shelf Dino model. We can see that AE-SVC (orange) significantly enhances the performance of the off-the-shelf Dino model (blue). Notice that it outperforms the complete Dino embedding. Applying (SS)2D on top of AE-SVC further improves retrieval performance at smaller embedding sizes (green). Section 5 experimentally shows the advantages of the proposed approaches in detail on four different datasets using different foundation models.

## 2 Related Work

Deep Metric Learning:
Image retrieval is modeled as a Deep Metric Learning (DML) problem (Wang et al., ). DML focuses on learning latent representations where similar items are close and dissimilar items are far apart. Various techniques have been proposed (Oh Song et al., ; Schroff et al., ; Ustinova & Lempitsky, ; Sohn, ; Wang et al., ) to learn meaningful latent representations for effective retrieval, including contrastive loss (Hadsell et al., ) and triplet loss (Schroff et al., ). More recent works use advanced N-pair loss (Sohn, ) or multi-similarity loss (Wang et al., ). Analogous to image retrieval, Visual Place Recognition (VPR) is also modeled as a DML problem (Garg et al., ). Various training objectives have been proposed for VPR (Ge et al., ; Xiao et al., ; Leyva-Vallina et al., ; Berton et al., ) to learn latent representations for effective retrieval. However, all of these approaches are dataset-specific, meaning they train specific neural networks for each dataset.

Foundation Models in Image Retrieval:
The rise of foundation models, such as DINOv2 (Oquab et al., ), CLIP (Radford et al., ), ViT (Dosovitskiy et al., ), has allowed recent works to explore the applicability of these off-the-shelf models in image retrieval. As shown in (Keetha et al., ; Lu et al., ), off-the-shelf models, in some cases, can have comparable retrieval performance to dataset-specific models. However, these approaches either use the off-the-shelf models as-is (Keetha et al., ) or adapt them using a labeled held-out dataset (Lu et al., ). In stark contrast, our approach, AE-SVC, exploits the inherent properties of the distribution to improve the performance of these off-the-shelf models without requiring any labeled data.

Distribution Constraints in Semi-supervised Representation Learning:
Semi-supervised learning has seen significant advances with methods that leverage self-supervised objectives to learn robust representations. A notable approach is Barlow Twins, which employs an additional cross-correlation loss to align representations while minimizing redundancy across dimensions, promoting feature decorrelation (Zbontar et al., ). Extending this idea, VICReg introduces an additional variance constraint that ensures sufficient spread in the learned features, preventing collapse and further enhancing representation quality (Bardes et al., ). However, to date, no work has specifically explored the impact of such constraints in the context of autoencoders for retrieval tasks. In this paper, we are the first to address this gap, providing a comprehensive theoretical analysis of these constraints and their impact on retrieval.

Similarity Preserving Knowledge Distillation:
Similarity preserving knowledge distillation (SPKD) focuses on transferring knowledge from a teacher model to a student model, similar to standard distillation (Hinton et al., ; Buciluǎ et al., ; Ba & Caruana, ; Zagoruyko & Komodakis, ; Zhang et al., ; Passalis & Tefas, ), while maintaining the structure of similarity of the data in the feature space. One of the pioneering works in this area is the FitNets approach, which uses intermediate representations to guide the student model (Romero et al., ). The attention transfer method extends this by aligning the attention maps of the student and teacher models (Zagoruyko & Komodakis, ). More recent advancements include (Tung & Mori, ), which explicitly preserves pairwise similarities between data points in the feature space during distillation. Contrastive Representation Distillation (CRD) further improves performance by using contrastive loss to maximize mutual information between teacher and student representations (Tian et al., ). Additionally, the Relational Knowledge Distillation (RKD) approach emphasizes the importance of preserving the relational knowledge among data points (Park et al., ). All of these works focus on training a lightweight student network to mimic a heavy teacher network.
We are the first to explore this idea in a dimensionality reduction setting for image retrieval.

Adaptive Feature Embeddings:
A core requirement for dimensionality reduction in retrieval systems is adaptive embeddings due to varying compute constraints. Previous works on adaptive feature embeddings have created efficient and adaptive embedding spaces for compression (Li et al., ) and image classification (Kusupati et al., ), while ours focuses on image retrieval. The key idea is to use only one neural network model to output a feature embedding ensuring that smaller sub-chunks of the embedding also perform well in retrieval tasks.

## 3 Methodology

We now formally define the image retrieval problem:

Suppose we have a query set $Q$ of $M$ query image feature vectors and a reference set $R$ of $N$ reference image feature vectors, where each vector is $d$-dimensional. Given $q_{j}\in\mathbb{R}^{d}$ randomly drawn from $Q$, the task is to identify the feature vector $r_{i}$ in $R$ that has the smallest distance or highest similarity with $q_{j}$. This can be mathematically expressed as:

$$ $r_{i}=\arg\min_{r\in R}\mathrm{dis}(q_{j},r),$ $$

with the distance $\mathrm{dis}$ generally based on cosine similarity, i.e., $\mathrm{dis}(q_{j},r_{i})=1-\mathrm{cos}(q_{j},r_{i})$, where $\mathrm{cos}$ denotes the cosine similarity.

We will now describe our approach in detail. First, we will explain the step-by-step procedure for AE-SVC. Next, we will present an in-depth explanation of (SS)2D. Note that both AE-SVC and (SS)2D utilize only the reference set $R$ during the training time. The query set $Q$ is assumed to be unavailable to the user during training. A standard assumption when doing dimensionality reduction in image retrieval settings (Keetha et al., ).

### 3.1 Autoencoders with Strong Variance Constraints ( AE-SVC )

Figure: Figure 2: In standard dimensionality reduction (say PCA), cosine similarity is disproportionately influenced by high-variance dimensions, leading to poor retrieval. Given a task to match the query with the correct clothing type. A query image of a white man in a tank-top may be incorrectly matched to a white man in a half-shirt due to the dominant person dimension. Ideally, both orthogonal dimensions should have an equal influence on cosine similarity.
Refer to caption: extracted/5913932/figs/method_vapca_motivation.png

Motivation: Our primary motivation for AE-SVC is that, while foundation models capture necessary subtleties for effective retrieval, the underlying distribution of their embedding space can negatively impact cosine similarity searches. In Sec. 4, we discuss in mathematical detail that the discriminative power of the retrieval task is maximized when the variance of the cosine similarity distribution is minimized. AE-SVC introduces three constraints on the latent space: an orthogonality constraint, a mean centering constraint, and a unit variance constraint.
In Sec. 4, we prove that these constraints on the latent space minimize the variance of the cosine similarity distribution. We now provide a more intuitive explanation of why these constraints are helpful. Consider the example illustrated in Fig. 2. We have a reference set comprising four categories of clothing: men’s tank-tops, men’s half-shirts, women’s tank-tops, and women’s half-shirts. The task is to match the query with the correct clothing type. After applying dimensionality reduction (like a standard PCA) to the reference set, we obtain two orthogonal dimensions: a person dimension (with high variance) and a shirt shape dimension (with low variance). If similarity is overly dependent on the high-variance dimension, as shown mathematically in Sec. 4, retrieval performance suffers. For instance, a query for a white man in a tank-top might incorrectly match with a white man in a half-shirt due to the dominant person dimension. Ideally, we need a space where both the person and shirt dimensions equally influence the distinction between men and women, as well as between half-shirts and tank-tops. The goal of AE-SVC is to learn a projection from the reference set $R$ such that the latent dimensions are orthogonal and have equal variance.

Approach:. AE-SVC (as shown in Fig. 1(A)) takes an input embedding $I\in\mathbb{R}^{d}$ coming from the foundation model, encodes it to a latent representation $z\in\mathbb{R}^{d}$, and produces a reconstruction $I^{\prime}\in\mathbb{R}^{d}$. We have three additional losses to enforce the constraints mentioned above: an orthogonality constraint, a mean centering constraint, and a unit variance constraint. Note that we can choose any size for the latent representation $z$. We show the results for multiple sizes of $z$. However, since AE-SVC is used to guide (SS)2D later, we will use the best performing embedding, i.e., $z\in\mathbb{R}^{d}$, for this guidance.

Reconstruction Loss: We minimize the Mean Squared Error (MSE) between the input embedding $I$ and the reconstruction $I^{\prime}$:

$$ $\mathcal{L}_{\text{rec}}=\frac{1}{n}\sum_{i=1}^{n}\|I_{i}-I^{\prime}_{i}\|^{2} _{2},$ (1) $$

where $n$ is the number of samples.

Covariance Loss: To promote orthogonality, we penalize off-diagonal terms of the covariance matrix:

$$ $\mathcal{L}_{\text{cov}}=\left\|\frac{1}{n}(Z-\mu)^{\top}(Z-\mu)-\mathbb{I} \right\|_{F}^{2},$ (2) $$

where $Z\in\mathbb{R}^{n\times d}$ is the matrix of latent representations, $\mu\in\mathbb{R}^{d}$ is the mean of $Z$, and $\mathbb{I}$ is the identity matrix.

Variance Loss: We constrain the variance of each latent dimension to be one:

$$ $\mathcal{L}_{\text{var}}=\frac{1}{d}\sum_{i=1}^{d}\left(\text{Var}(z^{i})-1 \right)^{2},$ (3) $$

where $z^{i}$ is the $i$-th latent dimension of $z$ and $\text{Var}(z^{i})$ is the variance of the $i$-th latent dimension across all $n$ samples.

Mean-Centering Loss: To ensure the latent space is centered around zero, we minimize the mean of the latent dimensions:

$$ $\mathcal{L}_{\text{mean}}=\frac{1}{d}\sum_{i=1}^{d}(\mu^{i})^{2},$ (4) $$

where $\mu^{i}$ is the mean of the $i$-th latent dimension across all $n$ samples.

Total Loss: The final objective combines all four losses:

$$ $\mathcal{L}_{\text{AE-SVC}}=\lambda_{\text{rec}}\mathcal{L}_{\text{rec}}+ \lambda_{\text{cov}}\mathcal{L}_{\text{cov}}+\lambda_{\text{var}}\mathcal{L}_{ \text{var}}+\lambda_{\text{mean}}\mathcal{L}_{\text{mean}}$ (5) $$

where $\lambda_{\text{rec}},\lambda_{\text{cov}},\lambda_{\text{var}},$ and $\lambda_{\text{mean}}$ are hyperparameters discussed in A.6. By optimizing this loss, AE-SVC learns a latent space that improves retrieval performance through accurate reconstruction, decorrelated and normalized feature dimensions, and a zero-centered mean.
The latent representation, $z\in\mathbb{R}^{d}$, is then used to guide the training of (SS)2D.

### 3.2 Single Shot Similarity Space Distillation ( (SS) 2 D )

Motivation: As previously mentioned, embedding size significantly impacts retrieval speed. For an embedding of size $d$ and a reference set with $N$ image vectors, the retrieval speed for a single query is $O(d\times N)$. Therefore, it is crucial to reduce the embedding size while maintaining performance.
Standard dimensionality reduction techniques, such as PCA, Autoencoders (AEs), and Variational Autoencoders (VAEs), are not optimized for retrieval. Additionally, AEs and VAEs require separate training for each embedding size, which is impractical given varying computational constraints.
Therefore, there is a need for adaptive embeddings that maintain retrieval performance and are flexible, ensuring that smaller sub-chunks of the embedding also perform well in retrieval tasks.

Approach: Previous works have employed Similarity-Preserving Knowledge Distillation (SPKD) both in hidden layer activations (Smith et al., ) and output layers (Wu et al., ). SPKD trains a student network to ensure that input pairs producing similar (or dissimilar) activations in the teacher network yield similar (or dissimilar) activations in the student network. However, these studies have not explored similarity-preserving distillation for efficient dimensionality reduction. Our approach is conceptually similar to SPKD, but rather than focusing on network distillation (teacher to student), we perform embedding-level distillation, reducing a large embedding to a smaller one while preserving similarity relationships. Using SPKD directly for dimensionality reduction requires training separate networks for each dimension size. Previous works on adaptive embeddings have utilized random projections in a distributed compression setting (Li et al., ) or optimized multi-class classification loss for each nested dimension (Kusupati et al., ). Following the approach in (Kusupati et al., ), we optimize a similarity-preserving loss for each nested dimension. By combining a similarity-preserving loss with an adaptive embedding training pipeline, we introduce our approach, Single Shot Similarity Space Distillation, or (SS)2D. Fig. 1(B) illustrates the central idea of (SS)2D which we will now describe in detail.

Let ${z}\in\mathbb{R}^{d}$ be the latent representation of the reference set obtained via AE-SVC. Our goal is to reduce the dimensionality of ${z}$ so that the performance of the smaller embedding approximates that of the full embedding. In essence, the complete embedding ${z}$ serves as a guide for creating smaller embeddings. Although (SS)2Dcould theoretically be applied directly to the foundation model embedding $I$, the latent representation ${z}$ is shown to perform better in retrieval tasks, thus providing stronger guidance.

Consider a set $\mathcal{M}\subset[d]$ of embedding sizes and a neural network $\mathcal{F}(\cdot;\theta)$ parameterized by $\theta$. We want to learn a projection $\hat{z}=\mathcal{F}(z;\theta)$ such that each of the first $m$ dimensions, for $m\in\mathcal{M}$, of the learned embedding vector $\hat{z}^{1:m}\in\mathbb{R}^{m}$ independently preserves the retrieval performance of ${z}$.
To learn the function $\mathcal{F}$, we first compute a cosine similarity matrix $\mathrm{C}\in\mathbb{R}^{N\times N}$ by calculating the cosine similarities of every $z$ with every other $z$ in the reference set $R$. We refer to this matrix as the cosine similarity space of the teacher $Z$. Let $\mathrm{C}_{i}$ denote the $i^{\text{th}}$ row of $\mathrm{C}$.

Next, for each $m\in\mathcal{M}$, we define a student similarity space $\tilde{\mathrm{C}}^{m}\in\mathbb{R}^{N\times N}$. We then introduce a Kullback-Leibler (KL) divergence loss $l^{m}$ between $\tilde{\mathrm{C}}^{m}$ and $\mathrm{C}$ for each $m$ as follows:

$$ $l^{m}=\sum_{i}\mathrm{D}_{\mathrm{KL}}(\tilde{\mathrm{C}}^{m}_{i}\|\mathrm{C}_ {i}).$ (6) $$

The overall loss $L$ is then expressed as:

$$ $L_{\texttt{(SS)${}_{2}$D}}=\sum_{m}l^{m}=\sum_{m}\sum_{i}\mathrm{D}_{\mathrm{ KL}}(\tilde{\mathrm{C}}^{m}_{i}\|\mathrm{C}_{i}).$ (7) $$

## 4 Theoretical Analysis of AE-SVC

In Sec. 3.1, we stated that the discriminative power of the retrieval task is maximized when the variance of the cosine similarity distribution is minimized. This discriminative power can be modeled as the probability that the cosine similarity between two randomly drawn vectors $S_{a}$ and $S_{b}$ from the same class exceeds a given threshold $\tau$:

$$ $P\left(\cos(S_{a},S_{b})>\tau\right)=\mathrm{erf}\left(\frac{\mathbb{E}[\cos(S _{a},S_{b})]-\tau}{\sqrt{\text{Var}[\cos(S_{a},S_{b})]}}\right),$ (8) $$

where $\mathrm{erf}$ is the error function. The vectors $S_{a}$ and $S_{b}$ are assumed to be drawn from the same Gaussian distribution. As evident from Eq. 8, the discriminative power increases as the variance of the cosine similarity distribution decreases. For a theoretical proof of this and alternative methods for modeling discriminative power, we refer the reader to Smith et al. ().

We now verify if the results of Gaussian assumptions also hold for real datasets.
Fig. 3 shows the distribution of cosine similarities of the reference set feature vectors on the Pittsburgh30k dataset (Torii et al., ). We compute cosine similarities between all pairs of feature vectors in the reference set and plot the probability mass function (PMF) with and without AE-SVC for two models: the off-the-shelf foundation model Dino (Fig. 3a) and the SOTA dataset-specific model Cosplace (Berton et al., ) (Fig. 3b).
Cosplace, which excels at retrieval, exhibits a PMF with significantly less variance compared to Dino, as seen in the blue curves in Fig. 3a and 3b. The lower variance indicates more discriminative feature vectors, which aligns with the findings from Eq. 8 that smaller variance enhances discriminative power and retrieval performance.
Applying AE-SVC reduces the variance in both models (orange curves), with a more pronounced shift in Dino than in Cosplace. This suggests that AE-SVC benefits the off-the-shelf foundation model more ($10\%$) than the dataset-specific model ($2\%$), as reflected in the retrieval performance plot in Fig. 3c. Thus, the remainder of this work focuses on off-the-shelf foundation models.

Figure: (a)
Refer to caption: extracted/5913932/figs/theory_simspace_dino.png

We have mathematically and empirically demonstrated that the discriminative power of the retrieval task is maximized when the variance of the cosine similarity distribution is minimized. Next, we will prove that the constraints introduced on the latent space in Sec. 3.1 indeed minimize this variance.
Let $Z$ be the latent space distribution of AE-SVC. By applying constraints in Eq. 2 and Eq. 4, the distribution $Z$ has a zero mean, $\mu=0$, and a diagonal covariance matrix $\Sigma\in\mathbb{R}^{d\times d}$, with eigenvalues $\lambda_{i}=\sigma_{i}^{2}$, where $\sigma_{i}^{2}$ is the variance of the individual dimensions of $Z$. Given two vectors $X$ and $Y$ drawn from $Z$, i.e., $X,Y\sim Z$, the cosine similarity between $X$ and $Y$ is:

$$ $\mathrm{cos}(X,Y)=\frac{\sum_{i=1}^{d}X_{i}Y_{i}}{\sqrt{\sum_{i}X_{i}^{2}} \sqrt{\sum_{i}Y_{i}^{2}}}=\frac{X^{T}Y}{\sqrt{(X^{T}X)(Y^{T}Y)}}=\frac{X^{T}Y} {\|X\|\|Y\|},$ (9) $$

where subscript $i$ denotes the $i$-th element of a vector
, and $\|X\|$ and $\|Y\|$ denote the Euclidean norms of vectors $X$ and $Y$ respectively.
Each term in the numerator of Eq. 9 is just a product of two random variables, but the denominator makes things complicated.
A closed-form solution of a ratio of two random variables is difficult to calculate.

Let us introduce a relaxation that can be used to approximate the denominator and simplify the calculation. Using cosine’s scale invariance (shown in A.2), we define a relaxation $r(Z)$ as:

$$ $r(Z)=\frac{Z}{m},$ (10) $$

where $m=\sqrt{\sum_{i}\sigma_{i}^{2}}$.
Relaxation $r$ has two properties. First, the expected value of the squared norm after relaxation, denoted as $\mathbb{E}[||r(Z)||^{2}]$, is equal to 1. Second, the ratio of variance to square of the mean is given by $\frac{\sum_{k}2\sigma_{k}^{2}}{(\sum_{k}\sigma_{k}^{2})^{2}}$, and it approaches 0 as the dimensionality $d$ tends to infinity. Under the condition that the contribution of any single covariance eigenvalue is sufficiently small. Smith et al. () show that these properties can then be used to approximate the cosine similarity as:

$$ $\mathrm{cos}(X,Y)=\mathrm{cos}(r(X),r(Y))=\frac{\sum_{i=1}^{d}r(X)_{i}r(Y)_{i} }{\|r(X)\|\|r(Y)\|}\approx\sum_{i=1}^{d}r(X)_{i}r(Y)_{i}.$ (11) $$

With this approximation, the moments of the cosine similarity can be easily calculated:

$$ $\mathbb{E}[\mathrm{cos}(X,Y)]=0,$ (12) $$

$$ $\mathrm{Var}(\mathrm{cos}(X,Y))=\sum_{i=1}^{d}\frac{\sigma_{i}^{4}}{(\sum_{j=1 }^{d}\sigma_{j}^{2})^{2}}.$ (13) $$

We can see from Eq. 13 that the cosine similarity distribution is highly dependent on the dimensions with high variance. Taking the gradient of Eq. 13 with respect to $\sigma_{i}^{2}$, we get:

$$ $\frac{\partial}{\partial\sigma_{i}^{2}}\mathrm{Var}(\mathrm{cos}(X,Y))=2\left( \sum_{j=1}^{d}\sigma_{j}^{2}\right)^{-3}\left[(\sum_{j=1}^{d}\sigma_{j}^{2}) \sigma_{i}^{2}-(\sum_{j=1}^{d}\sigma_{j}^{4})\right].$ (14) $$

We can see that the global minimum of the approximation for $\mathrm{Var}(\mathrm{cos}(X,Y))$ is achieved when all individual variances of the original distribution are equal, i.e., $\sigma_{i}^{2}=\sigma_{j}^{2}\ \forall\ i,j$. This is enforced in AE-SVC by the variance constraint (Eq. 3). Therefore, we theoretically and empirically justified that AE-SVC minimizes the variance of the cosine similarity distribution, improving retrieval performance.

## 5 Experiments

### 5.1 Experimental Setup

Datasets: We evaluate our approach on four distinct image retrieval datasets: InShop (Liu et al., ), Stanford Online Products (SOP) (Song et al., ), Pittsburgh30K (Torii et al., ), and TokyoVal (Torii et al., ). InShop and SOP are state-of-the-art (SOTA) image retrieval datasets commonly used in metric learning research, while Pittsburgh30K and TokyoVal are recognized as SOTA place recognition datasets, frequently used in robotics research.

Foundation Models Evaluated:
To demonstrate the broad applicability of our approach, we conducted experiments using four different foundational models trained with a variety of techniques and pre-training datasets: CLIP (Radford et al., ), DINO (Caron et al., ), DINOv2 (Oquab et al., ), and ViT (Dosovitskiy et al., ). The CLIP model was pre-trained on the LAION-2B dataset (Community, ) using a contrastive learning approach. Both DINO and ViT models were pre-trained in a self-supervised manner on the ImageNet dataset (Russakovsky et al., ). The DINOv2 model was pre-trained in a self-supervised manner on the LVD-142M dataset (Oquab et al., ). For the DINO and DINOv2 models, we evaluated different model sizes: specifically, DINO-S8 with 22.7 million parameters and DINO-B16 with 300 million parameters, as well as DINOv2-Small with 22.1 million parameters and DINOv2-Large with 300 million parameters.

Settings: Our work presents two contributions: the introduction of a novel space, AE-SVC, and a new method for learning an adaptive embedding, (SS)2D, on top of AE-SVC. Consequently, our experiments are divided into two parts. First, we demonstrate that AE-SVC significantly enhances the retrieval performance of foundation models across various datasets and model choices. Second, we show that (SS)2D facilitates more effective dimensionality reduction on top of AE-SVC, further improving performance at lower dimensions. We compare (SS)2D with the following baselines: a variational auto-encoder (VAE) applied to AE-SVC for each dimensional size separately. Additionally, we compare (SS)2D with using Similarity Space Distillation applied to each dimension separately, referred to as SSD. Note that SSD serves as a theoretical upper bound for (SS)2D. We use mean Average Precision at $\mathbf{k}$ (mAP@$\mathbf{k}$), a standard metric for evaluating retrieval performance. We show additional results on the Recall@$\mathbf{k}$ metric in Appendix A.5.

### 5.2 Results

Figure: Figure 4: AE-SVC significantly improves the retrieval performance of foundation models. AE-SVC (solid lines) consistently outperforms the off-the-shelf foundation models, i.e., PCA (dashed lines), on four datasets, achieving a 15.5% average improvement in retrieval performance.
Refer to caption: extracted/5913932/figs/exp_vapca_map.png

Fig. 4 shows the performance comparison between AE-SVC and PCA across $4$ datasets, InShop, Stanford Online Products (SOP), Pittsburgh30K, and TokyoVal, using $3$ foundation models (ViT, DINOv2-Large, and DINO-B16). The performance of PCA (dotted line) represents the default off-the-shelf performance of the foundation models, while the performance of AE-SVC is shown with solid lines. AE-SVC consistently outperforms PCA across all datasets and embeddings. For example, the DINOv2-Large embedding (yellow) shows a $24\%$ improvement on InShop, $10\%$ on SoP, $5\%$ on Pittsburgh30K, and $22\%$ on TokyoVal at full embedding size. Overall, AE-SVC achieves an average improvement of $15.5\%$ across all datasets and embeddings at full size. Also, AE-SVC consistently surpasses PCA at smaller embedding sizes, providing a better size-performance trade-off. For additional results with more foundation models, see Appendix A.3.

Fig. 5 illustrates the advantages of (SS)2D on two datasets: InShop and Pittsburgh30K, utilizing two foundation models, Dinov2-Large and Dino-S8. We compare (SS)2D with a non-linear dimensionality reduction technique, Variational Auto Encoder (VAE) (black crosses). Additionally, we compare it with Similarity Space Distillation (SSD), which represents the theoretical upper bound of (SS)2D. Note that unlike VAE (black crosses) and SSD (red crosses), which are trained for each dimension separately, (SS)2D learns an adaptive embedding in one shot. From the plot, we observe that AE-SVC (orange) initially enhances the performance of the original embedding (blue), and (SS)2D subsequently provides additional improvements of up to $10\%$ at smaller embedding sizes.
Also note that since SSD is trained for each dimension separately, it serves as the theoretical upper bound of (SS)2D. The results indicate that (SS)2D closely approaches this upper bound. For additional results involving more datasets and foundation models, please refer to Appendix A.4. As discussed in Sec. 1, a smaller embedding size directly translates to faster retrieval speeds. Therefore, the effective dimensionality reduction achieved with (SS)2D results in improved retrieval speeds, as demonstrated in Fig. 1(D).

Figure: Figure 5: Applying (SS)2D over AE-SVC leads to further performance boost at lower embedding sizes. Compared to VAE and SSD, (SS)2D offers superior single-shot dimensionality reduction, achieving up to a 10% enhancement at smaller embedding sizes, closely approaching SSD’s theoretical upper bound.
Refer to caption: extracted/5913932/figs/exp_ss2d.png

### 5.3 Limitations

While AE-SVC demonstrates significant improvements when applied to foundation models, its ability to enhance dataset-specific models is limited. The pre-trained models tailored for a specific dataset already capture the nuances required for effective retrieval, leaving less room for further optimization via AE-SVC.
Additionally, (SS)2D introduces a computational overhead during training due to the need to compute the loss functions for various embedding sizes. This overhead increases training time, though it is still substantially lower than the cost of retraining neural networks for different embedding sizes from scratch.

## 6 Conclusion and Future Work

In this paper, we proposed AE-SVC and (SS)2D, which improve retrieval scalability and efficiency.
Our results show a significant improvement on $4$ datasets, from image retrieval to place recognition tasks.
AE-SVC demonstrates up to a $16\%$ improvement in retrieval performance, and (SS)2D shows an improvement of $10\%$ for smaller embedding sizes, which further boosts retrieval efficiency.
Future work includes new retrieval models that orchestrate the variance-aware property in training loss of foundation models.
Although we observe the resulting difference in the cosine similarity distributions of the foundation models and the data set-specific models (Figure 3), the fundamental discrepancy between the embedding spaces remains unknown to the community.

## Reproducibility Statement

In the theoretical analysis section (Sec. 4) of this work, we clearly state all the assumptions as and when necessary. Appendix A.1 and A.2 provides additional details necessary to understand the proofs in Sec. 4. We also discussed all the hyperparameters in Appendix A.6. Lastly, we release a reproducible codebase in the supplementary details. The code includes dataloaders, execution code, and links to download all the datasets and models used. We also provide detailed instructions on how to replicate our experiment setup and results, ensuring complete transparency and reproducibility.

## Appendix A Appendix

### A.1 Invariance of Orthogonal Transformation

We show that cosine similarity is invariant to orthogonal transformation.
Define $T_{\mathrm{orth}}\in\{T:T^{\top}T=\mathbf{I}\}$ as an orthogonal transformation.
Following equation 9, we see that

$$ $\displaystyle\cos(T_{\mathrm{orth}}X,T_{\mathrm{orth}}Y)$ $\displaystyle=\frac{\left(T_{\mathrm{orth}}X\right)^{\top}T_{\mathrm{orth}}Y}{ \|T_{\mathrm{orth}}X\|\|T_{\mathrm{orth}}Y\|}$ (15) $\displaystyle=\frac{X^{\top}\mathbf{I}Y}{\sqrt{X^{\top}\mathbf{I}X}\sqrt{Y^{ \top}\mathbf{I}Y}}$ $\displaystyle=\frac{X^{\top}Y}{\|X\|\|Y\|}$ $\displaystyle=\cos(X,Y)~{}~{}~{}\square.$ $$

### A.2 Cosine Scale Invariance

Similar to the derivation in A.1, we now show that cosine similarity is invariant to scaling.
Again, we define a constant $c$ and follow equation 9:

$$ $\displaystyle\cos\left(\frac{X}{c},\frac{Y}{c}\right)$ $\displaystyle=\frac{X^{\top}Y/c^{2}}{\|X\|\|Y\|/c^{2}}$ (16) $\displaystyle=\frac{X^{\top}Y}{\|X\|\|Y\|}$ $\displaystyle=\cos(X,Y)~{}~{}~{}\square.$ $$

### A.3 AE-SVC Additional Results with more foundation models

In Fig. 4, we showed the advantage of using AE-SVC for embeddings of three models. Here, in Fig. 6, we show additional results with three more embeddings, namely: CLIP, DINO-V2-Small, and DINO-S8. We observe similar trends as previously noted. We release our codebase in the supplementary material for easy reproduction of all these results.

Figure: Figure 6: AE-SVC significantly improves the retrieval performance of foundation models. Here, we see a similar trend as discussed in the main paper but with three additional foundation models.
Refer to caption: extracted/5913932/figs/appendix_vapca.png

### A.4 (SS) 2 D Additional Results with more datasets and foundation models

In Fig. 5, we showed the advantage of using (SS)2D on top of AE-SVC for additional improvement in retrieval performance at smaller dimensions. Here, in Fig. 7, we show additional results for (SS)2D on SOP with DINO-B16 and on TokyoVal with ViT. We observe similar trends as previously noted. For trying (SS)2D with more combinations of datasets and foundation models, please refer to our codebase that we release in the supplementary material.

Figure: Figure 7: Applying (SS)2D over AE-SVC leads to further performance boost at lower embedding sizes. Here, we see a similar trend as discussed in the main paper but with two additional foundation models and datasets.
Refer to caption: extracted/5913932/figs/appendix_ss2d.png

### A.5 AE-SVC Additional Results on the Recall Metric

In Fig. 4, we show the advantage of using AE-SVC for embeddings of three models on the mean Average Precision at $\mathbf{k}$ (mAP@$\mathbf{k}$) metric. Here, in Fig. 8, we show additional results on the Recall@$\mathbf{k}$ metric. We observe very similar trends on the Recall@$\mathbf{k}$ metric as we did on the mAP@$\mathbf{k}$ metric. We release a reproducible codebase for readers to experiment with both metrics.

Figure: Figure 8: AE-SVC significantly improves the retrieval performance of foundation models on the recall metric. AE-SVC (solid lines) consistently outperforms the off-the-shelf foundation models, i.e., PCA (dashed lines), on four datasets.
Refer to caption: extracted/5913932/figs/exp_vapca_recall.png

### A.6 Hyperparameters

We discussed four hyperparameters in Sec. 3.1: $\lambda_{\text{rec}}$, $\lambda_{\text{cov}}$, $\lambda_{\text{var}}$, and $\lambda_{\text{mean}}$. We empirically determined the values of these hyperparameters, ensuring optimal performance across all datasets. The fixed values are: $\lambda_{\text{rec}}=25$, $\lambda_{\text{cov}}=1$, $\lambda_{\text{var}}=15$, and $\lambda_{\text{mean}}=1$.

## Peer Reviews

confidence: 4
contribution: 3
initial_rating: 6
presentation: 3
questions: 1. During training, does the back propagation happen after traversing the whole dataset or a batch of the images? Would the choice of two ways influence the performance?
2. Does the embedding size reduction influence the cosine similarity variance or other constraints for AE-SVC of the output features? 
3. Does the performance be changed if the order of SVC and (SS)_2D switch? Or would the performance be improved if append SVC on the resized embeddings?
soundness: 4
strengths: 1. The paper presents Autoencoders with Strong Variance Constraints (AE-SVC), an unsupervised method. By enforcing orthogonality, mean centering, and unit variance constraints in the latent space, AE-SVC shifts the cosine similarity distribution to be more discriminative. This advancement addresses the scalability issue in current SOTA methods.
2. The authors introduce Single Shot Similarity Space Distillation ((SS)_2D), that preserves the similarity structure of embeddings without requiring retraining for different dimensions. (SS)_2D produces adaptive embeddings where smaller segments maintain high retrieval performance, addressing the impracticality of separate training for varying embedding sizes found in methods like PCA or traditional autoencoders.
3. The paper's two-step approach, first enhancing embeddings with AE-SVC and then applying (SS)_2D, yields significant improvements in retrieval performance across multiple datasets and foundation models. Experimental results show that this method not only accelerates retrieval speed by reducing embedding dimensions but also surpasses the accuracy of existing unsupervised techniques and even the original full-sized embeddings.
summary: This paper proposes a method for facilitating the image retrieval task. It focuses on solving two challenges, scalability and efficiency, existed in the image retrieval. To tackle scalability, strong variance constraints are introduced to restrict the latent features. The variance of feature in different dimensions could be similar, so that improving the discriminative ability of the latent features. For the efficiency, a Single-shot Similarity Space Distillation method is introduced to generate adaptive size features, and keep the cosine similarity of sample features. The experiments show that the AE-SVC improve the retrieval performance, especially for the foundation models. The (SS)_2D can improve the performance for smaller embedding sizes.
weaknesses: 1. The detail of the projection function F in L280 may need more explanations. The size of the embedding \hat{z} is adaptive. The change of the embedding size whether influences the projection F. In other words, there are matched functions for different size m.
2. The figures in Fig. (3,4,5) are small. It is hard to distinguish data points of lower embedding dimensions. It could be better that the quantity results are listed in a table.

---

confidence: 3
contribution: 2
initial_rating: 5
presentation: 3
questions: 1. My major concern is whether the learned normal distribution is still effective in the biased target domain, which is common in dataset-specific tasks. Without the prior information of the target domain, it is ambiguous for neural networks to distinguish instances by discriminative features or global features ( discriminate shirt shape or visually more similar image in the example in Fig.2, the former would be right in commerce recommendation system and the latter is more likely to be the output of the common search engine)
2. The settings of Fig.4 and 5 are not consistent, and it is hard to discriminate results in different styles (color, line and dot). It would be better to list the results in tables for better comparison.
3. It is contrary to common sense that the performance of ViT > DINO-Base > DINO-Large in InShop results in Fig.4, but the results of Pittsburgh30K and TokyoVal are normal.
4. Without additional data, student models learned from teachers would get inferior results. Based on AE-SVC (orange) in Fig.5, it seems that applying feature distillation (green) further improves the retrieval performance. It would be better to add an analysis of this point in the experiment.
soundness: 2
strengths: The authors focus on major concerns about using the foundation model in image retrieval or even more CV tasks: how to promote foundation model features and balance model performance and efficiency. By aligning teacher and student similarity spaces, the proposed distillation with flexible output dimensions indeed reduces retrieval compute resources.
summary: This paper discusses two major concerns regarding image retrieval: scalability and efficiency. The concerns are more meaningful when existing retrieval methods use foundation models as feature extractors. To improve the foundation model's performance on retrieval tasks, the authors proposed Autoencoders with Strong Variance Constraints, which enforced three constraints on the learned latent space. Meanwhile, Single-shot Similarity Space Distillation outputs more compact embeddings with flexible dimensions for the model efficiency.
weaknesses: 1. The motivation of AE-SVC is worth discussing: it encourages the network to learn the normal embedding distribution (mean to 0, std to 1), but gt labels of dataset-specific tasks may biased.
2. The authors claim that they are the first to explore representation learning ideas in image retrieval, considering retrieval is essentially a feature learning task, the soundness is not as remarkable as claimed.
3. AE-SVC is only compared with the standard PCA in Sec.5.2, more competitors (e.g. PCA variants or AnyLoc ) are needed to demonstrate the effectiveness of AE-SVC, if possible.
4. Experiment settings in Fig.4 and Fig.5 are not consistent. Compared with AE-SVC, the results of (SS)2D seems incomplete.

---

confidence: 3
contribution: 2
initial_rating: 5
presentation: 3
questions: See weaknesses.
soundness: 3
strengths: - Overall, the paper is well-motivated and well-written, aiming to solve an important problem in large-scale image retrieval.
- The theoretical aspects are well-developed, though it is unclear if the approximation in Eq. 11 is sufficiently supported. It may benefit from references to statistics or mathematical literature rather than an Arxiv paper in biology.
- Experiments demonstrate the method's effectiveness, showing significant improvement.
summary: This paper addresses two key challenges in large-scale image retrieval systems: scalability and efficiency. To enhance scalability, the authors focus on improving the performance of existing foundation models. For efficiency, they propose an effective unsupervised dimension reduction technique.
weaknesses: - I am somewhat skeptical about the scenario described in L39-L42; in fact, self- or semi-supervised learning might address this issue (see related works [1, 2]).
- I have some doubts about the novelty of AE-SVC, as it combines several common losses. Additionally, there are four losses in Eq. 5, but no explanation is provided on how they are balanced. Experiments or theoretical support in this regard would be beneficial.
- The experimental results are somewhat limited. Is there additional related work beyond PCA for comparison? Also, I noticed a lack of ablation studies, as only two experiments, shown in Fig.4 and Fig.5, are provided.

[1] eProduct: A Million-Scale Visual Search Benchmark to Address Product Recognition Challenges

[2] V2L: Leveraging Vision and Vision-language Models into Large-scale Product Retrieval

---

confidence: 3
contribution: 2
initial_rating: 6
presentation: 3
questions: The paper focuses on improving the embedding from image foundation models by unsupervised training without any labels. However, given the recent advancements in multimodal LLMs, it seems possible to leverage these models (like GPT-4o) to automatically produce reasonable labels for fine-tuning image embedding models. If this is feasible, would the proposed approaches still be practically useful, given the fact mentioned in the paper that dataset-specific models are better than foundation models?
soundness: 2
strengths: 1. The proposed approaches are quite simple yet effective. AE-SVC is backed up by some reasonable empirical analyses in Sec.4. The adaptive dimension reduction brought by (SS)$_2$D is interesting and can be practically useful to avoid training multimodal models for different embedding dimensions.

2. The authors conduct thorough experiments to demonstrate the effectiveness of the proposed approaches: 4 image retrieval datasets and 4 visual foundation models are considered, and AE-SVC + (SS)$_2$D consistently show positive results for efficient and accurate image retrieval.

3. A codebase is provided in the supplementary material. Roughly looking through the code, all necessary model/data components are included, ensuring the reproducibility of the paper.
summary: This paper aims to improve image retrieval under the context of using pretrained image foundation models. Two concrete techniques are designed: (1) Autoencoders with Strong Variance Constraints (AE-SVC) regularizes the embeddings from foundation models to improve the the retrieval performance, (2) Single-shot Similarity Space Distillation ((SS)_2D) compresses the embedding in an adaptive way to accelerate the retrieval. The paper provides theoretical and empirical analyses to justify the effectiveness of AE-SVC. The experimental results show that the proposed approaches can effectively improve the retrieval performance/efficiency of embedding from four different visual foundation models across four datasets.
weaknesses: 1. The theoretical analysis in Sec.4 is only conducted based on the Gaussian distribution assumption, which does not necessarily hold for the embeddings from the visual foundation models. Based on the PDF visualization in Figure 3, the true distribution of cosine similarity between image embeddings is far from Gaussian.

2. In large-scale retrieval, approximate nearest neighbour (ANN) is commonly used to accelerate the retrieval process, so the complexity analysis of image retrieval in the paper is not necessarily right in real-world applications. 

3. The figure placement can be optimized. The method details should not be in the first figure, which increases the burden of understanding the introduction, and forces people to switch back and forth when reading the method section.

---

confidence: 5
contribution: 2
initial_rating: 3
presentation: 3
questions: 1. The mean and covariance quantities in (4) and (2) (also variance in (3), but this is redundant as discussed above), are global in the sense that they are measured on all $n$ samples of the test set. What happens when the test set is large and how are they treated when learning with a stochastic optimizer in mini-batches? Are mini-batches used in this work?

2. Can the authors confirm or refute my understanding that the variance of similarities is a concave function of the mean and is maximized when the mean is zero? In the latter case, how would they update the paper?
soundness: 2
strengths: The idea is well motivated. The methodology is sound. The writing is of high quality and mostly clear. The experiments support the claims on a number of datasets and models. The two contributions are shown to be effective comparing with simple baselines.
summary: This work uses a foundation model for image retrieval, making two contributions: (1) AE-SVC trains an autoencoder on top of the foundation model such that the latent vectors have statistical (mean and covariance) properties inspired or equivalent to centering, PCA and whitening. (2) SS2D performs dimensionality reduction jointly for multiple target dimensions by distilling a similarity matrix from the AE-SVC encoder to another student encoder. Both are learned on the test (reference) set, while the foundation model is pre-trained and frozen.
weaknesses: The novelty is incremental, while important related work and baselines are missing. The setting of learning on the test set is questionable. Comparisons with baselines are lacking, not fair and not informative enough. Section 4 fails to properly separate existing work by Smith et al. (2023) from this work and is partly unclear. More details follow.

1. The paper addresses image retrieval and performs evaluation on standard deep metric learning and place recognition datasets. It misses revelant foundational work on image retrieval, including methodologies, benchmarks and protocols:

    > [*1] Gordo et al. 2016, Deep Image Retrieval: Learning Global Representations for Image Search
    >
    > [*2] Radenovic et al. 2017, Fine-Tuning CNN Image Retrieval with No Human Annotation
    >
    > [*3] Radenovic et al. 2018, Revisiting Oxford and Paris: Large-Scale Image Retrieval Benchmarking
    >
    > [*4] Weyand et al. 2020, Google Landmarks Dataset v2: A Large-Scale Benchmark for Instance-Level Recognition and Retrieval

    This work should be repositioned against this work in terms of novelty and protocols used. The related work and experiments sections should be extended accordingly, considering additional training and test datasets and evaluation protocols from these works.

2. The motivation for AE-SVC is scalability. That is, it is challenging to have a labeled dataset-specific training set, hence it is suggested to learn in an unsupervised on the test (reference) set. On one hand, this setting is extremely problematic because it transfers the burden from training to testing, by having to learn something on the test set. The latter is in many practical applications constantly changing, hence the model would need to be constantly updated. It is a kind of overfitting to the test set that has never been used in standard work like the references above. To my knowledge, it has only been used in

    > [*5] Liu et al. 2019, Guided Similarity Separation for Image Retrieval

    Hence, this work should be a baseline for comparison if the authors insist in their setting of learning on the test set. On the other hand, if the motivation to learn in an unsupervised way, this can still be done on a separate training set. For example:

    > [*6] Kim et al. 2022, Self-Taught Metric Learning without Labels

    Besides, learning in [2*] may be supervised, but with labels obtained automatically without human annotation. Self-supervised learning should not be a reason to learn on the test set.

3. In AE-SVC, I believe variance loss (3) is redundant. It is merely the diagonal part of covariance loss (2). Other than that, the covariance loss (2) and centering loss (4) are inspired by centering, PCA and whitening. Hence, AE-SVC should be compared to a baseline where centering, PCA and whitening are performed as post-processing on features extracted by the foundation model, without learning. This baseline is missing. Whitening has a long history in image retrieval but ignored here. Also very successful is supervised discriminative whitening [*2], which could be adapted for self-supervision to be used as a baseline.

4. In SS2D, I believe the loss is the same as

    > [*7] Park et al. 2019, Relational Knowledge Distillation

   The authors cite this paper but do not discuss the actual relation with their work. I believe the only new component here is applying it for dimensionality reduction and learning for multiple target dimensions at once, which are pretty incremental contributions. Also very relevant is

    > [*8] Kalantidis et al. 2022, TLDR: Twin Learning for Dimensionality Reduction

    in the sense that it is also learning a kind of autoencoder on top of frozen feature for dimensionality reduction. Instead of a reconstruction loss, it uses the Barlow Twins loss on the output vectors, which is the same as the covariance loss here, but uses cross-correlation instead. Both the architecture and cross-correlation matrix should be compared to as baselines.

5. Section 4 is useful and interesting, but it is presented as new to a large extent while it actually all comes from Smith et al. (2023). The only new part is the experiment of Figure 3. In addition, parts are copied from Smith et al. (2023) without carrying over all relevant information, such that the resulting text is unclear. For example, (8) refers to "discriminative power" but there is nothing to discriminate here, whereas Smith et al. (2023) compare the distribution of target similarities to a "null" or "background" distribution where the term "discriminative" makes sense. As another example, (14) results in a "global minimum" without saying anything about the function being convex, which is shown indeed by Smith et al. (2023).

    In the analysis of Section 4, the data is taken to be centered and nothing is said about the effect of this choice to the distribution of similarities, either here or in Smith et al. (2023). To my understanding, the variance of similarities is a **concave** function of the mean and is **maximized** when the mean is zero. This is the opposite of what happens with whitening and raises questions on the motivation of minimizing the variance. To the very least, the experiment of Figure 3 should be extended to also show the distribution of similarities with centered data but without PCA and whitening. It should also include centering, PCA and whitening as post-processing steps rather than being learned with (2), (3) and (4).

6. In the experiments of Figures 4 and 5, the current work (AE-SVC, SS2D or SSD) uses an additional encoder on top of the foundation model, while the "default" and "PCA" are using the foundation model alone, hence are not comparable. These results tell us nothing about whether the benefit is coming from the properties encouraged by the loss functions, the additional learned parameters or the adaptation to the test set. These are three factors that should be evaluated independently in some way, for example by fine-tuning the last layers of the foundation model rather than adding an extra encoder.

    Whitening by post-processing after PCA should be definitely added as a baseline. TLDR or other state of the art dimensionality reduction methods should also be added, because centering, PCA, whitening and VAE are very basic baselines.

## Post-rebuttal

The following is an assessment after rebuttal and considering the other reviews.

The contribution is incremental, which is not necessarily a problem if there is a clear and fair positioning with respect to previous work, a clear message and good results. The situation has clearly improved during rebuttal and discussion, but I am afraid that the paper is still weak in these three aspects. In particular:

- *Losses*. The authors state in the dicussion that there is "stark contrast" with TLDR in that they use three losses rather than one. The covariance loss is very close to Barlow Twins loss used in TLDR, which is cross-correlation. An experimental comparison between the two is not there. The variance loss only refers to the diagonal of the covariance loss and treating it separately makes sense only because of using a different loss weight, which is not mentioned anywhere in the main text. The appendix A6 says $\lambda_{\text{cov}}=1$, $\lambda_{\text{var}}=15$, but the effect of each $\lambda$ is not shown. What if $\lambda_{\text{cov}}=15$ for example? A contribution is meaningful when effective, especially if incremental. The mean-centering loss is something ubiquitous. The relative benefit of each loss component is not shown anywhere. Under these conditions, the contribution claim of three losses is still not adequately supported by evidence.

- *Theoretical support*. The authors claim in the paper and discussion that providing theoretical analysis in section 4 is a contribution but this study is carried out by Smith at al. (2023) and section 4 does not clarify if there is anything new here except the experiment of Figure 3. This is kind of over-claiming contribution. Minimizing the variance of cosine similarity distribution is still not a clear narrative as motivation since, as a function of the mean, it is maximized. Using the term "discriminative" when there is nothing to discriminate from is still confusing.

- *Distillation*. The authors state in the paper that their approach "is conceptually similar to SPKD", but this is different from saying "we use RKD (Park et al., 2019)", for example. Again, this is kind of over-claiming contribution.

- *Supervision*. On one hand, there is a supervised PCA whitening approach that is common in image retrieval, as performed by Radenovic et al. (2017) mentioned in my review. This would be another baseline, plus the authors could use this kind of supervision on their own approach. On the other hand, there are more unsupervised metric learning methods like Kim et al. (2022) that I have mentioned in my review, which could be used for distillation or self-supervision, as competitors or as components of this method.

- *Two-step*. The authors state in the discussion that their approach is two-step in "stark contrast" with previous work, but they do not experiment with an one-step variant of their approach to show the benefit and they do not explain why two-step is better.

The most important additions during the rebuttal are the experiment of training on a held-out training set, the comparison with TLDR and applying the losses at different layers (Table 2 of rebuttal document). The last one possibly accounts partially for the performance improvement over TLDR but the effect of the losses is not shown. However, as discussed above, several things are still missing in terms of fair positioning with previous work, some rewriting to make the message clearer and more experiments. I believe all of these issues could be resolved in theory, in which case the paper would be interesting enough to publish despite the incremental contribution. In case the paper is accepted, I strongly recommend the authors to revise accordingly for the camera-ready.

Meta-review: The paper addresses the scalability and efficiency challenges in image retrieval. Scalability is tackled through strong variance constraints to restrict the latent features, while efficiency is improved via a distillation method that generates adaptive size features. The proposal aims to address problems in pre-trained (or "foundation") models commonly used in retrieval tasks, promoting a balance between performance and efficiency.

Strengths:
- The proposed method shifts the distribution of cosine similarity to be more discriminative.
- The method preserves the similarity structure of embeddings without requiring retraining for different dimensions.
- The paper demonstrates improvements in retrieval performance across multiple datasets and pre-trained models.
- The theoretical aspects are well-developed, with some exceptions.

Weaknesses:
- The contributions in the proposed autoencoder are limited since it relies on common losses. The combination of variance, covariance, and mean-centering constraints is interesting, but their application at the representation level is incremental.
- The learned normal distribution may not be effective in a biased target domain.
- The theoretical analysis is limited to the Gaussian distribution assumption, which does not necessarily hold for embeddings from different visual pre-trained models.
- The experiments require further analysis to better understand the results.
- The baselines used are limited. The paper would benefit from comparisons with other methods.
- The proposed setting is problematic because it shifts the burden from training to testing by requiring learning to be conducted on the test set, which is highly problematic.

Despite the raised concerns, the majority of the reviewers have a positive recommendation given the results presented in the paper.  The minority of the reviewers had concerns regarding the limited technical contribution of the paper.  Nevertheless, I recommend a weak acceptance due to the positive, yet somewhat limited, strengths of the paper.

Decision: Accept (Poster)