Learning Semantic Proxies from Visual Prompts for Parameter-Efficient Fine-Tuning in Deep Metric Learning
Authors: Li Ren
Chen Chen
Liqiang Wang
Kien A. Hua
Venue: ICLR 2024 poster (2024)
Keywords: metric learning
image retrieval
parameter-efficient fine tuning
Primary area: representation learning for computer vision, audio, language, and other modalities
TL;DR: We investigate parameter-efficient methods for fine-tuning the pre-trained model for DML tasks

Abstract: Deep Metric Learning (DML) has long attracted the attention of the machine learning community as a key objective. Existing solutions concentrate on fine-tuning the pre-trained models on conventional image datasets. As a result of the success of recent pre-trained models derived from larger-scale datasets, it is challenging to adapt the model to the DML tasks in the local data domain while retaining the previously gained knowledge. In this paper, we investigate parameter-efficient methods for fine-tuning the pre-trained model for DML tasks. In particular, we propose a novel and effective framework based on learning Visual Prompts (VPT) in the pre-trained Vision Transformers (ViT). Based on the conventional proxy-based DML paradigm, we augment the proxy by incorporating the semantic information from the input image and the ViT, in which we optimize the visual prompts for each class. We demonstrate that our new approximations with semantic information are superior to representative capabilities, thereby improving metric learning performance. We conduct extensive experiments to demonstrate that our proposed framework is superior and efficient by evaluating popular DML benchmarks. In particular, we demonstrate that our fine-tuning method achieves comparable or even better performance than recent state-of-the-art full fine-tuning works of DML while tuning only a small percentage of total parameters.

Published as a conference paper at ICLR 2024
LEARNING SEMANTIC PROXIES FROM VISUAL PROMPTS FOR GARAMETER-EFFICINT FINE-TUNING IN DEEP METRIC LEARNING
Li Ren', Chen Chen Liqiang Wang', Kien Hua* 1 Department of Computer Science University of Central Florida, USA
Center for Research in Computer Vision
[Li Ren, Shen.Chem, Liqiang. Wang, Kien Hua] @ucf edu
ABSTRACT
Deep Metric Learning (DML) has long attracted the attention of the machine learn- ing community key objective. Existing solutions concentrate the pre-trained models on conventional image datasets. As a result of the success of recent pre-trained models trained from larger-scale datasets, it is challenging to adapt the model to the DML tasks in the local data domain while retaining the previously gained knowledge. In this paper, we investigate parameter-effiient methods for tine-tuning the pre-trained model for DML tasks. In particular, we pro- pose novel and effective framework based on learning Visual Prompts (VPT) in the pre-trained Vision Transformers (VIT). Based on the conventional proxy-based DML. paradigm, we augment the proxy by incorporating the semantic information from the input image and the VIT, in which we optimize the visual prompts for each class. We demonstrate that our new approximations with semantic informa- tion are superior to representative capabilities, thereby improving metric learning performance. We conduct extensive experiments to demonstrate that our proposed framework is effective and efficient by evaluating popular DML benchmarks. In particular, we demonstrate that our fine-tuning method achieves comparable or even better performance than recent state-of-the-art full fine-tuning works of DML while tuning only a small percentage of total parameters. Code available at http:///httt..cooooaaaaaeeeettee
fine-tuning
DML
INTRODUCTION
Metric learning is crucial part of machine learning that creates distance functions based on the semantic similarity of the data points. Modern Deep Metric Learning (DML) uses deep neural networks to map data to embedding space where similar data closer. This is especially useful
for computer vision applications. including image retrieval (Lee et al., 2008 Yang et al. 2018). human re-identification (Wojke & Bewley 2018 Hermans et 2017). ) and image localization
et al. 2015; Ge et al.] 2020). Because of the success the development of deep neural networks,
most current DML works (Song al., 2016 Wang al. [201b]) Kim al. 22200] Roth al. 2022 Yang et al. 2022) recommend convolutional neural networks (CNN) (He et al.) 2016 Wang et al. 2019) pre-trained on conventional ImageNetIK dataset (Deng al. 2009) as their backbone of data
encoder. It was recently discovered that the more advanced vision transformers (ViT) are better in
terms of representation capability and performance on DML tasks (El-Nouby al. 202T Ramzi
et al.
2021
Patel et al. 2022 Ermolov et al. (022).
The recent foundation models have trends to scale up their parameters and training data. Utilizing larger backbone models and massive amounts of raw data, recent studies have demonstrated superior performance in major deep learning tasks (Radford et al. 2021). Considering their prohibitively expensive training costs, applying these huge models to downstream tasks becomes crucial and challenging research topic. Unfortunately, the most straightforward adaptation strategy, the full fine-tuning of the pre-trained model, has been criticized for its high training cost for tuning all parameters and the catastrophic forgetting caused by the difference between the scale and distribution of the raw data and the local dataset.

---

Published as a conference paper at ICLR 2024
Recent research looks into efficient techniques adapt large-scale models by fine-tuning just few parameters. A simple method for this is to optimize dask-specific heads, such as the linear classifier, commonly known as linear probing (Mahajan et al. 2018 Chen et al. 2021). People also investigated more efficient ways to adjust the Transformers, such as adjusting the bias factor (Zaken et al. 2021].
tuning soft prompts (Lester et al.) [022] Jia al. (022) to serve as additional input data, or tuning additional parameters besides the Transformers (Rebuffi et al. [0018] Hu et al.) [021] Chen et al.
2022a). Recent works also confirm that the ViT can also be effectively adapted with those methods to
downstream tasks in both image and video domains (Jia et al. 2022 Chen et al. 2023b; Yang et al. 2023).
However, the parameter-efficient method for fine-tuning a large model, particularly for DML tasks, has yet to be fully investigated. Recent works that propose full fine-tuning of the ViTs in the DML. paradigm continue to require substantial amount of computational resources. For example, the state- of-the-art method proposes training their model on multiple GPUs or a huge batch size for an extended period (Ermolov et al. 2022; Patel et al. 2022). In addition, it is still being determined whether the full fine-tuning is optimized for DML tasks, as inappropriate fine-tuning would easily lead to overfitting, resulting in catastrophic forgetting where previously learned tasks and presentations are lost. To overcome these difficulties, we investigate parameter-efficieet methods to fine-tune the pre-trained vision models DML tasks in this paper. Specifically, we compare existing parameter-efficient fine-tuning strategies based on proxy-based DML. paradigms. After that, we chose Visual-Prompt Tuning (VPT), the most efficient strategy, as our backbone to increase its efficiency on DML tasks further.
To this end, we propose an effective learning framework based on the VPT for fine-tuning pre-trained ViT's on DML tasks. Based on the traditional proxy-based DML system, we learn the representation of image samples, particularly by fine-tume tin-tuming additional learnable soft prompts while other parameters are fixed. Instead of the original proxies that are randomly initialized, we propose to initial proxies with semantic information by training a set of extra visual prompts for each image class. Then, the proxies are generated from the ViT with the class-based visual prompts in each class. Furthermore, we propose a novel method to integrate semantic embeddings in the same class progressively with fresh input samples by feeding them into recurrent neural network efficiently. We show that our technique outperforms the original proxy-basee loss regarding learning efficiency and metric learning performance. We further show that by tuning small proportion of parameters, our framework performs comparably to the ofae-of-thhe-a full fine-tuning techniques with a substantially reduced batch size and computing usage. Our main contributions this paper can be summarized follows:
1. We investigate and compare several parameter-efficient fine-tuning strategies on pre-trained ViTs for DML tasks using conventional proxy-based DML losses.
2. We propose a simple but effective approach that uses VPT to generate and integrate semantic proxies from image data to improve their representational capacity and DML performance.
3. Extensive experiments popular DML benchmarks show that our proposed parameter- efficient framework outperforms previous fully fine-tum seeetunnng state-of-the-ee while learning a small percentage of parameters (e.g.. 5.2% of 30M).
2 RELATED WORKS
Parameter-Efficient Fine- Fine-Tuning. Firstly introduced by Vaswani et al. (017)). Transformers have been widely applied in and CV areas. Recently, researchers have begun to develop parameter- efficient methods enhance and accelerate the training process of Transformers. |Houlsby al.) (0119) first propose fine-tuning a group of extra-lightweight models as adapters in addition to the backbones. Pfeiffer et al. (2000) designed the adapters with the extra structure that combines down and up projection networks. Jia et al. (2022) demonstrate that the prompts also perform well on ViT for vision tasks. Gao et al. (0023) further extends the prompt tuning in the video recognition area. Khattak et al. (2022) utilize the prompt tuning between the image and language domain, and Chen et al. (2023a) further propose the optimal transport solution to match the prompts between the two domains. Unlike the above works, we propose applying prompt tuning adapt the pre- pre-trained model specifically for DML tasks and generate proxies containing semantic information to enhance its tuning performance further.
2

---

Published as a conference paper at ICLR 2024
Pony
Seauntic Praay
Data Sample
Data Samule
Head
4 Head
Head
Pourppte
Clas Praggts
ViT
+ VIT
VIT
(a) Proxy-based DML framework
(b) Our proposed framework
Figure I: Overview of conventional proxy-based DML framework and our proposed framework in the training stage. (a) In typical proxy-based DMI. framework with the proxies that are randomly initialized, the ViT encoder fully fine-tuned. (b) In our proposed framework, we only fine-tune the linear head and additional learnable visual prompts. We also propose to generate proxies that contain semantic information by tuning parameters, while the snow label represents the fixed parameters.
prompts. The fire label represents the learnable
Deep Metric Learning. Deep Metric Learning seeks to develop representations and metrics for determining the separation between two data samples. The contrastive loss is applied to samples from various classes (Chopra et al. 20055 Hadsell et al. (2006). Another group of methods chooses an additional sample as an anchor to compare with both positive and negative samples using the triplet Cheng al. 2016 Hermans al., 2017). More recently, ProxyNCA 2017) utilized proxies, a collection of learnable representations, to represent
loss (Weinberger & Saul
2009
(Mosshovitz-Attias et al.
the data classes, [Teh al. (2020) further enhances the ProxyNCA by scaling the proxy gradient. Later, Proxy-Anchor (PA) (Kim et al., (2220) improves the proxy methods by setting the proxies as anchors rather than samples to learn the inter-class structure. In this paper. we adopt the PA as the backbone loss function of our framework for its high performance and reliability. Recent studies also investigated the application of domain adaptation image or textual retrieval tasks (Laradji & Babanezhad] 2020) Ren et al.) 2019 2021) Wang et al. (2017a) employ domain adaptation to align image and textural data using a single discriminator, whereas Ren et al. (2024) utilize the adversarial learning to align the distributions between data and proxies.
Deep Metrie Learning with Transformer: People have recently begun to employ pre-trained ViT to improve DML performance because of their greater data representation capacity and performance CV tasks. El-Nouby et al. (202I) initially proposed ViT on image retrieval tasks with conventional constructive loss. Enmolov al. (2022) propose comparing the data embeddings from ViT in hyperbolic space, whereas Patel et al. (2022) propose to utilize exceptionally large batch sizes to enhance the DML performance. More recently, Kim al. (2023) proposes to build hierarchical structure to enhance the DML performance on hyperbolic space. Wang et al. (2023) factorizes the training signal for different components in the backbone network, and Kotovenko constructs conditional hierarchical structure with cross attention in the training process. Considering the prompts. Chen al.] (2022b) apply prompts to NLP Transformers in DML paradigms, but they apply fixed prompt terms to enhance full fine-tuning on an NLP task. In contrast, we propose optimizing soft visual prompts for ViT' based on proxy-based DMI. paradigms image retrieval tasks, distinguishing us from the previous approaches.
(2023)
PRELIMINARY
3.1 VISION TRANSFORMERS FINE-TUNING
Dosovitskiy et al. (2000) suggested Vision Transformers (ViT) extension of the original NLP Transformers Vaswani et al. 2017). A standard ViT comprises an embedding layer and numerous successively connected blocks. The embedding layer divided input image sample I R #xxx3 into N flattened 2D patches æ of the size & x k. The patches are then transmitted into the attention blocks along with additional position embeddings. Like NLP Transformers, the embeddings combine
3

---

Published as a conference paper at ICLR 2024
with extra learnable [CLS] token Eets as the embeddings for downstream tasks. They are then processed within the transformer blocks, built with a multi-head self-attention (MHSA) architecture. Those embeddings are linearly projected into three vectors in MHSA: query Q, key K, and value V. va V) paired with an extra LayerNorm and an MLP layer is then used to construct the representation for the following attention layer. After transforming within several transformer blocks, the information from all other patches passed to linear head before being sent to the loss of downstream tasks. The size of typical transformer is determined by the number of block layers L and the dimension D of the hidden embedding within the Transformer.
The self-attention softmax( QK*
Adapter Fine Tuning (Rebuffi et al. 2018) is one major group of existing works that fine-tune group of MLP models besides the Transformer. A typical vision transformer adapter (Chen et al.) 2022ab consists of a series of linear layers that project the embeddings of dimension D into low- dimension space of dimension where d < D known as the bottleneck, which is linked to another linear layer that projects up to the original dimension after passing no-linear unit. An extra residual link connects the starting and finishing locations of this MLP configuration.
2021) is one of the simplest solutions adapt the pre-trained model to a downstream task. Except for the Bias factor of each linear projector, every other parameter is fixed. Since it is simple plug-and-play solution integrate with other strategies, we incorporate Bitfit into our framework to enhance its performance.
Bitfit Fine Tuning (aaen
Visual Prompt Tuning (Jia al. 2022) relies on additional learnable prompt vectors as the extra data input. Inspired by the prompts in the language model, visual prompts combine the original visual embeddings with additional n learnable prompts T [tt,..., tx] E R** D For the embedding of the image patches x in the kth layer, the output of this layer and the next layer can be described as follows:
[elts 21 k+1
Il BLK ([Fetss .i....,, 2b
(I)
where th th are prompts in the kth layer, BLK represents the transformer block described above, and the represents the empty position which is left for the prompts in the next layer. In the last layer, the class token Fets is taken as the output and passes through task-specific head combined with a LayerNorm and linear projector. In this paper, we also adopt the output of head from the classifier token Zeta as the representation of our deep metric learning task.
3.2 PROXY-BASED DEEP METRIC LEARNING
Deep Metric Learning (DML.) consider a set of data samples X = (m and their corresponding
class labels Y {3 J E {L...., .....,,H, C); our goal is to learn a projection function fo : X 4 X. which projects the input data samples a hidden embedding space X. The primal goal of DML is to refine the projection function (-). which is usually constructed with CNN or Transformers as the backbone, to generate the projected features that can be easily measured with defined distance metric d(z;, , based on the semantic similarity between image sample I, and I5. We adopt the conventional distance metric d(()) as the cosine similarity. Before passing features to any loss, we use L2 normalization to eliminate the effect of differing magnitudes.
To further boost the learning efficiency, state-of-the-art methods set up set of learnable repre- 1==1, named proxies, to represent subsets or categories of data samples. Typically, there is one proxy for each class, so the number of proxies is the same as the number of classes C. The proxies are also optimized with other network parameters. The first proxy-based method, Proxy-NCA (Movshovitz-Attue Attiasss et al. (017). or its improved version al. 2020), utilizes the Neighborhood Component Analysis (NCA) (Goldberger al. (004) loss to conduct this optimization. The later loss Proxy-Anchor (PA) (Kim et al.) 2020) alternatively sets the proxy as the anchor and measures all proxies for each minibatch of samples. We adopt the PA as our backbone loss since it shows robustness on various extended works (Venkataramanan et al., 2022 Roth al. 2022; Zhang et al. [2022] Yao al. 2022) Generally, for data sample set X and proxy set P the PA loss Lprozy X, P) can be presented as,
sentations P {p.- E RdiC
(Teh)
Eyroes (X, P)= p++
Σ log P-
1+
Σ c
1 PP|
Σ log PEP
14 Σ
(2)
x
zEXp
4

---

Published as a conference paper at ICLR 2024
where X² denotes the set of positive samples for proxy p: X; is its complement set; is the scale factor; and 5 the margin. Since the PA updates approximations for each mini-batch, the model more effective at learning the structure of samples beyond the mini-batch size.
4 PROPOSED METHODOLOGY
4.1 BASELINE ARCHITECTURE
In this paper, we propose optimizing the virtual prompts (VPT) to fine-tune the pre-trained ViT models for DMI tasks. For our backbone model, fix every parameter from the model except for the head projector. We send the prompts t from the final block layer of the ViT to the head layer in dimension D, which is typically equivalent to
Proxy
EMA/GRU
Pf
Qf
Sample
Semantic
Learnable praxy bios
0f
Embedding
the dimension of the hidden layer in the ViT. For DML tasks, we employ the Proxy-Anchor (PA) loss as our backbone loss, where the data samples compared with the proxy anchor for each class as opposed to each other sample. Our framework and the baseline framework for DML tasks can be revealed in Figure
VIT
Class Prompps
Image Sample
Prompts
X
Figure 2: Illustrate the architectures of our frame- work. Note that the parameters of the linear head are independent between the tower of the sample encoder and the semantic proxy. The tower of semantic proxy encodes all image samples the same class and accumulates them into single proxy for class e with EMA or GRU.
4.2 THE SEMANTIC PROXIES
One disadvantage of the proxy-basee DML loss that the proxies are randomly initialized and lack semantic information. Thus, the proxies are updated by comparing them to the sample embed- dings that were not generated optimally during the early stages of training. To address this issue, we attempt initialize the proxies from the sample encoder with the same input data. However, this solution introduces severe flaw: the proxies created by the data sample are close to the same as the data samples from the same images. To deal with this problem, we included some extra learnable prompts dependent on sample class labels. In other words, for class c, we append set of extra prompts I E RmxD to the prompts the data encoder each layer. ( the details of our proposed architecture.
4.3 PROXY UPDATE AND INTEGRATION
Due to the variable quality semantic proxies from each image in multiple batches at different training times, it is challenging to efficiently aggregate all of them into single representation as the proxy for a typical class. One straightforward solution is inspired from the exponential moving average (EMA), in which we integrate the semantic proxies by adding each proxy to the representation of its associated class with a specific ratio. Assuming the pe the semantic proxy in class e, for each generated proxy of in iteration
update pe in the following manner.
p.d normal + +(1 (1- A)pE)
(3)
where is hyperparameter between and I, and normal represents the 1.2 normalization. We collect the newly arriving semantic proxy by updating with larger ratio to depress the information from the previous iteration (i - 1) that needed to be better learned. Directly collecting the representations by EMA has several downsides. For example, the uniform decay to the vector may fail to capture the data structure within a short training time, and freshly produced proxies may contain noise irrelevant to the metric learning tasks.
To address these issues, we propose a novel and effective way to collect proxies in a recurrent manner, in which the semantic proxy is produced be updated based its current state and correlation
5

---

Published as a conference paper at ICLR 2024
the proxy vector. Inspired by the gated recurvent unit (GRU) update the pe iteration as follows:
Chung et al. 2014), propose to
2, oWW.p =14 + U2P;-1 bz)
(4)
oWW, + U
(5)
N = Relu +r , U 1
(6)
P. = normal((1 3
(7)
where W,U& E DDD are the learnable parameters to project the imput P. and previous proxy P
to hidden space of dimension D. Rº represents the bias vector, and @ denotes element multiplication. In our GRU approach, 24 E RR' in a range (0, 1) represents the update gate at iteration and ri R represents the reset gate iteration i. The N is the candidate representation generated with reset gate ri. Then the previous proxy P is further updated with the candidate N₁ and gate 24-
The GRU provides a more complex mechanism to update the accumulated proxy P. as a hidden state, which is controlled with learnable parameters based the class proxy p and the previous hidden state 1- The difference between our design and a normal GRU is that we propose applying the ReLU function rather than the Tanh function since we want retain the updating the vector scale. Note that we do not pass the gradient between the iterations or batches, and we feed the samples with the same class each batch in random order to the accumulator.
After accumulating, we also add the original proxies, initialed randomly, as bias with certain ratio construct the output representation. For an output semantic proxy P,, we construct the output representation as follows:
O = normal((1 a)P; + aOf))
(8)
where O represents the original proxies and Q; represents the output representations for class c. pre-defined pperrparrmeter. We further compare the performance of these two accumulation strategies in our experiments.
4.4 DISCUSSION
Amount of Tunable Parameters The amount of tunable parameters plays a crucial role in parameter- efficient research as it directly impacts the complexity of the training process. Based on the observa- tion the previous VPT work (Tia et al., 2022) where the prompts the shallow layer have higher efficiency than the deep layer, we empirically set the number of prompts for each layer as constant or decreasing number counting from the first layer. Specifically, from layer = 0, we set the number of prompts for each layer to N x where is the step that decreases the number of prompts based on the current layer. We also examine the impact of the number tunable parameters in the Appendix.
Memory Consumption Memory usage is another important factor in evaluating parameter-efficitn methods. In our approach, the number of prompts to construct semantic proxies increases linearly with the total number classes in the downstream task. This will be critical issue when the number of classes in this dataset grows. To reduce memory consumption, to organize prompts to conserve the GPU memory. In particular, we build buffeer in GPU memory to store the prompts that will be updated in its current batch. other words, before running an iteration, we temporally shift the prompts of the semantic proxies in corresponding classes that are sampled in the batch from CPU memory the GPU buffer and also update their statistic states to the training optimizer. Table the comparison between ours and other methods in parameters and memory usage.
develop effective technique
5 EXPERIMENTS
5.1 SETTINGS
In this paper, focus the standard vision transformer (VIT) of different scales (VIT-S, VIT-B) (Dosovitskiy et al. 2020). Without specific claim, our backbone models in this section are pre- trained on ImageNet-21k (Ridnik al. 2021). The selection of this pre-trained model was based

---

Published as a conference paper at ICLR 2024
on its training on sufficiently large dataset, which has been widely used in previous studies in the field of DML comparable to ours. Our method can be applied to larger pre-trained models, such as CLIP (Radford et al. 202I), which trained the gigantic Laion dataset. We also report the experiments with CLIP and other pre-trained models in our Appendix.
5.1.1 DATASETS AND METRICS
We utilize the conventional benchmarks
(CUB200) (Wah al., 2011) with 11,788 bird images and 200 classes, and CARS196 Krause et al. 2013) that contains 16, 185 car images and 196 classes. We also evaluate our method on larger Stanford Online Products (SOP) (Oh Song et al.) 2016) benchmark that includes 1 120,053 images with 22,63* product classes, and In-shop Clothes Retrieval (In-Shop) (Liu et al. 2016) dataset with 25,882 images and 7982 classes. We follow the data split consistent with the standard settings of existing DML works Teh et al. [2200] Kim et al. 2020, Ermolov et al. 2022). We also evaluate our methods on the iNaturalist (Van Horn et al., 2018).
larger dataset widely containing species in nature. We adopt the Recall @ K proposed in existing works to evaluate ranking accuracy. In addition, we also evaluate it with Mean Average Precision at R (MAP@) R), which is more informative DML metric proposed by Musgrave et al. (2020). More details of our training progress and sepeerpparameee searching are listed in the Appendix.
5.1.2 IMPLEMENTATION DETAILS
In this paper, we compare the existing parameter-effiiiet approaches and our baseline as follows:
1. Fully Fine-tuning: We fully update all parameters from the backbone, including the linear head with Proxy-Anchor loss.
2. Linear Prob: We only fine-tune the linear head with PA loss; all the rest of the parameters are frozen.
3. Adapter: We implement the Adapter by referring to the code released in the work (Chen et al. 2022a). We fine-tune the parameters belonging to the Adapter and the linear head while freezing all other parameters.
4. Bitfit: We fine-tuned all bias vectors and the linear head of the Transformer and fixed all
other parameters, proposed in Zaken al. 2222I.
5. VPT: This is the baseline setting following Jia al. (2022) where we fine-tune the visual prompts, the linear head, and the learnable original proxies O while all other parameters are fixed.
6. VPTSP-M and VPTSP-G:
The VPT with our proposed Semantic Proxies (VPTSP) is our framework, which we generate semantic proxies by tuning class-based prompts and combining them with proxy bias. -M represents the setting we use EMA to accumulate the proxies, and -G represents the corresponding accumulation with GRU.
Following recent DML studies (Kim et al. 2020 Roth et al., 2022 Venkataramanan et al., 2022, Ermolov et al., (022), we do some conventional data augmentation, which includes random horizontal flip, cropping, and scaling the image to 224 x 224 pixels using bicubic interpolation. We also follow standard ViT configurations where the images divided into 16 16 patches and keep the pre-trained [CLS] vector. We train our models in a machine that contains a single RTX3090 GPU with 24GB memory.
5.1.3 TRAINING DETAILS AND PARAMETER SEARCHING
We propose ViT-S/16 as our backbone for all the above settings to compare with other parameter- efficient methods. We fix the batch size to 64 with samples each class. We search the learning rate to optimize the Transformer encoder in a set of (0.05. 0.001, 0.0005, 0.0001, 0,00005). and the learning rate to optimize the proxies a set of (0.5, 0.1, 0.05, 0.01, 0.005 ). For optimizers, we search within SGD (Robbins & Monro 1951). Adam (Kingma & Ba 2014). and Adam (Loshchilov & Hutter 2017). Specifically, we search the architecture for both the Adapter and VPT, including the mid-dimension, position, and number of applied layers for the Adapter and the number of prompts and applied layers for VPT and our framework. More details of our search range and selected parameters can be referred to in the Appendix.
7

---

Published as a conference paper at ICLR 2024
Compare Archifecture and Performance on CUB200
Method
Parameter(tunable)
Memory
RI
MAP@R
Full Fine-tuning (PA)
30.2M (100%)
6.0G (100%)
85.5
51.1
Linear Prob
0.19M (0.63%)
1.7G (28.3%)
(-0.6)
48.1 (-3.0)
BitFit
0.26M (0.86%)
4.4G (73.3%)
85.7 (+0.2)
51.3 (40.2)
Adapter (I:77.::556)
1.57M (5.2%)
1.9G (31.7%)
(-2.4)
(-4.7)
Adapter (B) (I:1,d:256)
0.46M (1.52%)
2.4G (40%)
85.8 (+0.3)
49.7
VPT (L:12.N:10)
0.23M (0.76%)
2.2G (36.6%)
(-..4)
(+0.)
VPT (B) (L:12.N:10)
0.30M (0.99%)
2.2G (36.6%)
(+0.2)
(+0.)
VPTSP-M (ours)
0.60M (1.9%)
2.5G (41.7%)
(-.4(-
(40.3)
VPTSP-M (B) (ours)
0.67M (4.5%)
2.6G (43.3%)
84.9
51.8 (+0.7)
VPTSP-G (ours)
1,48M (4.9%)
2.5G (41.7%)
86.1 (+0.6)
52.1 (+1.0)
VPTSP-G (B) (ours)
1.56M (5.2%)
2.6G (43.3%)
86.6 (+1.1)
52.7 (+1.6)
Table Compare our framework and other parameter-efficient methods on the CUB200 dataset. For each approach, we look for hyperparameters in the range mentioned above. The Full Fine-tuning (PA) approach serves as our baseline, which builds upon the PA loss (Kim et al. 2020). L: denotes the number of layers to which the Adapter VPT should be applied, d: represents the Adapter's mid-dimension, and N: represents the number of prompts applied in each layer. Furthermore, -M represents approaches that used EMA fusion, -G represents our methods that used GRU fusion, and (B) represents applying the BitFit method. We introduce memory checkpoints in the transformer block layer for both Adapter and VPT to save memory. We apply the memory buffer indicated above for our proposed method.
5.2 QUALITATIVE RESULTS
Comparing with Other Parameter-efffcient Methods. The brief comparison results can be re- vealed in Table M It has been discovered that BitFit has universal advantage for all methods, resulting in 2. pp gain in @@ 1 and a 3.3pp increase in MAP@R R for the Adapter. Additionally, it leads to 0.6pp improvement in R@ and a 0.4pp increase for VPT. We found that the linear probing has slightly weak performance compared with full fine-tuning (-0.6yp on R@) I). Compared to Adapter and VPT, we find that VPT benefits more for DML. tasks and outperforms the full fine-tuning method (when combined with BitFit). This is the primary reason we chose VPT as the foundation for designing the framework for parameter-efffcicnt DML. Compared with our method, we demonstrate that our semantic proxies significantly improve compared with the baseline VPT method with 1. 1.5pp on R@1 and 0.8pp on MAP@R. Our method has 1.7pp better performance on the R@ metric than linear probing with slightly more parameters. Our approach also has better performance than the Adapter approaches.
Method
Settings
CUB-200
CARS-199 196
SOP
InShop
Name (Batch Size)
Arch/Dim
R@1
R@2
R@4
R@ 1
R@2
R@4
R@
R@ 10
R@ 100
R@ |
R@ 10
R@
Hyp- VIT (882/900
VIT-S16384
85.6
91.4
94.8
86.5
92.1
95.3
85.9
94.9
98.1
92.5
98.3
98.8
VPT-Base (64/32)
VIT-S16/384
85.1
91.1
94.0
86.3
91.8
95.5
82.1
92.5
97.1
88.4
97.5
98.4
VPTSP-M (64/32)
VIT-S163844
94.6
86.8
92.0
82.8
93.1
97.3
90.8
97.7
98.6
VPTSP G (64/32)
S16/384
86.6
91.7
94.8
87.7
93.3
96.1
84.4
93.6
97.3
91.2
97.6
98.4
RS@K (>200)
VIT-B16/512
89.5
94.2
96.6
88.0
96.1
98.6
VPTSP-G
VIT-B16/512
88.5
92.8
95.1
91.2
95.1
97.3
86.8
95.0
98.0
92.5
98.2
98.9
Table 2: Comparison with the state-of-the full fine-tuning methods, including Hyp- ViT (Ermolov et al. 2022) and RS@K K (Patel et al.) (022) applying ViT on the conventional benchmarks pre-trained on ImageNet-211 The second column shows the architecture of the backbone and the feature dimension selected to compare. We also indicate the batch size for training in the brackets. ViT-S16 and ViT-B 16 represent the small and basic size of ViT with patch size 16 x 16. We shade our proposed method, and the best under the same pretraining and architecture are bold.
Comparing with ttate-of-the-art. We further compare the performance of our method with the state-of-the-art methods based on the ViT that was pre-trained on ImageNet-21 as listed in Table 2 3 For both the CARS 196 and the CUB200 datasets, our method reaches better R@ 1, which has significantly improved over the previous oftte--o-the-arr our results with ViT-B/ B/16 outperform the RS@ K. (Patel et al., (022), the previous state-of-the- work ViT-B. For CUB200, framework outperforms the previous sfate-of-the-art Hyp-ViT
the same ViT-S pre-trained model. Also,
8

---

Published as a conference paper at ICLR 2024
Conventional Evaluation
iNaturalist R@4
Method
Architecture
R@1
R@
R@32
MAP@R
SAP (Brown et al. 2020)
VIT-B
79.1
89.0
94.2
95.8
Recall K (Patel et al. 2022)
ViT-B 16/512
83.9
92.1
95.9
97.2
VPTSP- (ours)
84.5
92.5
96.4
97.5
48.1
Table 3: Experimental results on iNaturalist. We compare with the state-of-the-art method (Patel et al. 2022) on this dataset under the same pre-trained dataset and architecture. We bold the higher results in the same setting and metric.
(Ermolov al. 2022). For larger datasets, SOP, InShop, and iNaturalist, we also have a comparable or better result with the existing state-of-the-ard full fine-tuning works. Notably, specific datasets, such as SOP, are marginally enhanced by our proposed framework. This phenomenon can be attributed to the greater number of classes, which complicates the process of quickly optimizing the parameters of each proxy. Note that most existing DML methods substantially rely on the full fine-tuning of pre-trained model with different sampling or optimization strategies. As a result, their computing costs and runtime latencies are close to our full fine-tuning baseline, which much larger than our proposed method. Compared to existing DML approaches, our proposed method demonstrates reductions in computational cost, batch size, and memory usage while achieving comparable DML performance.
CUB200
CARSI%
R@1
MAP@R
R@1
MAP@R
Full Fine-tuning (PA)
85.5
51.1
86.7
28.7
VPT+Bias (Baseline)
85.7
51.6
86.3
27.6
VPTSP (EMA)
85.4 (-0.3)
51.4 (-0.2)
86.4 (+0.1)
28.3 (+0.7)
VPTSP (EMA) +Bias
85.9 (+0.2)
51.8 (+0.2)
86.8 (+0.5)
28.7 (+1.1)
VPTSP (GRU)
86.1 (+0.4)
52.1 (+0.5)
87.1 (+0.8)
28.6 (+1.0)
VPTSP (GRU) Bias
86.6 (+0.9)
52.7 (+1.1)
87.7 (+1.4)
28.9 (+1.3)
Table 4: Ablation of components of framework CUB200 and CARS196. VPT is the conven- tional virtual prompts approach, and VPTSP is our proposed VPT with the Semantic Proxies method. Bias represents the Bitfit for all approaches, and all settings we show here are already combined with linear probing.
5.3 ABLATION STUDIES
We analyze the contribution of each component of our proposed framework based on the results of CUB200 and CARS 198. As listed in Table our GRU accumulation improves the performance on both R@! I and MAP@R. which conducts the efficiency the recurrent accumulation in the training progress. Also, it is clear that our proposed semantic proxies significantly improve the performance of both datasets. We also analyze the impact of the hyperparameters, including the fusion ratio for EMA, the ratio for fusion with the proxy bias, the number of the prompts, and the layers the prompts should append to in our Appendix.
CONCLUSION
In this paper, we investigate the application of parameter-efficient optimization strategies to various deep metric learning (DML) tasks. The performance of existing parameter-efficien strategies on DML. benchmarks is first evaluated. Then, we enhance the efficacy of metric learning using the most effective technique, visual prompt tuning (VPT). Specifically, we develop quick and efficient framework based the conventionally successful proxy-basee DML paradigm, in which improve the representation capability and metric learning performance by integrating the semantic information from the pre-trained ViT that has been modified with class-based prompts. Our evaluation demonstrates the learning effectiveness our method, and our extensive experiments reveal a superior intent to enhance metric learning and retrieval performance further while maintaining small number of tuning parameters.

---

Published as a conference paper at ICLR 2024
REFERENCES
Andrew Brown, Weidi Xie, Vicky Kalogeiton, and Andrew Zisserman. Smooth-ap: Smoothing the path towards large-scale image retrieval. In ECCV, pp. 677-694. Springer, 2020.
Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties in self-supervised vision transformers. In ICCV, pp. 9650- 9660, 2021.
Guangyi Chen, Weiran Yao. Xiangehen Song, Xinyue Li, Yongming Rao, and Kun Zhang, Plot: Prompt learning with optimal transport for vision-language models. 2023a.
Shoufa Chen, Chongjian Ge, Zhan Tong, Jiangliu Wang. Yibing Song, Jue Wang. and Ping Luo. Adaptformer: Adapting vision transformers for scalable visual recognition. NeurIPS, 2022a.
Xinlei Chen, Saining Xie, and Kaiming He. An empirical study of training superviue vision transformers. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 9640-9649, 2021.
Yanru Chen, Yanan Zheng, and Zhilin Yang. Prompt-based metric learning for few-shot ner. arXiv preprint arXiv:2211.0453553 2022b.
Zhe Chen, Yuchen Duan, Wenhai Wang. Junjun He, Tong Lu, Jifeng Dai, and Yu Qiao. Vision transformer adapter for dense predictions. ICLR, 2023b.
De Cheng, Yihong Gong, Sanping Zhou, Jinjun Wang, and Nanning Zheng. Person re-identification by multi-channel parts-based cnn with improved triplet loss function. In CVPR, pp. 1335-1344, 2016.
Sumit Chopra, Raia Hadsell, and Yann LeCun. Learning a similarity metric discriminatively, with application to face verification. In CVPR, volume 1, pp. 539-546. IEEE, 2005.
Junyoung Chung, Caglar Gulcehre, KyungHyun Cho, and Yoshua Bengio. Empirical evaluation of gated recurrent neural networks on sequence modeling. arXiv preprint arXiv:1412.3555, 2014.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In CVPR, pp. 248-255. Icee, 2009.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010.11929, 2020.
Alaaeldin El-Nouby, Natalia Neverova, Ivan Laptev, and Hervé Jégou. Training vision transformers for image retrieval. arXiv preprint arXhv:2102.05644, 2021.
Aleksandr Ermolov, Leyla Mirvakhabova, Valentin Khrulkov, Nicu Sebe, and Ivan Oseledets. Hyper- bolic vision transformers: Combining improvements in metric learning. In CVPR, pp. 7409-7419, 2022.
Kaifeng Gao, Long Chen, Hanwang Zhang, Jun Xiao, and Qianru Sun. Compositional prompt tuning with motion cues for open- vpecbvocayllary video relation detection. ICLR, 2023.
Yixiao Ge, Haibo Wang, Feng Zhu, Rui Zhao, and Hongsheng Li. Self-supervising fine- ane-grained region similarities for large-scale image localization. In ECCV, pp. 369-386. Springer, 2020.
Jacob Goldberger, Geoffrey E Hinton, Sam Roweis, and Russ Salakhutdinov. Neighbourhood components analysis. NIPS, 17, 2004.
Raia Hadsell, Sumit Chopra, and Yann LeCun. Dimensionality reduction by learning invariant mapping. In CVPR, volume 2, pp. 1735-1742. IEBE, 2006.
Kaiming He. Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In CVPR, pp. 770-778, 2016.
10

---

Published as a conference paper at ICLR 2024
Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. Masked autoencoders are scalable vision learners. CVPR, pp. 16000-16009, 2022.
Alexander Hermans, Lucas Beyer, and Bastian Leibe. In defense of the triplet loss for person re-identification. arXiv preprint arXiv: 10X3:v7770307777,, 2017.
Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin De Laroussilhe, Andrea Gesmundo, Mona Attariyan, and Sylvain Gelly. Parameter-efficien transfer learning for nlp. In International Conference on Machine Learning, pp. 2790-2290200 2799. PMLR, 2019.
Edward Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang. Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models. arXiv preprint arXiv:2 09685, 2021.
Menglin Jia, Luming Tang, Bor-Chun Chen, Claire Cardie, Serge Belongie, Bharath Hariharan, and Ser-Nam Lim. Visual prompt tuning. In ECCV, pp. 709-727. Springer, 2022.
Muhammad Uzair Khattak, Hanoona Rasheed, Muhammad Maaz, Salman Khan, and Fahad Shahbaz Khan. Maple: Multi-modal prompt learning. arXiv preprint urXiv:2210.03117, 2022.
Sungyeon Kim, Dongwon Kim, Minsu Cho, and Suha Kwak. Proxy anchor loss for deep metric learning. In CVPR, pp. 3238-3247, 2020.
Sungyeon Kim, Boseung Jeong. and Suha Kwak. Hier: Metric learning beyond class labels via hierarchical regularization. In CVPR, pp. 19903-19912, 2023.
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint UrXiv:1412.6990 2014.
Dmytro Kotovenko, Pingchuan Ma, Timo Milbich, and Bjöm Ommer. Cross-image-attentioon for conditional embeddings in deep metric learning. In CVPR, pp. 11070-11081, 2023.
Jonathan Krause, Michael Stark, Jia Deng, and Li Fei-Fei. 3d object representations for fine-grained categorization. In ICCV workshop. pp. 554-561, 2013.
Issam H Laradji and Reza Babanezhad. M-adda: Unsupervised domain adaptation with deep metric learning. Domain adaptation for visual understanding, pp. 17-31, 2020.
Jung-Eun Eun Lee, Rong Jin, and Anil K Jain. Rank-based distance metric learning: An application to image retrieval. In CVPR, pp. 1-8. IEEE, 2008.
Brian Lester, Rami Al-Rfou, and Noah Constant. The power of scale for efffmeter-efficient prompt tuning. arXiv preprint arXiv:2 100.000.00.0 08691, 2021.
Ziwei Liu, Ping Luo, Shi Qiu, Xiaogang Wang, and Xiaoou Tang. Deepfashion: Powering robust clothes recognition and retrieval with rich annotations. In CVPR, pp. 1096-1104, 2016.
Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. arXiv preprint
17 2017.
Guoyu Lu, Yan Yan, Li Ren, Jingkuan Song, Nicu Sebe, and Chandra Kambhamettu. Localize me anywhere, anytime: multi-task point-retrieval approach. In ICCV, pp. 2434 2444,2442,44 2015.
Dhruv Mahajan, Ross Girshick, Vignesh Ramanathan, Kaiming He, Manohar Paluri, Yixuan Li, Ashwin Bharambe, and Laurens Van Der Maaten. Exploring the limits of weakly supervised pretraining. In ECCV, pp. 181-196, 2018
Yair Movshovit--Atiaa,, Alexander Toshev, Thomas K Leung, Sergey Ioffe, and Saurabh Singh. No fuss distance metric learning using proxies. In CVPR, pp. 360-368, 2017.
Kevin Musgrave, Serge Belongie, and Ser-Nam Lim. A metric learning reality check. In ECCV 2020 pp. 681-699. Springer, 2020.
Hyun Oh Song, Yu Xiang, Stefanie Jegelka, and Silvio Savarese. Deep metric learning via lifted structured feature embedding. In CVPR, pp. 4004-4012, 2016.
11

---

Published as a conference paper at ICLR 2024
Yash Patel, Giorgos Tolias, and Jifí Matas. Recall@ k surrogate loss with large batches and similarity mixup. In CVPR, pp. 7502-7511, 2022.
Jonas Pfeiffer, Andreas Rücklé, Clifton Poth, Aishwarya Kamath, Ivan Vulić, Sebastian Ruder, Kyunghyun Cho, and Iryna Gurevych. Adapterhub: A framework for adapting transformers. arXiv preprint arXiv:2007.07773 077799 2020.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In ICML, pp. 8748-8763, PMLR, 2021.
Elias Ramzi, Nicolas Thome, Clément Rambour, Nicolas Audebert, and Xavier Bitot. Robust and decomposable average precision for image retrieval. NeurIPS, 34:23569-23588. 2021.
Sylvestre-Alvin Rebuffi, Hakan Bilen, and Andrea Vedaldi. Efficient parametrization of multi- domain deep neural networks. In CVPR, pp. 8119-8127, 2018.
Li Ren, Guo-Jun Qi, and Kien Hua. Improving diversity of image captioning through variational autoencoders and adversarial learning. WACV, pp. 263-272. IEEE, 2019.
Li Ren, Kai Li, LiQiang Wang, and Kien Hua. Beyond the deep metric learning: enhance the cross- modal matching with adversarial discriminative domain regularization. In ICPR, pp. 10165-10172. IEEE, 2021.
Li Ren, Chen Chen, Liqiang Wang, and Kien Hua. Towards improved proxy-based deep metric learning via data-augeetted domain adaptation. In AAAI, 2024.
Tal Ridnik, Emanuel Ben-Baruch, Asaf Noy, and Lihi Zelnik-Manor. Imagenet-21k pretraining for the masses. arXiv preprint 1xXi:2204.10972 2021.
Herbert Robbins and Sutton Monro. A stochastic approximation method. The annals of mathematical statistics, pp. 400-407, 1951.
Karsten Roth, Oriol Vinyals, and Zeynep Akata. Non-isotropy regularization for proxy-based deep metric learning. In CVPR, pp. 7420-7430, 2022.
Hyun Oh Song, Stefanie Jegelka, Vivek Rathod, and Kevin Murphy. Learnable structured clustering framework for deep metric learning. arXiv preprint arXiv: 1612.0111, 2016.
Eu Wern Teh, Terrance DeVries, and Graham W Taylor. Proxynca++: Revisiting and revitalizing proxy neighborhood component analysis. In ECCF, pp. 448-464. Springer, 2020.
Hugo Touvron, Matthieu Cord, Matthijs Douze, Francisco Massa, Alexandre Sablayrolles, and Hervé Jégou. Training data-efficient image transformers distillation through attention. In ICML, pp. 10347-10357. PMLR, 2021.
Grant Van Horn, Oisin Mac Aodha, Yang Song, Yin Cui, Chen Sun, Alex Shepard, Hartwig Adam, Pietro Perona, and Serge Belongie. The inaturalist species classification and detection dataset. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 8769-8778 2018.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. NIPS, 30, 2017.
Shashanka Venkataramanan, Bill Psomas, Yannis Avrithis, Ewa Kijak, Laurent Amsaleg, and Kon- stantinos Karantzalos. It takes two to tango: Mixup for deep metric learning. ICLR, 2022.
Catherine Wah, Steve Branson, Peter Welinder, Pietro Perona, and Serge Belongie. The caltech-uesd birds-200-2011 dataset. 2011.
Bokun Wang. Yang Yang, Xing Xu, Alan Hanjalic, and Heng Tao Shen. Adversarial cross-modal retrieval. In Multimedia, pp. 162- 2017a.
Chengkun Wang, Wenzhao Zheng, Junlong Li, Jie Zhou, and Jiwen Lu. Deep factorized metric learning. In CVPR, pp. 7672-7682, 2023.
12

---

Published as a conference paper at ICLR 2024
Jian Wang, Feng Zhou, Shilei Wen, Xiao Liu, and Yuanqing Lin. Deep metric learning with angular loss. In CVPR. pp. 0993-2601,20 2017b.
Xun Wang, Xintong Han, Weilin Huang, Dengke Dong, and Matthew R Scott. Multi-similarity loss with general pair weighting for deep metric learning. In CVPR, pp. 5022-5030, 2019.
Xun Wang. Haozhi Zhang, Weilin Huang, and Matthew Scott. Cross-batch memory for embedding learning. In CVPR, pp. 6388-6397, 2020.
Kilian Q Weinberger and Lawrence K Saul. Distance metric learning for large margin nearest neighbor classification. Journal of machine learning research, 10(2), 2009.
Nicolai Wojke and Alex Bewley. Deep cosine metric learning for person re-identification. In WACV, pp. 748-756. IEEE, 2018
Jufeng Yang, Dongyu She, Yu-Kun Lai, and Ming- Hsuan Yang. Retrieving and classifying affective images via deep metric learning. In AAAI, volume 32, 2018.
Ming-Hsuan
Taojiannan Yang, Yi Zhu, Yusheng Xie, Aston Zhang, Chen Chen, and Mu Li. Aim: Adapting image models for efficient video action recognition. ICLR, 2023.
Zhibo Yang, Muhammet Bastan, Xinliang Zhu, Douglas Gray, and Dimitris Samaras. Hierarchical proxy-based loss for deep metric learning. In CVPR, pp. 1859-1868, 2022.
Xufeng Yao, Yang Bai, Xinyun Zhang, Yuechen Zhang, Qi Sun, Ran Chen, Ruiyu Li, and Bei Yu. Pel: Proxy-based contrastive learning for domain generalization. In CVPR. pp. 7097-7107, 2022.
Elad Ben Zaken, Shauli Ravfogel, and Yoav Goldberg. Bitfit: Simple parameter-efficient fine-tuning transformer-based masked language-models. arXiv preprint arXiv:2106.10199 2021.
Borui Zhang, Wenzhao Zheng, Jie Zhou, and Jiwen Lu. Attributable visual similarity learning. In CVPR, pp. 1322, 2022.
13

---

Published as a conference paper at ICLR 2024
APPENDIX: LEARNING SEMANTIC PROXIES FROM VISUAL PROMPTS FOR PARAMETER-EFFICENT TUNING IN DEEP METRIC LEARNING
A EXTENSIVE DISCUSSIONS
In this section, we explain and discuss the insight behind our design of semantic proxies for the parameter-efficien DML.
A.1
WHY DO THE SEMANTIC PROXIES BENEFIT THE PROXY-BASED DML?
Proxy-based DML techniques have gained widespread acceptance because of their greater perfor- mance when compared to other current DML losses (Kim et al. 2020; Roth et al. 2022 Venkatara manan et al. 2022; Zhang et al. 2022; Yang et al., 2022). However, its learning process continues to be hindered by basic issue: the huge distribution gap between the initialized proxies and the data samples, in which the proxies are initialized randomly with no prior knowledge of their distribution (Ren al. 2024p. This would result in low convergence efficiency and bias in the learning process. To address this issue, we offer semantic proxies, in which semantic information is embedded into the proxies from the beginning. Even if the class-based prompts are not fully trained at the start, the pre-trained ViT still encodes rich semantic information to the initialized proxies, improving the efficiency of learning data embeddings. For example, the visualized comparison between the semantic and original proxies can be referred to in Figure.
A.2 WHY DO WE EXPLORE THE FINE-TUNING METHOD SPECIFICALLY FOR DML TASKS?
Given that existing works have explored several fine-tuning methods for CV tasks. However, the evaluation of their performance for deep metric learning and image retrieval tasks still needs to be completed. Our proposed method achieves a higher computational efficiency than full fine- tine-tuning methods and improves the overall accuracy. As highlighted in our paper, our technique has achieved state-of-ther -ott-----tteetttt-e results in several benchmarks. In times dominated by large models, our contribution invaluable model users who might need more computational resources but hope to achieve the best performance in their local image retrieval tasks. We remain convinced of the relevance and value of investigating faster and more effective alternative fine tine-tuning methods, especially for specific local image retrieval tasks. While the design of prompts for tuning has been subject of recent research, there remains a gap the exploration of prompt tuning methods tailored for deep metric learning and image retrieval. Our work draws inspiration from prompt tuning methodologies, but our primary emphasis lies in elevating image retrieval performance. We achieve this by innovating semantic proxies that mitigate the shortcomings of cxisting full fine-tuning-be based deep metric learning techniques.
A.3 WHY DO WE COMBINE THE ORIGINAL PROXIES TO THE SEMANTIC PROXIES AS BIAS?
One limitation of our suggested semantic proxies is the restricted range of parameters that can be updated for each class batch. We would only update the parameters of the semantic proxies whose classes sampled into the batch in each iteration, but the proxy- -hased loss compares each sample with proxies from all classes. As a result, the classes that were not sampled will need to be updated or have semantic meaning shift compared to the classes in the batch. When the number of classes is vast, updating all semantic proxies in each cycle might be difficult. To address this issue, we propose combining the original proxies, which can readily be updated outside of the mini-batch, as bias when the semantic proxies cannot be updated in time. In other words, inside the mini-batch, we would update both the semantic proxies and the bias, whereas, outside the mini-batch, we would update the bias. This would balance the trade-off between each batch's 's parameter update frequency and training delay.
A.4 WHAT IF WE DIRECTLY ACCUMULATE THE SAMPLE REPRESENTATIONS AS SEMANTIC PROXIES?
This is one of our early, primitive attempts. is valid when the sample representations are directly accumulated semantic proxies (without fine-tuning using virtual prompts). However, discovered
14

---

Published as a conference paper at ICLR 2024
CUB200
CARST96
Settings
R@1
MAP@R
R@T
MAP@R
Sample
85.7
51.2
86.2
28.9
Encoder
86.1
51.5
86.9
28.4
Fixed Encoder
85.4
50.5
86.4
28.7
VTPSP-G (ours)
86.6
52.7
87.7
28.9
Table 5: Experimental results the settings evaluate the efficiency of our proposed class-based virtual prompts. Note that we do not consider the contribution of proxy bias in this comparison.
Method
Arch
Parameters
R@L
R@2
R@4
MAP@R
Proxy.CAA++ (Teh et al.
2020
R50
26.5M
69.0
79.8
87.3
NIR (Roth et al.] 2022)
R50
26.5M
69.1
79.6
XBM+PA [Wang
2020)
BN
12M
65.8
75.9
84.0
PA (vanilla) (Kim et al.
2020
R50
26.5M
69.7
80.0
87.0
26.5
Hyp (Ermolov al.
2022
ViT-S/16
30.2M
85.6
91.4
94.8
Full fine-tuning (PA)
ViT-S/16
30.2M
85.1
91.1
94.0
51.1
VPTSP (ours)
ViT-S/16
1.56M
86.6
91.7
94.8
52.7
Table 6: In this study, we present a detailed comparison with tfate-of-the-ar works in the field, as outlined in the CUB200. Specifically, the second column lists various backbone architectures: R50 denotes ResNet50, BN refers to InceptionBN, and ViT-S/16 signifies the small Vision Transformer utilizing 16 16 patches. However, it is essential note that this comparison may be entirely equitable due to differences in the backbone architectures and the datasets used for pretraining.
that this method cannot resolve the change distribution between the pre-trained and fine-tuning datasets. The semantic proxies built in this situation can benefit the sample encoder during the early training stage but will mislead the encoder training later. To further demonstrate the contribution of our class-based prompts, we propose extended experiment with the following settings:
1. Sample: Semantic Proxies accumulated from encoder samples (no class-based prompts).
2. Encoder: Semantic Proxies from encoder samples with same prompts from encoder.
3. Fixed Encoder: Proxies from encoder samples with same prompts from encoder, but do not pass gradient.
The Setting Sample is proposed to evaluate the semantic shift we discussed. In Setting Encoder, we improve the weakness of setting Sample by involving trainable sample encoder. For Setting Fixed Encoder, we evaluate whether the learning signal from the proxy side limits the capability of representations. The result of this comparison can be referred to Table 5] The results demonstrate that generating the semantic proxies with the original ViT encoder does not have enough capability, Generating proxy from the same trainable encoder has higher R @ but has low MAP@R R, which reflects overall retrieval performance. Compared with the fixed encoder, we also conclude that the gradient from the samples and proxies encoder does not have the same direction. Based on this conclusion, our proposed framework, which provides an isolated proxy encoder for each class, is the optimized solution compared with these settings with simply few more parameters.
A.5 COMBINE THE VPT AND ADAPTER
Conducting the settings where the VPT and Adapter are combined is straightforward. Unfortunately, we did not find any benefit in combining these two architectures. Only with minimal parameters, Adapter has close performance with the pure VPT approach. This may be because the Adapter unsuitable for DML tasks where the extra trainable parameters need to provide more learning capability while increasing the training diffieulties.
15

---

Published as a conference paper at ICLR 2024
$1.0
97.5
$7.3
CUR260 CHASIN
80.5
85.0
CUR250
CAR5200
100
153
202
230
255
301
201
Number parameters
Sies
Dimension
(a) Illustrate of the performance when the tunable parameters grows up.
(b) Illustrate the performance when batch size increased.
Illustrate the performance with different hidden dimension of the model.
Cu2203
17.00
09.75
00.30
06:25
30.00
15
Tuumber append prompts
number promets layer
(d) Impact of the
(c) Impact of the layers that ap- pend prompts
(f)
Impact of the number of prompts in each layer
Figure 3: We compare the performance with different tunable parameters in Figure (3ap. Our VPTSP has optimal range in which the amount and layers of prompts appropriate for fine-tuning the model. When the number of prompts increases, the learnable capacity is reached, and the extra prompts will be tuned well. Figure (3b) shows that our proxy-based DML technique has an appropriate batch size range. When the batch size increases too large, the overall performance suffers. In Figure (3c). we also compare the various dimension values with similar features the batch size. We also illustrate the impact of a, which is the fusion ratio between the original proxies and our semantic proxies, the number of layers that we apply the prompts as additional, and the number of
prompts in each layer in Figure (3d) (3e) and 31
A.6 IMPACT OF HYPEPARAMETERS
We also analyze the impact of the hyperparameters, including the fusion ratio À for EMA, the ratio for fusion with the proxy bias, the number of prompts, and the layers where the prompts are set up. As illustrated 3 when the number of prompts larger than limit, which varies between datasets, the overall performance starts to drop. The layers with prompts also have similar properties. These observations are consistent with other vision tasks reported previous works (Jia al. (002)).
B EXTENTED EXPERIMENTS
B.1 COMPARE THE METHOD TO INTEGRATE PROXIES
In this section, we compare the performance of different RNN architectures and other networks to integrate the semantic proxies. We searched the architecture of recurrent units in small space and compared the recurrent neural network with cross aotss-attention networks. For the cross-attention approach, we only integrate representations within the mini-batch. critical weakness of attention and cross-attention in our case, where we integrate the representations from different batches and iterations, is that they have to integrate all data within the same iteration. we add too much data in the same iteration for cross attentioon there will be external time complexity and memory. Table lists the evaluation results on CUB200, and we can see that our proposed GRU approach performs better.
16

---

Published as a conference paper at ICLR 2024
Random Initialize
Proxy
Provy
Semantic Proxies
Composed
Semantic
Prosies
Sample Emhedding
Sample Embedding
Sample Embedding
Semantic Initialize
Semantic Initialize
Semantic Initialize
Semantic Inittalize
(a) The conven- tional proxy-based
(b) The semantic proxies can only be update within the mini-batch
(c) Our proposed framework update the semantic proxies that combined with the original proxies bias
DML
Figure 4: Illustrate the space of the original proxy-based DML, our semantic proxy, and our proposed metric space. The triangles represent the generated image embeddings, and the circles represent the proxy. The + represents the positive (in the same class), and the represents the negative (not in the same class). The green circle represents our composed semantic proxies.
Integration Architecture
R@
MAP@R R
LSTM
85.1
49.8
RNN Unit
84.6
49.2
Cross Attention
82.2
47.8
GRU with Tanh
85.7
51.4
GRU with Relu
86.1
52.1
Table 7: Comparing the network architecture to integrate semantic proxies on CUB200.
B.2 EXTENSION ON PRE-TRAINING MODELS
ImageNet-1K pretraining is widely applied fine-tun in CNN-based architecture. Although people started to focus on larger datasets for pre- pre-training after moving to Transformers, there are still a few high-quality pretraining strategies and models in ImageNet-Ik K that show promising results in CV tasks. Here, we also provide a comparison of DML performance between parameter-efficient fine-tun methods on two popular pre-trained models, DeiT (Touvron al. 2021 and DINO (Caron al. 2021). as an additional reference.
MAE He al. 2022) an interesting unsupervised learning method where the representations are directed by guessing the blank patches. Since it is also pre-trained on ImageNet-IK. we expect it to perform better than other ImageNet- 1K pretraining methods.
CLIP Vision Tower (Radford et al. (021) is well-known large-scale model pre-trained on massive dataset containing billions image-text pairs. We test our method on the single vision tower of CLIP to evaluate the performance of our method in large-scale model. The evaluation result can be referred Table
The results are presented in Table We believe that the VPT approach and other parameter efficiency methods are influenced by the differences in distributions between the pre-trained and fine-tuning datasets. In contrast to the large datasets, smaller pretraining datasets, such as ImageNet-1K often show larger distributional variance. Consequently, the full fine-tuning strategies have better performance smaller pretraining datasets. However, the overall performance of the full fine-tuning methods on small pretraining datasets is still limited compared with larger pretraining datasets of their limited general ability.
17

---

Published as a conference paper at ICLR 2024
Architecture
Model
Pre-training Set
CUB200
CARST9
SOP
InShop
Vit-S/16
DINO
ImageNet 1K
82.1
86.9
82.6
90.1
Vit-S/I6
DeiT
ImageNet IK
80.2
84.5
81.2
88.7
Vit-B/16
MAE
ImageNet IK
85.1
83.2
77.3
90.3
Vit-I/114
CLIP (vision)
Laion2b
86.7
97.4
90.1
96.5
Table 8: We list the supplemental results from framework with different architecture and pre- trained models. report the R@ result for each pre-trained model and dataset.
Search Attribute
Search Range
Global Parameters
optimizer
(SGD, Adam, AdamW)
Batch Size
[32, 64)
Number of Sample per Class (n)
[1,
Ir Ir for proxy bias
(0.05, 0.01, 0.005, 0.001, 0.0005, 0.0001)
(0.5, 0.1, 0.05, 0.01, 0.005, 0.001)
Adapter
Number of Layers (L)
[1,2. 11, 12
mid-dimension (d)
{1, ... 512]
Adapter Position
Sequential Parallel
Initialization
uniform kaiming uniform
Adapter Position
Pre Post
VPT
Number of Layers (L)
[1,2, 11, 12
Number of Prompts (N)
(1,5, 10, 30, 50, 70, 90)
Decreasing Step (7)
20, 10)
Our Method
Number of Layers for Class Based Prompts (CLS-LL)
{2, 4, 6, 10, 12)
Number of Class Based Prompts (CLS-N)
3, 10, 40, 80)
Ratio between Semantic Proxy and Bias (A)
(0.1,0.2.....0.9)
Table 9; We list the search space for each method compare with our framework. For each method, we first search a pair of proper lr and Ir for proxy and then the corresponding architectures with grid based on the ir and Ir for proxy.
C ANALYSIS ON LATENCY
To compare the training efficiency between parameter-efffcient methods in DML tasks more intuitively, we compare the overall running time of each iteration under the same machine. For all running times, we average the running time over 100 trials. From the results in Table I0 we can conclude that our proposed framework takes slightly more time than other parameters, but we still have faster training speed than the full fine-tuning process. It is fair trade from the training latency (training complicity) the overall performance.
Methods
Latency (B=32)
Latency (B=64)
Full Fine-tuning (PA)
191ms
345ms
linear
115ms
120ms
BitFit
130ms
210ms
Adapter
120ms
190ms
VPT
140ms
226ms
VPTSP
166ms
260ms
Table 10: List of the running time single forward- forward-backward step on Cars 16 with different batch sizes (average over 100 times). The Full Fine-tuning (PA) approach serves baseline, which builds upon the Proxy-Anchor method (Kim et al. (2000). However, incorporates a ViT backbone that received pre-training the larger ImageNet-21a dataset.
18

---

Published as a conference paper at ICLR 2024
D LIMITATION AND BOARDER IMPACT
Limitation Our baseline and proposed framework clearly do not outperform the most recent state-of- the-art on SOP and InShop. This is because there are many more classes in these two more extensive datasets. will be challenging to properly train the parameters based on each class due to the large number of classes. This widespread flaw with various proxy-based methods, be sure. By attempting to reduce the parameters for proxy and semantic proxies or finding other potential ways to boost their learning efficiency, we will strive to address this challenge our subsequent work. Because the fate-of-theear full fine-tuning works (Ermolov et al. 2022) propose to train their model with distributed settings on multiple GPUs with large memory (e.g.. 8x 100) for long time, we still propose framework with close results but with much smaller batch size, faster learning efficiency, and lower GPU cost which can be efficiently completed on a single GPU.
Boarder Impact We thoroughly compare the most popular parameter-efficient algorithms on the essential deep metric learning tasks as the first effort to investigate the parameter-efin efficiien framework for retrieval tasks. We also suggest existing proxy-based approaches where the initialed proxies lack semantic information. We expect that these contributions will give ideas and inspiration for future efforts that will investigate how to efficiently adapt large-scale models to deep metric learning or image retrieval applications.
framework based the VPT to address the issue with
19


Reviews:
[{"review_id": "knNEXukVvK", "reviewer": "Reviewer_hvFX", "summary": "This paper introduces visual prompt tuning (VPT) to improve proxy-based deep metric learning. As VPT could generate and integrate semantic proxies to improve the representations in deep metric learning, the proxies generated by VPT is considered to be better than random proxy which is generally used in proxy-based deep metric learning. The experiments are conducted several classification benchmark datasets including CUB-200-2011 (CUB200), CARS196, and retrieval benchmark datasets including Stanford Online Products (SOP), In-shop Clothes Retrieval (In-Shop). The proposed method is compared and shown to outperform or perform on par with various parameter-efficient fine-tuning methods such as Adapter Fine Tuning, and Bitfit.", "questions": "What is the computational cost of the proposed method as compared to (1) vanilla proxy-based DML method, (2) other fine-tuning methods such as BitFit and Adapter? Please discuss the comparison on computation cost for model training and inference. \n\nHow does the proposed method compare to the state-of-the-art deep metric learning methods? \n\nHow does the proposed method compared to the vanilla proxy-based deep metric learning in terms of model performance and compute cost?", "limitations": "", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "Although the proposed method is simple, the proposed idea of introducing visual prompt tuning to improve proxy-based deep metric learning is well motivated and valid.", "weaknesses": "The main concern of this paper lies in the lack of evaluation/comparison:\n\n-- The paper claims the proposed method to be parameter-efficient for deep metric learning (DML) tasks, but there is not comparison in term of efficiency as compared to existing DML methods. \n\n-- The proposed method targets to solve deep metric learning (DML) tasks but the compared methods are mostly fine-tuning methods. More DML methods should be considered for comparison. \n\n-- A baseline using default/vanilla proxy-based deep metric learning (without visual prompt learning) is missing to convince the effectiveness of introducing visual prompt tuning.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "IGXEsoEuWT", "reviewer": "Reviewer_HvDC", "summary": "This paper proposes an efficient method for fine-tuning the pre-trained models for the DML image retrieval tasks. The authors propose a method based on visual prompts (VPT) to partially fine-tune the model instead of tuning all parameters. Based on the proxy-based DML methods, they also initial the proxies with input images and visual prompts based on classes.", "questions": "Please see the weaknesses.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 4, "strengths": "1 The contribution is solid. As far as I know, this may be the first work to explore the parameter-efficient (PEFT) methods specifically for deep metric learning.\n\n2 Novelty is fine. Although the VPT fine-tuning is not something new, the local design to improve it specifically for DML is interesting and novel.\n\n3 The writing is clear and easy to follow.\n\n4 Experiments is well organized and convincible. \n\nThe authors evaluate their method widely on popular DML datasets, and the results seem to be strong and solid. They also comprehensively compare other PEFT methods on DML datasets, which might be interesting to the community.", "weaknesses": "I didn't see significant weakness. Based on the limited results on some datasets compared with SOTA, I hope the authors can provide more analysis and possible ideas to improve it.\n\nIt seems the results are good on large pre-training datasets but fair on small pre-training. Could you explain more about this phenomenon? \n\nIt is suggested to move some results in the Appendix, like results on other datasets and pre-trained models, to the main paper. \n\nMinor:\nSome typo or grammar errors:\n\"which includes a random horizontal flip, cropping at random...\"\n\"we found that it linear probing...\"\n\nSome recent works missing: \n\nWang et al. \"Deep Factorized Metric Learning.\" CVPR 2023\nKim et al. \"HIER: Metric Learning Beyond Class Labels via Hierarchical Regularization\" CVPR 2023\nKotovenko et al. \"Cross-Image-Attention for Conditional Embeddings in Deep Metric Learning\", CVPR 2023", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "8nhjXEjkf6", "reviewer": "Reviewer_1Fvy", "summary": "This paper addresses the challenge of adapting pre-trained models for Deep Metric Learning (DML) tasks while preserving prior knowledge. Existing solutions often fine-tune models on standard image datasets, making it difficult to apply them to local data domains. The paper introduces a novel approach, the Visual Prompts (VPT) framework, which augments the conventional proxy-based DML method. VPT optimizes visual prompts for each class by integrating semantic information from input images and Vision Transformers (ViT). The experimental results demonstrate that this approach outperforms existing methods in DML benchmarks, achieving comparable or better performance with minimal parameter fine-tuning. This parameter-efficient technique offers a promising avenue for improving DML without requiring extensive model retraining.", "questions": "See weakness.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The paper introduces a novel approach to Deep Metric Learning (DML) by proposing the Visual Prompts (VPT) framework. This framework addresses the challenge of fine-tuning pre-trained models for DML while incorporating semantic information and optimizing visual prompts. This approach represents an innovative combination of existing ideas, extending the traditional proxy-based DML paradigm with an efficient and effective method.\n\n- The paper is well-written and clearly articulates the motivation, methodology, and experimental results. The authors effectively communicate their approach, making it accessible to a broad audience of readers, including those familiar with DML and those new to the field. The paper's clarity enhances its potential for adoption and understanding by the research community.\n\n- The proposed framework has the potential to streamline and improve DML tasks, as it achieves comparable or better performance compared to full fine-tuning approaches, all while adjusting only a small percentage of total parameters.", "weaknesses": "- The paper lacks a comprehensive comparison with existing DML methods, especially those using transformer based backbone. Additionally, it would be beneficial to compare the proposed Visual Prompts (VPT) framework not only against full fine-tuning but also against other state-of-the-art parameter-efficient methods. This would provide a more complete assessment of its effectiveness and novelty.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)