Search-Adaptor: Embedding Customization for Information Retrieval (2024)

Abstract: Embeddings extracted by pre-trained Large Language Models (LLMs) have significant potential to improve information retrieval and search. Beyond the zero-shot setup in which they are being conventionally used, being able to take advantage of the information from the relevant query-corpus paired data can further boost the LLM capabilities. In this paper, we propose a novel method, SearchAdaptor, for customizing LLMs for information retrieval in an efficient and robust way. Search-Adaptor modifies the embeddings generated by pre-trained LLMs, and can be integrated with any LLM, including those only available via prediction APIs. On multiple English, multilingual, and multimodal retrieval datasets, we show consistent and significant performance benefits for Search-Adaptor – e.g., more than 5% improvements for Google Embedding APIs in nDCG@10 averaged over 14 BEIR datasets.

heading: 1 Introduction
text: Information retrieval is broadly considered as the task of searching for information via querying corpus database that might consist many different types of data, such as documents, webpages or logs. It has a wide range of applications across many industries, including retail, healthcare, and finance, with a significant portion of the world’s economy is built on. Particularly, language modeling is the key part of information retrieval as in most cases, query and corpus data are in text form. Large language models (LLMs) have demonstrated significant achievements for a variety of text processing tasks, including question answering, summarization, and mathematical reasoning (Brown et al., 2020; Chowdhery et al., 2022; Zhang et al., 2022a). One critical enabler for the success on these has been transforming raw text into meaningful representations that preserve semantic meanings
in the representation space (Ouyang et al., 2022). For a wide range of applications, from recommendations to anomaly detection, tasks are defined as explicit operations on the learned representations. This makes the quality of the text mapping into embeddings become of paramount importance. Information retrieval systems commonly utilize the text embeddings, with relevant corpora being ranked based on the similarity between queries and corpus (Wang et al., 2022; Izacard et al., 2021a).
Various LLMs have been proposed to extract embeddings from raw text, with the notable ones including the Sentence T5 (Ni et al., 2021a), OpenAI Embedding APIs (ope) and Google Embedding APIs (gcp). However, one fundamental limitation of pre-trained LLMs is that they cannot utilize retrieval-specific target task data, that are in the form of positively relevant query-corpus pairs. Even with a small amount, using such data for tuning is expected to significantly improve information retrieval capabilities. Conventional fine-tuning (Howard and Ruder, 2018) can be one straightforward way of utilizing the paired query-corpus information. However, if the number of paired samples is small, tuning all the weights of a model might yield overfitting and thus poor generalization (Lin et al., 2023), especially in the presence of distribution shifts. In addition, it can be costly from a computational perspective as it requires large memory. There are multiple parameter-efficient tuning methods such as prompt tuning (Lester et al., 2021; Li and Liang, 2021), LoRA (Hu et al., 2021), partial fine-tuning (Zaken et al., 2021), and various adapters (Houlsby et al., 2019; Rücklé et al., 2020). These approaches only tune a subset of the parameters of LLMs, aiming to reduce the risks of overfitting and bringing computational gains. As a common bottleneck, all of these methods need full access to the LLM’s parameters to tune the model, which may not be possible for LLMs with API-based inference-only access.
12230
In this paper, our focus is customizing LLMs to obtain superior embeddings for information retrieval, particularly from the angle of how to best take advantage of the available retrieval-specific tuning data and obtain robust improvements in wide range of regimes, with a tuning method that is low cost and even applicable to LLMs with APIbased inference-only access. Fig. 1 overviews the proposed approach, Search-Adaptor. Improving the tuning performance in this setup requires a set of innovations. We propose integration of a low capacity adapter module (to be customized for the target dataset) on top of fixed LLMs to modify the pre-trained embeddings. For efficient tuning, we introduce a novel differentiable ranking loss that can directly utilize the information of positive query and corpus pairs. In addition, we include multiple regularizers to improve generalization in this small data regime where without intervention, the pretrained LLMs would end up with catastrophic forgetting. Enabled by such design approach, one major advantage of Search-Adaptor is that it does not require access to the parameters of the pre-trained LLMs – only the inference outputs of the LLMs are needed. Commercial embedding APIs that show state-of-the-art performance usually do not provide access to their model parameters. In such cases,
Search-Adaptor can still be used to further improve those API-based embedding models, in contrast to alternative tuning methods. We demonstrate the effectiveness of Search-Adaptor across 14 BEIR datasets (Thakur et al., 2021) and 17 MIRACL multilingual datasets (Zhang et al., 2022b) with Google and OpenAI embedding APIs, applying the Search-Adaptor on top. In addition, we evaluate Search-Adaptor’s performance improvements with open-source Sentence T5 models (Ni et al., 2021a). Overall, Search-Adaptor provides significant improvements over alternatives, consistently across different datasets and models. The contributions of this paper are:
• We propose a novel adaptation framework for information retrieval applications that can significantly improve the pre-trained LLMs.
• We introduce a novel ranking loss and multiple regularizers that reduce overfitting and forgetting and thereby improve the retrieval performance even in the small data regime.
• We provide consistent and significant improvements for retrieval performance with a range of datasets (from 1K to 500K positive query-corpus training data pairs).
• We show that Search-Adaptor on smaller LLMs can approach the performance of larger LLMs in zero-shot setting, underlining its potential for model distillation.
• We extend the application of Search-Adaptor to multimodal learning and tool use scenarios, demonstrating its significant benefits.

heading: 2 Related Work
text: Pre-trained LLMs for zero-shot retrieval. LLMs to extract general text embeddings are commonly used in both academia and industry. AI solution providers like Google (gcp) and OpenAI have productionized general text embeddings that can be directly used via simple APIs for zero-shot retrieval applications. In addition, many previous efforts have introduced new general text embedding models with various pre-training methods and datasets. GTE (Li et al., 2023) proposes a multi-stage pre-training of embedding models with diverse naturally paired text datasets. E5 (Wang et al., 2022) pre-trains the embedding models by weakly-supervised contrastive learning, utilizing consistency-based filter to generate high
quality text pairs for pre-training. Note that SearchAdaptor can be applicable on top of any pre-trained LLM embedding models to customize their embeddings for superior retrieval performances.
Embedding customization. Instead of using one unified model for zero-shot retrieval, the embeddings can be customized for each dataset or task. Instruction-based embedding customization is one popular method. TART (Asai et al., 2022) builds a retrieval system that adapts the retrieval based on the instruction. Different retrieval tasks (e.g., code, question, or answer) are given as the instruction to further improve dense embedding retrieval. InstructOR (Su et al., 2022) integrates the task and domain descriptions prior to the input to fine-tune the embeddings for retrieval. However, these do not directly utilize the provided relevant query-corpus pairs. Full or parameter-efficient fine-tuning (such as LoRA (Hu et al., 2021) and (IA)3 (Liu et al., 2022)) can also be considered for embedding customization. Pre-trained LLMs can be fine-tuned with contrastive loss using positive query-corpus paired data. Promptagator (Dai et al., 2022) utilizes in-context learning to generate synthetic query-corpus pairs using a few number of original query-corpus pairs, and subsequently using those synthetic pairs to fine-tune the pre-trained LLMs. However, all these are only applicable when there is full access to the parameters of pre-trained LLMs, which is often not possible for state-of-theart commercial text embedding models. On the other hand, Search-Adaptor can be applied without full access to the LLM parameters.

heading: 3 Problem Formulation
text: We formulate the retrieval problem with a given query-corpus paired dataset. Assume a query set denoted as Q = {q1, ..., qN} ∈ Q and a corpus set denoted as C = {c1, ..., cM} ∈ C. Each positive relationship between a query and corpus is represented as the triplet rij = (qi, cj , yij) with yij > 0 as the strength of the relationship between qi and cj . We treat all other triplets as negative relationships (yij = 0). The set of all query-corpus relationships is denoted as R = {(qi, cj , yij)}i=1:N,j=1:M = Rp∪Rn, whereRp = {(qi, cj , yij) ∈ R|yij > 0} is the set of positive relationships and Rn = {(qi, cj , yij) ∈ R|yij = 0} is the set of negative relationships. Note that yij can be either binary or continuous.
The retrieval system aims to find the relationship
between the given query (qi) and corpus (cj) such that the predicted relationship is highly correlated with the ground truth relationship (yij). The scoring function f : Q×C → R takes queries and corpus data as inputs and outputs a score estimate on the relationship between them. The optimal score is the one that has the same order as the ground truth relationship for each query.

heading: 4 Methods
text: Fig. 2 overviews the components of SearchAdaptor, that are described in following sections.

heading: 4.1 Adapting pre-trained LLMs
text: Major real-world constraints for tuning the LLM embedding models shape our methodological design. The tuning operation for high-capacity models can be very costly, and one often does not have access to the parameters and the gradients of the pre-trained models (e.g. LLMs with API-based inference-only access). This motivates the need for an adaptation method that can operate with
fixed pre-trained embedding models with an efficient adaptation module, to extend to the LLMs with API-based inference-only access. In SearchAdaptor, we propose modifying the embeddings extracted from pre-trained LLMs for superior search and information retrieval.
Consider the query and corpus embeddings extracted using the pre-trained embedding model E: QE = {qe1, ..., qeN} ∈ Rd and CE = {ce1, ..., ceN} ∈ Rd where qei = E(qi) and cej = E(cj). Note that both query and corpus embeddings are in the same embedding space.
The objective of Search-Adaptor is to modify embeddings extracted from pre-trained LLMs in a way that maximizes retrieval performance. A learnable adaptation function is defined as f : Rd → Rd, which maps the original embedding to a new embedding that is more favorable for retrieval applications. The modified embeddings are denoted as Q̂E = {q̂e1, ..., q̂eN} ∈ Rd and ĈE = {ĉe1, ..., ĉeM} ∈ Rd where q̂ei = f(qei) and ĉej = f(cej). The relevance scores between modified query and corpus embeddings are defined as follows:
ŝij = Cosine-Similarity(q̂ei, ĉej) = q̂ei · ĉej ||q̂ei||||ĉej || .
Search-Adaptor consists of the following components (see Fig. 2 for details):
• Adaptation function f . This function is used to modify the query and corpus embeddings. We add a skip connection to f so that it can only learn the residual between the original and adapted embeddings as follows: q̂ei = qei + f(qei) and ĉej = cei + f(cei). Note that we use the shared adapter for both query and corpus (see Sec. 6 for ablation studies). The ranking loss, reconstruction loss, and prediction loss are used to train f .
• Query predictor p. This function is used to predict the query embedding using the adapted corpus embedding. The prediction loss is used to train p.
At inference, we only use the adaptation functions (f ) to modify the query and corpus embeddings. We then compute the cosine similarity between the modified query and corpus embeddings to estimate the relevance between query and corpus. Query predictor is not used at inference.

heading: 4.2 Ranking objective
text: As explained in Sec. 3, the objective of the retrieval is to predict the correct order of the relevance between queries and corpus. Therefore, the most critical part is to properly design the ranking loss. We propose a ranking loss as follows:
LRank = N∑
i=1
M∑
j=1
M∑
k=1
I(yij > yik) · (yij − yik)
· log(1 + e(sik−sij)),
where I(yij > yik) is an indicator function that is equal to 1 if yij > yik and 0 otherwise. sij = Cosine-Similarity(E(qi), E(cj)) is the relevance score between query text (qi) and corpus text (cj).
The ranking loss penalizes the model more when it predicts a lower score for a pair of query and corpus that has a higher ground truth relevance (i.e., sij < sik even though yij > yik). The amount of penalization is proportional to the difference in ground truth relevance (yij − yik) and the difference in estimated scores log(1 + e(sik−sij)). Note that log(1 + e(sik−sij)) can be replaced with any monotonic function such as linear function. In general, the ranking loss encourages the model to predict higher scores for pairs of query and corpus that have a higher ground truth relevance. Table 4 shows a comparison of this ranking loss to alternatives and demonstrates its effectiveness.

heading: 4.3 Regularization
text: Introducing proper inductive biases via regularization is important to improve adaptation from pre-trained LLM embeddings without forgetting too much information from the pre-trained LLMs. Towards this end, we propose two regularization methods:
Recovery. To increase generalizability, we postulate avoiding modification of the adapted embedding too far away from the original embedding. As such, we propose minimization of the difference between the original and adapted embeddings using a recovery regularizer:
LRec = 1
N
N∑
i=1
||q̂ei−qei||1+ 1
M
M∑
j=1
||ĉei−cei||1
where q̂ei is the adapted query embedding and qei is the original query embedding. Similarly, ĉei is the adapted corpus embedding and cei is the original corpus embedding. The recovery regularizer
encourages the adapted embeddings to be not too far from the original embeddings.
Prediction. Intuitively, if the query and corpus are highly relevant, we can use the corpus to predict the query. Building upon this intuition, we propose a regularizer in the form of prediction loss between the query and corpus, calculated as follows:
LPred = ∑N i=1 ∑M j=1 yij · ||q̂ei − p(ĉej)||1∑N
i=1 ∑M j=1 yij
where p : Rd → Rd is a function that predicts the query given the corpus, and yij is a weight that is assigned to the loss if the query and corpus are correlated. The prediction loss encourages the model to predict the query well given the corpus, especially if the query and corpus are correlated. Note that we do not include the prediction function from query to corpus because usually corpora are significantly longer than queries which would render the task challenging.

heading: 4.4 Training
text: Using the proposed ranking loss, recovery loss, and prediction loss, we optimize the adaptation function f and prediction function p by minimizing the following loss function:
min f,p LRank(f) + αLRec(f) + βLPred(f, p),
where α ≥ 0 and β ≥ 0 are the hyper-parameters that control the relative importance of the different loss terms.1 Table 4 shows the results of ablation studies on the effectiveness of the different loss terms. All hyper-parameters are provided in Appendix C.
Note that the ranking loss compares all possible pairs between queries and corpus which needs NM2 times computations per one epoch (M >> N ). For efficient computation, we randomly subsample the corpus for each query batch. While doing so, we always include the corpus which has positive relevance to queries in that batch.

heading: 5 Experiments
text: We evaluate the performance of Search-Adaptor in multiple scenarios on numerous datasets. We demonstrate that Search-Adaptor is modelagnostic, applying it both on top of API-based
1In the experiments, we tune these hyper-parameters based on validation set (α ∈ {0.0, 0.1, 1.0} and β ∈ {0.0, 0.01, 0.1}).
LLMs (merely via access to Google & OpenAI APIs) and open-sourced LLMs (e.g., Sentence T5 (Ni et al., 2021a)). We also demonstrate that it is data-agnostic by evaluating Search-Adaptor on English, multilingual and multimodal datasets.

heading: 5.1 Experimental settings
text: We first consider the 14 retrieval datasets from the BEIR repository (bei) to evaluate the performance in English data, with corpus sizes ranging from 3.6K to 8.8M, and training pairs ranging from 0.7K to 532K. For the datasets with only test data (e.g., Arguana, SciDocs), we split the data into disjoint train and test sets with a 50/50 ratio, based on the sorted query IDs. We also use MIRACL data (mir) which consists of 17 multilingual datasets including Japanese, Chinese, French, and Indonesian. More dataset details can be found in Appendix A.
We use nDCG@10 as the main metric to quantify the retrieval performance (see Appendix B for more details). For model selection, we divide the training data into disjoint training and validation splits with an 80/20 ratio, and select the model with the highest validation nDCG@10 value.
We consider both API-based and open-sourced LLMs. As the API-based LLM, we use OpenAI embedding API (ope) and Google embedding API (gcp). As the open-sourced LLM, we use Sentence T5 models2 of two different sizes, GTE-Large (Li et al., 2023), GTR-Large (Ni et al., 2021b) and Condenser-Retriever (Gao and Callan, 2021).

heading: 5.2 Adapting with API-based LLMs
text: One of the biggest advantages of Search-Adaptor is that it can be applied on top of any API-based LLM – without having access to the parameters of LLMs, Search-Adaptor can further improve the retrieval performance. This is particularly important as the state-of-the-art LLMs are actually API-based models (owned by companies).
As can be seen in Fig. 3 and Table 1, we demonstrate the retrieval performance improvements on top of API-based text embedding models across 14 datasets from the BEIR repository. On average, Search-Adaptor achieves 0.0475 and 0.0349 nDCG@10 improvements for both Google and OpenAI text embedding APIs. The improvements of some datasets are quite significant indeed – e.g., 0.1739 with Arguana, 0.0856 with Scifact datasets.
2https://tfhub.dev/google/sentence-t5/ st5-base/1
We also compare with OpenAI’s embedding customization solutions (OpenAI’s solution)3. Table 1
3https://github.com/openai/openai-cookbook/ blob/main/examples/Customizing_embeddings.ipynb
shows worse performance compared to not only Search-Adaptor but also Zero-shot OpenAI embedding in terms of ranking metrics (nDCG@10). OpenAI’s solution tries to solve classification problems between positive and negatively correlated text pairs. It adds “random negatives” to their positive paired samples and try to distinguish between positive and “random negative” pairs using MSE loss. This problem is much easier than the retrieval problem (that Search-Adaptor tries to solve), where the task is to identify the positive pairs from all possible negative pairs (including hard negatives). Also, Search-Adaptor utilizes ranking loss while OpenAI’s solution utilizes MSE loss (which is beneficial for classification or regression problems). Lastly, OpenAI’s solution does not have regularizations that mitigate the high risks of overfitting when the number of query-corpus pairs is small.

heading: 5.2.1 Search-Adaptor on multilingual data
text: Search-Adaptor is also applicable on non-English multilingual data. In Fig. 4, Search-Adaptor shows consistent performance improvements on top of Google Embedding API across 17 different languages (on average 0.0396 nDCG@10 improvement). For some languages, it is particularly significant, e.g. the improvement is 0.0687 for Thai. These highlight Search-Adaptor being a model-agnostic and data-agnostic approach. More experiments with additional embedding models (e.g., GTR-Large (Ni et al., 2021b) and CondenserRetriever (Gao and Callan, 2021)) can be found in Appendix D. Qualitative analyses can be also found in Appendix E.

heading: 5.3 Adapting with open-sourced LLMs
text: Beyond API-based LLMs, Search-Adaptor can be applied to open-sourced LLMs. Here, we consider Sentence T5-Base (Ni et al., 2021a) and GTELarge (Li et al., 2023) models as the open-sourced LLMs to demonstrate the performance improvements over the baselines.
As shown in Table 2, Search-Adaptor shows consistent improvements over zero-shot ST5-Base model. For the open-sourced LLMs, we can also utilize full fine-tuning (with contrastive loss) and LoRA (Hu et al., 2021) as alternatives of SearchAdaptor, albeit the higher training cost. The experimental results on the considered benchmarks show that on average, full fine-tuning and LoRA performances can indeed be worse than SearchAdaptor. Surprisingly, the performance of full fine-tuning and LoRA can even be much worse than the zero-shot baseline (e.g., for Trec-Covid, Touche with full fine-tuning and NFCorpus, SciFact with LoRA) which is attributed to fine-tuning being prone to overfitting and poor generalization
(Lin et al., 2023). With a limited number of query-corpus pairs, Search-Adaptor performs better than fine-tuning methods due to lower risk of overfitting. Also, training cost (both memory and computations) is much lower with Search-Adaptor than fine-tuning methods. On the other hand, with enough query-corpus pairs, fine-tuning methods may perform better than Search-Adaptor with open-source retrievers. Based on these pros and cons, the appropriate customization method can be selected given the specific application scenario. If black-box retrievers work better than open-source retrievers or when the number of query-corpus pairs is small, then Search-Adaptor would be a superior choice of customization. On the other hand, if open-source retrievers work better than black-box retrievers and the number of querycorpus pairs is large, we can utilize fine-tuning methods to customize their retrievers.

heading: 5.4 Search-Adaptor with multimodal data
text: Search-Adaptor makes consistent and significant improvements when applied on text embeddings. We also extend Search-Adaptor from textonly to multimodal data, with image and text, using
Google Cloud’s multimodal embedding API 4. We use the Fashion-200K dataset (Han et al., 2017) for the text to image retrieval task to show how much Search-Adaptor can make further improvements on top of base multimodal embedding API.
Table 3 shows that Search-Adaptor can achieve 20-30% of performance improvement in nDCG@10 across 5 sub datasets of Fashion-200K datasets. Relevant qualitative analyses can be found in Appendix F.

heading: 5.5 Search-Adaptor for tool retrieval
text: In this subsection, we further extend SearchAdaptor to tool retrieval application (Patil et al., 2023) where agents choose which actions to perform to automate execution of a task given the input query. The objective of tool retrieval is to retrieve the proper tools for the new query based on the descriptions of tools. On two datasets, ToolE (Huang et al., 2023), ToolBench (Qin et al., 2023), we study the potential of Search-Adaptor to improve the tool
4https://cloud.google.com/vertex-ai/ generative-ai/docs/embeddings/ get-multimodal-embeddings
retrieval performance, that would yield superior agents. Table 5 shows that with Search-Adaptor, significant retrieval performance improvements, 15- 50%, are obtained.

heading: 6 Discussions

heading: 6.1 Ablation studies
text: Search-Adaptor proposes multiple innovations to improve the adaptation performance. We quantify the contributions of proposed constituents to the retrieval performance on various datasets with as ST5-Base as the base embedding model, with the results in Table 4. We consider various modifications to Search-Adaptor: (i) altering the architecture, (ii) applying different regularizations, and (iii) applying different losses. Using different losses yields the largest performance degradation, underlining the importance of the proposed ranking loss. Aside from the losses, if we use separate adapters for query and corpus, it also yields a noticeable performance drop. This shows the importance of ‘shared embedding space’ between the query and corpus for retrieval. The skip connection and the two regularization functions bring additional performance gains but their impact is lower than the ranking losses. Table 4(c) shows the impact of the proposed ranking loss in comparison to alternatives: (i) point-wise sigmoid cross entropy, (ii) contrastive loss (Izacard et al., 2021b), (iii) softmax cross entropy (Bruch et al., 2019) and (iv) RankNet loss (Burges et al., 2005). With the proposed ranking loss of the original Search-Adaptor, significant outperformance is observed, compared
to the alternative ranking losses.

heading: 6.2 Small LLMs with Search-Adaptor outperform zero-shot large LLMs
text: One bottleneck for real-world LLM deployment can be the prediction latency and cost, that are highly dependent on the LLM model size. We demonstrate that Search-Adaptor can achieve better or comparable retrieval performance even with much smaller LLMs, compared to to zero-shot retrieval with larger LLMs.
As shown in Table 6, Search-Adaptor with ST5Base model (110M parameters) performs much better than ST5-Large (335M parameters). SearchAdaptor can achieve better results with much smaller encoders, positioning it as an effective distillation mechanism to significantly decrease the serving cost and latency of retrieval systems. It also reiterates the benefits of Search-Adaptor being model agnostic.

heading: 7 Conclusions
text: In this paper, we focus on pushing the capabilities of LLMs for information retrieval and search. We propose a canonical efficient adaptation method, Search-Adaptor, that can also be applied to LLMs even with inference-only access. Search-Adaptor is a low-cost tuning method that brings significant and consistent improvements in retrieval performance across diverse regimes of training data size, encoder type, and corpus set. This is enabled by the judicious design of its adaptor module, along with training objectives and approaches. We have also studied the extension of Search-Adaptor to multimodal learning and tool use scenarios, highlighting
the importance of embedding customization for such applications.

heading: 8 Limitations and Future Works
text: Important future directions might include generalizing the propose adaptation method to include partial tuning of the embedding models as a way to increase trainable degrees of freedom; extensions to embedding tasks beyond retrieval; and extensions to multimodal learning with many modalities. There is no specific risk of the proposed method other than the general risks of tuning methods that they can lead to overfitting to certain tasks and they can absorb the biases present in the target tuning data.

heading: A Data Statistics
text: A.1 BEIR datasets
A.2 MIRACL datasets
A.3 Fashion-200K datasets
A.4 Tool retrieval datasets

heading: B Metrics
text: For tasks that involve retrieving information, normalized discounted cumulative gain (nDCG) (Järvelin and Kekäläinen, 2002) is a standard metric for evaluating performance. To define nDCG, we first consider discounted cumulative gain (DCG):
DCG(y, s) = ∑
i
2yi
log2(rank(si) + 1) ,
where s is the relevance score computed by the model and y is the ground truth label. nDCG is then defined as nDCG(y, s) = DCG(y,s)DCG(y,y) , where the denominator assumes the optimal case where the ranking of the scores (s) are exactly the same as the ranking of the ground truth label (y). nDCG@k is a widely used variation of nDCG where only the top k scores are considered. In this paper, we use nDCG@10 as the main retrieval metric.

heading: C Hyper-parameters
text: We summarize the hyper-parameters used to train Search-Adaptor. In all experiments, we utilize the fixed hyper-parameters (except α, β) that enable applying Search-Adaptor without extensive hyper-parameter tuning. We use multi-layer perceptron as the adaptor architecture for both the encoder and the predictor.

heading: D Additional Experiments
text: We include the additional results of Search-Adaptor with GTR-Large5 (Ni et al., 2021b) and CondenserRetriever 6 (Gao and Callan, 2021) as the base embedding models. As can be seen in Table. 12, the results are consistent with the above results that Search-Adaptor shows consistent and significant improvements on top of both GTR-Large and Condenser-Retriever models. For the Condenser-Retriever model, we apply pooling and normalization on the token embeddings to extract the final text embeddings.
5https://huggingface.co/sentence-transformers/gtr-t5-large 6https://huggingface.co/Luyu/co-condenser-marco-retriever

heading: E Qualitative Analysis
text: To understand the impact of Search-Adaptor, we first analyze the cosine similarity between query and corpus, before and after Search-Adaptor training.
As can be seen in Fig. 5, after Search-Adaptor training, the distribution differences between relevant and irrelevant pairs’ cosine similarity are larger which means that we can identify the relevant corpus per each query better.
To further understand the distribution difference of query and corpus embeddings before and after Search-Adaptor training, we plot tSNE graphs of them.
Fig. 6 shows the impact of Search-Adaptor. The left figure shows that the original query and corpus embeddings are quite distinct. Most query embeddings are located in the restricted region. On the other hand, after training with Search-Adaptor, query embedding distribution is observed to better overlap with the corpus embedding distribution, which could result in more robust retrieval.
We further investigate the success and failure cases of Search-Adaptor in comparison to the zero-shot baseline. Bold represents the relevant corpus to the query.
As can be seen in Table. 13 and 14, in failure cases, Search-Adaptor still can retrieve the relevant corpus in the top-3 corpus but the ranking is lower than the baseline. For the success cases, Search-Adaptor can retrieve the correct corpus even though the baseline is completely failed. Quantitatively, with 300 test samples, there are 9 cases where Search-Adaptor can retrieve the correct corpus in top-3 but Baseline cannot retrieve any correct corpus in top-3. But there is no case for the opposite.

heading: F Qualitative Analysis on Multimodal Data
text: Fig. 7 shows 6 examples in Fashion-200K datasets where Search-Adaptor makes better retrieved output than the zero-shot baseline (Google Cloud’s multimodal embedding API). First column represents the given text query. Second column represents the ground truth relevant image for the given text query. Third column shows the top-2 retrieved outputs based on the zero-shot baseline. Last column shows the top-2 retrieved outputs based on Search-Adaptor. Note that the ground truths are included in top-2 retrieved outputs by Search-Adaptor but not included in the baseline.

heading: Future Work
text: Important future directions might include generalizing the propose adaptation method to include partial tuning of the embedding models as a way to increase trainable degrees of freedom; extensions to embedding tasks beyond retrieval; and extensions to multimodal learning with many modalities. There is no specific risk of the proposed method other than the general risks of tuning methods that they can lead to overfitting to certain tasks and they can absorb the biases present in the target tuning data