Venue: NAACL 2025

# Search Query Embeddings

via User-behavior-driven Contrastive Learning

Sosuke Nishikawa  Jun Hirako Nobuhiro Kaji Koki Watanabe Hiroki Asano Souta Yamashiro Shumpei Sano LY Corporation

Equal contribution.

###### Abstract

Universal query embeddings that accurately capture the semantic meaning of search queries are crucial for supporting a range of query understanding (QU) tasks within enterprises. However, current embedding approaches often struggle to effectively represent queries due to the shortness of search queries and their tendency for surface-level variations. We propose a user-behavior-driven contrastive learning approach which directly aligns embeddings according to user intent. This approach uses intent-aligned query pairs as positive examples, derived from two types of real-world user interactions: (1) clickthrough data, in which queries leading to clicks on the same URLs are assumed to share the same intent, and (2) session data, in which queries within the same user session are considered to share intent. By incorporating these query pairs into a robust contrastive learning framework, we can construct query embedding models that align with user intent while minimizing reliance on surface-level lexical similarities. Evaluations on real-word QU tasks demonstrated that these models substantially outperformed state-of-the-art text embedding models such as mE5 and SimCSE. Our models have been deployed in our search engine to support QU technologies.

## 1 Introduction

Query understanding (QU) tasks, such as query classification and suggestion, play a crucial role in improving user search experiences by interpreting users' search intents and supporting search behavior [30, 4]. Embedding-based approaches have gained prominence in addressing these tasks due to their robustness to lexical variations [7]. Building tailored embeddings for every QU task is costly, making universal query embeddings essential. Such universal embeddings enable accurate representation of search intent and provide a versatile solution applicable across QU tasks, which is highly valuable for enterprises.

Despite their importance, developing query embeddings that well reflect users' intent presents unique challenges. Since search queries are typically short, they lack rich contextual information, making it difficult to precisely capture users' search intent [8]. This shortness also means that minor wording changes in queries, e.g., replacing even a couple of words with their synonyms, can noticeably alter their appearances. For example, "buy car" and "purchase an automobile" express the same intent but differ substantially in wording. These challenges highlight the need to consider suitable learning embedding approaches for search queries.

A widely recognized approach for learning robust text embeddings is contrastive learning, which has demonstrated notable success in this field. State-of-the-art (SOTA) contrastive learning approaches typically use large-scale weak supervision from web sources, such as question-answer

Figure 1: Illustrations of user interactions used to construct positive query pairs in UBIQUE.



pairs from QA forums or title-passage pairs from encyclopedic articles (Wang et al., 2024a,b; Li et al., 2023b). However, as these datasets primarily consist of longer, contextually detailed sentences, models trained on them struggle to handle short, context-poor queries.

An alternative approach is to use unsupervised contrastive learning models, such as Unsup. SimCSE Gao et al. (2021), directly on a large corpus of search queries. Unsup. SimCSE generates pseudo-positive examples by encoding the same sentence twice with different dropout noise. A model trained on such positive examples tends to overemphasize lexical overlap as a cue for semantic equivalence; we observed that Unsup. SimCSE struggles to capture semantic similarities between queries with different appearances but the same intent, such as "buy car" and "purchase an automobile" (SS5.1).

Overall, current contrastive learning approaches are suboptimal when creating effective positive examples for search query embeddings: typical weakly supervised approaches struggle to generalize to context-poor queries, while the representative unsupervised approach results in models that are overly sensitive to surface-level variations. To address these problems, we propose **U**ser **B**ehavior-driven contrastive learning with **I**nent alignment for search **QU**ery **E**mbeddings (UBIQUE). UBIQUE directly aligns embeddings according to user intent, using intent-aligned query pairs derived from real-world user interactions as positive examples. As shown in Figure 1, we explore two types of user interactions. (1) **Clickthrough data** are records of users' clicking on web pages after submitting search queries. Queries are considered to have the same intent if they lead to clicks on the same URL, as users tend to click on results that satisfy similar information needs. (2) **Session data** are sequences of queries a single user takes on a search engine within a given time frame. Queries within the same user session are assumed to share the same search intent. By using a robust contrastive learning framework (Chen et al., 2020) on these intent-aligned query pairs, UBIQUE constructs models that precisely capture the inherent intent of context-poor queries. This approach also minimizes reliance on appearances, as these intent-aligned query pairs are constructed independently of surface-level similarities.

For our experiments, we built four practical QU datasets using real-world search queries to evaluate UBIQUE from multiple perspectives (SS4). The results indicate that our click-based model (\({\text{UBIQUE}_{\text{click}}}\)) and session-based model (\({\text{UBIQUE}_{\text{session}}}\)) substantially outperformed baselines such as mE5large and Unsup. SimCSE. Specifically, compared to mE5large, \({\text{UBIQUE}_{\text{click}}}\) achieved an average improvement of 8.7 points in task-performance metrics across all tasks, while \({\text{UBIQUE}_{\text{session}}}\) showed strengths in a query-suggestion task, achieving an improvement of 5.3 points in NDCG@10 score. Our analysis also confirmed their robustness to lexical variations, effectively capturing semantic similarities where unsupervised models fail (SS5.1). These findings highlight the effectiveness of leveraging user behavior data in learning universal query embeddings.

## 2 Related Work

Query UnderstandingQU aims to enhance search experiences by effectively processing user queries (Shneiderman et al., 1997; Lau and Horvitz, 1999). Due to the shortness and challenges in capturing their intent, user behavior logs have traditionally supported each QU task before the emergence of deep learning. For instance, mutual query suggestions have been derived from co-occurring session queries (Huang et al., 2003). Similarly, query classification and clustering have leveraged clicked URLs Cao et al. (2009); Beeferman and Berger (2000).

More recently, pre-trained language models have advanced QU. Jiang et al. (2022) mitigated context absence in queries via extended token classification, while Li et al. (2023a) proposed a pre-training framework using a query-URL bipartite graph. We fine-tuned pre-trained language models using user interactions to construct fixed-size text embeddings for general QU tasks. Our approach can be combined with these pre-training techniques.

Closely related is the study by Zhang et al. (2019), who proposed a Bi-GRU-based GEN encoder to compute intent similarity using clickthrough data and task-specific human annotations. Unlike their method, UBIQUE constructs general-purpose search query embedding models that rely solely on automatically collected user interactions.

Contrastive LearningContrastive learning has proven effective for learning text embeddings by pulling similar pairs closer and pushing dissimilar pairs apart Hadsell et al. (2006). Prior research typically focused on constructing positive examples. Early studies relied on annotated datasets, 

such as the NLI dataset (Gao et al., 2021; Zhang et al., 2021), while more recent studies used large-scale weak supervision from web resources, achieving SOTA results (Wang et al., 2024a,b; Li et al., 2023b). While these datasets consist of longer texts, we focused on handling short-text queries using weak supervision from user interactions.

To reduce reliance on annotated data, unsupervised approaches have also been explored. A prominent example is Unsup. SimCSE, which uses dropout as minimal noise to generate positive pairs (Gao et al., 2021; Liu et al., 2021). While Wu et al. (2022) addressed the length biases inherent in Unsup. SimCSE, we examined its ineffectiveness with search queries, particularly its sensitivity to surface-level variations.

## 3 Ubique

This section introduces UBIQUE for constructing universal query-embedding models.

### Overview

UBIQUE uses query pairs \((q,q^{+})\) as positive examples, which match the same search intent, regardless of differences in their surface forms. These pairs are mined from user-interaction logs, which capture detailed records of search activities and engagement patterns with a search engine (SS3.2 and SS3.3). Given a set of query pairs \(D=(q_{i},q_{i}^{+})_{i=1}^{m}\), UBIQUE models are trained using the InfoNCE loss over in-batch negatives (Chen et al., 2017):

\[L_{i}=-\textrm{log}\frac{e^{\textrm{sim}(\mathbf{u_{i}},\mathbf{q_{i}^{+}})/ \tau}}{\sum_{j=1}^{N}e^{\textrm{sim}(\mathbf{u_{i}},\mathbf{q_{j}^{+}})/\tau}}, \tag{1}\]

where \(N\) denotes the mini-batch size, \(\tau\) the temperature hyperparameter, sim\((\cdot)\) the cosine similarity, and \(\mathbf{q_{i}}\) and \(\mathbf{q_{i}^{+}}\) the embeddings of \(q_{i}\) and \(q_{i}^{+}\), respectively. In the following sections, we explain the construction of these positive examples \((q,q^{+})\) from user interactions. 1

Footnote 1: We also experimented with hard negative sampling, but it did not yield improved results (see Appendix A).

### Clickthrough Data

Clickthrough data consist of records of user clicks on web pages after submitting search queries. Queries leading to clicks on the same URL are presumed to share similar search intent, as user clicks generally reflect fulfillment of informational needs (Beeferman and Berger, 2000; Croft et al., 2009).

However, simply mining query pairs that co-clicked on a single URL can produce false positive pairs, as records include unreliable information such as user misclicks or clicks to generic sites (e.g., news portals) that attract diverse queries. To mitigate these types of noise, we mined query pairs in which sets of clicked URLs are similar. By leveraging set similarity, we reduce the impact of noise, as the reliable click information within the sets helps identify appropriate query pairs. Following previous literature (Beeferman and Berger, 2000; Huang et al., 2023), we used the Jaccard coefficient as the measure of set similarity:

\[\small\textsc{Simclick}(q_{1},q_{2})=\frac{\mathrm{U}(q_{1})\cap\mathrm{U}( q_{2})}{\mathrm{U}(q_{1})\cup\mathrm{U}(q_{2})}, \tag{2}\]

where \(q_{1}\) and \(q_{2}\) denote the search queries, and \(\mathrm{U}(q_{i})\) denotes the set of URLs associated with \(q_{i}\). Query pairs exceeding a similarity threshold \(\theta\) were selected as positive pairs.

### Session Data

Session data comprise sequences of queries submitted by a single user within a specific time frame \(t\). Queries within the same session are assumed to have similar search intent, as they may involve reformulating queries, adding further information to previous queries, or searching for different aspects of the same topic (Huang et al., 2003).

Simply mining query pairs that co-occurred within a session can introduce noise, as users may also search with different intents within a session, such as aimless web surfing or addressing multiple informational needs. To address this, we aggregated the co-occurrence frequencies of adjacent queries from each session across multiple sessions (Fonseca et al., 2005), assuming that query pairs with similar search intent are more prevalent than those with different search intents. Since high-frequency queries, such as "YouTube", can bias simple co-occurrence frequencies, we used the Jaccard coefficient that accounts for individual query frequencies (Huang et al., 2003):

\[\small\textsc{Simsession}(q_{1},q_{2})=\frac{c(q_{1},q_{2})}{f(q_{1})+f(q_{2} )-c(q_{1},q_{2})}, \tag{3}\]

where \(c(q_{1},q_{2})\) denotes the co-occurrence frequency of \(q_{1}\) and \(q2\), and \(f(q_{i})\) the frequency of query \(q_{i}\). This measure ensures that even if two queries frequently co-occur, they receive a low similarity score if one of them is popular across different contexts. Query pairs with similarity above a threshold \(\phi\) were selected as positive pairs 

Examples of the constructed query pairs are presented in Table 1.

## 4 Experiment

We evaluated UBIQUE on four real-world QU tasks using Japanese search query logs.

### Evaluation

A multifaceted evaluation across various QU tasks is essential to assess the effectiveness of universal embeddings, as performance on one task may not correlate with performance on others Muennighoff et al. (2023). Due to privacy and proprietary restrictions, comprehensive benchmarks covering multiple QU tasks are not publicly available. Therefore, we constructed a QU benchmark comprising the following four distinct tasks, including one with a public dataset.

Query-Synonym Retrieval (QR)This task retrieves queries that express the same intent despite lexical differences Li and Xu (2014). For each source query, retrieval was conducted by calculating cosine similarity against all other queries in the test set, excluding the source query itself. Mean Reciprocal Rank (MRR) was used as the evaluation metric.

Query Suggestion (QS)This task aims to retrieve contextually related queries that users may consider next. Related queries are sourced from related search keywords in our search system, curated by human evaluators for quality assurance. For evaluation, we retrieved the top ten queries from the full set of related queries, ranked by cosine similarity to the source query. We computed Normalized Discounted Cumulative Gain (NDCG)@10 by assigning a gain value of 1.0 to related queries and 0.0 to all others for each source query.

Query Classification (QC)This task involves categorizing geolocation-related queries into four classes: landmarks, chain stores, addresses, and station names. We trained a linear classifier on the embeddings and evaluated its performance using five-fold cross-validation following Conneau and Kiela (2018) and reported the average of macro F1 score.

Short-Text Reranking (SR)This task re-ranks product names linked to user queries using the publicly available ESCI dataset Reddy et al. (2022). Each query corresponds to multiple products with graded relevance labels: Exact, Substitute, Complement, and Irrelevant. We assigned gain values of 1.0, 0.1, 0.01, and 0.0 to these labels, respectively, for computing NDCG. We ranked all the product names by cosine similarity to the source query.

Statistics of the QU benchmark are shown in Table 2.

\begin{table}
\begin{tabular}{l r c} \hline \hline
**Task** & **\#Samples** & **\#Associated** \\ \hline Query-Synonym Retrieval & 5,000 & 1 \\ Query Suggestion & 951 & 8.2 \\ Query Classification & 1,456 & N/A \\ Short-Text Reranking & 4,667 & 25.5 \\ \hline \hline \end{tabular}
\end{table}
Table 2: Statistics of the QU benchmark. #Samples denotes the size of the dataset, and #Associated denotes the average number of associated items per source query. The associated items were created based on human annotations.

 



### Training Details

The training data, sourced from user logs of Yahoo! JAPAN Search2 in April 2024, includes 50 million query pairs. For clickthrough data, we set \(\theta\) to 0.4, while for session data, we set \(\phi\) to 0.2 and \(t\) to 300 seconds.3 Queries containing predefined adult terms were excluded, as such queries often trigger diverse URL clicks or shifts in intent within a short time frame, resulting in the generation of irrelevant query pairs.

Footnote 2: [https://search.yahoo.co.jp](https://search.yahoo.co.jp)

Footnote 3: Performance improved with higher thresholds for \(\theta\) and \(\phi\), reaching a plateau at these values.

We used Japanese DistilBERT Koga et al. (2023) as the base model, a lightweight model well-suited for practical deployment. The [CLS] representation was used as the query embedding. The batch size was set to 1,024, with a maximum sequence length of \(16^{4}\). The learning rate was 2e-4, using linear decay and a warmup for the initial 1% of steps, with the AdamW optimizer. Training was conducted over 5 epochs, and we selected the best checkpoint on the basis of evaluations conducted every 4,000 steps. We implemented our code using Transformers Wolf et al. (2020) and ran the training on four NVIDIA V100 GPUs, which took 16 hours. To leverage a large number of in-batch negatives crucial for model performance Wang et al. (2024a), we used DeepSpeed ZeRO-2 Rajbhandari et al. (2020) to reduce memory usage and scale up batch size (see Appendix B for details).

### Baselines

We compared UBIQUEclick and UBIQUEsession with SOTA general domain text embedding models and unsupervised models trained on search queries.

We used five SOTA models: Japanese Sup. SimCSELarge Tsukagoshi et al. (2023), Rurilarge Tsukagoshi and Sasano (2024), mE5Large Wang et al. (2024b), Sarashina1.lbNg Thuitions (2024), and the commercial model OpenAI3-large OpenAI (2024). We also used Japanese DistilBERT (UBIQUE's base model), Rurismall, and mE5small as similar-scale models for fair comparison.

For unsupervised models, we used fastText Bojanowski et al. (2017) and Unsup. SimCSE, both trained on 50 million queries. For fastText, we tokenized queries with MeCab Kudo (2006) and trained a 300-dimensional vector model using Skipgram, with default hyperparameters. For UnSup. SimCSE, we used Japanese DistilBERT as the base model, with a learning rate of 3e-5, dropout rate of 0.2, and the same settings as our UBIQUE models for the remaining parameters (see Appendix C for details).

### Results

Table 3 presents the evaluation results on the QU benchmark. UBIQUEclick and UBIQUEsession substantially outperformed all similar-scale models on most tasks and even surpassed the larger SOTA models on average. For instance, UBIQUEclick achieved high scores on average, outperforming Rurilarge by 6.6% in average performance (89.7% vs. 83.1%).5 DBIQUEsession also surpassed Rurilarge with an average score of 2.3% and demonstrated exceptional strength in the QS task, achieving an NDCG@10 score of 92.6%, which is a 6.3% absolute improvement over the baseline's 86.3%. It is worth noting that these SOTA models are not solely based on contrastive learning but involve complex two-stage training pipelines using rerankers Wang et al. (2024a); Li et al. (2023b). These results underscore the importance of constructing positive examples specialized for search queries.

\begin{table}
\begin{tabular}{l c c c c c c} \hline \hline
**Model** & **Params** & **QR** & **QS** & **QC** & **SR** & **Avg.** \\ \hline \multicolumn{8}{c}{_General_} \\ \hline
**SOTA** & & & & & \\ Sup. SimCSELarge & 337M & 40.9 & 81.3 & 85.4 & 88.4 & 74.0 \\ Rurilarge & 337M & 67.7 & 86.3 & **88.0** & 90.5 & 83.1 \\ mE5large & 560M & 63.1 & 87.3 & 82.4 & 91.1 & 81.0 \\ Sarashina1.lb & 1.2B & 73.9 & 89.2 & 84.5 & 91.3 & 84.7 \\ OpenAI3-large & & 65.9 & 89.9 & 80.8 & **91.4** & 82.0 \\ \hline
**Similar Scale** & & & & & \\ DistilBERT & 68M & 20.3 & 79.7 & 83.4 & 87.3 & 67.7 \\ Rurismall & 68M & 54.5 & 87.6 & 84.1 & 90.8 & 79.3 \\ mE5small & 118M & 59.5 & 87.7 & 71.5 & 90.8 & 77.4 \\ \hline \multicolumn{8}{c}{_Search Logs_} \\ \hline
* & 22.9 & 84.8 & 82.5 & 87.7 & 69.5 \\ Unsup. SimCSE & 68M & 28.5 & 84.8 & 83.2 & 88.2 & 71.2 \\ \hline
**Ours** & & & & & \\ UBIQUEclick & 68M & **91.4** & 91.2 & 85.8 & 90.5 & **89.7** \\ UBIQUEsession & 68M & 71.9 & **92.6** & 86.9 & 90.3 & 85.4 \\ \hline \hline \end{tabular}
\end{table}
Table 3: Performance comparison of models on QU benchmark. Metrics: QR (MRR), QS (NDCG@10), QC (F1), SR (NDCG). Avg. is the macro average across tasks. Bold: best, Underline: second best.



UBIQUE\({}_{\text{click}}\) and UBIQUE\({}_{\text{session}}\) substantially outperformed unsupervised models trained on search queries. For example, Unsup. SimCSE achieved an MRR of 28.5% on the QR task, whereas UBIQUE\({}_{\text{click}}\) achieved 91.4%. This notable performance gap in QR task scores indicates that these unsupervised models struggle to capture semantic relationships between search queries with different appearances (see also SS5.1), resulting in limited performance improvements.

To evaluate the effectiveness of the Jaccard coefficient in query pair selection, we conducted an ablation study. We trained UBIQUE\({}_{\text{click}}\) and UBIQUE\({}_{\text{session}}\) without applying Jaccard similarity thresholds (i.e., using query pairs that simply co-clicked on a single URL [30] or just co-occurred in a session). As shown in Table 4, incorporating the Jaccard coefficient led to consistent performance improvements in both our models across most tasks. This suggests the importance of integrating a robust query-pair-mining approach based on the Jaccard coefficient to mitigate noise and irrelevant pairs.

## 5 Analysis

To understand the effectiveness of UBIQUE models, we conducted comparative analyses with representative baseline models.

### Robustness to Lexical Variations

By leveraging user interactions for contrastive learning, UBIQUE\({}_{\text{click}}\) and UBIQUE\({}_{\text{Session}}\) avoid reliance on appearances alone and capture the semantic meaning of search queries, which are often short and thus prone to lexical variations. To verify this property, we evaluated their performance on a query-synonym retrieval task across different edit distances.

As shown in Figure 2, we observed that all models achieved decent MRR scores for lexically similar pairs (e.g., "colour palette" and "color palette"). However, as the lexical difference increased (e.g., "purchase an automobile" and "buy car"), the scores of the baseline models, especially Unsup. SimCSE, decreased dramatically, whereas our models maintained their performance. These findings indicate that, while Unsup. SimCSE is highly sensitive to lexical variations, our models are robust against such variations and can appropriately capture the intent of queries. This robustness can be attributed to using user interactions as weak supervision, which enables the models to focus on semantic similarities rather than appearances.

### Qualitative Analysis

To understand how our models improve query embeddings, we analyzed nearest neighbor queries for each model in the embedding space6. Representative nearest neighbor queries are shown in Table 5.

Footnote 6: Using Faiss Douze et al., 2024), we conducted approximate nearest neighbor search on 10 million random queries.

With mE\({}_{\text{large}}\) and Unsup. SimCSE, the nearest neighbors often had similar appearances but different intents. For example, when given a query "  (LA travel expenses)", these baseline models retrieved ""  (Swiss travel expenses)" because they were affected by the lexical overlap "  (Travel expenses)", even though the destination differed. In contrast, our models succeeded in retrieving queries that share similar intents regardless of lexical differences, such as ""  (Cost of a trip to Los Angeles)." UBIQUE\({}_{\text{click}}\) tended to retrieve paraphrases of queries that more precisely matched the intent while UBIQUE\({}_{\text{session}}\) retrieved queries with broader or transitional intents, such as ""  (LA local time)." These observations align with the characteristics of each data source.

\begin{table}
\begin{tabular}{l c c c c} \hline \hline
**Model** & **QR** & **QS** & **QC** & **SR** \\ \hline UBIQUE\({}_{\text{click}}\) & **91.4** & **91.2** & **85.8** & **90.5** \\ w/o Jaccard & 89.4 & 90.8 & 85.6 & 90.4 \\ \hline UBIQUE\({}_{\text{Session}}\) & **71.9** & **92.6** & 86.9 & **90.3** \\ w/o Jaccard & 58.7 & 92.1 & **87.1** & 90.1 \\ \hline \hline \end{tabular}
\end{table}
Table 4: Ablation study on the QU benchmark.

Figure 2: MRR scores on the QR task across different bins of normalized Levenshtein similarity.

 



## 6 Conclusion and Future Work

We proposed UBIQUE, a simple yet effective approach to address the challenges of learning universal search query embeddings by harnessing user behavior data through contrastive learning. UBIQUE constructs positive query pairs from clickthrough and session data, enabling the model to align embeddings based on user intent rather than surface-level similarities. The empirical results on four practical QU tasks demonstrated that UBIQUE models outperformed strong baselines, particularly in their robustness to lexical variations in search queries.

While our study focused on a Japanese search system, we recognize that search styles can vary across languages Chu et al. (2012). Since UBIQUE is theoretically applicable to other languages, evaluating its effectiveness in diverse linguistic contexts is an exciting future direction. Although we constructed our models separately using clickthrough and session data, combining these data sources may lead to further performance improvements. Incorporating additional information from search results, such as titles and documents, could further enhance UBIQUE, provided the potential increase in inference latency is acceptable.

## 7 Ethics Statement

Throughout UBIQUE's training data generation process (SS3) and the creation of evaluation datasets (SS4.1), all user information was rigorously anonymized to ensure that neither researchers nor reviewers could identify individual users. Specifically, user IDs were replaced with hashed strings, guaranteeing that personal identities remain undisclosed. Additionally, all annotation tasks were conducted by internal senior reviewers who had access only to the queries themselves, without any user information.

In our qualitative evaluation (SS5.2), we included only queries that appeared at least ten times in the logs to further protect user privacy.

## Acknowledgments

We would like to thank Dr. Satoshi Akasaki for his valuable comments and suggestions on an earlier version of this manuscript.

## References

* Beeferman and Berger (2000) Doug Beeferman and Adam Berger. 2000. Agglomerative clustering of a search engine query log. In _Proceedings of the Sixth ACM SIGKDD International Conference on Knowledge Discovery and Data Mining_, KDD '00, page 407-416.
* Bojanowski et al. (2017) Piotr Bojanowski, Edouard Grave, Armand Joulin, and Tomas Mikolov. 2017. Enriching Word Vectors with Subword Information. _Transactions of the Association for Computational Linguistics_, 5:135-146.
* Cao et al. (2009) Huanhuan Cao, Derek Hao Hu, Dou Shen, Daxin Jiang, Jian-Tao Sun, Enhong Chen, and Qiang Yang. 2009. Context-aware query classification. In _Proceedings of the 32nd International ACM SIGIR Conference on Research and Development in Information Retrieval_, SIGIR '09, page 3-10.
* Chen et al. (2020) Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. 2020. A Simple Framework for Contrastive Learning of Visual Representations. In _Proceedings of the 37th International Conference on Machine Learning_, volume 119 of _Proceedings of Machine Learning Research_, pages 1597-1607.
* Chen et al. (2017) Ting Chen, Yizhou Sun, Yue Shi, and Liangjie Hong. 2017. On Sampling Strategies for Neural Network-based Collaborative Filtering. In _Proceedings of the 23rd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining_, KDD '17, page 767-776.
* Chu et al. (2012) Peng Chu, Eszter Jozsa, Anita Komlodi, and Karoly Hercegfi. 2012. An exploratory study on search behavior in different languages. In _Proceedings of the 4th Information Interaction in Context Symposium_, IIIX '12, page 318-321.

\begin{table}
\begin{tabular}{p{56.9pt} p{113.8pt} p{113.8pt} p{113.8pt}} \hline \hline
**Model** & **1st Query** & **2nd Query** & **3rd Query** \\ \hline Unsup. SimCSE & ✓ ✓ ✗  & ✓✓  \\  & (Caims travel expenses) & (Switzerland travel expenses) & (Singapore travel expenses) \\ \cline{2-4} mES\({}_{\text{large}}\) & ✓ ✓ ✗  & ✓✓  & ✓✓  \\  & (Switzerland travel expenses) & (Cost of a trip to Los Angeles) & (Thailand travel expenses) \\ \cline{2-4} UBIQUE\({}_{\text{eak}}\) & ✗  & ✗  & ✗  \\  & Trip to LA) & (Cost of a trip to Los Angeles) & (Los Angeles cost of living) \\ \cline{2-4} UBIQUE\({}_{\text{esasion}}\) & ✏  & ✏  & ✏  \\  & (Cost of a trip to Los Angeles) & (LA Haneda Airport) & (LA local time) \\ \hline \hline \end{tabular}
\end{table}
Table 5: Nearest neighbors in embedding space for “ ⚎ 

* Conneau and Kiela (2018) Alexis Conneau and Douwe Kiela. 2018. SentEval: An Evaluation Toolkit for Universal Sentence Representations. In _Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018)_.
* Croft et al. (2009) Bruce Croft, Donald Metzler, and Trevor Strohman. 2009. _Search Engines: Information Retrieval in Practice_, 1st edition.
* Douze et al. (2024) Matthijs Douze, Alexandr Guzhva, Chengqi Deng, Jeff Johnson, Gergely Szilvasy, Pierre-Emmanuel Mazare, Maria Lomeli, Lucas Hosseini, and Herve Jegou. 2024. The Faiss library. _Preprint_, arXiv:2401.08281.
* Fonseca et al. (2005) Bruno M. Fonseca, Paulo Golgher, Bruno Possas, Berthier Ribeiro-Neto, and Nivio Ziviani. 2005. Concept-based interactive query expansion. CIKM '05, page 696-703.
* Gao et al. (2021) Tianyu Gao, Xingcheng Yao, and Danqi Chen. 2021. SimCSE: Simple Contrastive Learning of Sentence Embeddings. In _Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing_, pages 6894-6910.
* Hadsell et al. (2006) R. Hadsell, S. Chopra, and Y. LeCun. 2006. Dimensionality Reduction by Learning an Invariant Mapping. In _2006 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'06)_, volume 2, pages 1735-1742.
* Hashemi (2016) Homa Baradaran Hashemi. 2016. Query intent detection using convolutional neural networks.
* Huang et al. (2003) Chien-Kang Huang, Lee-Feng Chien, and Yen-Jen Oyang. 2003. Relevant term suggestion in interactive web search based on contextual information in query session logs. _J. Am. Soc. Inf. Sci. Technol._, 54(7):638-649.
* Huang et al. (2023) Yupin Huang, Jiri Gesi, Xinyu Hong, Han Cheng, Kai Zhong, Vivek Mittal, Qingjun Cui, and Vamsi Salaka. 2023. Behavior-driven query similarity prediction based on pre-trained language models for e-commerce search. In _SIGIR 2023 Workshop on eCommerce_.
* Jiang et al. (2022) Haoming Jiang, Tianyu Cao, Zheng Li, Chen Luo, Xiaufeng Tang, Qingyu Yin, Danqing Zhang, Rahul Goutam, and Bing Yin. 2022. Short Text Pre-training with Extended Token Classification for E-commerce Query Understanding. _Preprint_, arXiv:2210.03915.
* Karpukhin et al. (2020) Vladimir Karpukhin, Barlas Oguz, Sewon Min, Patrick Lewis, Ledell Wu, Sergey Edunov, Danqi Chen, and Wen-tau Yih. 2020. Dense Passage Retrieval for Open-Domain Question Answering. In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_, pages 6769-6781.
* Koga et al. (2023) Kobayashi Koga, Shengzhe Li, Akifumi Nakamachi, and Toshinori Sato. 2023. LINE DistilBERT Japanese.
* Kudo (2006) Taku Kudo. 2006. MeCab: Yet Another Part-of-Speech and Morphological Analyzer.
* Lau and Horvitz (1999) Tessa Lau and Eric Horvitz. 1999. Patterns of Search: Analyzing and Modeling Web Query Refinement. In _Proceedings of the Seventh International Conference on User Modeling, Banff, Canada, June 1999_, pages 119-128.
* Li and Xu (2014) Hang Li and Jun Xu. 2014. Semantic Matching in Search. _Found. Trends Inf. Retr._, 7(5):343-469.
* Li et al. (2023a) Juanhui Li, Wei Zeng, Suqi Cheng, Yao Ma, Jiliang Tang, Shuaiqiang Wang, and Dawei Yin. 2023a. Graph Enhanced BERT for Query Understanding. In _Proceedings of the 46th International ACM SIGIR Conference on Research and Development in Information Retrieval_, SIGIR '23, page 3315-3319.
* Li et al. (2023b) Zehan Li, Xin Zhang, Yanzhao Zhang, Dingkun Long, Pengjun Xie, and Meishan Zhang. 2023b. Towards General Text Embeddings with Multi-stage Contrastive Learning. _Preprint_, arXiv:2308.03281.
* Liu et al. (2021) Fangyu Liu, Ivan Vulic, Anna Korhonen, and Nigel Collier. 2021. Fast, Effective, and Self-Supervised: Transforming Masked Language Models into Universal Lexical and Sentence Encoders. In _Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing_.
* Muennighoff et al. (2023) Niklas Muennighoff, Nouamane Tazi, Loic Magne, and Nils Reimers. 2023. MTEB: Massive Text Embedding Benchmark. In _Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics_, pages 2014-2037.
* OpenAI (2024) OpenAI. 2024. New embedding models and API updates.
* Rajbhandari et al. (2020) Samyam Rajbhandari, Jeff Rasley, Olatunji Ruwase, and Yuxiong He. 2020. ZeRO: Memory Optimizations Toward Training Trillion Parameter Models. _Preprint_, arXiv:1910.02054.
* Reddy et al. (2022) Chandan K. Reddy, Lluis Marquez, Fran Valero, Nikhil Rao, Hugo Zaragoza, Sambaran Bandypodhyay, Arnalo Biswas, Anlu Xing, and Karthik Subbian. 2022. Shopping Queries Dataset: A Large-Scale ESCI Benchmark for Improving Product Search. _Preprint_, arXiv:2206.06588.
* IR intuitions (2024) SB Intuitions. 2024. Sarashina-embedding-v1-lb.
* Shneiderman et al. (1997) Ben Shneiderman, Don Byrd, and W. B Croft. 1997. Clarifying Search: A User-Interface Framework for Text Searches. Technical report.
* Tsukagoshi and Sasano (2024) Hayato Tsukagoshi and Ryobei Sasano. 2024. Ruri: Japanese General Text Embeddings. _Preprint_, arXiv:2409.07737.
* Tsukagoshi et al. (2023) Hayato Tsukagoshi, Ryobei Sasano, and Koichi Takeda. 2023. Japanese SimCSE Technical Report. _Preprint_, arXiv:2310.19349 

Bin Wang, C.-C. Jay Kuo, and Haizhou Li (2022)Just bank: rethinking evaluation with word and sentence similarities. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 6060-6077. Cited by: SS3.
* L. Wang, N. Yang, X. Huang, B. Jiao, L. Yang, D. Jiang, R. Majumder, and F. Wei (2024a)Text embeddings by weakly-supervised contrastive pre-training. Preprint arXiv:2212.03533. External Links: Link, Document Cited by: SS1, SS2.
* L. Wang, N. Yang, X. Huang, L. Yang, R. Majumder, and F. Wei (2024b)Multilingual E5 Text Embeddings: A Technical Report.. PreprintarXiv:2402.05672. External Links: Link, Document Cited by: SS1, SS2.
* T. Wolf, L. Debut, V. Sanh, J. Chaumond, C. Delangue, A. Moi, P. Cistac, T. Rault, R. Louf, M. Funtowicz, J. Davison, S. Shleifer, P. von Platen, C. Ma, Y. Iernite, J. Phu, C. Xu, T. Le Scao, S. Gugger, M. Drame, Q. Lhoest, and A. Rush (2020)Transformers: state-of-the-art natural language processing. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pp. 38-45. Cited by: SS1, SS2.
* X. Wu, C. Gao, L. Zang, J. Han, Z. Wang, and S. Hu (2022)ESimCSE: enhanced sample building method for contrastive learning of unsupervised sentence embedding. In Proceedings of the 29th International Conference on Computational Linguistics, pp. 3898-3907. Cited by: SS1, SS2.
* D. Zhang, S. Li, W. Xiao, H. Zhu, R. Nallapati, A. O. Arnold, and B. Xiang (2021)Pairwise supervised contrastive learning of sentence representations. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, Cited by: SS1, SS2.
* H. Zhang, X. Song, C. Xiong, C. Rosset, P. N. Bennett, N. Craswell, and S. Tiwary (2019)Generic intent representation in web search. In Proceedings of the 42nd International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR'19, pp. 65-74. Cited by: SS1, SS2.

## Appendix A Limitations of Hard Negative Sampling

To capture more fine-grained information with our models, we aimed to incorporate hard negatives --negative examples that are challenging to distinguish from the anchor query. Following a prior study Karpukhin et al. (2020), we selected hard negative queries that are lexically similar to the anchor query (i.e., with small edit distances) but have non-overlapping sets of clicked URLs. Specifically, we applied string matching using SimString 7 to a dataset of 10 million queries, treating the anchor query from clickthrough-based training pairs (SS3.2) as the search string. To avoid false negatives due to missing click information, we ensured that all 10 million queries in this dataset were associated with click data. We empirically set the similarity range to 0.45-0.60 to avoid selecting queries that are too lexically similar as hard negatives. We then filtered out extracted queries with any overlapping clicked URLs, treating the remaining queries as hard negatives. Using these hard negatives, we constructed a triplet dataset (i.e., anchor, positive, hard negative) and conducted additional contrastive learning using UBIQUE\({}_{\text{click}}\).

Footnote 7: [https://www.chokkan.org/software/simstring/index.html.en](https://www.chokkan.org/software/simstring/index.html.en)

Despite this effort, overall task performance slightly declined (see Table 6). While this model showed a slight improvement in distinguishing lexically similar negatives, it struggled overall to recognize semantically equivalent queries. This decline in performance may be attributed to the inherent difficulty of consistently using lexically similar queries as negatives, as surface features can also serve as cues for query representation. Future work will focus on refining the negative sampling strategy beyond simple edit-distance measures.

## Appendix B Training Details

To construct the training data, we conducted dedublication to prevent overfitting and excluded query pairs included in the test set to prevent leakage. The learning rate was explored from {2e-4, 3e-4, 3e-5}, and we chose the best one, 2e-4, based on the dev set. For evaluation during training to select the best checkpoint, we used query-synonym retrieval, as the symmetric retrieval task exhibits a strong correlation with downstream tasks Wang et al. (2022). We used a dev set consisting of 5,000 queries for evaluation.

We also tried using \(\text{Ruri}_{\text{small}}\) as the base model for UBIQUE models. \(\text{Ruri}_{\text{small}}\) was initialized with Japanese DistilBERT and further trained using contrastive learning with weak supervision on large-scale web data. While \(\text{Ruri}_{\text{small}}\)-based UBIQUE models' performance was relatively higher than

\begin{table}
\begin{tabular}{l c c c c} \hline \hline
**Model** & **QR** & **QS** & **QC** & **SR** \\ \hline UBIQUE\({}_{\text{click}}\) & **91.4** & **91.2** & 85.8 & **90.6** \\ w/hardnegatives & 90.2 & 90.1 & **86.2** & 90.1 \\ \hline \hline \end{tabular}
\end{table}
Table 6: Results of introducing hard negatives.

 

that of Japanese DistilBERT-based UBIQUE models in the initial stages of training, the final performance showed a negligible difference. This result underscores the importance of using user-behavior data rather than general web data for constructing query embedding models.

## Appendix C Baseline Details

We used [CLS] pooling for Sup. SimCSElarge, mean pooling for DistilBERT, Ruri, and mE5, and last-token pooling for Sarashina1.lb, with a maximum sequence length of 512 used across all models. For mE5 and Ruri, it is necessary to add a prefix to the input sentence, indicating whether it is a source text (query) or a target text (passage) to differentiate the embeddings. We added a query prefix to the source query across all tasks. For the target query, the prefix was added according to the task: a query prefix was used for the symmetric task QR, while a passage prefix was used for the asymmetric tasks QS and SR.

For fastText, we obtained query embeddings by applying mean pooling to the vectors of each token. In Unsup. SimCSE, we explored learning rates from {2e-4, 3e-4, 3e-5} and dropout rates from {0.05, 0.1, 0.2} on the dev set, choosing the best ones, 3e-5 and 0.2, respectively