Venue: NAACL 2025

# CPRM: A LLM-based Continual Pre-training Framework for Relevance

Modeling in Commercial Search

Kaixin Wu1, Yixin Ji1, Zeyuan Chen1, Qiang Wang2, Cunxiang Wang3

**Hong Liu1, Baijun Ji4, Jia Xu1, Zhongyi Liu1, Jinjie Gu1, Yuan Zhou1, Linjian Mo1 1Ant Group, \({}^{2}\)Hithink RoyalFlush AI Research Institute \({}^{3}\)Westlake University, \({}^{4}\)Soochow University daniel.wkx@antgroup.com**

###### Abstract

Relevance modeling between queries and items stands as a pivotal component in commercial search engines, directly affecting the user experience. Given the remarkable achievements of large language models (LLMs) in various natural language processing (NLP) tasks, LLM-based relevance modeling is gradually being adopted within industrial search systems. Nevertheless, foundational LLMs lack domain-specific knowledge and do not fully exploit the potential of in-context learning. Furthermore, structured item text remains underutilized, and there is a shortage in the supply of corresponding queries and background knowledge. We thereby propose **CPRM** (**C**ontinual **P**re-training for **R**e**levance **M**odeling), a framework designed for the continual pre-training of LLMs to address these issues. Our CPRM framework includes three modules: 1) employing both queries and multi-field item to jointly pre-train for enhancing domain knowledge, 2) applying in-context pre-training, a novel approach where LLMs are pre-trained on a sequence of related queries or items, and 3) conducting reading comprehension on items to produce associated domain knowledge and background information (e.g., generating summaries and corresponding queries) to further strengthen LLMs. Results on offline experiments and online A/B testing demonstrate that our model achieves convincing performance compared to strong baselines.

+
Footnote †: Corresponding author.

_Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Industry Track)_, pages 998–1008 April 30, 2025 ©2025 Association for Computational Linguistics_

to be highly structured and difficult to leverage, which prevents LLMs from fully realizing their potential with such data.

To address the above problems, we investigate a **C**ontinual **P**re-training approach of LLMs for **R**elevance **M**odeling, CPRM for short. Initially, we introduce a pre-training technique using pairs of queries and multi-field item as inputs. This method enables the LLMs to explicitly model the semantic representations between queries and items, thus bridging the semantic gap between them. Subsequently, we collect sets of semantically similar queries and items based on user click logs, then further refine these sets through semantic modeling to filter out semantically irrelevant cases. Following that, we reorder these refined sets according to semantic similarity and ultimately construct in-context pre-training instances via prompting techniques. Utilizing this approach to data construction, LLMs are able to make better predictions within such contexts during the training process, which benefits the efficient learning current domain knowledge for LLMs. Lastly, we employ a larger parameter LLM (teacher LLM) to conduct reading comprehension on structured item data, facilitating the generation of relevant domain knowledge for pre-training, which can be considered as a secondary development and exploitation of the item. More specifically, we leverage teacher LLM to summarize and paraphrase item to produce fluent domain knowledge, while also guiding teacher LLM to produce background information related to the items. Additionally, we prompt teacher LLM to create diverse queries and provide further reasons for their generation. In summary, the contributions of this paper are as follows:

* To our knowledge, we are the first to systematically propose a continual pre-training approach of LLMs specifically designed for relevance modeling tasks.
* We propose a CPRM framework with three components. Firstly, the joint pre-training of queries and multi-field item to enhance domain knowledge of LLMs. Secondly, in-context pre-training by constructing collections of semantically similar queries or items. And thirdly, reading comprehension of structured items employed to strengthen the capabilities of LLMs further.
* Our approach has been validated on real-world industry data, outperforming strong baselines significantly in both offline experiments and online A/B testing.

## 2 Related Work

Relevance modeling in search corresponds to the semantic matching task in NLP. With the advancement of neural network and pre-trained models, deep semantic matching models have become mainstream. Deep semantic matching models are categorized into two types: representation-based Shen et al. (2014); Palangi et al. (2015); Rao et al. (2019) and interaction-based methods Chen et al. (2016); Hu et al. (2014); Pang et al. (2016); Parikh et al. (2016). The former focuses on learning low-dimensional representations, while the latter emphasizes capturing the interactions between inputs. The representation-based model with independently encoded inputs struggles to capture complex correlations, whereas interaction-based methods that concatenate the two inputs for semantic computation can alleviate this issue.

In recent years, pre-trained models like BERT Devlin et al. (2019) has show its superiority on natural language understanding (NLU) tasks. Consequently, both representation-based and interaction-based methods have begun leveraging the capabilities of these pre-trained models for semantic modeling. Most recently, LLMs like GPT-3 Brown et al. (2020), GLM Du et al. (2022), LLaMA Touvron et al. (2023a), Qwen Bai et al. (2023); Yang et al. (2024) pre-trained extensive volumes of data with numerous parameters have garnered significant performance in language understanding, generation and reasoning tasks. Compared to traditional pre-trained models like BERT Devlin et al. (2019), LLMs possess significant advantages in both the scale of pre-training data and the quantity of model parameters, leading to their evident superiority in performance across a variety of downstream tasks. Recent research work Sun et al. (2023); Spatharioti et al. (2023); Zhu et al. (2024) indicates that combining LLMs with downstream applications presents sig

Figure 1: Joint queries and multi-field item for pre-training. An example of the mini-app search scenario.

 

nificant potential, LLMs can achieve competitive or even superior results compared to traditional supervised methods on information retrieval benchmarks. Some research leverage LLMs for relevance modeling in search engines, adopting approaches such as behavior-augmented Chen et al. (2023); Xu and others (2024) or robust learning Liu et al. (2024) to improve the capability of relevance modeling. Our work mainly focuses on enhancing LLMs from the perspective of continual pre-training for relevance modeling. LLMs are pre-trained on a wide variety of data sources Brown et al. (2020); Du et al. (2022); Touvron et al. (2023a, b) without pay more attention on specific domains, resulting in a lack of domain knowledge. On the other hand, the pre-training phase of LLMs is task-agnostic Brown et al. (2020), making it difficult to direct connect with downstream tasks. This also means we can't easily customize the pre-training process to better fit those tasks Min et al. (2022); Gu et al. (2023). Previous work of injecting domain knowledge involves continued training of pre-trained models on domain-specific data Gururangan et al. (2020); Shi et al. (2023), as well as incorporating knowledge graphs Liu et al. (2019) or selectively masking important information Gu et al. (2020); Xu et al. (2023); Sanyal et al. (2023); Zhou et al. (2023). Another line of research aims to enhance the pre-training for downstream tasks, which simple concatenate relevant documents together for in-context pre-training Min et al. (2022); Gu et al. (2023); Shi et al. (2024). However, these approaches assume that downstream tasks contain only a single domain representation, neglecting the possibility of there being multiple or more. For instance, in commercial search relevance tasks, queries and items belong to two distinct domains with substantial differences. Our research work involves injecting domain knowledge and conducting in-context pre-training simultaneously, while being able to establish a connection between the two domains.

## 3 Problem Formulation

Given a query \(Q\) and an item \(I\), LLM-based relevance modeling in search engines aims to predict the relevance degree between them. Essentially, referring to PET Schick and Schutze (2021), we first design the prompt P\((Q,I)\), and LLM determines which verbalizer \(v\) (e.g., "no" or "yes") is the most likely candidate for "[Mask]" conditioned on the likelihood M\((v|{\text{P}}(Q,I))\). The above process is defined as follows:

\[{\text{P}}(Q,I)=Is~{}[Q]~{}and~{}[I]~{}related?~{}~{}\text{[ Mask]} \tag{1}\] \[y=\text{M}(v~{}|~{}{\text{P}}(Q,I)),\quad\text{for}~{}v\in\{ \text{no},\text{yes}\}, \tag{2}\]

where relevance label \(y\in\{0,1\}\) can be associated with a verbalizer (e.g., "no" or "yes") from the vocabulary of LLMs to represent "irrelevant" and "relevant" between \(Q\) and \(I\) respectively. To enable the adaptation of general LLMs to the relevance modeling task, SFT operation is selected for training with the cross-entropy loss function. Consequently, the relevance degree could be given from LLMs for subsequent applications in search scenarios.

## 4 Methodology

In this section, we present the details of the CPRM framework, including Domain Knowledge Enhancement (DKE), In-Context Pre-training (ICP) and Reading Comprehension Distillation (RCD).

### Domain Knowledge Enhancement (DKE)

Different from conventional pre-training methods, we jointly pre-train the structured item data with multiple queries as shown in Figure 1. Each item encompasses multiple fields, including title, keywords, category, description, etc., with the query being the most frequently searched top query for a given item. For each query or item, we employ segment embeddings to distinguish between different texts. For convenience, we add special tokens "<lstartofpiece>" and "<lendofpieceb>" between the queries and item as segment embeddings to differentiate them. Furthermore, queries and item are combined for position encoding, thereby allowing LLMs to explicitly model the relationships between them during pre-training process. Due to constraints on response time for online services, when calculating relevance scores between queries and items, only a limited number of item fields (such as title and keywords) are considered. Consequently, domain knowledge from other unused item fields, such as description, can be incorporated through continual pre-training. Considering that relevance modeling is a NLU task, we adopt both token-level masked language modeling (MLM) Devlin et al. (2018) and segment-level MLM pre-training strategies for LLMs. Therefore, the optimization objective is:

\[\mathcal{L}(\theta)=\underset{\theta}{\text{min}}\,\alpha\mathcal{L}_{\text{ t-MLM}}(\theta)+(1-\alpha)\mathcal{L}_{\text{s-MLM}}(\theta),\] (3) 

where \(\theta\) is the parameters of the model, \({\cal L}_{\text{t-MLM}}(\theta)\) and \({\cal L}_{\text{s-MLM}}(\theta)\) represent token-level MLM loss and segment-level MLM loss respectively, we set \(\alpha=0.7\) in our experiments.

### In-Context Pre-training (ICP)

We construct in-context pre-training instances using historical click logs from real-world business search scenario. The overall idea is to build collections of semantically similar queries and items as pre-training data to further stimulate in-context learning capabilities of LLMs. The detailed data construction methodology is as follows:

**Coarse Screening.** Utilizing the click logs, we establish a mapping from Queries to Items (denoted as \(Q2I\) and from Items to Queries (denoted as \(I2Q\)), sorting them by the number of clicks in descending order. Consequently, within the \(Q2I\) mapping, for a query \(Query\) there is an associated set of items \(I^{Query}=\{I_{1},I_{2},...,I_{N}\}\). These items can be considered as a preliminarily semantically related collection under the specific constraint of query \(Query\). Vice versa for \(I2Q\) mapping.

**Fine Screening.** Following described above, cases may be introduced that received clicks but are semantically unrelated. We employ _Contriever_[18], a semantic model, to encode text into vectors, and then calculate the similarity between various text representations for semantic filtering. For set \(I^{Query}\), when the following condition is met:

\[Sim(Query,I_{k})<\sigma,\quad\text{for $k\in[1,N]$}, \tag{4}\]

it signifies that \(Query\) and \(I_{k}\) are semantically unrelated and require filtering, where \(Sim(\cdot)\) is similarity function, \(\sigma\) is a threshold. **Data Construction.** As shown in the left of Figure 2, we subsequently obtain a collection of items that are semantically relevant to the query, then sort these items by semantic similarity in ascending order. Finally, we concatenate the query with the sorted collection of items to create an ICP instance via prompting. The right of Figure 2 illustrates how to construct ICP instances under \(I2Q\) mapping, namely, obtaining a set of semantically related queries given the constraint of an item.

Why adopt this construction manner? By assembling collections of items under a specified query or collections of queries under a specified item, LLMs can make better predictions based on the context during the pre-training process, enabling more efficient learning within the current domain. Moreover, the reordering in ICP instances implicitly indicates the strength of relevance between queries and items, enabling LLMs to model the degree of their association effectively. On the other hand, by linking queries and items in our ICP instances, we enable LLMs to model their semantic representations directly.

Figure 3: Prompt for reading comprehension on item.

Figure 2: In-context pre-training instances construction. The left and right figures represent the ICP instances constructed from similar item sets and similar query sets respectively.

 



### Reading Comprehension Distillation (RCD)

We employ the teacher LLM for reading comprehension on items, with the prompt design shown in Figure 3. Assume that in a mini-app search scenario, we need to provide the mini-app's structured information like title, keywords, category, and description, and utilizing prompt template instructions to invoke teacher LLM. This generates the reading comprehension pre-training instances.

Why design the prompt in this way? We have several reasons for this design choice. Firstly, item text is structured data and difficult to utilize, lacking in relevant background knowledge. Through summarizing and rephrasing with Prompt 1, fluent domain knowledge can be generated. Additionally, understanding and analyzing items can instruct teacher LLM in generating relevant background knowledge. Secondly, by using Prompt 2 and Prompt 3 guide teacher LLM to generate related and diversified queries, enriching the supply of suitable queries. We also instruct teacher LLM to provide further explanations for the generated queries. This approach not only facilitates the generation of relevant domain knowledge but also allows downstream models to significantly improve their understanding and handling of the item when utilizing these data. Employing teacher LLM for reading comprehension on items can be considered a secondary development and utilization of item data, enriching the domain knowledge further. Pre-training LLMs on the above data can also be seen as a process of knowledge transfer from teahacher LLM to LLMs.

## 5 Experiments

### Experimental Settings

Dataset & Evaluation Metrics.We utilize the real-world mini-app search scenario data for verification. The pre-training data includes three parts: DKE data (4M), ICP data (4M) and RCD data (500K). The first part is sampled from the mini-app search scenarios and consist of structured items containing multiple fields. For top 500K most frequently visited items, we sample 5 top queries based on click logs for each item, which are then concatenated with multi-field item to serve as pre-training examples for adapting relevance tasks. The second part is in-context pre-training data, where we construct these examples based on the real-world search click logs using the method described in Section 4.2, and subsequently randomly sample 4M from them. The third part is reading comprehension data, for which we utilize teacher LLM (e.g. Qwen2-72B Yang et al. (2024)) to perform reading comprehension on item data. The SFT data consists of three parts: train set (625K), valid set (35K) and test set (35K). These data are sourced from real mini-app search results and then are generated through manual annotation. The human-annotated data for relevance tasks are in format of triples <_Query_, _Item_, _Label>_, the data statistics as shown in Table 1. The annotated data have only two levels of relevance: "#Relevant" and "#Irrelevant". For evaluation, we employ three widely used metrics Acc., F1 and AUC to evaluate model performance, with higher values indicating better performance. Note that AUC serves as the most important metric in relevance tasks while the others provide auxiliary supports for our analysis.

Baselines.We selected classic NLU-based models and commonly used LLMs as our baseline models: DSSM Shen et al. (2014), ReprBERT Yao et al. (2022), BERT Devlin et al. (2018), GLM Du et al. (2022), Quen2Yang et al. (2024), ChatGPT & GPT-4 Team (2024).

Implementation Details.All our pre-training experiments are conducted on the GLM-2B. The model configuration set to 36 layers, hidden size of 2048, FFN size of 8192 and 32 attention heads. We utilize adam Kingma and Ba (2017) optimizer and the warmup steps and learning rate set 28K and \(e^{-4}\). All models are pre-trained on 8 A100 GPUs for 2 epochs and the batch size set 64. During SFT, all models are trained for 5 epochs on 8 A100 GPUs and the batch size is 8. The adam optimizer is employed and warmup steps and learning rate set to 5K and \(2e^{-5}\) respectively. When constructing the ICP instances, we utilize facebook's open-source multilingual _Contriever_1 Izacard et al. (2022) model for semantic filtering.

Footnote 1: [https://github.com/facebooksearch/contriever](https://github.com/facebooksearch/contriever)

tinual pre-training models on the relevance task. From the experimental results, GLM demonstrates strong competitiveness, achieving superior performance even with similar parameter numbers compared to BERT-Large (line 7 vs. line 4). We also conducts experiments on the GLM of various parameter sizes, and the results show that as the model size increases, its performance gradually improves. However, with further increases in model size, the performance gains become progressively smaller. Specifically, GLM-10B achieves only a 0.21% improvement in AUC over GLM-5B (line 10 vs. line 9). Compared to other latest LLMs such as Qwen2, our GLM also shows impressive performance at a similar parameter scale (line 7 vs. line 5, line 8 vs. line 6). ChatGPT & GPT-4 performed poorly compared to other SFT-based baseline systems; this is because the task data belongs to a proprietary domain, and models without SFT operations have relatively poor discriminative ability. We conducts continued pre-training experiments on GLM-2B, and the experimental results demonstrate that all three different methods result in performance improvements compared to the baseline model. Notably, the DKE and ICP methods achieve significant performance enhancements, with respective gains of 1.11% and 1.54% in AUC. This is because both methods are constructed based on domain-specific data and jointly training semantically related queries or items can further enhance model performance. The experimental results also indicate that integrating different continued pre-training methods can further strength model performance (line 17 and line 18), with the combination of all three methods leading to the greatest performance gain, making it comparable to that of GLM-10B (line 18 vs. line 10). Our CPRM model achieves the highest F1 score (94.42%) among all models compared.

**Analysis on Query Length.** We compare the performance of different models at various query lengths on test set. As shown in Figure 4, our CPRM model outperforms the baseline model across all length intervals, especially on longer queries (when the length greater than 15), where the CPRM model demonstrates a significant improvement performance gains, surpassing the baseline model by 15.85% in AUC (92.27% vs.

\begin{table}
\begin{tabular}{l l|c c|c c|c c} \hline \hline \# & **Model** & **Acc. (\%)** & \(\Delta_{\text{Acc}}\) & **F1 (\%)** & \(\Delta_{\text{FI}}\) & **AUC (\%)** & \(\Delta_{\text{AUC}}\) \\ \hline \hline \multicolumn{8}{c}{_Only fine-tuning on supervised datasets_} \\ \hline
1 & DSSM & & 70.32 & - & 71.21 & - & 70.64 & - \\
2 & ReprBERT & 80.65 & - & 82.77 & - & 80.24 & - \\
3 & BERT-Base (0.1B) & 82.24 & - & 84.33 & - & 81.79 & - \\
4 & BERT-Large (0.3B) & 83.47 & - & 85.65 & - & 83.01 & - \\ \hline
5 & Qwen2-0.5B & 80.48 & - & 79.52 & - & 81.63 & - \\
6 & Qwen2-1.5B & 91.17 & - & 92.08 & - & 90.90 & - \\ \hline
7 & GLM-0.3B & 85.93 & - & 87.32 & - & 85.64 & - \\
8 & GLM-2B & 91.16 & - & 91.95 & - & 91.04 & - \\
9 & GLM-5B & 93.53 & - & 94.12 & - & 93.41 & - \\
10 & GLM-10B & **93.70** & - & 94.26 & - & **93.62** & - \\ \hline \hline \multicolumn{8}{c}{_Without fine-tuning_} \\ \hline
11 & ChatGPT (+8-shot) & 62.93 & - & 59.78 & - & 64.88 & - \\
12 & GPT-4 (+8-shot) & 61.89 & - & 67.44 & - & 60.83 & - \\ \hline \hline \multicolumn{8}{c}{_Continual pre-training LLM and then fine-tuning on supervised datasets_} \\ \hline
13 & GLM-2B & 91.16 & +0.00 & 91.95 & +0.00 & 91.04 & +0.00 \\
14 & + DKE & 92.28 & +1.12 & 92.99 & +1.04 & 92.15 & +1.11 \\
15 & + ICP & 92.72 & +1.56 & 93.40 & +1.45 & 92.58 & +1.54 \\
16 & + RCD & 91.59 & +0.43 & 92.36 & +0.41 & 91.46 & +0.42 \\
17 & + DKE + ICP & 93.33 & +2.17 & 93.93 & +1.98 & 93.23 & +2.19 \\
18 & + DKE + ICP + RCD (a.k.a. **CPRM**) & 93.64 & +2.48 & **94.42** & +2.47 & 93.49 & +2.45 \\ \hline \hline \end{tabular}
\end{table}
Table 2: Performance of different baselines and various continual pre-training models on the relevance task. **bold** and underline represent the best and second best result respectively. Improvements over variants are statistically significant with p ¡ 0.05.

 

76.42%). This suggests that the CPRM model possesses a superior ability to understand and deal with long queries. We speculate that this advantage may be attributable to the ICP and RCD methods. Since the ICP method semantically aggregates historical search queries, allowing the model have the possibility encountered related long queries and to understand their semantics in the in-context pre-training process. On the other hand, the RCD method generates diverse queries, thereby enriching the model's understanding of various long query types.

**Impact of Training Steps.** As shown in Figure 5, we compare models' performance with different pre-training methods across various training steps. The experimental results show that models trained with all three different pre-training methods surpasses the baseline across various training steps. The CPRM model, which combines all three methods, achieves the best performance at each step. These evidence highlights the robustness of our proposed approach. Interestingly, an phenomenon observed from the figure is that the baseline model's performance significantly decreases at the 16K training step before it gradually increases thereafter. The reason is due to the significant difference between the current task data and the data previously seen by the LLM, resulting in challenges for the LLM in fitting this domain-specific data. None of the other pre-training methods exhibits this phenomenon; instead, the performance of these models steadily improved at each training step. This indicates that our proposed methods are beneficial for the domain adaptation of LLM.

## 6 Online A/B Testing

We deploy the proposed model on the online search platform to provide search services for mini-apps, and conduct a two-week online A/B testing with 5% proportion of the experiment traffic. The experimental results show that, compared to the baseline system (GLM-2B), our CPRM method yields a statistically significant increase of 0.32% in valid PVCTR2 at a 95% confidence level. Human evaluation indicates a 0.75% reduction in Badcase@10 metric and a 4.71% decrease in the Error Filtering Rate3. The model has now been serving search functions to mini-apps for over nine months. These results suggest that our proposed method can effectively enhance relevance models' performance in real-world search systems.

Footnote 2: Page view click-through rate, the number of valid clicks divided by the number of searches.

Footnote 3: The number of relevant items that are incorrectly filtered out divided by the total number of filtered items.

## 7 Conclusion

In this paper, we have investigated CPRM framework, a continued pre-training approach of LLMs tailored to relevance modeling tasks, which comprises three methods: DKE, ICP and RCD. Both offline experiments and online A/B testing results demonstrate that our proposed method boosts the search relevance of LLMs effectively. Our model has been successfully deployed online search platform.

## References

* Bai et al. [2018] Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang, Xiaodong Deng, Yang Fan, Wenbin Ge, Yu Han, Fei Huang, Bingyuan Hui, Luo Ji, Mei Li, Junyang Lin, Runji Lin, Dayiheng Liu, Gao Liu, Chengqiang Lu, Keming Lu, Jianxin Ma, Rui Men, Xingzhang Ren, Xuxencheng Ren, Chuanqi Tan, Sinan Tian, Jianhong Tu, Peng Wang, Shijie Wang, Wei Wang, Shenggung Wu, Benfeng Xu, Jin Xu, An Yang, Hao Yang, Jian Yang, Shusheng Yang, Yang Yao, Bowen Yu, Hongyi Yuan, Zheng Yuan, Jianwei Zhang, Xingsxuan Zhang, Yichang Zhang, Zhenru Zhang, Chang

Figure 4: Performance of different query lengths.

Figure 5: Performance of different training steps.

 

Zhou, Jingren Zhou, Xiaohuan Zhou, and Tianhang Zhu. 2023. Qwen technical report.
* Brown et al. (2020) Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Torn Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. 2020. Language models are few-shot learners.
* Chen et al. (2016) Qian Chen, Xiaodan Zhu, Zhenhua Ling, Si Wei, Hui Jiang, and Diana Inkpen. 2016. Enhanced lstm for natural language inference. _arXiv preprint arXiv:1609.06038_.
* Chen et al. (2023) Zeyuan Chen, Wei Chen, Jia Xu, Zhongyi Liu, and Wei Zhang. 2023. Beyond semantics: Learning a behavior augmented relevance model with self-supervised learning. In _Proceedings of the 32nd ACM International Conference on Information and Knowledge Management_, pages 4516-4522.
* Chen et al. (2024) Zeyuan Chen, Haiyan Wu, Kaixin Wu, Wei Chen, Mingjie Zhong, Jia Xu, Zhongyi Liu, and Wei Zhang. 2024. Towards boosting llms-driven relevance modeling with progressive retrieved behavior-augmented prompting.
* Cui et al. (2023) Jiaxi Cui, Zongjian Li, Yang Yan, Bohua Chen, and Li Yuan. 2023. Chatlaw: Open-source legal large language model with integrated external knowledge bases.
* Devlin et al. (2018) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. Bert: Pre-training of deep bidirectional transformers for language understanding. _arXiv preprint arXiv:1810.04805_.
* Devlin et al. (2019) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. Bert: Pre-training of deep bidirectional transformers for language understanding. In _North American Chapter of the Association for Computational Linguistics_.
* Du et al. (2022) Zhengxiao Du, Yujie Qian, Xiao Liu, Ming Ding, Jiezhong Qiu, Zhilin Yang, and Jie Tang. 2022. Glm: General language model pretraining with autoregressive blank infilling. In _Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_, pages 320-335.
* Gu et al. (2023) Yuxian Gu, Li Dong, Furu Wei, and Minlie Huang. 2023. Pre-training to learn in context.
* Gu et al. (2020) Yuxian Gu, Zhengyan Zhang, Xiaozhi Wang, Zhiyuan Liu, and Maosong Sun. 2020. Train no evil: Selective masking for task-guided pre-training.
* Gururangan et al. (2020) Suchin Gururangan, Ana Marasovic, Swabha Swayamdipta, Kyle Lo, Iz Beltagy, Doug Downey, and Noah A. Smith. 2020. Don't stop retraining: Adapt language models to domains and tasks.
* Hu et al. (2014) Baotian Hu, Zhengdong Lu, Hang Li, and Qingcai Chen. 2014. Convolutional neural network architectures for matching natural language sentences. _Advances in neural information processing systems_, 27.
* Izacard et al. (2022) Gautier Izacard, Mathilde Caron, Lucas Hosseini, Sebastian Riedel, Piotr Bojanowski, Armand Joulin, and Edouard Grave. 2022. Unsupervised dense information retrieval with contrastive learning.
* Kingma and Ba (2017) Diederik P. Kingma and Jimmy Ba. 2017. Adam: A method for stochastic optimization.
* Kumar and Sarkar (2021) Lakshya Kumar and Sagnik Sarkar. 2021. Neural search: Learning query and product representations in fashion e-commerce.
* Lian et al. (2019)Yijiang Lian, Zhijie Chen, Jinlong Hu, Kefeng Zhang, Chunwei Yan, Muchenavan Tong, Wenying Han, Haniju Guan, Ying Li, Ying Cao, Yang Yu, Zhigang Li, Xiaochun Liu, and Yue Wang. 2019. An end-to-end generative retrieval method for sponsored search engine -decoding efficiently into a closed target domain.
* Liu et al. (2024) Hong Liu, Saisai Gong, Yixin Ji, Kaixin Wu, Jia Xu, and Jinjie Gu. 2024. Boosting llm-based relevance modeling with distribution-aware robust learning. In _Proceedings of the 33rd ACM International Conference on Information and Knowledge Management_, pages 4718-4725.
* Liu et al. (2019a) Weijie Liu, Peng Zhou, Zhe Zhao, Zhiruo Wang, Qi Ju, Haotang Deng, and Ping Wang. 2019a. K-bert: Enabling language representation with knowledge graph.
* Liu et al. (2019b) Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019b. Roberta: A robustly optimized bert pretraining approach.
* Min et al. (2022) Sewon Min, Mike Lewis, Luke Zettlemoyer, and Hannaneh Hajishirzi. 2022. Metaicl: Learning to learn in context.
* Palangi et al. (2015) H. Palangi, L. Deng, Y. Shen, J. Gao, X. He, J. Chen, X. Song, and R. Ward. 2015. Semantic modelling with long-short-term memory for information retrieval.
* Pang et al. (2016) Liang Pang, Yanyan Lan, Jiafeng Guo, Jun Xu, Shengxian Wan, and Xueqi Cheng. 2016. Text matching as image recognition. In _Proceedings of the AAAI Conference on Artificial Intelligence_, volume 30.
* Parikh et al. (2016) Ankur P Parikh, Oscar Tackstrom, Dipanjan Das, and Jakob Uszkoreit. 2016. A decomposable attention model for natural language inference. _arXiv preprint arXiv:1606.01933_.
* Qi et al. (2020) Weizhen Qi, Yu Yan, Yeyun Gong, Dayiheng Liu, Nan Duan, Jiusheng Chen, Ruofei Zhang, and Ming Zhou. 2020. Prophetnet: Predicting future n-gram for sequence-to-sequence pre-training 

Jinfeng Rao, Linqing Liu, Yi Tay, Wei Yang, Peng Shi, and Jimmy Lin. 2019 Bridging the gap between relevance matching and semantic matching for short text similarity modeling. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pp. 5370-5381. Cited by: SS4.
* S. Sanyal, Y. Xu, S. Wang, Z. Yang, R. Pryzant, W. Yu, C. Zhu, and X. Ren (2023)Apollo: a simple approach for adaptive pretraining of language models for logical reasoning. External Links: Link Cited by: SS4.
* T. Schick and H. Schutze (2021)Exploiting cloze questions for few shot text classification and natural language inference. External Links: Link Cited by: SS4.
* Y. Shen, X. He, J. Gao, L. Deng, and G. Mesnil (2014)Learning semantic representations using convolutional neural networks for web search. In Proceedings of the 23rd international conference on world wide web, pp. 373-374. Cited by: SS4.
* W. Shi, S. Min, M. Lomeli, C. Zhou, M. Li, G. Szilvasy, R. James, A. Victoria Lin, N. A. Smith, L. Zettlemoyer, S. Yih, and M. Lewis (2024)InContext pretraining: language modeling beyond document boundaries. External Links: Link Cited by: SS4.
* Z. Shi, F. Tonolini, N. Aletras, E. Yilmaz, G. Kazai, and Y. Jiao (2023)Rethinking semi-supervised learning with language models. External Links: Link Cited by: SS4.
* S. Eleni Spatharioti, D. M. Rothschild, D. G. Goldstein, and J. M. Hofman (2023)Comparing traditional and llm-based search for consumer choice: a randomized experiment. External Links: Link Cited by: SS4.
* W. Sun, L. Yan, X. Ma, P. Ren, D. Yin, and Z. Ren (2023)Is chatgPT good at search? investigating large language models as re-ranking agent. arXiv preprint arXiv:2304.09542. Cited by: SS4.
* O. Team (2024)Gpt-4 technical report. External Links: Link Cited by: SS4.
* H. Touvron, T. Lavril, G. Izacard, X. Martinet, M. Lachaux, T. Lacroix, B. Roziere, N. Goyal, E. Hambro, F. Azhar, et al. (2023a)LIMA: open and efficient foundation language models. arXiv preprint arXiv:2302.13971. Cited by: SS4.
* H. Touvron, L. Martin, K. Stone, P. Albert, A. Almahairi, Y. Babaei, N. Bashlykov, S. Batra, P. Bhargava, S. Bhosale, D. Bilkel, L. Blecher, C. Canton Ferrer, M. Chen, G. Cucurull, D. Esiboglu, J. Fernandes, J. Fu, W. Fu, B. Fuller, C. Gao, V. Goswami, N. Goyal, A. Hartshorn, S. Hosseini, R. Hou, H. Inan, M. Kardas, V. Kerkez, M. Khabsa, I. Klounann, A. Korenev, P. Singh Kouara, M. Lachaux, T. Lavril, J. Lee, D. Liskovich, Y. Lu, Y. Mao, X. Martinet, T. Mihaylov, P. Mishra, I. Molbyog, Y. Nie, A. Poulton, J. Reizenstein, R. Rungaik, K. Saladi, A. Scheiten, R. Silva, E. Michael Smith, R. Subramanian, X. Ellen Tan, B. Tang, R. Taylor, A. Williams, J. Xiang Kuan, P. Xu, Z. Yan, I. Zarov, Y. Zhang, A. Fan, M. Kambandur, S. Narang, A. Rodriguez, R. Stojnic, S. Edunov, and T. Scialom (2023b)Llama 2: open foundation and fine-tuned chat models. External Links: Link Cited by: SS4.
* S. Wu, O. Irsoy, S. Lu, V. Dabravolski, M. Dredze, S. Gehrmann, P. Kampbadur, D. Rosenberg, and G. Mann (2023)Bloombergopt: a large language model for finance. External Links: Link Cited by: SS4.
* H. Xiong, S. Wang, Y. Zhu, Z. Zhao, Y. Liu, L. Huang, Q. Wang, and D. Shen (2023)Doctorglm: fine-tuning your chinese doctor is not a hervelcan task. External Links: Link Cited by: SS4.
* Y. Xu, M. Namazifar, D. Hazarika, A. Padmakumar, Y. Liu, and D. HakkaniTur (2023)XILM: knowledge injection into encoder-decoder language models. External Links: Link Cited by: SS4.
* A. Yang, B. Yang, B. Hui, B. Zheng, B. Yu, C. Zhou, C. Li, C. Li, D. Liu, F. Huang, G. Dong, H. Wei, H. Lin, J. Tang, J. Wang, J. Yang, J. Tu, J. Zhang, J. Ma, J. Xu, J. Zhou, J. Bai, J. He, J. Lin, K. Pang, K. Lu, K. Chen, K. Yang, M. Li, M. Xue, N. Ni, P. Zhang, P. Wang, R. Peng, R. Men, R. Gao, R. Lin, S. Wang, S. Bai, S. Tan, T. Zhu, T. Li, T. Liu, W. Mei, X. Deng, X. Ren, X. Zhang, X. Ren, Y. Yao, Y. Zhang, Y. Wan, Y. Chu, Y. Liu, Z. Cui, Z. Zhang, and Z. Fan (2024)QWEN2 technical report.. External Links: Link Cited by: SS4.
* Z. Yang, Z. Dai, Y. Yang, J. Carbonell, R. Salakhutdinov, and Q. V. Le (2020)XLNet: generalized autoregressive pretraining for language understanding. External Links: Link Cited by: SS4.
* S. Yao, J. Tan, X. Chen, J. Zhang, X. Zeng, and K. Yang (2022)RepBERT: distilling bert to an efficient representation-based relevance model for e-commerce. In Proceedings of the 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, Vol. , pp. 4363-4371. External Links: Document, ISSN 1574-4695, Link Cited by: SS4.
* W. Zhou, R. L. Bras, and Y. Choi (2023)Commonsense knowledge transfer for pretrained language models. External Links: Link Cited by: SS4.
* Y. Zhu, H. Yuan, S. Wang, J. Liu, W. Liu, C. Deng, H. Chen, Z. Dou, and Ji-Rong Wen (2024)Large language models for information retrieval: a survey 

Baselines

We compare our proposed CPRM model with the following baselines:

* **DSSM**[21] is a classic two-tower structure text matching model that constructs representations for the query and item independently, using cosine similarity to measure their relevance.
* **ReprBERT**[30] is a representation-based BERT model that utilizes novel interaction strategies to balance performance and latency.
* **BERT**[47] has achieved great success on NLP tasks as an interaction-based model. Here, we concatenate the query and item as the model input for relevance modeling.
* **GLM**[56] is a powerful LLM architecture with various parameter sizes to suit different business scenarios. Our LLM online system is developed based on the GLM, thus all our experiments are mainly conducted on the GLM.
* **Qwen2**[38] is currently one of the newest and the state-of-the-art (SOTA) open-source LLMs for Chinese NLP tasks.
* **ChatGPT\({}^{4}\) & GPT-4**[9] are the SOTA closed-source LLMs. We employ the direct generation approach for relevance task evaluation.

## Appendix B Model Deployment

LLMs have achieved significant performance improvements in relevance tasks, but their large parameter size leads to low inference efficiency, thus affecting their deployment online. We have designed a solution that allows for the real-time use of LLMs' relevance scores. As shown in Figure 6, the online relevance model for search consists of two parts: the GLM-0.3B model serves as the online model to respond to search queries in real-time, while the GLM-2B model employ a T+1 update strategy to score historical Q-I pairs and cache them offline. The online relevance service gives priority to using the cached scores from GLM-2B; if the cache does not exist, it calls on the GLM-0.3B online model. Currently, using GLM-2B's offline caching scoring has covered over 60% of mini-app search requests, significantly alleviating the request pressure on the online model.

## Appendix C Case Study

As shown in Figure 7, we provide several cases to compare the relevance output results between the baseline (GLM-2B) and CPRM models. From these cases, we can observe that the CPRM method is able to supplement additional domain knowledge to correct erroneous prediction results. Furthermore, CPRM demonstrates a stronger understanding of long and complex queries (query length greater than 15) compared to the baseline. The SFT data format can be referred to in Figure 8.

Figure 6: Deployment of the CPRM relevance model.

 

* [105] M. C. B. K. S. Chan, and J. A. L. Friedman, "A new approach to the design of a neural network for learning a class of features," in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_, pp. 237-246, 2009.

[MISSING_PAGE_POST]