# Towards Robust Neural Rankers with Large Language Model: A Contrastive Training Approach
## 1 Introduction  
In recent years, a plethora of work has demonstrated the astonishing performance of pre-trained models (PLMs) [1; 2; 3; 4] in natural language processing (NLP). The new research paradigm, which involves fine-tuning PLMs on self-supervised tasks for specific applications, has been rapidly adopted by researchers in the community [5]. The paradigm shift in NLP research has also brought new perspectives to information retrieval (IR) studies, which are similarly focusing on text processing. Existing research [6] has indicated that leveraging PLMs with remarkable performance on NLP tasks could significantly enhance the performance of IR models. Although these models exhibit formidable capabilities, including ranking effectiveness and efficiency, they tend to be more vulnerable to adversarial attacks compared to traditional models [7]. For example, upon the introduction of perturbations to the query, the performance of neural rankers may potentially deteriorate. These perturbations, such as word typos and synonym word replacement, are frequently encountered by IR models deployed in the wild. Despite the existence of various proposals aimed at enhancing the robustness of retrieval models, the majority of these approaches [8; 9; 10; 11] are either task-specific or solely reliant on data augmentation. While methods for resisting perturbations at the word or sentence level (such as those addressing typos and synonym substitutions) have been well-developed, there was a notable scarcity of work focusing onquery reformulations with the same intent. A key reason for this is the lack of adequate high-quality augmented data. In light of the significant performance improvement of large language models (LLMs) in generative tasks, we endeavored to employ LLMs to generate query reformulations while maintaining the same intent. Research in computer vision (CV) has demonstrated that representations obtained under a contrastive self-supervised setting exhibited greater robustness when faced with out-of-distribution (OOD) data and image corruption [12]. In enhancing the robustness of information retrieval models with respect to query reformulations maintaining the same intent, it is fundamentally essential to focus on the robustness of query representations under perturbations from diverse distributions. Based on the definition of adversarial text [13], we posit that the semantics of various reformulations of a single query under the same intent should tend to be consistent, while the semantics between different queries should tend to diverge. Guided by this intuition, we employ a contrastive training approach, utilizing LLM-enhanced data to optimize the text representations in PLMs within information retrieval systems. In this study, we primarily focus on the defensive ability against query attacks for IR models based on PLMs, which constitutes an essential subtask in assessing the robustness of IR. We concentrate on ensuring that information retrieval models produce stable and accurate results when faced with different query expressions under the same intent. We propose a novel contrastive learning approach to fine-tune PLM-based IR models and investigate its benefits for enhancing model robustness. Experiments are conducted on both different IR model architectures, employing multi-objective optimization loss functions. In terms of ranking, we train the neural rankers by optimizing ranking loss, which compares scores between different documents under the same query and maximizes the score for the positive candidate. Additionally, we utilize contrastive loss to optimize the representation of queries with the same intent in the latent space (i.e., within an input batch, different query expressions with the same intent should be close in the latent space, while those with different intents should be distant).