# Contrastive Refinement for Dense Retrieval Inference in the Open-Domain Question Answering Task
## 1 Introduction  
Since 2019, there has been a significant development in pre-trained language models, making dense retrieval [1, 2, 3, 4, 5, 6] (which projects text into a low-dimensional mathematical space) more effective than traditional sparse retrieval (which is based on term retrieval) in OpenQA. The retrieval module is a crucial step in OpenQA and typically adopts a dual-encoder model architecture to learn dense representations of queries and passages. The dot product operation is then used to calculate the similarity between query-passage pairs for sorting purposes. Currently, many studies aim to improve this dense retrieval architecture; however, they have mainly focused on improving the query side, such as the model-level improvement [4, 7] or instance-level improvement [8]. Despite these efforts, [PERSON] et al. [9] has experimentally demonstrated that DPR [1], which is a dense retrieval baseline, performed significantly worse than sparse retrieval on the entity-centric dataset EntityQuestions (EQ), as shown in Table 1. Their study indicated that DPR [1] could only improve questions with common entities and specific sentence templates, and text augmentation of queries alone could not solve this problem. Therefore, there is a need to build a more robust passage encoder. Based on the findings of [PERSON] et al. [9], we contend that the refinement of passage encoding is a crucial problem that needs to be addressed in open-domain question-answering (OpenQA) research. Thus, to enhance retrieval performance in such tasks, we \begin{table}
\begin{tabular}{c c c c} \hline \hline
**Dataset** & **DPR-Single 1** & **DPR-Multi 2** & **BM25** \\ \hline NQ & 80.1 & 79.4 & 64.4 \\ EQ & 49.7 & 56.7 & 72.0 \\ \hline \hline \end{tabular} \({}^{1}\) DPR model trained on NQ only. 2 DPR model trained on 4 datasets (NQ, TQA, WebQ, TREC) combined.  
\end{table} 
Table 1: Retrieval results (Top-k) on NQ and EQ datasets. propose optimizing the dense representations of both queries and passages. However, before proceeding with a specific implementation, we encountered two major challenges. Firstly, encoding millions of passages in the entire training set with a dense encoder can be compute-intensive and time-consuming (8.8 h on 8 64 GB GPUs). Secondly, as demonstrated by [PERSON] et al. [9], there is currently no universal and robust passage encoder that can effectively encode long sequence texts from different domains and contain the required semantic information. This paper proposes a method for optimizing text representations based on the pseudo-relevance feedback (PRF) algorithm to address potential issues in retrieval. The proposed method constructs pseudo-labeled data with the query or passage as the center and builds a text representation optimization model called AOpt using the gradient descent algorithm to optimize the dense representations of queries and passages at the instance level. This approach avoids repeated text encoding, thus improving efficiency. To improve the general PRF algorithm, we use a knowledge-filtering strategy and conduct binary classification on query-passage pairs to obtain positive and negative pseudo-labels. Our positive and negative sampling strategy is based solely on whether the answer is contained in the passage, which is the most direct strategy. We believe that using the most relevant labels is beneficial for the AOpt model, as the text representation will be optimized through multiple iterations. After constructing the pseudo-labeled data, we first experiment with improving dense retrieval performance using a simple linear weight calculation method under different hyperparameters of pseudo-labeled data. This verifies the effectiveness of pseudo-labeled data in optimizing text representation. We then build the text representation optimization model AOpt to iteratively optimize text-dense representations. AOpt makes the relevant query-passage pairs closer in the mathematical space while making the irrelevant ones farther.