MrRank: Improving Question Answering Retrieval System through Multi-Result Ranking Model (2024)

Abstract: Large Language Models (LLMs) often struggle with hallucinations and outdated information. To address this, Information Retrieval (IR) systems can be employed to augment LLMs with up-to-date knowledge. However, existing IR techniques contain deficiencies, posing a performance bottleneck. Given the extensive array of IR systems, combining diverse approaches presents a viable strategy. Nevertheless, prior attempts have yielded restricted efficacy. In this work, we propose an approach that leverages learning-to-rank techniques to combine heterogeneous IR systems. We demonstrate the method on two Retrieval Question Answering (ReQA) tasks. Our empirical findings exhibit a significant performance enhancement, outperforming previous approaches and achieving state-of-the-art results on ReQA SQuAD.

text: Findings of the Association for Computational Linguistics ACL 2024, pages 4750–4762 August 11-16, 2024 ©2024 Association for Computational Linguistics
Large Language Models (LLMs) often struggle with hallucinations and outdated information. To address this, Information Retrieval (IR) systems can be employed to augment LLMs with up-to-date knowledge. However, existing IR techniques contain deficiencies, posing a performance bottleneck. Given the extensive array of IR systems, combining diverse approaches presents a viable strategy. Nevertheless, prior attempts have yielded restricted efficacy. In this work, we propose an approach that leverages learning-to-rank techniques to combine heterogeneous IR systems. We demonstrate the method on two Retrieval Question Answering (ReQA) tasks. Our empirical findings exhibit a significant performance enhancement, outperforming previous approaches and achieving state-of-the-art results on ReQA SQuAD.

heading: 1 Introduction
text: Large Language Models (LLMs) have demonstrated proficiency in comprehending human language across diverse domains. Among these advancements, question answering (QA) systems integrated with LLMs have gained significant attention. Nonetheless, concerns persist regarding the susceptibility of LLMs to hallucinations, potentially leading to inaccuracies in their responses.
To address the aforementioned challenge, the Retrieval Augmented Generation (RAG) (Lewis et al., 2020) is proposed. The RAG system generally consists of two parts: an Information Retrieval (IR) engine which proposes the top k candidate pieces from a document corpus for the query, and the QA engine which is responsible for deriving the final answer from retrieved information.
Although LLMs demonstrate exceptional proficiency in QA tasks, contemporary text embedding techniques employed in IR remain underdeveloped. This deficiency poses a performance bottleneck for
RAG, as shown by the results in the Massive Text Embedding Benchmark (Muennighoff et al., 2023).
To improve retrieval performance, advanced finetuning techniques applied to individual retrieval systems (Zhao et al., 2023) or the enhancement of model components (Zhao et al., 2021) emerge as a prominent area of investigation. While finetuning approaches have demonstrated efficacy in this domain, they commonly require substantial resources for training, posing a challenge to realworld adoption.
Combining results from multiple IR systems can be quite effective, as demonstrated in the work of Cormack et al. (2009), Reciprocal Rank Fusion (RRF) is a technique that utilizes ranking information to rerank the overall ranking, which has become the industry standard for combining multiple ranking systems. Although the implementation of RRF is straightforward, a notable limitation arises due to its failure to leverage output scores from individual retrievers. Modern techniques have been developed to overcome this limitation; FlexNeuART (Boytsov and Nyberg, 2020) employs the Learning to Rank (LTR) framework to amalgamate scores from diverse retrievers into a unified score. However, the application of LTR in the QA retrieval setting requires a modification of the training process to avoid irrelevant samples. Moreover, the exploration and optimization of these techniques within the domain of question answering retrieval settings remain largely unaddressed.
In the work of Liang et al. (2021), a routing method for combining multiple retrievers in the QA setting is introduced. This model employs a routing classifier to select between a neural retriever and a traditional BM25 retriever. However, this method heavily relies on the score generated by BM25 and disregards ranking information from the neural model. Consequently, its performance is limited in modern question answering systems where the neural model typically outperforms BM25.
4750
We propose re-ranking techniques specifically trained to improve question answering retrieval performance. Our investigations involve combining two different types of model architecture (term weighting and neural networks). Additionally, we conducted experiments on two distinct styles of ReQA datasets to demonstrate that combining multiple models using this setup can leverage the advantages of both models through ranking loss objective training. Our findings show that the proposed methodology yields enhanced retrieval performance and ultimately improves the downstream QA task. Our approach outperforms the current state-of-the-art (Zhao et al., 2023) on ReQA SQuAD, surpassing all individual retrieval models, RRF, and the statistical routing strategy, yielding an average enhancement of 13.6% in the mean reciprocal rank (MRR) across datasets.

heading: 2 Related Work

heading: Massive Text Embedding Benchmark
text: Massive Text Embedding Benchmark (MTEB) proposed by Muennighoff et al. (2023) offers comprehensive insights and evaluations across a wide range of modern text embedding models. This benchmark is particularly crucial for understanding the out-of-the-box performance of various models. The evaluation incorporated established text embedding models available on the Hugging Face platform (Wolf et al., 2020). The assessment encompassed retrieval tasks across 15 distinct datasets, each representing diverse domains and dataset characteristics. The evaluation of re-ranking tasks was carried out using four specifically curated reranking datasets, providing a robust assessment of performance in real-world applications.
As shown in Table 1, GPT Sentence Embeddings for Semantic Search (SGPT) (Muennighoff, 2022) is one of the top performing models in retrieval tasks. Note that sentence embeddings constructed by simply averaging Global Vectors for Word Representation (GloVe) (Pennington et al., 2014) outperforms BERT (Devlin et al., 2019) and OpenAI’s Ada Similarity. This observation underscores the challenges inherent in retrieval tasks, where mainstream text embedding models still exhibit a deficiency in text retrieval proficiency.
In the context of re-ranking tasks, MPNET (Song et al., 2020) has demonstrated superior performance, highlighting the challenges in achieving a universally best solution in the current landscape.
Notably, a model exhibiting proficiency in retrieval tasks may not necessarily excel in re-ranking tasks.

heading: Model Retr. Rerank.

heading: Retrieval Question Answering
text: Retrieval question answering (ReQA) tasks are proposed in works such as Ahmad et al. (2019) and Chen et al. (2017). They encompass a type of information retrieval system that takes a question as input and responds with the most relevant sentence containing the answer. This task distinguishes itself from traditional Question Answering (QA), where the paragraph containing the answer is provided upfront, leading to notable achievements in machine learning systems, some of which surpass human performance (Rajpurkar et al., 2016). QA retrieval systems typically rely on either neural or term weighting methods. Recent studies, such as SPARTA (Zhao et al., 2021), exhibit significant improvements in retrieval performance by adhering to the concept of tokens-level matching from traditional IR systems. However, the exploration of QA retrieval systems that incorporate hybrid retrieval mechanisms remains relatively limited.

heading: Hybrid Method for Evidence Retrieval for Question Answering
text: Previous attempts to combine a term weighting model with a neural-based model simply involve routing between retrieval results from each model. This is achieved by training a routing classifier that determines whether to use the traditional IR method (BM25) or the neural-based method (USEQA) based on statistics derived from the training dataset (Liang et al., 2021). The study demonstrates that the hybrid approach outperforms individual retrieval strategies in ReQA SQuAD and ReQA NQ. However, this methodology has limitations in its support for combining multiple neural models, the integration of more than two models, and heavily relies on BM25. These limitations may deter real-world adoption, where there are numerous retrieval models specialized in their own
areas. Notably, USE-QA (Yang et al., 2020), employs fine-tuning techniques to tackle an asymmetric matching task present in QA retrieval. Combining the specialized abilities of these models could lead to better performance.
Thus, we emphasized an approach enhancing model retrieval performance through model-level combination that aims to capitalize on the strengths of multiple model types. Our objective is to present a robust methodology that can effectively combine state-of-the-art or fine-tuned models for better performance improvement.

heading: 3 Proposed Method
text: To construct a cohesive framework for merging multiple sets of ranking information, we adopt a re-ranking approach. We have designated a neural-based model as the primary retriever, complemented by BM25 as a supporting model. This choice is based on the observed consistency of the neural-based model, which demonstrates superior performance on R@10. This metric is particularly crucial for assessing the upper limits of performance gain in re-ranking performance. The rankings obtained from the main retriever serve as the initial rankings, subsequently augmented by incorporating scores derived from the support retriever. These concatenated scores are then fed into the re-ranker model to predict the final rankings. The overall process is illustrated in Figure 1.
The frameworks consist of two stages: 1) Retrieval Stage: off-the-shelf retrievers are used in order to generate a candidate pool. We select top candidate retrievers from MTEB benchmarks within the retrieval and re-ranking category. The performance is shown in Table 2.
2) Re-ranking Stage: a re-ranking network is used to construct the final ranking from the candidate pool.

heading: 3.1 Retrieval Stage
text: Our approach focuses on incorporating multiple QA retrieval models. These models have demonstrated significant performance improvements in question answering retrieval tasks compared to base retrieval models. To maximize the applicability of our approach, we have chosen widely recognized and publicly accessible models. In accordance with Muennighoff et al. (2023) results, we also included SGPT-5.8B-msmarco and MPNET. These models represent the current state-of-the-art publicly accessible models in the domains of retrieval and re-ranking tasks, respectively.

heading: Model MRR R@1 R@5 R@10
text: BM25 Retriever The token statistics for BM25 in the original ReQA paper were computed over the first 10,000 questions of each dataset. This led to significant performance losses, as shown in Table 3. In our implementation of BM25, we have made modifications that include the following key distinctions:
1) The statistics are derived from all documents within the training set, instead of solely utilizing the first 10,000 questions.
2) Our BM25 model is constructed using the gensim library (Rehurek and Sojka, 2011) with default parameters.
The original ReQA paper provided the rationale behind using only the initial 10,000 questions as to reduce the scoring time. We found that there is no discernable disparity in inference time when comparing the utilization of all question statistics versus using only the first 10,000 statistics for constructing
BM25. Consequently, we recommend adopting our approach to implement the BM25 version instead, which yields a notable increase in performance. It is worth mentioning that the training and inference duration of our model remains unaffected by these variations in BM25 configurations.

heading: 3.2 Re-ranking Stage
text: To effectively integrate ranking information derived from multiple retrieval systems, we opted for a neural-based architecture with a pairwise learningto-rank approach. The method is motivated by two primary factors.
1) QA retrieval models are initially pre-trained with a specific focus on enhancing the ranking of question-answer document pairs. Consequently, applying the pointwise method to train the re-ranker model would lead to redundant training.
2) The unavailability of comprehensive ranking in the question-answer retrieval dataset poses limitations on our ability to fully utilize the advantageous attributes of listwise methods.
Architecture The re-ranker model adopts the RankNet (Burges et al., 2005) architecture, which comprises of two layers of Siamese feed-forward neural network. (Koch et al., 2015) The network consists of 10 hidden units with leaky ReLU (Maas et al., 2013) and a single sigmoid output node.
Training Hyperparameters We optimized the re-ranker network using the Adam optimizer, (Kingma and Ba, 2015) using a learning rate of 1e− 3, 1024 batch size, and 100 epochs for every model setting.
Data Preparation The training features are derived from the top 64 ranking scores of the bestperforming retrieval model during the retrieval stage (main retriever). Subsequently, the scores from support models are computed and concatenated to form a set of re-ranking features. For the term weighting model, we utilize the scoring output directly from the model. For the neural-based
model, the score is generated by calculating the cosine similarity between the encoded query and the candidate document vectors.
We chose a number of re-ranked documents (k) equal to 64 to facilitate a direct comparison with the routing method. We experimented on how the value of k affects the performance and inference time. The findings indicate that augmenting the value of k beyond 16 yields only marginal improvements in MRR and is not worth trading off with prediction speed. Detailed experimental outcomes are presented in Figure 3 and Figure 4, as outlined in the Appendix.
Training Objective The pairwise LTR framework objective involves training the model to produce a value closer to one when it determines the ranking of a particular document to be higher than another. Conversely, it generates a value closer to zero for the opposite scenario. In the event of a tied situation, the conventional RankNet framework designates a target value of 0.5. Since the ReQA dataset has a lot of non-target documents, selecting random pairs will usually result in a tie. To customize the training objective for ReQA tasks, our approach prioritizes sentences containing the correct answer over those that do not. We have modified the conventional RankNet learning-to-rank algorithm by excluding the training sample with a target value of 0.5. The reduction in unnecessary training pairs resulted in a significant decrease in training time while simultaneously enhancing performance.
Inference Phase During the inference phase, the re-ranker model generates ranking information for each candidate document pair to determine which document should rank higher {(Di > Dj), (Dj > Dk), (Di > Dk)}. The final ranking is derived using a topological sorting algorithm {Di > Dj > Dk}. In cases where the outcome fails to yield a conclusive ranking, the original results from the main retriever are used as the final ranking. All operations are performed using the networkx library (Hagberg et al., 2008).

heading: 4 Results
text: We assess different retrieval models, including BM25, USE-QA, SGPT-5.8B-msmarco and MPNET-QA (Song et al., 2020). Our baselines include average scoring, thresholding, and Rank Reciprocal Fusion (RRF), a technique commonly employed within modern multi-retriever systems.
We compare our proposed approach mainly on the ReQA SQuAD and ReQA NQ, aligning with the original ReQA paper. Additionally, we conduct experiments on seven question answering datasets from MTEB to assess its effectiveness in contemporary retrieval tasks.
The Stanford Question Answering Dataset (SQuAD) (Rajpurkar et al., 2016) consists of questions on a set of Wikipedia articles, whereas Natural Questions (NQ) (Kwiatkowski et al., 2019) involves user questions issued to Google search, with corresponding answers sourced from Wikipedia by annotators. These two datasets represent distinct characteristics related to lexical overlap between questions and contexts, with SQuAD representing a low-overlap scenario while NQ exhibits the opposite. The publicly available version of both datasets consists of two partitions: a training set and a development set. The number of questions and documents is shown in Table 4.
The development set was used as the evaluation set for both datasets. We focused on the sentencelevel retrieval task, following the method outlined in the original ReQA paper. However, our implementation of BM25 included the complete vocabulary of the training documents, deviating slightly from the original ReQA paper which only used the first 10,000 questions. This difference resulted in better performance for our BM25 implementation. See "BM25 Retriever" in Section 3.1 for additional details.

heading: Dataset Question Document
text: ReQASQuAD (train) 87,599 91,707 ReQASQuAD (dev) 11,426 10,250 ReQANQ (train) 74,097 239,013 ReQANQ (dev) 1,772 7,020
Table 4: The number of questions and candidate sentences on the ReQA SQuAD and ReQA NQ datasets.
We use the Mean Reciprocal Rank (MRR) and Recall at k (R@k) as evaluation metrics. The experiments were replicated five times, and the mean value is reported. We performed significant testing between routing and our method, which showed significant improvements in all settings (one sample t-test, with p < 0.05). See Appendix for summary statistics of the results.
Table 5 presents the performance of both individual and combined models. Our model demonstrates better performance across all datasets and evaluation metrics, particularly in ReQA SQuAD where BM25’s performance is comparable to those of the neural-based models. Our approach exhibits a significant improvement compared to individual retrieval methods, RRF and routing strategies. Our model (SGPT+BM25) outperforms MPNET-QA by 48.5%, USE-QA by 22.6%, BM25 by 21.6%, SGPT-5.8B-msmarco by 4.1%, and routing techniques on the same model combination by 7.0%.
On the ReQA NQ dataset, the performance of BM25 is significantly inferior compared to the
neural-based systems. Our approach demonstrates the capability to effectively integrate ranking information, resulting in an average performance uplift of 2.6% from individual neural models. This highlights the efficacy of our approach, even in situations where less potent models are combined with their more superior counterparts.

heading: Common Ensemble Techniques
text: We assess standard ensemble techniques including 1) Average scoring, re-rank outputs based on normalized average scores from each retriever. 2) Thresholding with a predefined criterion set at 0.5, routing between main and support retriever. Our results indicate that common ensemble techniques demonstrate inferior performance compared to our approach across all settings, as shown in Table 6.

heading: Neural Model Combinations
text: Our experiments were assessed within the framework of term weighting-neural combination in direct comparison with the routing method. We also included the neural-neural combination experiments. The results demonstrate the efficacy of our approach. The MPNET-QA and SGPT-5.8Bmsmarco combination demonstrated notable improvements, surpassing the best performer in the main experiments on ReQA NQ. This is consistent with our prior observations that the fusion of competitive models yields greater performance gains. The results of the model combination are shown in Table 7.

heading: Overall Question Answering Performance
text: In this experiment we investigated how enhancing retrieval performance translates to the overall question answering system performance. We integrated retrieval systems with the BERT reader model and found that an increase in retrieval performance benefits the overall QA system performance in every setting. We reported answer quality using two metrics, including the Exact Match score (EM), where the answer is strictly required to match the label, along with the less strict ROUGE-L score (Lin, 2004). The results show a strong correlation coefficient of 0.865 between R@1 and ROUGE-L scores, confirming that enhancements in retrieval performance positively impact overall QA performance. However, there is a substantial gap between the best retrieval method and the oracle retriever, showing the potential gain for further improvements. The results are presented in Table 8.

heading: Effect of Sample Mining Technique
text: We trained the the combination model of BM25 and SGPT-5.8B-msmarco using the original pairwise learning-to-rank approach, and compared it to our modified version which includes sample mining. Table 9 shows that by reducing unnecessary training pairs, there is a substantial 97.0% reduction in training samples. At the same time, it yields an average 0.6% marginally improvement in performance across datasets.

heading: Low-resource Setting
text: To emulate a low-resource scenario, we ran an experiment using only 10% of the training data. The findings demonstrate that the method can work even in scenarios with limited data. This suggests that our method also works on a smaller training size, as shown in Table 10.

heading: Combining More Than Two Models
text: Our framework also supports extensions beyond two models. We conducted experiments to combine three retrieval models. The results indicate that combining a model with a different specialization leads to a substantial performance boost, as demonstrated by the combination of SGPT-5.8Bmsmarco, MPNET-QA, and BM25. This combination surpasses the current state-of-the-art in ReQA SQuAD, yielding an improvement from 0.801 to 0.823 on the MRR metric (Zhao et al., 2023). The results are shown in Table 11.

heading: Performance on Modern QA Datasets
text: We performed experiments using QA datasets from MTEB to assess the efficacy of our methods in modern retrieval tasks. We incorporated MPNET-QA and SGPT-125M-msmarco, utilizing the development set for training and the test set for evaluation. The results presented in Table 12 show the effectiveness of our approach across datasets.

heading: End-to-end Evaluation on RAG
text: We conducted Retrieval-Augmented Generation (RAG) experiments to assess the end-to-end performance of our method using the mistral-7b-instructv0.2 (Jiang et al., 2023) as a reader model. The outcomes were evaluated based on METEOR (Banerjee and Lavie, 2005), BERT-F1 (Zhang et al., 2020), and ROUGE-L scores. Our systems demonstrate performance improvements, as shown in Table 13.

heading: Multilingual Settings
text: We conducted a non-English language experiment to assess the performance of our method when utilizing a multilingual model setup. We performed the experiment in the context of Thai language, utilizing the iapp-wiki-qa-squad datasets (Viriyayudhakorn and Polpanumas, 2021) which adhere to the SQuAD styles. Our modifications to support the Thai language include:
1) Changing USE-QA, which only supports English language, to Multilingual Universal Sentence Encoder Question Answering (mUSE-QA).
2) Substituting the NLTK (Bird and Loper, 2004) tokenizer with the PyThaiNLP (Phatthiyaphaibun et al., 2023) newmm tokenizer.
3) Using BM25 as the main retriever instead of the neural model due to a significant gap in performance on the R@10 metric.
The result shows that our method outperforms routing techniques. Notably, routing strategies only selected the ranking from the BM25 models, yielding a 0% uplift in performance. The results are shown in Table 14.

heading: Model MRR R@1 R@10

heading: 5 Conclusion
text: In this paper, we propose an effective method to combine term weighting and neural-based ReQA systems. Our approach involves training re-ranker neural networks using a modified pairwise reranking objective. Through empirical evaluations, the combined system exhibits a significant performance improvement over other combining strategies, particularly when confronted with significant performance disparities among the individual models. It is noteworthy that this technique is not restricted to a specific type or quantity of retrieval models. Our method focuses on improving the QA system by enhancing the retrieval which we identified as a bottleneck. Further investigation into improving the overall performance by directly optimizing the reader along with the retrieval model might improve the overall pipeline.

heading: Limitations
text: In our work, we have focused on implementing our methodology within the context of the retrieval question answering task. Nonetheless, it is reasonable to anticipate that the application of our techniques to a broader retrieval task would yield favorable results. Our methodology necessitated the selection of a main retriever. This is helpful in reducing the computing load of the supporting retriever, yet it can introduce a cap on the final performance. In the future, we may try to eliminate the need for main retrieval model selection. We also focus our experiment on techniques that do not require full-weight updates. Nevertheless, our approach can complement other full-weight update fine-tuning techniques to further enhance performance. Lastly, our model’s computational cost scales with the number of re-ranking indexes fed through the re-ranker (k), as the process requires the prediction of k2 pairs of attributes to determine the final ranking. This aspect may present challenges when deploying the model in situations with a tight compute budget.

heading: Acknowledgements
text: This work is under the joint research initiative between KASIKORN Business-Technology Group (KBTG) and the Faculty of Engineering, Chulalongkorn University. We would like to thank KASIKORNBANK PUBLIC COMPANY LIMITED (KBank) and KBTG for supporting the research and providing computational resources to conduct the experiments.

heading: A Effect of the Amount of Re-ranked Documents
text: We conducted an experiment to study the effect of the number of re-ranked documents on the retrieval performance (Figure 3) and runtime (Figure 4). The processing was conducted on a standalone machine featuring Intel(R) Core(TM) i7-7700 CPU @ 3.60GHz, with no GPU acceleration.
To measure the latency, we ran our trio model with k = 16. The re-ranker network adds a latency of 0.0133 seconds per query on average across datasets.

heading: B Correlation Between Retrieval and Overall QA Performance
text: We assess the correlation between retrieval and overall QA performance. The findings reveal a strong correlation across metrics, suggesting that improvements in retrieval performance have a positive influence on overall QA performance.

heading: Future Work
text: In the future, we may try to eliminate the need for main retrieval model selection. We also focus our experiment on techniques that do not require full-weight updates. Nevertheless, our approach can complement other full-weight update fine-tuning techniques to further enhance performance. Lastly, our model’s computational cost scales with the number of re-ranking indexes fed through the re-ranker (k), as the process requires the prediction of k2 pairs of attributes to determine the final ranking. This aspect may present challenges when deploying the model in situations with a tight compute budget