PRP-Graph: Pairwise Ranking Prompting to LLMs with Graph Aggregation for Effective Text Re-ranking (2024)

Abstract: Pairwise Ranking Prompting (PRP) demonstrates impressive effectiveness in zero-shot document re-ranking tasks with large language models (LLMs). However, in the existing methods, PRP only outputs the same label for the comparison results of different confidence intervals without considering the uncertainty of pairwise comparison, which implies an underutilization of the generation probability information of LLMs. To bridge this gap, we propose PRP-Graph, a novel pairwise re-ranking approach, based on a refined scoring PRP unit that exploits the output probabilities of target labels to capture the degree of certainty of the comparison results. Specifically, the PRP-Graph consists of two stages, namely ranking graph construction and ranking graph aggregation. Extensive experiments conducted on the BEIR benchmark demonstrate the superiority of our approach over existing PRP-based methods. Comprehensive analysis reveals that the PRPGraph displays strong robustness towards the initial ranking order and delivers exceptional re-ranking results with acceptable efficiency. Our code and data are available at https: //github.com/Memelank/PRP-Graph.

heading: 1 Introduction
text: Large Language Models (LLMs) such as GPT3 (Brown et al., 2020), FlanT5 (Wei et al., 2021), and PaLM (Chowdhery et al., 2023) have demonstrated exceptional performance across a diverse array of natural language processing tasks within the zero-shot paradigm (Agrawal et al., 2022; Kojima et al., 2022). Consequently, these LLMs have been adapted for zero-shot document ranking tasks, showcasing remarkable capabilities (Liang et al., 2022; Pradeep et al., 2023; Sun et al., 2023). The methodologies utilized LLMs in zero-shot ranking tasks focus on prompting the LLMs to provide relevance estimations for a query along with candidate documents (Liang et al., 2022; Qin et al., 2023).
Notably, recent research has shown that Pairwise Ranking Prompting (PRP) is effective for zeroshot re-ranking with moderate-sized, open-sourced LLMs and can produce state-of-the-art re-ranking performance with simple prompting and scoring mechanism (Qin et al., 2023). PRP involves building a pairwise prompting to determine which of two documents is more relevant to query with the help of LLMs, then using the PRP as a basic scoring unit to serve different ranking mechanisms.
Despite their effectiveness, we posit that existing PRP does not fully capitalize on the potential of LLMs. Firstly, current practices utilize PRP to simply ascertain the more relevant document relative to the query, disregarding a more granular analysis of generation probabilities, which represent the LLM’s confidence in its judgment. Secondly, the inherent sensitivity of LLMs to the text orders in prompt (Lu et al., 2021) has led to the verification of consistency by alternating the pairwise order of documents within the prompt and consulting the
5766
LLM twice. Only consistent outputs from both inquiries are valid, which does not take full advantage of the capabilities of LLMs. Finally, the comparison selection strategies in use, which are derived from sorting algorithms, lack the flexibility to enhance ranking quality through increased comparisons and are highly sensitive to the sequence order of input documents.
To this end, we propose scoring Pairwise Ranking Prompt (PRP) unit and a novel pairwise approach, termed PRP-Graph, building upon the scoring PRP unit’s performance. As illustrated in Figure 1, unlike the current PRP unit (Qin et al., 2023), which yields discrete judgments, our scoring PRP unit captures more granular correlation information for each pairwise order in prompt through the generation probability of judgment labels by LLMs, thus effectively utilizing the generative capability and order sensitivity of LLMs. PRP-Graph operates in two phases. The first phase continuously identifies document pairs for comparison, representing them as vertices connected by bidirectional edges within a ranking graph. The weights of vertices and edges are iteratively refined based on the scoring PRP unit’s outputs. The second phase aggregates these signals across the ranking graph to produce a cohesive final document ranking that encapsulates the entire graph’s sorting information.
We conduct experiments to evaluate our proposed PRP-Graph on Flan-T5 models (Chung et al., 2022). Our results on BEIR (Thakur et al., 2021) indicate that PRP-Graph surpasses existing PRPbased zero-shot re-ranking techniques. Further ablation studies confirm the effectiveness of individual components within our method, and our analyses demonstrate the influence of comparison rounds on PRP-Graph’s performance as well as its robustness against initial ranking order.
Our contributions are threefold: 1) We introduce the PRP-Graph, a novel pairwise approach underpinned by an innovative scoring PRP unit that exploits the full potential of LLMs by leveraging the probability scores from pairwise ranking prompts. 2) Benefiting from the graph-based nature of PRP-Graph, which relies less on the initial rankings produced by the first-stage retriever, it demonstrates a more effective interpolation with BM25 than current PRP-based methods. 3) Extensive experimental validation shows that our PRP-Graph method significantly outperforms contemporary approaches, establishing it as a superior choice for zero-shot document re-ranking tasks.

heading: 2 Preliminaries
text: As aforementioned, several recent text re-ranking approaches implement Pairwise Ranking Prompting (PRP) as an atomic ranking unit to extract specific textual labels via the generative capabilities or logit outputs of LLMs. This section provides an initial overview of the PRP construct, subsequently outlining the landscape of text re-ranking approaches that integrate the basic PRP unit.
Basic PRP unit. The prompt in PRP is shown in Figure 2(a), wherein it asks the LLM to answer which document is more relevant to the query. Akin to (Qin et al., 2023), we denote this PRP prompt text as u(q, di, dj) for a query q and two documents di and dj . As seen in Figure 2(b), the basic PRP unit, applied in existing work (Qin et al., 2023; Zhuang et al., 2023b), is only designed to obtain an answer to a question in the prompt. Specifically, there are two modes to obtain the answer, the first is generation mode wherein LLM directly generates the passage label ("Passage A" or "Passage B"), and the second is scoring mode wherein the loglikelihood of LLM generates the target label is used for a scoring comparison for the answer. As it is known that LLMs can be sensitive to text orders in the prompt (Lu et al., 2021), the order of these two documents within the prompt will be swapped and inquire the LLM twice, namely u(q, di, dj) and u(q, dj , di). Only if the output of two inquiries is consistent, the LLM’s judgment will have a clear preference as shown in Figure 2(b).
Text re-ranking via basic PRP unit. Given a set of N candidate documents, two principal approaches leveraging the PRP unit to achieve reranking have emerged: the exhaustive Allpair method and the sorting-based method (Qin et al., 2023). The Allpair approach applies the PRP unit to every possible document pair, calculating a cumulative score for each document based on pairwise preferences, yet it incurs a substantial O(N2) computational cost and does not assure the most effective result according to previous work (Qin et al., 2023). On the other hand, sorting algorithms such as Heapsort and Bubblesort utilize LLM-derived pairwise preferences for ordering documents. These methods improve efficiency (e.g. Heapsort operates at O(N logN) complexity), but their effectiveness is bounded by the reduced comparative scope and sensitivity to the initial ranking order (Zhuang et al., 2023b).

heading: 3 Method
text: Our PRP-Graph introduces a refined scoring PRP unit that leverages LLMs’ output probabilities for target labels, transforming these outputs into numerical scores for enhanced text ranking. The PRPGraph operates in two main stages: ranking graph construction and ranking graph aggregation. In the first stage, document pairs are selectively compared to form a ranking graph with documents as vertices linked by bidirectional edges, where the scoring PRP unit informs the iterative updating of weights. The second stage performs an aggregation of these weighted interactions to derive the final document ranking, fully encapsulating the sorting information across the graph.

heading: 3.1 Scoring PRP Unit
text: Although a scoring mode exists within the basic PRP framework, as outlined in Section 2, it employs log-likelihood merely to reach a comparative decision, without utilizing the full extent of the numerical information provided. In contrast, our new scoring PRP unit, illustrated in Figure 2(c), capitalizes on two probability values from the LLM outputs to gauge the degree of comparison between documents.
Mirroring the basic PRP, our scoring PRP unit also prompts the LLM twice with both u(q, di, dj) and u(q, dj , di), swapping the document positions. During decoding, a softmax function is applied to the LLM’s logit outputs. The resulting probability of generating label "A" serves as the score of "Passage A" over "Passage B" in relevance to the query q. For instance, given the input u(q, di, dj), sj→i represents the score that document di is more rele-
vant compared to dj . Hence, the scoring PRP unit yields a pair of scores reflecting relative relevance as described in Eq. 1, in contrast to the basic PRP unit’s singular comparative outcome.
sj→i = softmax(LLM(u(q, di, dj)))["Passage A"]
si→j = softmax(LLM(u(q, dj , di)))["Passage A"] (1) wherein ["Passage A"] corresponds to the probability that LLM generates "A", as "Passage" serves as conditional probability or the input to the Decoder.

heading: 3.2 Ranking Graph Construction
text: Addressing the challenge of quantifying the significance of pairwise comparison outcomes in a full ranking list, we draw parallels from the world of international chess tournaments. In these tournaments, players partake in multiple rounds of one-on-one matches, with the Swiss-system (Csató, 2013) tournament being a prevalent format. Here, players are matched in each round according to their current standings, and over successive rounds, they accumulate points that contribute to their final rankings. Such rules can allow Swiss-system to achieve a reasonable player ranking list with limited rounds of matches being played. Inspired by this system, we develop a ranking graph construction strategy, encapsulated in Algorithm 1, which mirrors the dynamic pairing and point accumulation process of the Swiss-system tournament to discern the importance of pairwise comparisons for document ranking.
Given a query q and an initially retrieved topN document ranking list D = [d1, ..., dN ] using a retrieval algorithm like BM25 (Robertson et al., 2009), we aim to construct a ranking graph G to
collect pairwise ranking relationships between documents. This is carried out by performing pairwise comparisons of documents, facilitated by our scoring PRP unit that leverages preference scores from an LLM. For this purpose, we conduct R rounds of comparisons across the ranking list D. Each document’s ranking score in the r-th round is denoted as Sr = [Sr1 , ..., SrN ], with the initial scores set as S0 = [1, 1 − 1N , 1 − 2N , . . . , 1N ] to reflect their starting position in the ranking list.
In each round, a document di is selected in sequence from the ranking list D, and paired with the closest subsequent document dj within the range [di+1, . . . , dN ] that has neither been compared with di in any previous round nor within the current round r. Upon selecting a pair of documents di and dj , they are input into the scoring PRP unit, as described in Eq. 1, to obtain the preference scores, denoted as sj→i and si→j , from the LLM. Consequently, two edges are constructed within the graph G, connecting dj to di and di to dj , each weighted by sj→i and si→j respectively.
After the pairwise comparison between documents di and dj in the r-th round, we utilize the preference scores, sj→i and si→j , derived from the LLM to adjust the vertex weights, Sri and S r j . The update adheres to the predefined score accumulation rule presented in Eq. 2.
Sri = S r−1 i + sj→i × Sr−1j r
Srj = S r−1 j + si→j × Sr−1i r
(2)
The rationale for this score updating process is predicated on the notion that sj→i represents the score secured by the pre-ranked document di against its subsequent counterpart dj . To account for the varying challenges posed by different documents,
the difficulty level associated with dj , denoted as Sr−1j /r, is incorporated into the calculation. This quantity reflects the normalized weight of dj from the previous round and scales the points added to di based on the relative difficulty of the comparison. In cases where the PRP scores are equivalent, the adjustment rewards the documents proportionally to the strength of their adversaries, thereby ensuring a more nuanced representation of each comparison’s contribution within the global ranking context.
After each comparison round, the document list D is updated and sorted by the new scores Sr from highest to lowest. Once all R rounds are complete, we create the final ranking graph G. In this graph, documents are the vertices, and the comparison scores sj→i and si→j are the weights of the two directed edges, representing the pairwise ranking relationships, between them. These edge weights are fixed after being set, since each document pair is compared only once. Figure 3 illustrates how this ranking graph is constructed.

heading: 3.3 Ranking Graph Aggregation
text: In the development of a ranking graph G for a particular query q across a suite of documents D, initial ranking outcomes are derived from the cumulative scores SR. However, these results are not yet refined by the structural nuances of the graph, which are essential for weighting each document’s ranking score more effectively. To address this, we introduce a graph-based aggregation method, conceptualized from the weighted PageRank algorithm (Xing and Ghorbani, 2004). Unlike the traditional application of PageRank, which assigns importance to web pages based on the quantity of links, our adaptation focuses on the interplay of edge weights in the graph to revise the scores of
Algorithm 1 Ranking Graph Construction Input: a query q, a top-N document ranking list D = [d1, . . . , dN ] for q, the scoring PRP unit based on LLM Parameter: the number of documents N , the total rounds of comparisons R, the scores of D in r-th round Sr = [Sr1 , . . . , SrN ] initialized with S0 = [1, 1− 1N , 1− 2N , . . . , 1N ] Output: a ranking graph G on D for q
1: for r from 1 to R do 2: for i from 1 to N do 3: if di has not been compared in r-th round yet then 4: for j from i+ 1 to N do 5: if dj has not been compared in r-th
round and has also not been compared with di before then
6: break 7: end if 8: end for 9: sj→i, si→j = Scoring PRP Unit (q, di,
dj), as in Eq. 1 10: Add two edges between di and dj into
G with sj→i and si→j as the weights of edges
11: Sri = S r−1 i + sj→i × Sr−1j r 12: Srj = S r−1 j + si→j × Sr−1i r 13: end if 14: end for 15: Re-sort D by Sr from highest to lowest 16: end for 17: return the ranking graph G
the vertices according to Eq. 3. Within the constructed ranking graph G for query q, each document is represented as a vertex i, corresponding to document di. The vertex value s(i), indicative of the aggregated score of di, is initially set by its BM25 score to incorporate precise matching signals. Subsequently, vertex values are iteratively recalculated following the order of descending BM25 scores as per Eq. 3.
s(i) = df×[ ∑
j∈In(i)
s(j)∑ k∈Out(j)wjk ×wji]+ 1− df N
(3) where N denotes the total number of vertices within G, df is the damping factor, In(i) signifies the set of vertices with edges directed towards
vertex i, Out(j) includes vertices with edges emanating from vertex j, and wij is the weight of the edge linking vertex i to vertex j.
The iterative process persists until the fluctuation in vertex ranking scores is below a specified threshold δ, whereupon the results are deemed to have stabilized. The final vertex scores then serve as the basis for re-ranking the documents in D. In line with the methods outlined in (Wang et al., 2021), our approach, inherently semantic and generative, is best utilized in conjunction with a precision-oriented keyword-matching method like BM25. This hybridized strategy is designed to achieve a balance between semantic coherence and keyword matching. In situations where a development set is not available, we determine the interpolation ratio through cross-validation.

heading: 4 Experiments

heading: 4.1 Experimental Setup
text: Benchmark and metric To evaluate the effectiveness of our PRP-Graph on zero-shot text reranking, we carry out a series of experiments on BEIR benchmark (Thakur et al., 2021). BEIR is a well-established document ranking benchmark composed of multiple datasets, which cover a wide range of domains, like news and medicine, and involve various retrieval tasks, such as question answering, fact-checking, and entity search. The relevance labels of the documents include binary and multi-level. The statistics of the datasets are displayed in Table 1. In the evaluation, each approach is tasked with re-ranking 100 documents initially retrieved by a first-stage BM25 retriever (Robertson et al., 2009). The performance of various approaches is assessed using NDCG@10, which serves as the official evaluation metric for the employed BEIR benchmark.
Baselines We consider two prevailing PRP-based methods as our comparative baselines: Allpair and Sorting-based comparisons as in (Qin et al., 2023; Zhuang et al., 2023b). For sorting-based comparisons, we have selected HeapSort and BubbleSort algorithms, which are founded on time-honored sorting techniques and have been validated for their exceptional effectiveness (Zhuang et al., 2023b). Our implementation replicates the experimental framework employed by (Zhuang et al., 2023b). For a fair comparison, we interpolate both the baseline methods and our PRP-Graph with sparse retrieval model BM25, and the interpolation parameters are determined by ten rounds of crosschecking, which determines parameters on a separate validation set in each fold.
Implementation details For the retrieval of the top-100 candidate documents, we employ the Pyserini Python library (Lin et al., 2021) with default settings to utilize the BM25 model. In line with previous works (Qin et al., 2023; Zhuang et al., 2023b), we conduct evaluations on Flan-T5 models of three sizes 1, including Flan-T5-Large (L) with 780M parameters, Flan-T5-XL with 3B parameters, and Flan-T5-XXL with 11B parameters, to explore the effectiveness of our method on LLMs with different parameter sizes. Besides, we use the same prompt following the existing literature, as introduced in Section 2, for all PRP-based re-ranking methods. Regarding the number of comparison
1https://huggingface.co/docs/transformers/ model_doc/flan-t5
rounds R performed at the ranking graph construction step, we mainly report the evaluation results when R = 10, 20, and 40, and put the exploration of the setting of R for different comparison rounds in the Analysis Section 4.3. For the ranking graph aggregation step, the damping factor df in Eq. 3 is conventionally set to 0.85, and the convergence threshold δ is set to 1e-6.

heading: 4.2 Results
text: Table 2 presents an empirical evaluation of different PRP-based re-ranking approaches on the BEIR benchmark. In the table, HeapSort, BubbleSort, and Allpair are the existing PRP-based re-ranking methods as baselines. We report the results of our PRP-Graph method for R = 10, R = 20, and R = 40, denoted as PRP-Graph-10, PRP-Graph-20, and PRP-Graph-40, respectively. To compare the results before and after interpolation with BM25 results, we also include the average uninterpolated results, labeled as "w/o IB".
PRP-Graph demonstrates a clear advantage over current PRP approaches in different model sizes. On three different sizes of models Flan-T5L, Flan-T5-XL, and Flan-T5-XXL, PRP-Graph-10 with a small number of comparison rounds can perform on par with the best baseline Allpair, while PRP-Graph-40 with more comparison rounds outperforms Allpair by margins of +1.4, +0.9, and +0.9 in terms of average NDCG@10, respectively. An exploration of the number of comparison rounds R in the ranking graph construction step can be found in Section 4.3.
PRP-Graph demonstrates a more effective interpolation with BM25. As shown in Table 2, the interpolation with BM25 provides a certain improvement for all PRP methods. However, compared to all baselines, PRP-Graph demonstrates a more effective interpolation with BM25. For example, PRP-Graph-40 obtains a +3.4 gain, while HeapSort, BubbleSort, and Allpair obtain +1.2, +0.1, and +2.3 at Flan-T5-XXL model after the interpolation. This enhancement can be attributed to the graph-based nature of PRP-Graph, which relies less on the initial rankings produced by BM25. In contrast, HeapSort and BubbleSort are dependent on the BM25 rankings throughout their entire sorting process. This also explains the outstanding performance of Allpair among baselines, since it does not take initial ranking into account.

heading: 4.3 Analysis
text: Ablation study. Herein, to verify the effectiveness of different components or settings in our PRP-Graph method, we design the following experiments as shown in Table 3. During the construction of the ranking graph, "w/o nearest selection" denotes the document comparison pairs in each round are randomly selected during the graph construction rather than finding the nearest documents in the ranking list (as in lines 4 through 8 in Algorithm 1), which exhibits a discernible reduction of effectiveness in NDCG@10; "w/o bidirectional edge (larger)" and "w/o bidirectional edge (subtract)" denote unidirectional edges rather than bidirectional edges are conducted in graph G using the larger weight and the subtraction between two weights, respectively. Their performance drops substantially, which indicates the benefit of our choice of bidirectional edges to build the ranking graph. As for the ranking graph aggregation, "w/o exact matching initialization" denotes initializing the weights of vertices in graph G with the final score SR from the graph construction phase rather than the BM25 exact matching scores; "w/o ranking
aggregation" denotes the results only after the construction of the ranking graph without the aggregation step, namely using the ranking scores in SR interpolated with BM25. Both suffered the same degree of performance decline.
The impact of comparison rounds. During the ranking graph construction stage, we performed R rounds of pairwise comparison of candidate documents. To understand the influence of the number of R on the ranking efficacy, we conduct a comprehensive empirical study, the results of which are encapsulated in Figure 4. This figure illustrates the ranking performance (average NDCG@10) on the BEIR benchmark when subjected to varying comparison round configurations for different model sizes. We also mark the results of Allpair, the best-performing baseline, in the figure to illustrate how many rounds of comparison PRP-Graph needs to perform to outperform it. Our investigation revealed that there is a clear trend demonstrating an increase in the number of comparison rounds, progressing from 2 up to around 15. After 15 rounds, the ranking effect exhibits an improvement in oscillation with the increase of R.
The trade-offs between effectiveness and efficiency. As depicted in Table 4, HeapSort is efficient and presents acceptable trade-offs among baselines. BubbleSort, on the other hand, faces challenges in both efficiency and effectiveness domains. Allpair, while demonstrating superior effectiveness compared to the two methods mentioned earlier, incurs significant overhead and still has room for improvement. In contrast, our PRP-Graph significantly surpasses these methods by achieving a better balance between ranking effectiveness and computational efficiency. Notably, PRP-Graph-10 provides a +0.6 improvement in NDCG@10 over the leading Allpair baseline while concurrently reducing the computational overhead by 90.0%. Furthermore, for scenarios where ranking effectiveness is paramount, the PRP-Graph provides flexibility to increase the number of comparison rounds, as evidenced by PRP-Graph-20 and PRP-Graph-40, which yield even more precise ranking outcomes.
The sensitivity to the initial ranking. The effectiveness of sorting methods is notably influenced by the order of the input rankings (Qin et al., 2023; Zhuang et al., 2023b). To delve into this aspect concerning our approach, we explore three types of orderings of the initial document list. Our results on Flan-T5-L are illustrated in Figure 5(a). Specifically, "BM25" denotes the ranking order retrieved
by a first-stage BM25 retriever, "RandomBM25" denotes the randomly shuffled BM25 ranking, and “InverseBM25” denotes the inverted BM25 ranking. It is observed that diverse initial ranking orders hurt both the sorting method and our PRP-Graph. But PRP-Graph’s overall impact is smaller and still maintains the best results. We also show the uninterpolated results Figure 5(b). Compared with the results after interpolation, different initial rankings improved after interpolation. Our PRP-Graph-40 maintains a consistent distribution of results on both uninterpolated and interpolated results and has the best results for both "RandomBM25" and "InverseBM25" among the uninterpolated results. This further indicates the robustness of our method to the initial ranking order.
The ranking similarity between BM25 and reranking results. To further investigate why our method can interpolate with BM25 more effectively compared to sorting-based methods, we conducted ranking similarity experiments on the Flan-T5-L model, using Rank-biased Overlap (RBO) (Webber et al., 2010) as the metric. Table 5 shows the ranking similarity between different re-ranking methods (without BM25 interpolation) and initial BM25. BubbleSort has the most similar ranking result to the BM25 with a 0.96 RBO value but achieves the highest NDCG@10. Our PRP-Graph-40 exhibits the lowest similarity to the BM25 ranking but outperforms HeapSort in performance. These results indicate that sorting-based methods only make small adjustments based on the BM25 ranking but our method reconstructs the ranking relationship. Therefore, our graph-based approach is more orthogonal to the initial BM25 ranking and, hence benefits more from the interpolation.
The efficiency on TREC-DL datasets. The TREC Deep Learning 2019 (Craswell et al., 2020) and 2020 (Roberts et al., 2020) datasets are wellregarded benchmarks in the field of text re-ranking.
Table 6 shows the results on Flan-T5-L on DL19 and DL20, our PRP-Graph-40 achieves the best average results with and without interpolation. It is worth noting that the BubbleSort method performs poorly, which is inconsistent with its competitive results on the BEIR benchmark, indicating the instability of the sorting-based method.

heading: 5 Related Work
text: In the realm of zero-shot text re-ranking with LLMs, there are three main prompting approaches categorized broadly into Pointwise (Liang et al., 2022; Sachan et al., 2022), Listwise (Ma et al., 2023; Pradeep et al., 2023; Sun et al., 2023), and Pairwise (Qin et al., 2023; Zhuang et al., 2023b). Pointwise approaches rank candidate documents based on prompting LLMs for query-document pair relevance judgment, such as prompting LLMs to generate whether the provided query-document pairs are relevant (Liang et al., 2022; Zhuang et al., 2023a) and using the LLMs generation likelihood of query for the corresponding document as ranking score (Muennighoff, 2022; Sachan et al., 2022). Listwise approaches involve sorting a list of documents at once, wherein LLMs are prompted to generate a ranked list of document labels based on the relevance of their corresponding documents to the query. Current listwise approaches (Sun et al., 2023; Ma et al., 2023) use sliding window approaches to work within the input length limits of LLMs, involving re-ranking the candidate document window from the bottom of the entire candidate documents list and progressing upwards.
Pairwise approaches determine the relevance order of two documents for a query at a time, wherein LLMs are prompted with a query alongside a pair of documents to choose the label indicating which document is more relevant to the query, namely
Pairwise Ranking Prompting (PRP). Exiting methods for obtaining the final re-ranking through pairwise comparisons between candidate documents can be classified into aggregation and sorting methods. Allpair is a basic aggregation method (Qin et al., 2023), which conducts pairwise comparisons on all possible pairings within the candidate documents and aggregates the result of each comparison into the ranking score of the document. Other aggregation methods exist focusing on sampling from all possible pairings to increase efficiency (Mikhailiuk et al., 2021; Gienapp et al., 2022), but it hasn’t been used in LLMs yet. Sorting methods rely on pairwise comparisons as a basic comparison unit to run sorting algorithms, such as Heapsort and Bubblesort (Qin et al., 2023; Zhuang et al., 2023b). These algorithms make use of efficient data structures to compare document pairs selectively.
However, existing methods only utilize PRP to obtain the same label for the comparison results of different confidence intervals without considering the uncertainty of pairwise comparisons. In contrast, our proposed PRP-Graph, based on a new scoring PRP unit that quantifies the certainty of the pairwise comparisons from LLMs, exploits the potential of LLMs on zero-shot re-ranking.

heading: 6 Conclusion
text: In our work, we propose a novel PRP-Graph approach based on a refined scoring Pairwise Ranking Prompting (PRP) unit to promote the potential of LLMs on text re-ranking tasks. Benefiting from the consideration of the uncertainty of pairwise comparison in scoring PRP unit, the PRP-Graph builds a ranking graph via the comparison results of the scoring PRP unit and aggregates global signals of the ranking graph, surpassing existing PRP-based zero-shot re-ranking techniques on the BEIR benchmark. In future work, we plan to explore a more efficient ranking graph aggregation strategy.
Limitations
Firstly, in the ranking graph aggregation stage of PRP-Graph, we only use an aggregation method based on weighted PageRank, more other graph learning strategies could be included to extend the effectiveness of PRP-Graph, such as the HITS algorithm (Sherin et al., 1998). Secondly, in the ranking graph construction step of PRP-Graph, the number of comparison rounds R we explore is limited to 40 for efficiency reasons. Even though our analysis of
comparison rounds shows the effectiveness of PRPGraph, more rounds are still unexplored. Moreover, although some closed-source models (such as GPT3.5 and GPT-4 (OpenAI, 2024)) provide methods to access logit values, if the closed-source models do not provide any access to logit values, it undeniably constrains the implementation of our method. Finally, we conducted experiments on the Flan-T5 series of models in line with previous works (Qin et al., 2023; Zhuang et al., 2023b). We can also extend the experiments to a more diverse range of LLMs, such as LLaMA (Touvron et al., 2023), to evaluate our approach.

heading: Acknowledgements
text: This work is supported by the National Natural Science Foundation of China (62272439), and the Fundamental Research Funds for the Central Universities.

heading: Future Work
text: In future work, we plan to explore a more efficient ranking graph aggregation strategy. Limitations
Firstly, in the ranking graph aggregation stage of PRP-Graph, we only use an aggregation method based on weighted PageRank, more other graph learning strategies could be included to extend the effectiveness of PRP-Graph, such as the HITS algorithm (Sherin et al. , 1998). Secondly, in the ranking graph construction step of PRP-Graph, the number of comparison rounds R we explore is limited to 40 for efficiency reasons. Even though our analysis of
comparison rounds shows the effectiveness of PRPGraph, more rounds are still unexplored. Moreover, although some closed-source models (such as GPT3. 5 and GPT-4 (OpenAI, 2024)) provide methods to access logit values, if the closed-source models do not provide any access to logit values, it undeniably constrains the implementation of our method. Finally, we conducted experiments on the Flan-T5 series of models in line with previous works (Qin et al. , 2023; Zhuang et al. , 2023b). We can also extend the experiments to a more diverse range of LLMs, such as LLaMA (Touvron et al. , 2023), to evaluate our approach