FAA: Fine-grained Attention Alignment for Cascade Document Ranking
Year: 2023

Abstract:
Document ranking aims at sorting a collection of documents with their relevance to a query. Contemporary methods explore more efficient transformers or divide long documents into passages to handle the long input. However, intensive query-irrelevant content may lead to harmful distraction and high query latency. Some recent works further propose cascade document ranking models that extract relevant passages with an efficient selector before ranking, however, their selection and ranking modules are almost independently optimized and deployed, leading to selecting error reinforcement and sub-optimal performance. In fact, the document ranker can provide fine-grained supervision to make the selector more generalizable and compatible, and the selector built upon a different structure can offer a distinct perspective to assist in document ranking. Inspired by this, we propose a fine-grained attention alignment approach to jointly optimize a cascade document ranking model. Specifically, we utilize the attention activations over the passages from the ranker as fine-grained attention feedback to optimize the selector. Meanwhile, we fuse the relevance scores from the passage selector into the ranker to assist in calculating the cooperative matching representation. Experiments on MS MARCO and TREC DL demonstrate the effectiveness of our method.

heading: 1 Introduction
text: Document ranking aims at ranking the candidate documents according to their relevance to an input query, and it has been widely applied in many natural language processing (NLP) and information retrieval tasks, such as search engines (Hofstätter et al., 2021) and question answering (Chen and Yih, 2020). Due to the powerful representation ability of large-scale pre-trained language models (PLMs) (e.g., BERT (Devlin et al., 2019) and
∗Corresponding author.
RoBERTa (Liu et al., 2019)) that have achieved impressive performance in a large number of NLP tasks, several researchers have considered making use of pre-trained models for document ranking (MacAvaney et al., 2019; Li and Gaussier, 2021; Fu et al., 2022).
One major challenge in applying PLMs for neural document ranking is their difficulty in handling long texts due to high computational complexity and memory requirements, such as the 512 token limit for BERT. In fact, documents typically contain long text, for example, the mean length of documents in 2019 TREC Deep Learning Track Document Collection is 1600 (Hofstätter et al., 2021). To address this issue, various studies have been conducted to develop more efficient attention mechanisms in transformers (Beltagy et al., 2020; Hofstätter et al., 2020a), by simply truncating the document to meet the requirement for the relevance model (Boytsov et al., 2022), or by breaking down the long document into smaller segments or passages that can be processed individually by the pre-trained models (Dai and Callan, 2019; Rudra and Anand, 2020; Li et al., 2020; Chen et al., 2022).
Actually, long documents often contain a vari-
1688
ety of subjects, as evidenced by the scope hypothesis (Robertson et al., 2009) from traditional information retrieval. An illustration from the MS MARCO dataset (Nguyen et al., 2016) is presented in Figure 1, and it is noted that only a small part of the document (e.g., p2 and p3) is relevant to the given query and different parts may be unequally informative to the query. Thus even though existing techniques for modeling long documents have been demonstrated to be effective and efficient, utilizing the entire document can result in high query latency and intensive query-irrelevant content can be a distraction and negatively impact performance. Consequently, some recent studies propose cascade document ranking models (Li et al., 2020; Hofstätter et al., 2021; Li and Gaussier, 2021) that extract relevant passages with an efficient selector before performing the ranking. However, their selection and ranking modules are almost independently optimized and deployed, leading to selecting error reinforcement and sub-optimal performance. Moreover, these models do not differentiate between the passages or segments taken from a document while matching with the query.
In fact, the document ranker can provide finegrained supervision to enhance the generalizability and compatibility of the selector. Conversely, the selector, built upon a heterogeneous structure, can offer a distinct perspective to assist in document ranking. Taking inspiration from this, we propose a Fine-grained Attention Alignment approach (FAA) to jointly optimize a cascade document ranking model. Specifically, we initialize the passage selector as an efficient dual encoder and the document ranker with an effective crossencoder. To better optimize and make use of both worlds, we leverage the attention activations over the passages from the ranker as fine-grained attention feedback to optimize the selector. Simultaneously, we incorporate the relevance scores from the passage selector into the ranker to assist in calculating the final cooperative matching representation. We conduct experiments on three public benchmarks: MS MARCO (Nguyen et al., 2016), TREC-DL 2019 (Craswell et al., 2020), and TRECDL 2020. The evaluation results show that our proposed model is better than several competitive baselines and our FAA can bring significant improvement to the cascade model. To sum up, our contribution is three-fold:
• We propose a Fine-grained Attention
Alignment approach to jointly optimize a cascade document ranking model.
• We explore fusing the passage-level relevance scores into the document ranker to produce the cooperative matching representation.
• We conduct extensive experiments and analysis on three benchmarks and the evaluation results show the effectiveness of our model.

heading: 2 Related Works
text: Neural models for document ranking In the early stages, traditional algorithms like BM25 (Robertson et al., 2009) and TF-IDF were commonly employed for ranking documents in information retrieval. With the development of neural network technology (Cho et al., 2014; Gu et al., 2018), some neural-based ranking models have been proposed (Huang et al., 2013; Guo et al., 2016; Hui et al., 2017, 2018; MacAvaney et al., 2020). Xiong et al. (2017) proposed a kernelbased neural ranking model (K-NRM) which used a kernel-pooling layer to combine word pair similarities with distributed representations. Dai et al. (2018) extended K-NRM to Conv-KNRM which used Convolutional Neural Networks to model ngram embedding. Hofstätter et al. (2020b) proposed a Transformer-Kernel model which used a small number of transformer layers to contextualize query and document sequences independently and distilled the interactions between terms. Compared to traditional methods, neural ranking models produce a dense representation of the queries and documents which improves the ranking performance.
Pre-trained models for document ranking Recently a large number of transformer-based pretrained models have been proposed (Devlin et al., 2019; Lewis et al., 2020; Radford et al., 2019; Raffel et al., 2020) and shown their effectiveness in natural language processing tasks. Therefore many works have utilized pre-trained models in document ranking tasks. Nogueira et al. (2019) used a sequence-to-sequence transformer model with document terms as input and produced the possible questions that the document might answer to expand document for document retrieval. Finally this work used BERT to re-rank these retrieved documents. Yan et al. (2019) used a pre-trained BERT model to classify sentences into three categories
and then fine-tuned the model using a point-wise ranking method for ranking documents.
Passage-level document ranking Since the high demand for memory space and computing resources, pre-trained models usually have a limit on the input length, and the length of actual long documents is always beyond this limitation. To this end, some works proposed to split the long documents into multiple passages which satisfy the limitation of the input length of the pre-trained models (Li et al., 2020; Hofstätter et al., 2020a; Yang et al., 2019). The studies applied pre-trained models to each passage individually and then combined the relevance scores at the passage level to generate the relevance scores for the entire document. For example, Dai and Callan (2019) determined the relevance score of the document by utilizing the score of the first passage, the top-performing passage, and the summation of all passages, respectively. Fu et al. (2022) proposed a Multi-view interpassage Interaction based Ranking model (MIR) with intra-passage attention and inter-passage attention, and used a multi-view aggregation layer to produce the document-level representation across multiple granularities. These works took all passages into document ranking which may introduce noise from the query-irrelevant passages and increase the query latency. To address this problem, some works proposed to pre-select query-relevant passages from all passages before aggregating. In this work, we propose a cooperative distillation and representation cascade ranking model which uses an efficient model as a passage selector to calculate passage-level relevance scores and select top-k passages, while uses an effective model as the document ranker to calculate the document-level relevance scores with the selected passages.

heading: 3 Methodology
text: In this section we first formalize the document ranking task, then we introduce the model architecture and the proposed Fine-grained Attention Alignment (FAA) approach for model optimization.
Task Formalization Given a query q and a set of candidate documents C = {d1, d2, ..., dm} including both the ground-truth document and negative documents, where m is the number of the candidate documents, the task is to train a document ranking model R(q, d) with the training data D. When provided with a new query and its corre-
sponding candidate documents, the ranking model assesses the relevance between the query and each candidate document by computing relevance scores. Subsequently, it can arrange the documents in order based on these scores.
Model Overview Inspired by previous work on passage-level evidence for document ranking (Hofstätter et al., 2021; Li and Gaussier, 2021), in this paper we adopt the efficient and effective cascade document ranking paradigm which first extracts relevant passages with an efficient selector and then performs the ranking with a smart document ranker based on the pruned content. To better optimize and make use of both worlds, we propose a finegrained attention alignment approach to jointly optimize a cascade document ranking model. Specifically, we utilize the attention activations over the passages from the ranker as fine-grained attention feedback to optimize the selector. Additionally, in the process of document ranking, the passage-level relevance scores in the selector are fused in the document ranker to produce the cooperative matching representation for calculating the final matching score. By this means, the document ranker can provide fine-grained supervision to make the selector more generalizable and compatible, and the selector built upon a heterogeneous structure can conversely offer a distinct view to help the ranker. Figure 2 presents the high-level architecture of the proposed method.

heading: 3.1 Passage Selector
text: To satisfy the input length limit of the pre-trained models, the candidate documents are first split into multiple passages with a sliding window in the size of l words and a stride in the size of s words. The set of passages of document d can be formalized as
P = {d0: l, ds: s+l, d2∗s: 2∗s+l, ...} (1)
In the phase of passage selection, the passage selector identifies and extracts a subset of passages that are highly relevant to the given query. We adopt the simple and efficient dual-encoder structure built on a small pre-trained model as the passages selector which has a lower query latency. Given the query q and the set of passages P = {p1, p2, · · · , pw} where w is the number of passages, q and each pi ∈ P are fed into the passage selector and encoded as d-dimensional vectors respectively which are denoted as Encpsg(q) and Encpsg(pi). With the representative vectors of
query and passages, the passage selector calculates the dot product between Encpsg(q) and Encpsg(pi):
Rpsg(q, pi) = Encpsg(q) T Encpsg(pi)√ d
(2)
The passage-level relevance scores are scaled by dividing by √ d. Next, the passage selector selects the k passages with the highest relevance score to form P̄, which is formalized as:
P̄ = argmax P̄⊂P, ∥P̄∥=k
∑ pi∈P̄ Rpsg(q, pi) (3)
Passages in P̄ contain informative content for query and are used for document ranking. By selecting the most relevant top-k passages P̄ from all the passages, the passage selector filters out a large number of irrelevant passages for document ranking processes, which can reduce the query latency and avoid the noise caused by irrelevant passages.

heading: 3.2 Document Ranker
text: We adopt a cross-encoder based on pre-trained models as the document ranker to calculate the document-level relevance score with P̄. The architecture performs full attention across the query and the extracted passages and has been proven to be effective for ranking (Hofstätter et al., 2021). Formally, all selected passages in P̄ are first spliced together as P̂ = {p̄1; p̄2; · · · ; p̄k}, and then we concatenate query and the spliced passages P̂ as the input of the document ranker with [CLS] and [SEP] tokens, which is denoted as u:
u = {[CLS]; q; [SEP]; P̂; [SEP]} (4)
The document ranker performs semantic interaction through multi-layer attention blocks and
outputs a sequence of contextualized representations. Typically, the output representation of the first token [CLS] is adpoted the encoded vector of u, namely Encdoc(u) = E[CLS]. Then the vector is fed to a multilayer perceptron (MLP) to calculate the document-level relevance score:
Rdoc(q, d) = MLP(Encdoc(u)) (5) Since the dataset provides the positive document for each query, the loss function we use to optimize the document ranker is defined below following the previous works (Wu et al., 2018; Oord et al., 2018):
Lrank = − ∑
q∈D log exp(Rdoc(q, d+))∑ d∈C exp(Rdoc(q, d)) (6)
where d+ is the ground-truth document for the query q and C is a set of document candidates (including both the ground-truth document and negative documents) for q.

heading: 3.3 Cooperative Matching Representation
text: Considering the passage selector is based on heterogeneous dual-encoder architecture, we think that the selector can offer a distinct view to help document ranking. Therefore, different from traditional encoding which only uses the encoded vector of the first token [CLS] as the representation of sequence, we propose to fuse the selected passagelevel relevance scores from the passage selector to produce the cooperative matching representation Encdoc(u) of input sequence u. Specifically, we denote the embedding vector of [CLS] as E[CLS] and denote the embedding vector of tokens in P̂ as {E11 , E21 , · · · , Eji , · · · }, where E j i represents the embedding vector of the j-th token in the i-th selected passage p̄i. To produce Encdoc(u), we first
calculate the average embedding vector of each selected passage:
MeanPool(p̄i) = ∑l z=1E z i /l (7)
where l is the length of p̄i. We then calculate the product of the passage-level relevance scores from the selector and the average vector of the passage in the ranker, and take the summation of the results as the passage-selector guided vector EPGV, formalized as:
EPGV = k∑
t=1
MeanPool(p̄i) · Rpsg(q, p̄i) (8)
Finally we fuse the passage-selector guided vector EPGV with E[CLS] to get the cooperative document-level matching representation:
Encdoc(u) = E[CLS] + λ · EPGV (9)
where λ is a parameter to control the weight of EPGV. Then we can feed the above Encdoc(u) into a multi-layer perceptron to calculate the final document-level relevance score, as formalized in Equation 5. We can find that the more relevant a passage is, the greater its proportion in the fusion, which causes the document ranker to pay more attention to it.

heading: 3.4 Fine-grained Attention Alignement
text: As mentioned above, the passage selector is initialized by dual-encoder architecture which is efficient but performance sub-optimal compared with cross-encoder. It is not so compatible in ranking model and need to be tuned. Besides, there are no passage-level labels in most document ranking tasks. Inspired by knowledge distillation (Hinton et al., 2015; Wang et al., 2020), we use the complicated and effective document ranker as the teacher model to provide fine-grained supervision for optimizing the passage selector which is regarded as a student model to make the selector more generalizable and compatible. To be specific, with the self-attention mechanism in the transformerbased model, we use fine-grained attention activation scores over the passage as the pseudo labels of passages for optimization. We consider that if one passage is more informative to query, the query will provide more attention to it when document ranking which results in a higher attention score for this passage.
For the input u, the representation output by the previous layer is denoted as H ∈ Rlu×d where lu is
the length of u. The self-attention module produces queries Q, keys K, and values V matrices through linear transformations (Vaswani et al., 2017), and then the attention map can be calculated as:
M = softmax( QKT√
d ) (10)
where d is the dimension of vectors in Q. We denote αi→j = Mi,j as the attention score from i-th token to j-th token in u. Following the calculation of the attention score from one token to another token, we calculate the attention activation score of each selected passage p̄i (∈ P̄) as the maximal attention score from all tokens in query q and all tokens in the p̄i:
αq→p̄i = MaxPool(M̃),
M̃ = Mx:x+lq ,y:y+lpi
(11)
where x, y is the starting token of q and pi, and lq, lpi is the length of q and pi respectively. M̃ is the attention map between q and pi, where M̃i,j is the attention score from i-th token in q to j-th token in pi. We also experimented with the meanpooling operation to calculate attention scores and found that it performed worse than max-pooling. Following previous knowledge distillation methods based on pre-trained language models (Wang et al., 2020), we also calculate the attention score of p̄i in the last self-attention layer of the document ranker. Taking into account the multi-head attention mechanism in the transformer-based model, we select the maximal attention score through all attention heads as the final scores.
We use KL-divergence between the relevance scores of passages output by the passage selector and the attention scores as the loss function of the passage selector:
Lalign = ∑
q∈D KL-Div(Hpsg(q, P̄),Adoc(q, P̄))
(12) where Hpsg(q, P̄) is the output distribution over the relevance scores of passages in P̄ from the selector, Adoc(q, P̄) is the distribution of the aggregated attention scores in ranker. Hpsg(q, p̄k) and Adoc(q, p̄k) are the k-th item in Hpsg and Adoc respectively, which can be calculated as below:
Hpsg(q, p̄k) = exp(Rpsg(q, p̄k)/τ)∑ p̄∈P̄ exp(Rpsg(q, p̄)/τ) (13)
Algorithm 1 The proposed FAA Require: Training set D, selector parameters ϕpsg,
ranker parameters ϕdoc Initialize parameters ϕpsg, ϕdoc repeat
Sample a batch B from D Compute passage relevance scores by Eq (2) Select top-k relevant passages P̄ by Eq (3) Compute document relevance scores with P̄ Compute Lrank on B and optimize ϕdoc Compute attention score Patt(p̄i) by Eq (14) Compute Lalign on B and optimize ϕpsg
until Convergence Return ϕpsg, ϕdoc
Adoc(q, p̄k) = exp(αq→p̄k/τ)∑ p̄∈P̄ exp(αq→p̄/τ)
(14)
where τ is the temperature hyper-parameter. Above all, in our overall ranking model, the loss function can be described as the combination of the loss for the document ranker and the attention alignment loss:
Lfinal = Lalign + Lrank (15)
In this work, we tried to jointly train the passage selector and document ranker. Particularly, we update the ranker with only Lrank, and the gradient from Lalign is stopped. Algorithm 1 gives a pseudo-code of our training process.

heading: 4 Experiments
text: In this section, we first introduce the datasets we use, the evaluation metrics, the baselines, and the implementation details of our experiment. Then we introduce the evaluation results and further analysis of our method.

heading: 4.1 Datasets and Evaluation
text: In line with previous studies on this task (Hofstätter et al., 2021; Li and Gaussier, 2021), we conduct an evaluation of our proposed model on three publicly available document ranking datasets: MSMARCO (Nguyen et al., 2016), TREC-DL 2019 (Craswell et al., 2020), and TREC-DL 2020. The MS-MARCO dataset comprises 3.2 million documents and 367,013 training queries, sourced from web pages. For evaluation, we utilize the MSMARCO DEV set, which consists of 5,193 queries.
The evaluation metrics employed are NDCG@10, MAP, and MRR@10. Both the TREC-DL 2019 and TREC-DL 2020 datasets share the same document collection as MS-MARCO and include 43 and 45 queries, respectively. For both TREC-DL datasets, we employ NDCG@10 and MAP as the evaluation metrics. Across all datasets, we perform document re-ranking based on the top 100 documents retrieved by BM25.

heading: 4.2 Baselines
text: We compare our model with traditional and neural document ranking models:
• BM25 (Robertson et al., 2009) is a widelyused unsupervised text-retrieval algorithm based on IDF-weighted counting.
• BERT-MaxP (Dai and Callan, 2019) uses BERT to encode passages split from the document to calculate the relevance score and choose the best passage-level score as the document-level score.
• Sparse-Transformer (Child et al., 2019) introduces several sparse factorizations of the attention matrix.
• LongFormer-QA (Beltagy et al., 2020) extends Sparse-Transformer by attaching two global attention tokens to the query and the document as their settings for QA.
• Transformer Kernel Long (Hofstätter et al., 2020a) proposes a local self-attention mechanism with the kernel-pooling strategy.
• Transformer-XH (Zhao et al., 2020) introduces an extra hop attention layer that can produce a more global representation of each piece of text.
• QDS-Transformer (Jiang et al., 2020) proposes a query-directed sparse transformerbased ranking model which uses sparse local attention to obtain high efficiency.
• KeyBLD (Li and Gaussier, 2021) proposes using local query-block pre-ranking to choose key blocks of a long document and aggregates blocks to form a short document which is further processed by BERT.
• PARADE (Li et al., 2020) truncates a long document into multiple passages and uses different strategies to aggregate the passage-level relevance scores. PARADEMax-Pool uses maxpooling to obtain document-level relevance scores and PARADETF uses a transformer encoder for passages aggregation.
• IDCM (Hofstätter et al., 2021) uses a fast model (ESM) for passage selection and a effective model (ETM) for document ranking, where optimizes the ESM with the knowledge distillation from ETM to ESM.

heading: 4.3 Implementation Details
text: Our proposed model is implemented by the transformer library provided by hugging face1. We use DistilBERT (Sanh et al., 2019) to initialize our passage selector which is more efficient and has comparable performance with BERT-base. For document ranking, we use the publicly trained model2 to initialize our document ranker. We set the length of the sliding window and stride as 72. The query length is set as 30 and the number of selected passages is set as 3. We use Adam optimizer (Kingma and Ba, 2015) to train our model with batch size set as 4. The initial learning rate of the passage selector and document ranker are set as 5e-7 and 7e-6 respectively. We vary λ (Equation (9)) in {0.1, 0.2, 0.5, 1.0} and find that 0.2 is the best choice. τ in Equation (13) and Equation (14) is set as 0.2.
1https://huggingface.co/docs/transformers/ 2https://huggingface.co/cross-encoder/
ms-marco-MiniLM-L-6-v2

heading: 4.4 Evaluation Results
text: The evaluation results of our proposed model and all baselines on MS MARCO, TREC-DL 2019 and TREC-DL 2020 are reported in Table 1. First, compared with the models with more efficient attention mechanisms in transformer (e.g. SparseTransformer, Transformer-XH, QDS-Transformer, and Transformer Kernel Long), our method and other cascade document ranking models (e.g. KeyBLD and IDCM) can achieve better performance on almost all metrics. The phenomenon indicates the superiority of the cascade document ranking paradigm. Second, compared with two previous cascade methods3 (e.g. IDCM) that select passage before ranking, our model has better performance than them on MS-MARCO and TREC DL 2020, and shows comparable performance on TREC DL 2019. Different from these baselines which optimize the selector and ranker independently, our model jointly optimizes the selector and ranker with fine-grained attention alignment. Meanwhile, we utilize the passage-level relevance scores in document ranking to obtain cooperative fusion representation. The evaluation results demonstrate the effectiveness of our proposed methods.

heading: 4.5 Discussions
text: Ablation study Table 2 presents the findings from our ablation study, where we systematically remove specific components to assess their impact on performance. Firstly, we eliminate the finegrained attention alignment for the passage selec-
3As of now, a direct comparison with KeyBLD is not feasible due to the lack of reported results on the MS-MARCO DEV dataset and the unavailability of the source code.
tor, denoted as "w/o. Lalign". Next, we remove the passage-level multi-vector fusion during document ranking, denoted as "w/o. EPGV". The results reveal that removing either Lalign or EPGV leads to a drop in performance, indicating the effectiveness of our fine-grained attention alignment approach and the importance of utilizing cooperative fusion representation to enhance the ranker’s capabilities. Notably, removing both components simultaneously results in an even greater performance decrease. Furthermore, we examine the use of average pooling in representation fusion, denoted as "Rpsg(q, p̄i) = 1/k," which replaces Rpsg in Eq. 8 with 1k . Our findings indicate that simply incorporating average pooling of passage representations does not yield substantial gains, as it only achieves comparable performance to the model without EPGV. Notably, the performance of "Rpsg(q, p̄i) = 1/k" and the model without EPGV are inferior to that of our model, illustrating the utility and superiority of cooperative fusion of relevance scores from selectors over independent representation fusion. Lastly, we explore the use of mean-pooling operation for calculating attention scores and observe that it performs worse than maxpooling.
The impact of passage length When constructing the training data, the length of the split passage plays a vital role as it also indirectly controls the

heading: 1 0.389 0.340 0.331

heading: 2 0.440 0.384 0.377

heading: 3 0.453 0.397 0.390

heading: 4 0.451 0.390 0.387
text: number of passage candidates for each document. To investigate the impact of the passage length, we test the performance of our method across different passage lengths and the results are shown in Figure 3. We can find that the performance of our model improves until the passage length reaches 72, and then drops when the passage length keeps increasing. The reason might be that the selector needs to rank fewer candidates as the passage length increases at first and it could select more accurate passages that are relevant to the query for matching, but when the length of the passage becomes larger enough, the noise will be brought to matching as some content in each passage could be irrelevant to the query.
The impact of the number of selected passages We are also curious about the impact of the number of selected passages. We test the performance of our method with different numbers of selected passages and the evaluation results are illustrated
in Table 3. We can observe that when the performance of our model was significantly improved as the number of selected passages increased at the beginning (≤ 3), and then began to drop when the number kept increasing. The results are rational because more passage entries can provide more useful information for response matching, but when the passage becomes enough, query-irrelevant noise will be brought to matching.
Case study To verify the effectiveness of our cascade model in document ranking, we show a ranker example from MS-MARCO dataset in Table 4. For the input query how many mm is a nickel coin, our FAA ranks the positive document at first and it is split into 24 passages. We show the top-3 passages selected by our passage selector and a random passage that is not selected. We can find that the top 2 passages harbor a significant amount of valuable query-relevant information, encompassing terms like "nickel" and "diameter." Conversely, the final passage, which displays lesser relevance to the query, receives a lower relevance score as determined by the passage selector. This case serves as an illustration of our model’s proficiency in selecting pertinent content within the document and ranking it based on query relevance.

heading: 5 Conclusion
text: In this work, we propose FAA, a cascade ranking model with a fine-grained attention alignment and cooperative matching representation. Our model utilizes the fine-grained attention alignment approach to train the passage selector and fuses the passage-level relevance scores into document ranking to produce cooperative matching representation. The evaluation results on MS MARCO and TREC DL demonstrate the effectiveness of FAA.

heading: 6 Limitations
text: While our approach effectively mitigates query latency through a cascade ranking paradigm, it necessitates additional computational resources during training due to the need for attention score calculation and alignment in the optimization process. Additionally, our model incorporates passage-level relevance scores into the ranker, generating a cooperative matching representation during document ranking, which could marginally augment the inference time. In our future endeavors, we aim to explore more efficient methodologies that can further improve ranking efficiency. Furthermore, it
is worth noting that our approach has been tested using specific backbone models. To fully evaluate the effectiveness of our method, it is essential to conduct experiments with a diverse range of backbone models, which remains an avenue for further exploration.

heading: Acknowledgements
text: We would like to thank the anonymous reviewers for their constructive comments. This work was supported by the National Key Research and Development Program of China (No. 2021YFC3340304).
Ethical Statement
Our paper centers around the document ranking task, a well-established and widely applicable problem. In conducting our research, we have exclusively utilized queries and documents sourced from open public datasets, with proper citation and adherence to licensing agreements. We have taken great care to ensure that our experiments have no bearing on privacy security, discrimination, or bias. We affirm that our work aligns with ethical principles and regulations, and it does not infringe upon any established ethical codes or guidelines.

heading: A For every submission:

heading: 3 A1. Did you describe the limitations of your work?
text: Limitation section
7 A2. Did you discuss any potential risks of your work? The topic of the paper deals only with document retrieval

heading: 3 A3. Do the abstract and introduction summarize the paper’s main claims?
text: Abstract and Introduction section
7 A4. Have you used AI writing assistants when working on this paper? Left blank.
B 3 Did you use or create scientific artifacts?

heading: 4 Experiments section

heading: 3 B1. Did you cite the creators of artifacts you used?
text: 4 Experiments section
7 B2. Did you discuss the license or terms for use and / or distribution of any artifacts? MS-MARCO and Trec DL are open-source datasets
7 B3. Did you discuss if your use of existing artifact(s) was consistent with their intended use, provided that it was specified? For the artifacts you create, do you specify intended use and whether that is compatible with the original access conditions (in particular, derivatives of data accessed for research purposes should not be used outside of research contexts)? Our use of MS-MARCO and Trec DL was consistent with their intended use.
B4. Did you discuss the steps taken to check whether the data that was collected / used contains any information that names or uniquely identifies individual people or offensive content, and the steps taken to protect / anonymize it? Not applicable. Left blank.
B5. Did you provide documentation of the artifacts, e.g., coverage of domains, languages, and linguistic phenomena, demographic groups represented, etc.? Not applicable. Left blank.
3 B6. Did you report relevant statistics like the number of examples, details of train / test / dev splits, etc. for the data that you used / created? Even for commonly-used benchmark datasets, include the number of examples in train / validation / test splits, as these provide necessary context for a reader to understand experimental results. For example, small differences in accuracy on large test sets may be significant, while on small test sets they may not be. 4 Experiments section
C 3 Did you run computational experiments?

heading: 4 Experiments section

heading: 3 C1. Did you report the number of parameters in the models used, the total computational budget
text: (e.g., GPU hours), and computing infrastructure used? 4 Experiments section
The Responsible NLP Checklist used at ACL 2023 is adopted from NAACL 2022, with the addition of a question on AI writing assistance.

heading: 3 C2. Did you discuss the experimental setup, including hyperparameter search and best-found
text: hyperparameter values? 4 Experiments section

heading: 3 C3. Did you report descriptive statistics about your results (e.g., error bars around results, summary
text: statistics from sets of experiments), and is it transparent whether you are reporting the max, mean, etc. or just a single run? 4 Experiments section
C4. If you used existing packages (e.g., for preprocessing, for normalization, or for evaluation), did you report the implementation, model, and parameter settings used (e.g., NLTK, Spacy, ROUGE, etc.)? Not applicable. Left blank.
D 7 Did you use human annotators (e.g., crowdworkers) or research with human participants? Left blank.
D1. Did you report the full text of instructions given to participants, including e.g., screenshots, disclaimers of any risks to participants or annotators, etc.? Not applicable. Left blank.
D2. Did you report information about how you recruited (e.g., crowdsourcing platform, students) and paid participants, and discuss if such payment is adequate given the participants’ demographic (e.g., country of residence)? Not applicable. Left blank.
D3. Did you discuss whether and how consent was obtained from people whose data you’re using/curating? For example, if you collected data via crowdsourcing, did your instructions to crowdworkers explain how the data would be used? Not applicable. Left blank.
D4. Was the data collection protocol approved (or determined exempt) by an ethics review board? Not applicable. Left blank.
D5. Did you report the basic demographic and geographic characteristics of the annotator population that is the source of the data? Not applicable. Left blank.

heading: Future Work
text: In our future endeavors, we aim to explore more efficient methodologies that can further improve ranking efficiency. Furthermore, it
is worth noting that our approach has been tested using specific backbone models. To fully evaluate the effectiveness of our method, it is essential to conduct experiments with a diverse range of backbone models, which remains an avenue for further exploration