Longtriever: a Pre-trained Long Text Encoder for Dense Document Retrieval
Authors: Junhan Yang
Zheng Liu
Chaozhuo Li
Guangzhong Sun
Xing Xie
Venue: EMNLP 2023 Main (2023)
Keywords: dense retrieval
document retrieval

Abstract: Pre-trained language models (PLMs) have achieved the preeminent position in dense retrieval due to their powerful capacity in modeling intrinsic semantics. However, most existing PLM-based retrieval models encounter substantial computational costs and are infeasible for processing long documents. In this paper, a novel retrieval model Longtriever is proposed to embrace three core  challenges of long document retrieval: substantial computational cost, incomprehensive document understanding, and scarce annotations. Longtriever splits long documents into short blocks and then efficiently models the local semantics within a block and the global context semantics across blocks in a tightly-coupled manner. A pre-training phase is further proposed to empower Longtriever to achieve a better understanding of underlying semantic correlations. Experimental results on two popular benchmark datasets demonstrate the superiority of our proposal.

Longtriever: a Pre-trained Long Text Encoder for Dense Document
Retrieval
Junhan Yang*
Zheng Liu'
Chaozhuo Li MSRA Beijing, China c1 crosof...ooccommm
USTC
MSRA
Hefei, China
Beijing, China
yangjun2@mail. ustc. cn zhengliu@microsoffi com
Guangzhong Sun
Xing Xie MSRA Beijing, China xingxêmicrosoft. com
USTC
Hefei, China
gzsun@ustc. edu. cn
Abstract
the intrinsic semantics, pre-trained language mod- els (e.g., BERT (Devlin et al., 2018), RoBERTa (Liu et al., 2019) and DeBERTa (He et al., 2020)) have attained preeminent position in dense re- trieval. Due to the high computational complexity (Vaswani et al., 2017; Lepikhin et al., 2020), most existing PLM-based retrieval models (Gao and Callan, 2021a.b; Liu and Shao, 2022) are designed for passage retrieval (passages are short texts gen- erally no longer than 100 words (Karpukhin et al., 2020)). Nevertheless, long documents are ubiqui- tous in real life. For example, the average number of words and tokens in the documents of the MS MARCO dataset (Nguyen et al., 2016) are 1,165.46 and 1,631.30 respectively, significantly larger than the maximum input length (e.g. 512) of passage retrieval. In this paper, we aim to investigate the crucial task of long document retrieval, which is challenging due to the following three reasons.
Pre-trained language models (PLMs) have achieved the preeminent position dense re- trieval due their powerful capacity in mod- eling intrinsic semantics. However, most ex- isting PLM-based retrieval models encounter substantial computational costs and are infeasi- ble for processing long documents. this pa- per, novel retrieval model Longtriever is pro- posed to embrace three core challenges of long document retrieval: substantial computational cost, incomprehensive document understand- ing, and scarce annotations. Longtriever splits long documents into short blocks and then ef- ficiently models the local semantics within block and the glohal context semantics across blocks in a tightly-coupled manner. pre- training phase is further proposed to empower Longtriever to achieve a better understand- ing of underlying semantic correlations. Ex- perimental results on two popular benchmark datasets demonstrate posal. The source code is released at https: //gi tig.. com/ Samuel SangelYangl /Longtriever
superiority of our pro-
(1) Substantial computational cost. One straight- forward solution directly employ short passage retrievers on long documents. Such approaches suf- fer from the rapidly growing computational costs due to the quadratic time complexity of vanilla transformers (O(L2d). where L is the input se- quence length. and d denotes the dimension of latent embeddings). Truncating the documents into short passages is common workaround, which may lead the potential information loss (Karpukhin et al., 2020; Zhan et al., 2021b).
1 Introduction
Document retrieval aims at retrieving the most rel- evant documents from vast corpus in response to an input query (Guo et al., 2022), facilitating myriad of applications such web search (Xiao al., 2022a) and question answering (Karpukhin al., 2020). Recent works (Karpukhin al., 2020; Xiong ct al., 2020) generally first embed queries and documents into low-dimensional dense vec- tors, and then subsequently calculate their rele- vance based on these vectors, dubbed dense re- trieval. Due to their powerful capacity in modeling
(2) Incomprehensive document understanding. Another typical method is to employ efficient trans- formers, such as sparse attention transformers (Child al., 2019; Beltagy et al., 2020; Zaheer et al., 2020) and hierarchical transformers (Zhang et al., 2019; Lin et al., 2021; Wu et al., 2021; Tian et al., 2023). Such approaches contribute to reduc-
WWrk was done during Junhan's internship MSRA "Correpponding author

---

ing the computational cost while suffering from severe incomprehensive document understanding, For instance, sparse attention transformers sparsify the full attention via masking, in which several virtual tokens are proposed as the global tokens attended by all tokens (Beltagy et al., 2020; Za- heer et al., 2020). After aggregating the informa- tion from all tokens, these global tokens tend to be overloaded, leading to the information mess (Child et al., 2019; Li al., 2022). addition, the popu- lar heuristic masking strategy (e.g., random mask- ing) may further aggravate the risk of information loss. The hierarchical transformers usually first split the document into short blocks. The seman- tics of different blocks are modeled independently, which are further fed into a readout layer in a cas- caded manner. Semantics from different blocks are loosely coupled as a token can only attend to the other tokens in the same block and rich long- range context information is largely ignored. Such a loosely-coupled paradigm might be insufficient in modeling the sophisticated cross-block relations and thus cannot learn the comprchensive document representations.
simultaneously under a desirable time complexity. Different from previous fine-tuning based methods (Wu et al., 2021; Li et al., 2020; Xiong et al., 2020), here we further design a pre-training phase to gain a better understanding of inherent semantics within the long documents. A novel pre- training task, lo- cal masked autoencoder (LMAE), is proposed to mine the intrinsic semantics by reconstructing the raw input of each block based on the global doc- ument representations, the block representations, and the original context tokens. The pre-training phase empowers Longtriever with the capability of capturing the unsupervised semantic correlations and contributes to alleviating the reliance on the annotations. Longtriever is extensively evaluated over two popular benchmark datasets, and the ex- perimental results demonstrate the superiority of our proposal.
2 Related Work
2.1 Dense Retrieval
In the field of dense retrieval models, bi-encoder architecture is typically employed to separately en- code queries and documents, thereby improving search efficiency. This architectural design has found its use in a multitude of applications, such as search engines (Karpukhin et al., 2020), advertis- ing (Lu et al., 2020), and recommendation systems (Xiao et al., 2022b). Karpukhin et al. (2020) illus- trated that incorporating in- negatives during training could substantially boost the performance of dense retrieval models compared the tradi- tional BM25 model. Consequently, Xiong et al. (2020) introduced ANCE, proposing the utilization of approximate nearest neighbors as negative sam- plcs. To refine the selection of negative samples, Qu et al. (2020) presented RocketQA, which used more precise cross-encoder. Zhang et al. (2021a) demonstrated the AR2 model, which concurrently trained the bi-encoder and cross-encoder.
(3) Scarce annotations. Existing dense retrieval models generally rely on annotations to fine-tune the PLMs. The scarcity of annotations is further ex- acerbated in long document retrieval. Compared to short passage retrieval, more training signals are in- dispensable to achieve an accurate understanding of long documents. However, manually labeled anno- tations are usually expensive and time-consuming. Hence, the elaborate unsupervised training signals are expected to facilitate the modeling of long doc- uments and alleviate the challenge of annotation scarcity in the fine-tuning phase.
In this paper, we propose a novel dense retrieval model for long documents, dubbed Longtriever. Longtriever follows the hicrarchical paradigm, in which the long document is split into multiple short blocks to ensure the model's efficiency. Each layer of Longtriever consists of two core mod- ules: the intra-block encoder to convey messages among tokens belonging to the same block, and the inter-block encoder to pass information across different blocks. After iteratively stacking multiple Longtriever layers, the tokens in a block are capa- ble of attending tokens in other blocks, resulting in a tightly-coupled paradigm. Longtriever models the local semantics within a single block and the global context correlations across multiple blocks
Advancements in the quality of dense retrieval are largely attributed to the recent progress in pre- trained language models (Karpukhin et al., 2020; Luan et al., 2021). A common approach entails deploying a universally pre-trained model as the bi-encoder. These models usually pre-trained using one or multiple masked language model- ing (MLM) tasks. For instance, BERT (Devlin et al., 2018) and RoBERTa (Liu et al., 2019) pre- dict masked tokens based on their context. In Emnc (Sun et al., 2019) () and Spanbert (Joshi et al., 2020),

---

Inter-Bock
80
B
H
Inte-BBock Encoder
Intra- Block Encoder
Intra-Blod
Intra-Bloc
Encodlr
Encoder
X L
00
Hi
n
11:
[DCC]
Intra-Block Block
Intra
Intra-Block Encieer
Intra-Block
[CLS)
Enunder
41
Figure 1: Architecture of Longtriever.
masked entities and spans are predicted, leading to enhanced performance in entity typing and ques- tion answering tasks. However, these generic mod- els, usually pre-trained on token-level tasks, might not effectively cultivate the ability to represent sen- tences (Chang et al., 2020).
(Zhang et al., 2019) and Hi- Transformer (Wu et al., 2021) are representative examples of this category. Recurrence transformers, generally used for gen- eration tasks, also partition the document into short blocks and process them using recurrence mecha- nism (Dai et al., 2019). Examples of such models are Transformer- XI. and XLNet (Yang et al., 2019), which keep the recurrence mechanism and intro- duce a permutation language modeling objective to capture bidirectional contextual information. In re- sponse to the problem of individual blocks lacking contextual information. ERNIE-Doc (Ding et al., 2020) implements retrospective feed mechanism, simulating human reading behavior.
To address this limitation, recent studies pro- posed two primary types of pre-training tasks. The first one is self-contrastive learning (SCL) (Wu et al., 2020; Zhang et al., 2021b; Yan et al., 2021; Liu et al., 2021; Gao et al., 2021), which trains the language models using relevant sentence pairs from unlabeled corpus. The second is auto-encoding (AE) (Gao and Callan, 202 Lu al., 2021; Liu and Shao, 2022: Pang al., 2022), which primar- ily trains language models to reconstruct the input sentence based on the sentence embedding.
3
Methodology
3.1 Longtriever
2.2 Efficient Transformers
Figure/ 1 illustrates the framework of the pro- posed Longtriever model. The input long docu- ment is tokenized into a sequence tokens X [I] via WordPiece tokenizer (Wu
Efficient transformers have recently gained sub- stantial attention for their ability to model long documents. Three primary types of efficient trans- formers have been proposed: sparse attention trans- formers, hierarchical transformers, and recurrence transformers. Sparse attention transformers, such Sparse Transformer (Child et al., 2019) and Reformer (Kitaev al., 2020), sparsify the self- attention matrix to reduce computational costs. Longformer (Beltagy et al., 2020) and BigBird (Zaheer et al., 2020), meanwhile, replace dense attention with a combination of random attention, window attention, and global attention.
{x1, 22, .. ,LL} et al., 2016; Li et al., 2021). This token sequence is further split into a set of blocks (2,,T2, ,Tx) where T; {=((-I)xa++1 notes the maximum number of tokens within a single block, which pre-defined hyperparame- ter (e.g., 512). N = [L/M] denotes the number of blocks. Each block is appended with a special token [CLS] as the representation of this entire block. In addition, special token [DOC] is fur- ther padded in the front of the input text as the document representation. Assume h, E mdx1 de- notes the representation of i-th token, S, E Rdx11 the document representation, and c Rdxx rep- resents the [CLS] token of i-th block. Each layer of Longtriever consists of two major modules: the block-block encoder to exchange information across
M. M de-
Hierarchical transformers, significant for tasks such as document summarization (Zhang et al., 2019) and document ranking (Lin et al., 2021), typ- ically divide long document into shorter blocks and aggregate these block representations to pro- duce the overall document representation. HIBERT

---

blocks, and the intra-block encoder convey mes- sages between tokens within a single block. Next, we will introduce the details of the I-th Longtriever layer.
where
Qe
(1)
HO
Qct
K (0)
-HOW
(6)
Ke
Vi
-HOW
())
Ve
3.1.1 Inter-block encoder.
in which w-()) W
Qe Ke w E Rdddd are learnable
The inter-block encoder aims to depict the global semanties via collecting and dispatching messages for different blocks. The document representation s and the [CLS] tokens c.") from all blocks ure combined into an embedding matrix:
matrices. The [CLS] embedding c!0) preserves the global signals from other blocks, which are incor- porated to facilitate the modeling of tokens in the belonging block. Different from previous loosely- coupled hierarchical transformers, the modeling of different blocks is tightly coupled together. Each token is capable of attending tokens in other blocks with the [CLS] token as the intermediary, leading to a comprehensive understanding the long document semantics. Longtriever layers are stacked as the model architecture, and the em- bedding of [DOC] is output as the final document representation.
B ([++))dd
[8 o
(1)
Matrix (!) essentially preserves the global seman- tics of the entire long document. Next, a multi- head transformer is employed on the matrix ( to transmit information across different blocks. For arbitrary attention head, inter-block passing is formalized as:
11)
K
()T
For the document retrieval task, Longtriever is utilized as both query encoder and document en- coder. Following (Karpukhin et al., 2020; Gao and Callan, 202 Liu and Shao, 2022), the dot product selected as the similarity metric. The relevance score between query and a candidate document is calculated as:
B softmax
V!
(2)
Vd
where
B(
ge U1 Ke
K
B
(3)
v -BOW!!
Ve
in which matrices W will w!ll Qe Ke: Ve E ddxd are learnable parameters. The information from dif- ferent blocks is fused and exchanged via the self- attention mechanism of the inter-block encoder, en- suring the (CLS) token of single block capable of attending to the global semantics.
relg, (T)
(7)
where is the query, represents the document and function LT denotes the feed-forward process of Longtriever.
A training sample is defined < q, d ,d,""., >, in which d' is the relevant (positive) candidate and di is the irrele- vant (negative) candidate. To ensure the training efficiency, here we adopt the in-batch negative sampling strategy (Karpukhin et al., 2020). The loss function of fine-tuning phase is formalized as the negative log-likelihood of the positive candidates:
3.1.2 Intra-block encoder.
The embeddings of [CLS] tokens learned by the inter-block encoder are dispatched back into the corresponding blocks. Given a single block, the embedding matrix fed into the intra-block encoder is formally defined as follows:
( (M+1)xx + ( h
h- (4)
rel guet
C, = Σ - log
(8)
in which c0 the [CLS] embedding dispatched from matrix (1) learned by the inter-block encoder. h represents the embedding of the i-th token of this block. Similar to the inter-block encoder, a multi-head transformer is applied on the matrix ( to conduct token-wer wise information propaga- tion:
c rel g,tt +
> rel
The time complexity of the inter-block encoder is O((N + 1)) The complexity of single intra- block encoder is O(MM + 1)2d). Since the in- put document is split into N blocks, the time cost of intra-block encoders is O(M + + d x N). Overall, the time complexity Longtriever is ( + N2d), which is more efficient than the vanilla transformers
( Qa
K! Vd
H softmax
V(I)

---

Decoder
Global ipprssatationn
Local rnpessetationn
Original token
Minked tolum
Encoder
Figure 2: Workflow of LMAE.
3.2 Pre-training for long document retrieval
H E Rxd and key inputs H E (M+2)xd for the multi-head attention layer within the trans- former:
Conventional dense retrieval models solely rely on the annotations to fine-tune PLMs, which is unsuit- able for long document retrieval since longer texts generally require more training signals. Thus, here we add a pre-training phase to encode the unsu- pervised semantic correlations into the proposed Longtriever model.
H +- [S] (1.) P(- + PixM]:
(1)
H < 1s (11) (14) ho (I)MMM+ he.,hh.at h
(9)
where and he denote the positional embeddings and the orignial token embeddings. After that, a vanilla transformer is employed the constructed matrices:
3.2.1 Masked Language Modeling
We adopt the vanilla masked language modeling (MLM) task to pre-train the Longtriever model. The input text X is randomly masked and the objec- tive is defined to predict the masked tokens based on the last hidden states (L).
Q Vd
H softmax
A
Vd.
(10)
3.2.2 Local Masked Autoencoder
where A is random mask matrix and
To address the annotation scarcity, we propose novel pre-training task for long document retrieval called local masked autoencoder (LMAE). Shown figure 2, LMAE first learns two types of repre- sentations: the global representation capturing the general semantics of the whole document, and the local representation preserving the specific seman- tics of the local block. A shallow decoder is further integrated into the Longtriever. LMAE aims to re- construct the original input tokens via the decoder based on the global and local representations.
Q H. Qa:
K -H Wkd. kd.
(11)
V -H Wvd
in which Woa, WKd. Wva E Rddd are learnable matrices.
Finally, the output hidden states of the decoder. denoted by H.ec are processed by the token pre- diction head vi. and the following objective is opti- mized:
In Longtriever, the last hidden state of [DOC], (I+1) (denoted by s//) ), is the global representa- tion, and the last hidden states of [CLS] in each block, of (de=ttddentted by e!'')), are the local repre- sentations. We utilize transformer layer as our de- coder. For each block T,, we generate distinct query
LLMAE Σ
(E(r))((DDe)) Dec))
(12)
ZLCX
where CE the cross-entropy loss.

---

MARCO Doc MRR@ 100
TREC 2019
Method
R@
NDCG@ R@ 100
Sparse refriever
BM25 (Rohertson al., 2009)
0.277
0.808
0.519
0.395
DeepCT (Dai and Callan, 2019)
0.320
0.544
TRECTrad (Craswell et al., 2020)
0.549
Dense retriever
JPQ (Zhan et al., 2021a)
0.384
0,905
0.608
0.302
RepCONC (Zhan ct al., 2022)
0.399
0.911
0.600
0.305
ANCE (Xiong ct al., 2020)
0.377
0.894
0.610
0.273
STAR (Zhan ct al., 2021b)
0,390
0.913
0.605
0.313
ADORE (Zhan et al., 202
0.405
0.919
0.628
0.317
BERT (Ma et al., 2022)
0.389
0,877
0.594
0.301
PROP (Ma al., 2021a)
0.394
0,884
0.596
0.298
B-PROP (Ma al., 2021b)
0.395
0,883
0.601
0.305
ICT (Lee et al.. 2019)
0.396
0,882
0,605
0.303
SEED (Lu et al., 2021)
0.396
0,902
0.605
0.307
COSTA (Ma et al., 2022)
0.422
0.919
0.626
0.320
Longtriever (ours)
0.434
0.940
0.645
0.356
Table I: Experimental results of retrieval methods.
4 Experiments
et al., 2022) are recent vector compression meth- ods. ANCE (Xiong et al., 2020), STAR (Zhan et al., 202 and ADORE (Zhan et al., 2021b) are complicated fine-tuning methods to enhance dense retrievers.
4.1 Experimental Settings
Datasets. In order to evaluate the performance of our proposal, extensive experiments are con- ducted on two popular document retrieval datasets: 1) MARCO Dev Doe (MS MARCO Document Ranking) (Nguyen et al., 2016) is a large-scale benchmark dataset for web document retrieval, comprising about 3 million documents, 0.4 mil- lion training queries, and 5 thousand development queries. TREC 2019 Doc (TREC 2019 Doc- ument Ranking) (Craswell et al., 2020) is a test set in MS Marco document ranking task produced by TREC, consisting of 43 queries with more com- prehensive labeling. We use the official metrics of these two benchmarks (Nguyen et al., 2016; Craswell et al., 2020). For the MS MARCO doc- ument ranking task. report the mean recipro- cal rank at 100 (MRR @ 100) and recall at 100 (R @ 100). For the TREC 2019 document ranking task, we report normalized discounted cumulative gain at 10 (NDCG@ 10) and recall 100 @R@ 100). Longtriever is pre-trained on the BookCorpus (Zhu et al., 2015) and English Wikipedia (Devlin et al., 2018).
Several popular PLMs are also introduced as baselines: 1) BERT (Devlin al.. 2018) is the most popular pre-trained language model in NLP tasks. RetroMAE (Liu and Shao, 2022) is pre-training paradigm that focuses on dense pas- sage retrieval tasks. It utilizes the masked autoen- coder technique and exhibits remarkable perfor- mance in these types of tasks. 3) Parade (Li et al., 2020) is method for aggregating passage repre- sentations into a document representation. It em- ploys transformer layer to achieve this aggrega- tion. 4) Longformer (Beltagy et al., 2020) uses technique called "local attention". which allows the model to process much longer sequences than traditional transformer models. 5) BigBird (Za- heer et al., 2020) is an efficient transformer model with several sparse attention mechanisms. 6) Hi- Transformer (Wu et al., 2021) is a hierarchical interactive transformer for efficient long document modeling. 7) XLNet (Yang et al., 2019) is extension of Transformer-XI. (Dai et al., 2019), pre-trained using an autoregressive method learn bidirectional contexts. utilizes the recurrent mem- ory mechanism to handle long text.
Baselines. Multiple SOTA passage retrieval and document retrieval methods selected the base- lines. BM25 (Robertson et al., 2009), DeepCT (Dai and Callan, 2019) and TRECTrad (Craswell et al., 2020) are classic sparse document retriev- ers. JPQ (Zhan et al., 2021a) and RepCONC (Zhan
Implementation details. Longtriever employs 24 transformer layers, 12 layers as intra-block en-

---

MARCO MRR@ 100
TREC 2019 Doc
Method
R@ 100
NDCG@ @@ 100
Passage-basee Models
Passage (Devlin et al., 2018)
0.296
0,859
0.285
RetroMAE. Passage
A-MMasaggeegeeeeeee (Liu and Shao, 2022)
0.311
0.883
0.547
0.325
BERT-PAR (Devlin et al., 2018; 1.i et al., 2020)
0.306
0.852
0.531
0.282
RetroMAE PARADE (Liu and Shao, 2022; Li et al., 2020)
0.310
0.870
0.572
0.298
Long-Doocunn Document Models
Longformer (Beltagy et al., 2020)
0.291
0.859
0.520
0.280
BigBird (Zaheer et al., 2020)
0.293
0.859
0.544
0.281
Hi- Ti-Transformer (Wu et al., 2021)
0.279
0.848
0.,11
0.283
XLNet (Yang et al., 2019)
0.279
0.833
0.479
0.261
Longtriever (ours)
0.329
0.893
0.572
0.345
Table 2: Experimental results of different pre-trained models.
Method
Time (ms)
Memory (GiB)
with hard negatives. Longtriever is trained five times on each dataset, and the average perfor- mance is reported. Table 1 presents the retrieval performance of Longtriever and SOTA models. One can clearly see that Longtriever consistently achieves the best performance all metrics over all datasets. Specifically, Longtriever surpasses the strongest baselines by 84. (MRR 100) and +2.29% (Recall @ 100) on MARCO Dev Doc, and by +2.71% % (NDCG@ 10) and +11.25% (Re- call @ 100) on TREC 2019 Doc. By enjoying the merits of nested inter-block and intra-block aggre- gations, Longtriever is capable of precisely mod- eling the semantics within each block, and com- prehensively capturing the global semantic correla- tions between different blocks, leading to superior performance.
Passage- based Models
BERT-PARADE, (4*)12)
674.47
3.85
Moo-DDoemment Models
BEKT-Document (2048)
1083.98
8.42
Longformer (2048)
970.99
6,00
BigBird (2048)
970.06
7.09
Hi-Transformer (4*512)
674.77
4,00
XLNet (4*512)
2463.49
3.99
Longtriever (4*512)
696.10
4.56
Table 3: Time and memory costs of different models (batch size is 16).
coders, and 12 layers as inter-block encoders. The dimension the hidden states is 768, and the vo- cabulary size is 30,522. The masking ratios for the masked language modeling (MLM) and local masked autoencoder (LMAE) are 30% and 50%. The maximum text length for each block is 512, and the maximum number of blocks is 8. The model is continuously pre-trained from the BERT checkpoint on 8x NVIDIA A100 (40GB) GPUs for 8 epochs with batch size of (per device), which takes about 3 days. We use the AdamW (Loshchilov and Hutter, 2017) as the optimizer. The peak learning rate is set to le-4, with linear warmup over 0.1 ratio and linear decay. The weight decay is set to 0.01.
In order to further investigate the superiority of Longtriever on long document retrieval, also present the results of SOTA pre-trained models in Table 2. All models are fine-tuned using in in-batch negatives for a single iteration. The notation "*_ Passage" indicates that the model takes the first 512 tokens of the text as the input (Ma et al., 2022). The results clearly demonstrate that Longtriever consistently achieves superior performance on both datasets.
The time and memory costs of various models are evaluated on single NVIDIA V100 GPU with 32GB memory. To ensure fair comparison, all models given batch of 16 documents in- put, each document comprised of 2048 tokens 4 blocks of 512 length. BERT-Passage is modified to BERT-Document for this comparison (The input length is expanded from 512 to 2048). RetroMAE is omitted, as shares the same backbone with
4.2 Main Results
Following previous work (Ma et al., 2022), Longtriever first fine-tuned the MS MARCO Passage Ranking dataset and subsequently on the MS MARCO Document Ranking dataset. Each fine-tuning stage includes three iterations: one it- eration with BM25 negatives, and two iterations

---

MARCO Dev Doc
0.330
No.
Method
@RR@ @ 100
@@ 100
0.892
I.
Longtriever
0.329
0.893
MRRØ 100
0.325
II.
I w/o LMAE
0,307
0,852
0.889
III.
II w/ MLM
0,280
0,845
Ri
IV.
III w/o [DOC] token
0.272
0,844
0.320
V.
IV w/ inter-block encoder
0.249
0.823
100
100
0.883
Table 4: Ablation studies on Longtriever.
0.315
61 128
256
512
Block size
(a) Performance vs, Block size
BERT. The table 3 shows: The time cost analy- sis reveals that BERT-PARADE, Hi-Transformer, and Longtriever are the fastest models, taking less than 700ms to process a batch, due to their effi- cient hierarchical architectures. On the other hand, BERT-Document, Longformer, and BigBird take approximately 1000ms due to their similar atten- tion strategies. XLNet is the slowest model with over 2000ms, due the low parallelism of its re- current architecture. 2) The memory cost analysis shows that BERT-PARADE, Hi- Ti-Transformer, and XLNet consume no more than 4 GiB, Longtriever is slightly higher, as concatenates an extra to- ken at the beginning of each block in every intra- block encoder layer. As the attention matrix be- comes denser, Longformer, BigBird, and BERT- Document consume more memory in that order. Based on the above analysis, it can be concluded that Longtriever provides an appropriate trade-off between performances and costs.
0.323
MRR 100
0.320
R 100
0.317
0.889
MRR @ 100
0.314
R@ 100
0.1
0.3
0.5
0.7
0.9
LMAE's masking ratio
(b) Performance LMAE's masking ratio
Figure 3: (a) and (b) show the performance of Longtriever with varying block sizes and LMAE's mask- ing ratios on MARCO Dev Doc.
ture the general semantics of the whole document through the LMAE task. After removing the [DOC) token, the model's performance presents a decline in Experiment III, which reveals that to generate high-quality document representation, informa- tion collector among blocks is helpful. In addition, the inter-block encoders are also essential for fa- cilitating interactions between different blocks as shown in the comparison between Experiments IV and V. Therefore, integrating inter-block interac- tions in the modeling of document representation is crucial.
4.3 Ablation Study
Table 4 presents the results of the Longtriever model the MARCO Dev Doc after removing dif- ferent components. Various components are gradu- ally removed from Longtriever. In Experiment II, the model is pre- trained with only the MLM task. In Experiment III, the model is fine-tuned without continuous pre-training (just the BERT checkpoint is loaded). In Experiment IV, the [DOC] token is removed, and mean pooling is used to aggregate the [CLS] hidden states of each block obtain the document representation.
4.4 Hyperparameter Study
The influence of block size and LMAE's masking ratio on the performance of Longtriever is depicted in Figure 3. By maintaining a maximum input length of 2,048, the block size is increased from 64 to 512. One can see that model performance consistently increases with larger block sizes. This is reasonable larger block size brings more rich intra-block correlations while suffering from infe- rior model efficiency. We also increase LMAE's masking ratio from 0.1 to 0.9, while MLM's s mask- ing ratio kept at 0.3. The performance improves with LMAE's masking ratio and reaches a peak at
Based on the results of Experiments I and III, the proposed pre-training phase is quite useful, yielding 17.50% and 5.68% improvements on MRR @ 100 and Recall@ 100. The scarcity of an- notations has been effectively alleviated. More- over, comparing the results of Experiments and II, one can see that the LMAE task brings a 7.17% improvement on MRR @ 100 and 4.81% on Re- call @ 100. It means that Longtriver is able to cap-

---

0.5. After that, the performance declines, which demonstrates that proper masking ratio of LMAE is suitable for Longtriever's pre-training. but a too- aggressive masking ratio harms Longtriever's abil- ity to represent the entire document.
2019. Transformer-xl Attentive language mod- els beyond a fixed-length context. arXiv preprint arXiv: 1eee:990100000
Jacob Devlin, Ming- Wei Chang, Kenton Lec, and Kristina Toutanova. 2018. Bert: Pre-training of deep bidirectional transformers for language understand- ing. arXiv preprint arXiv:1810.0480.
5 Conclusion
We propose Longtriever, a novel dense retrieval model for long documents, Longtriever effectively incorporates both local and global semantic model- ing, while maintaining a desirable time complexity. Besides, Longtriever is pre-trained with novel task LMAE gain better understanding of inher- ent semantics within the long documents. Exper- imental results demonstrate that Longtriever con- sistently outperforms existing retrieval methods on various document retrieval datasets.
Siyu Ding, Junyuan Shang, Shuohuan Wang. Yu Sun, Hao Tian, Hua Wu, and Haifeng Wang. 2020. Ernie- doc: A retrospective long- document modeling trans- former. arXiv preprint arXiv:2012.15666
Luyu Gao and Jamie Callan, 2021a. Condenser: a pre-training architecture for dense retrieval. arXiv preprint arXiv:2104.0822
Luyu Gao and Jamie Callan. 2021b. Unsupervised cor- pus aware language model pre-training dense pas- sage retrieval. arXiv preprint arXiv: 20kv:2100.00 105.0000..
Tianyu Gao, Xingcheng Yao, and Danqi Chen. 2021.
Limitations
Simese: Simple contrastive learning of sentence em- beddings. arXiv preprint 2x4x::214.0088/
One limitation of our current study the unexam- ined performance of the BERT architecture when utilized as a document encoder (BERT-Document) This deficiency is attributed to the substantial GPU memory requirements integral to the fine-tuning process. Our future work endeavours aim to extend the input length of BERT 2048, which is antic- ipated to function as referential upper bound to inform and optimize our methodological approach.
Jiafeng Guo, Yinqiong Cai, Yixing Fan, Fei Sun, Ruqing Zhang, and Xueqi Cheng. 2022. Semantic models for the first-stage retrieval: A comprehensive review. ACM Transactions on Information Systems (TOIS), 40(4):1-42.
Pengcheng He, Xiaodong Liu, Jianfeng Gao, and Weizhu Chen. 2020. Deberta: Decoding-enhanced bert with disentangled attention. arXiv preprint arXiv:2006.03654.
Mandar Joshi, Danqi Chen, Yinhan Liu, Daniel S Weld, Luke Zettlemoyer, and Omer Levy. 2020. Spanbert: Improving pre-training by representing and predict- ing spans. Transactions of the Association for Com- putational Linguistics, 77.
References
Beltagy, Matthew E Peters, and Arman Cohan, 2020. Longformer: The long- documentt transformer. arXiv preprint 2rX::::00440000555550
Vladimir Karpukhin, Barlas Oğuz, Sewon Min, Patrick Lewis, Ledell Wu, Sergey Edunov, Danqi Chen, and Wen-tau Yih. 2020. Dense passage retrieval for open-domain question answering. arXiv preprint arXiv:2004.049000
Wei-Cheng Chang, Felix Yu, Yin-Wen Chang, Yim- ing Yang, and Sanjiv Kumar. 2020. Pre-training tasks for embedding- large-scale retrieval. arXiv prepriat 2000:2200...3332.
Rewon Child, Scott Gray, Alec Radford, and Ilya Sutskever. 2019. Generating long se- quences with sparse transformers. arXiv preprint arXiv:7904 10509.
Nikita Kitaev, Lukasz Kaiser, and Anselm Levskaya. 2020. Reformer: The efficient transformer. arXiv preprint (rXh::2001.045.1
Nick Craswell, Bhaskar Mitra, Emine Yilmaz, Daniel Campos, and Ellen M Voorhees. 2020. Overview of the tree 2019 deep learning track. arXiv preprint arXiv:2003.07820
Kenton Lee, Ming-Wieeee Bei Chang, and Kristina Toutanova. 2019. Latent retrieval for weakly supervised open domain question answering. arXiv preprint 19X0009000000000
Zhuyun Dai and Jamie Callan. 2019. Context-aware sentence/pussage term importance estimation for first stage retrieval. arXiv preprint arXiv: 1910..100.00 10687.
Dmitry Lepikhin, HyoukJoong Lee. Yuanzhong Xu. Dehao Chen, Orhan Firat, Yamping Huang, Maxim Krikun, Noam Shazeer, and Zhifeng Chen. 2020. Gshard: Scaling giant models with conditional com- putation automatic sharding. arXiv preprint arXiv:2006, 166688
Zihang Dai, Zhilin Yang, Yiming Yang, Jaime Car- bonell, Quoe V Le, and Ruslan Salakhutdinov.

---

Canjia Li, Andrew Yates, Sean MacAvaney, Ben He, and Yingfei Sun. 2020. Parade: Passage representation aggregation for document reranking. arXiv preprint 0rXiv:2008.00093
Xinyu Ma, Jiafeng Guo, Ruqing Zhang, Yixing Fan, Xi- ang Ji, and Xueqi Cheng. 2021a. Prop: pre-training with representative words prediction for ad-hoc re- trieval. In Proceedings the 14th ACM International Conference on Web Search and Data Mining, pages 283-2911
Chaozhuo Li, Bochen Pang. Yuming Liu, Hao Sun, Zheng Liu, Xing Xie, Tianqi Yang, Yanling Cui, Liangjie Zhang, and Qi Zhang. 2021. Adsgnn: Behavior-graph augmented relevance modeling in sponsored search. In Proceedings of SIGIR.
Xinyu Ma. Jiafeng Guo, Ruqing Zhang. Yixing Fan, Yingyan Li, and Xueqi Cheng. 2021b. B-prop: boot- strapped pre- pre-training with representative words pre- diction for ad-hoe retrieval. In Proceedings of the 44th International ACM SIGIR Conference Re- search and Development in Information Retrieval pages 1513-1522 1522.
Rui Li, Jianan Zhao, Chaozhuo Li. Di He, Yiqi Wang. Yuming Liu, Hao Sun, Senzhang Wang, Weiwei Deng, Yanming Shen, et al. 2022. House: Knowl- edge graph embedding with householder parameteri- zation. In Proceedings of ICML.
Tri Nguyen, Mir Rosenberg, Xia Song, Jianfeng Gao. Saurabh Tiwary, Rangan Majumder, and Li Deng. 2016. Ms marco: A human generated machine read- ing comprehension dataset. In CoCo@ NIPs.
Jimmy Lin, Rodrigo Nogucira, and Andrew Yates. 2021, Pretrained transformers for text ranking: Bert and beyond. Synthesis Lectures on Human Language Technologies, 14(5).1-225.
Bochen Pang, Chaozhuo Li, Yuming Liu, Jianxun Lian, Jianan Zhao, Hao Sun, Weiwei Deng. Xing Xie, and Qi Zhang. 2022. Improving relevance modeling via heterogeneous behavior graph learning in bing ads. In Proceedings of KDD.
Fangyu Liu, Ivan Vulić, Anna Korhonen, and Nigel Collier. 2021. Fast, effective, and self-supervised: Transforming masked language models into univer- sal lexical and sentence encoders. arXiv preprint arXiv:2104.080
Yingqi Qu Yuchen Ding, Jing Liu, Kai Liu, Ruiyang Ren, Wayne Xin Zhao, Daxiang Dong. Hua Wu, and Haifeng Wang. 2020. Rocketqa: An opti- mized training approach to dense passage retrieval for open-domain question answering. arXiv preprint arXiv:2010.08191.
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Man- dar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019.
Roberta: robustly optimized bert pretraining ap- proach. arXiv preprint vrrXiv:1907.11 1692..
Stephen Robertson, Hugo Zaragoza, et al. 2009. The probabilistic relevance framework: Bm25 and be- yond. Foundations and Trends@ in Information Re- trieval, 33):333-33..
Zheng Liu and Yingxia Shao. 2022. Retromae: Pre- training serrival-orienter transformers via masked auto-encoder. arXiv preprint arXiv: 2205.12033
Yu Sun, Shuohuan Wang. Yukun Li, Shikun Feng, Xuyi Chen, Han Zhang, Xin Tian, Danxiang Zhu, Hao Tian, and Hua Wu. 2019. Ernic: Enhanced represen- tation through knowledge integration. arXiv preprint 1904 904.09242....
Ilya Loshchilov and Frank Hutter. 2017. Decou- pled weight decay regularization. arXiv preprint
17
Shuqi Lu. Chenyan Xiong, Di He, Guolin Ke, Waleed Malik, Zhicheng Dou, Paul Bennett, Tie- Liu, and Amold Overwijk. 2021. Less is more: Pre-training a strong siamese encoder using weak decoder.
Zhoujin Tian, Chaozhuo Li, Zhiqiang Zuo, Zengx- uan Wen, Lichao Sun, Xinyue Hu, Wen Zhang, Haizhen Huang, Senzhang Wang. Weiwei Deng, et al. 2023. Pass: Personalized advertiser-aware sponsored search. In Proceedings of the 29th ACM SIGKDD Conference on Knowledge Discovery and Data Min- ing.
Wenhao Lu, Jian Jiao, and Ruofei Zhang. 2020. Twin- bert: Distilling knowledge to twin-structured com- pressed bert models for large-scale retrieval. In Pro- ceedings of the 29th ACM International Conference on Information & Knowledge Management, pages 2645-2652.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. Advances in neural information processing systems, 30.
Yi Luan, Jacob Eisenstein, Kristina Toutanova, and Michael Collins. 2021. Sparse, dense, and attentional representations for text retrieval. Transactions of the Association for Computational Linguistics, 9:329- 345.
Chuhan Wu, Fangzhao Wu, Tao Qi, and Yongfeng Huang. 2021. Hi-transfformrr: hierarchical interac- tive transformer for efficient and effective long docu- ment modeling. arXiv preprint arXiv:2106.01044
Xinyu Ma, Jiafeng Guo, Ruqing Zhang, Yixing Fan, and Xueqi Cheng. 2022. Pre-train discriminative text encoder for dense retrieval via contrastive span prediction. arXiv preprint arXiv:2204. 10641.
Yonghui Wu, Mike Schuster, Zhifeng Chen, Quoe V Le, Mohammad Norouzi, Wolfgang Macherey, Maxim Krikun, Yuan Cao, Qin Gao, Klaus Macherey, et al.

---

2016. Google's neural machine translation system: Bridging the gap between human and machine trans- lation. arXiv preprint arXiv: 1609.08744.
the Fifteenth ACM International Conference on Web Search and Data Mining, pages 1328 1336.
Hang Zhang, Yeyun Gong, Yelong Shen, Jiancheng Lv, Nan Duan, and Weizhu Chen. 2021a. Adver- sarial retriever-ranker for dense text retrieval. arXiv preprimi 10..0661..
Zhuofeng Wu, Sinong Wang, Jiatao Gu, Madian Khabsa, Fei Sun, and Hao Ma. 2020. Clear: Contrastive learning for sentence representation. arXiv preprint arXiv:2012.15466 15466.
Xingxing Zhang, Furu Wei, and Ming Zhou. 2019. Hib- ert: Document level pre- rre-trainin of hierarchical bidi- rectional transformers for document summarization. arXiv preprint 1XXX:::99S.06666
Shitao Xiao, Zheng Liu, Weihao Han, Jianjin Zhang. Yingxia Shao. Defu Lian, Chaozhuo Li, Hao Sun, Denvy Deng, Liangjic Zhang, et al. 2022a. Progres- sively optimized bi- bi-granular document representation for scalable embedding based retrieval. In Proceed- ings of the ACM Web Conference 2022, pages 286- 296.
Yan Zhang, Ruidan Hc, Zuozhu Liu, Lidong Bing, and Haizhou Li. 2021b. Bootstrapped unsupervised sen- tence representation learning. In Proceedings of the 59th Annual Meeting of the Association for Compu- tational Linguistics and the 11th International Joint Conference on Natural Language Processing ( (Vol- ume I: Long Papers), pages 5168-5180.
Shitao Xiao, Zheng Liu, Yingxia Shao, Tao Di, Bhu- Middha, Fangzhao Wu, and Xing Xie. 2022b. Training large-scale news recommenders with pre- trained language models in the loop. In Proceedings of the 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, pages 4215-4225.
Yukun Zhu, Ryan Kiros, Rich Zemel. Ruslan Salakhut- dinov, Raquel Urtasun, Antonio Torralba. and Sanja Fidler. 2015. Aligning books and movies: Towards story-like visual explanations by watching movies and reading books. In The IEEE International Con- ference on Computer Vision (ICCV).
Lee Xiong. Chenyan Xiong. Li, Kwok-Fung Tang, Jialin Liu, Paul Bennett, Junaid Ahmed, and Arnold Overwijk. 2020. Approximate nearest neighbor neg- ative contrastive learning for dense text retrieval. arXiv preprint urXiv:2007.00800
Yuanmeng Yan, Rumei Li, Sirui Wang. Fuzheng Zhang, Wei Wu, and Weiran Xu. 2021. Consert: A con- trastive framework for self supervised sentence repre- sentation transfer. arXiv preprint arXiv:2105.11 11741.
Zhilin Yang. Zihang Dai, Yiming Yang, Jaime Car- bonell, Russ R Salakhutdinov, and Quoe V Le. 2019. Xlnet: Generalized autoregressive pretraining for lan- guage understanding. Advances in neural informa- tion processing systems, 32.
Manzil Zaheer, Guru Guruganesh, Kumar Avinava Dubey, Joshua Ainslie, Chris Alberti, Santiago On- tanon, Philip Pham, Anirudh Ravula, Qifan Wang. Li Yang, et al. 2020. Big bird: Transformers for longer sequences. Advances in Neural Information Processing Systems, 33: 17283-17777.
Jingtao Zhan, Jiaxin Mao, Yiqun Liu, Jiafeng Guo, Min Zhang. and Shaoping Ma. 2021a. Jointly optimiz- ing query encoder and product quantization im- prove retrieval performance. In Proceedings of the 30th ACM International Conference on Information Knowledge Management, pages 2487-2496.
Jingtao Zhan, Jiaxin Mao, Yiqun Liu, Jiafeng Guo, Min Zhang, and Shaoping Ma. 202 lb. Optimizing dense retrieval model training with hard negatives. In Pro- ceedings of the 44th International ACM SIGIR Con- ference on Research and Development Information Retrieval, pages 1503-1512.
Jingtao Zhan, Jiaxin Mao, Yiqun Liu, Jiafeng Guo, Min Zhang, and Shaoping Ma. 2022. Learning discrete representations via constrained clustering for effec- tive and efficient dense retrieval. In Proceedings of


Reviews:
[{"review_id": "DmRhA3Zspm", "reviewer": "Reviewer_Yhi7", "summary": "", "questions": "1. In Section 4.4, the examination of block size effects raises a question about the choice to halt the block size at 512. Is this decision influenced by computational limitations? The trend suggests that expanding the block size could potentially yield performance improvements, given the capacity to include more tokens.", "limitations": "", "rating": 4, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks a lot for your recognition of our work! We highly value the opportunity to improve our work based on your feedback. In the following part, clarifications are made to address your concerns.\n- “The ablation studies shed some light on the importance of the author's choices along the way but do not provide any deep insight into the method's mechanism. I encourage the authors to provide such analysis to help readers to understand the proposed method.”\n\nThe analysis is made for the two components within Longretriever: the model architecture and the pre-training method.\nFirstly, the proposed model architecture, i.e. the joint utilization of Inter-block encoder and Intra-block encoder, helps to comprehensively capture long document semantics in a computationally efficient way.\nThe computation cost v.s. the vanilla transformer (namely BERT-Document in Table 3) was evaluated in our paper, where Longtriever achieves a notable reduction in terms of both time cost and memory consumption.\n\n| Method             | Time(ms) | Memory |\n|----------------------|----------|--------|\n| BERT-Document (2048) | 1083.98  | 8.42   |\n| Longretriever(4*512) | 696.10   | 4.56   |\n\nThe underlying mechanism is that the vanilla transformer is limited by its O(N^2) time complexity when dealing with long documents. With Longretriever, the long document is partitioned into blocks, and the interaction between different blocks is established by the lightweight inter-block encoder; thus, it achieves an almost O(N) time complexity, which significantly contributes to the working efficiency.\n\nThe proposed model architecture is not only efficient but also expressive. The underlying mechanism is that although the long document is partitioned into blocks, it may still preserve the long-range dependency on top of the Inter-block encoder. The Inter-block encoder can be regarded as a \"message exchanger\" between different blocks, which enables the encoding of one block to be conditioned on the context of the entire document. To verify this point, we evaluate the block-to-block similarity between Longretriever and BERT-PARADE (where each block is independently encoded).\n\n|Method \t  |Average Similarity (cosine)|\n|--------|------|\n|BERT-PARADE  |0.645\t\t\t\t\t\t  |\n|Longretriever|0.812 \t\t\t\t\t\t  |\n\nIt can be observed that Longretriever results in a much higher block-to-block similarity, indicating that different blocks are more closely related during the encoding process.\n\n\n- \"While Longtriever presents a novel contribution, it's important to note that the LMAE method may not be entirely novel. The fundamental concept behind LMAE resembles a pre-training task found in autoencoders, such as BART, which involves sentence reconstruction through masking.\"\n\nWe agree that LMAE resembles BART considering that both of them perform reconstruction for the masked input, i.e. the autoencoding operation. However, LMAE differs from BART in terms of how the autoencoding is conducted. Particularly, BART is mostly for generation tasks; thus, the autoencoding is performed in a Seq2Seq manner. LMAE is for representation tasks (encoding a long document into its embedding); thus, the reconstruction is made purely based on the embedding; in other words, it can be thought to be \"embedding-to-sequence\". The value of LMAE can be observed from the following empirical gain:\n|Method|MRR@100|\n|----------------------|----------|\n|Longtriever|0.329|\n|Longtriever w.o. LMAE|0.307|\n|Longtriever w.o. LMAE & MLM|0.280|\n\nLongtriever w.o. LMAE indicates that LMAE task is disabled and only MLM is preserved for pre-training (which is more similar to the Seq2Seq as performed in BART). Longtriever w.o. LMAE & MLM means LMAE & MLM are both disabled. The above result shows that the autoencoding is helpful (i.e. by performing MLM only as Longtriever w.o. LMAE). However, the performance can be further improved with the incorporation of both LMAE and MLM (as Longtriever).\n\n- \"You don't state you will provide the code for your paper upon acceptance (if you had, and I missed it, please let me know), I believe one could reproduce your results without it, but I (strongly) encourage you to provide it for future researchers.\"\n\nThe source code was submitted in our supplementary material together with the paper. They will also be open to public upon the acceptance. We apologize for making no clear statement about this point in the paper.\n\n- “The organization of Section 5 is very hard to follow. Notably, Tables 1 and 2 are positioned within Section 3, making it difficult for readers to follow the flow of experimental results seamlessly.”\n\nWe are sorry for having the experiment results placed in different parts of the paper. The placement of the experiment results will be adjusted for the camera-ready manuscript. Particularly, we will move backward Table 1/2 by one page, and put together Table 3/4 and Figure 3 in the same page. By doing so, all the results will be fully included by the section on experiment.\n\n- \"In Section 4.4, the examination of block size effects raises a question about the choice to halt the block size at 512. Is this decision influenced by computational limitations? The trend suggests that expanding the block size could potentially yield performance improvements, given the capacity to include more tokens.\"\n\nIndeed the performance will probably benefit from the further increasing of block size. However, knowing that our intra-block encoder is based on BERT whose max position embedding is 512, we cannot improve the size of the intra-block encoder any further beyond 512. Besides, just as you mentioned, the increasing block size will considerably add to the computation cost, which also prevents the block size from unlimited growth.\n\nWe will update our paper accordingly based on your suggestions and our newly presented analysis. Thanks again for your precious comments!", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The paper is well written, providing a clear motivation and view of the comparative landscape.\n2. The goals/challenges of document retrieval (e.g., computational cost, document understanding, scare annotations) are very valuable for readers. It is noteworthy that the authors have effectively tackled each of these aspects within their work.\n3. Longtriever introduces a novel framework and architectural design for dense retrieval. While the concept of integrating document knowledge into dense retrieval is not new, the authors innovate by presenting a fresh paradigm for aggregating representations through the utilization of inter-intra mechanisms.\n4. The authors provide a (very) comprehensive experimental setting, showing their model is competitive with SOTA methods.\n", "reasons_to_reject": "1. The ablation studies shed some light on the importance of the author's choices along the way but do not provide any deep insight into the method's mechanism. I encourage the authors to provide such analysis to help readers to understand the proposed method.\n2. While Longtriever presents a novel contribution, it's important to note that the LMAE method may not be entirely novel. The fundamental concept behind LMAE resembles a pre-training task found in autoencoders, such as BART, which involves sentence reconstruction through masking. \n3. You don't state you will provide the code for your paper upon acceptance (if you had, and I missed it, please let me know), I believe one could reproduce your results without it, but I (strongly) encourage you to provide it for future researchers.\n4. The organization of Section 5 is very hard to follow. Notably, Tables 1 and 2 are positioned within Section 3, making it difficult for readers to follow the flow of experimental results seamlessly.\n", "questions_to_authors": "1. In Section 4.4, the examination of block size effects raises a question about the choice to halt the block size at 512. Is this decision influenced by computational limitations? The trend suggests that expanding the block size could potentially yield performance improvements, given the capacity to include more tokens.", "paper_topic_and_main_contributions": "The authors present Longtriever, an innovative dense retrieval approach that capitalizes on the hierarchical paradigm, leveraging both inter- and intra-mechanisms to deepen the comprehension of document semantics. Additionally, the authors introduce a novel pre-training task, LMAE, designed to enhance the overall generalizability of document inputs. Remarkably, Longtriever surpasses all benchmark methods across two dense retrieval datasets, MS MARCO and TREC.", "excitement": 4, "reproducibility": 2, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "Y4U3y7LSiv", "reviewer": "Reviewer_kLS1", "summary": "", "questions": "As mentioned in \"Reasons To Reject\" above", "limitations": "", "rating": 3, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We are sincerely grateful for your recognition of our work and your constructive suggestions. We would like to take this opportunity to address all your questions with the following clarifications.\n\n- \"The proposed LMAE approach bears a strong resemblance to LexMAE [1], yet the paper fails to provide a comparative analysis or discussion of the two methods.\"\n\nThe relationship and difference between LMAE and LexMAE are discussed as follows. LMAE resembles LexMAE given the fact that both methods leverage autoencoding to strengthen the representation capability of the document encoder. Particularly, the output embeddings from the document encoder are used to reconstruct the original document on top of a decoding network. In this sense, both works are related to MAE [1], which can be regarded as MAE's derivatives in the text domain. All these methods indicate that the masked autoencoding style pre-training can be highly effective in helping the encoding models generate discriminative embeddings.\n\n[1] Masked Autoencoders Are Scalable Vision Learners\n\nThe distinction between LMAE and LexMAE is that the two methods target different representation capabilities of the document encoder, which leads to a completely differentiated pre-training workflow. LMAE is for dense retrieval, where the pre-training targets improving the semantic representation capability for the [CLS]'s embedding. Therefore, the reconstruction is made based on the [CLS]'s embedding. In contrast, LexMAE focuses on sparse retrieval, where the pre-training targets improving the lexical representation capability of the document encoder. To this end, it transforms the entire output embeddings into the vocabulary space and reconstructs the original document on top of it.\n\n- \"The experimental results presented in the paper do not include comparisons with other state-of-the-art models, such as LexMAE and FDG.\"\n\nDespite that LexMAE and FDG are also dedicated to document retrieval, they are two parallel works to Longtriever. LexMAE is about improving the sparse retrieval of documents, and FDG is about leveraging knowledge distillation to improve document retrieval. By comparison, Longtriever is about dense retrieval, where new model architecture and pre-training methods are proposed. That's to say, these methods target different aspects of document retrieval, and it is likely to combine them for a better performance. In this place, we add one simple group of experiments by a collaboration of LMAE and LexMAE (since there is no publicly available checkpoint for the original LexMAE, we use the reproduced model in our experiment). According to the following result, LMAE and LexMAE's ensemble leads to a notable improvement over the performance of each single method, which verifies our point of view.\n| |MARCO Dev Doc| |TREC 2019 Doc| |\n|---|----|---|----|--|\n| |MRR@100|R@100|NDCG@10|R@100|\n|LexMAE|0.444|0.925|-|-|\n|LMAE|0.432|0.942|0.651|0.344|\n|LMAE + LexMAE(our reproduction)|0.445|0.949|0.659|0.391|\n\n- \"The description of the experimental setup is unclear, leaving readers uncertain about key experimental details. For instance, it is not specified in Table 1 whether all methods, including the proposed approach, use a sequence length of 512 as input.\"\n\nThe baselines in Table 1 (all based on vanilla BERT) use a sequence length of 512 (by truncating the first 512 tokens of the document) due to the max-length limitation of BERT. The baselines in Table 2 (based on hierarchical or sparse transformers) use a sequence length of 2048, which is the same as Longtriever. The experiment result in Table 1 reflects that Longtriever effectively utilizes the additional context of the document (beyond the first 512 tokens) to achieve a more accurate retrieval performance; while the experiment result in Table 2 indicates that Longtriever makes better utilization of the long context compared with other efficient transformer architectures.\n\nWe will update our paper in light of your suggestions and our newly presented analysis. Thanks once again for your invaluable feedback!", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- The work proposes a novel pre-training task, the Local Masked Autoencoder (LMAE), which enhances the model's ability to capture the underlying meaning of long texts by leveraging both local and global representations.\n- The writing and organization of the paper are well, making it easy to follow and understand.", "reasons_to_reject": "- The proposed LMAE approach bears a strong resemblance to LexMAE [1], yet the paper fails to provide a comparative analysis or discussion of the two methods.\n- The experimental results presented in the paper do not include comparisons with other state-of-the-art models, such as LexMAE [1] and FDG [2].\n- The description of the experimental setup is unclear, leaving readers uncertain about key experimental details. For instance, it is not specified in Table 1 whether all methods, including the proposed approach, use a sequence length of 512 as input.\n\n[1] Lexicon-Bottlenecked Pretraining for Large-Scale Retrieval. ICLR 2023.\\\n[2] Fine-Grained Distillation for Long Document Retrieval.", "questions_to_authors": "As mentioned in \"Reasons To Reject\" above", "paper_topic_and_main_contributions": "This paper presents Longtriever, an innovative model that seamlessly integrates local and global semantics for long text retrieval while maintaining computational efficiency. The authors introduce a novel pre-training task, the Local Masked Autoencoder (LMAE), which enhances the model's ability to capture the underlying meaning of long texts by leveraging both local and global representations.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Missing_References": "[1] Lexicon-Bottlenecked Pretraining for Large-Scale Retrieval. ICLR 2023.\\\n[2] Fine-Grained Distillation for Long Document Retrieval."}, "extra_text": {}}, {"review_id": "wHTFPFeSEu", "reviewer": "Reviewer_fFzZ", "summary": "", "questions": "1. Have you evaluated Longtriever’s effectiveness on other NLP tasks of long document used in Longformer, BigBird, Hi-Transformer? It would be nice to see if Longtriever not only outperform other models in retrieval tasks but also other NLP tasks.\n2. Where the numbers of PROP and B-PROP come from? I’ve checked the original papers and cannot find the numbers reported in Table 1.", "limitations": "", "rating": 3, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We are sincerely grateful for the reviewer's recognition and the precious comments towards our work! We would like to take this opportunity to address all the questions within the review comments.\n\n- \"From the experiments, it seems the main factor for Longtriever to outperform other models comes from LMAE pre-training, which the other models do not experience, rather than the proposed Longtriever architecture (by comparing III in Table 4 with other long-document models in Table 2). It is possible that if we apply the same LMAE pre-training strategy to other models, we can get similar ranking effectiveness.\"\n\nIt is indeed necessary to compare different model architectures based on the same pre-training recipe. Note that LMAE depends on the model architecture of Longtriever, i.e. intra & inter block encoder, making it non-trivial to be applied to models like BigBird. However, we may find common ground by ablating LMAE and having all the methods pre-trained by MLM (XLNET is the only exception, whose pre-training method GAR was found to be more effective in its original work). Based on this unified pre-training condition, the experiment is performed on MSMARCO-Doc Dev:\n\n| |MRR@100|R@100|\n|--|---|---|\n|Longformer|0.291|0.859|\n|BigBird|0.293|0.859|\n|Hi-Transformer|0.279|0.848|\n|XLNet|0.279|0.833|\n|Longtriever (w. MLM only)|0.307|0.852|\n|Longtriever|0.329|0.893|\n\nIt can be observed that although Longtriever's performance drops if LMAE is ablated, i.e. Longtriever (w. MLM only), it still outperforms baselines other baselines like Longformer, BigBird, which follow the same pre-training strategy as Longtriever (w. MLM only). Therefore, the effectiveness of Longtriever's model architecture can be verified. In fact, the corresponding result was included in the original paper but marked as Longtriever \"w/o MLM\" by mistake. We will correct this mistake in our manuscript.\n\n- “The evaluation is only conducted on MS MARCO document ranking dataset. However, there are many other document ranking datasets, such as Robust04, ClueWeb or Gov2.”\n\nWe analyze Longtriever's performance on Robust04 (the other two call for specific applications from the dataset owner, which is not time feasible within the response period). Due to the highly limited time and computation resources, we directly apply our saved checkpoints from the previous experiment.\nWe use the saved checkpoints in Table 2 where all methods are briefly fine-tuned with MSMARCO query (without using enhancement strategies, like hard negatives). Same as our results on MSMARCO Doc Dev and TREC'19 Doc, Longtriever maintains a notable advantage over the strongest baselines in our original experiment.\n| |Robust04(NDCG@20)|\n|--|--|\n|RetroMAE-PARADE|0.331|\n|BigBird|0.306|\n|Longtriever|0.388|\n\nWe also apply Longtriever's saved checkpoint fine-tuned with hard negatives. For this case, it achieves a NDCG@20 of 0.468, which is much better than the above checkpoint from brief fine-tuning as well as Indri initial rankings' 0.409. However, knowing that the optimization of fine-tuning strategy is not the focus of our work, the relative improvement may already well support the effectiveness of Longtriever.\n\n- \"Have you evaluated Longtriever’s effectiveness on other NLP tasks of long document used in Longformer, BigBird, Hi-Transformer? It would be nice to see if Longtriever not only outperform other models in retrieval tasks but also other NLP tasks.\"\n\nNote that Longtriever is designed mainly for the retrieval of long documents, given its proposed model architecture and pre-training method. Although it can still be applied to deal with short texts (e.g., sentences or passages) on general representation tasks (e.g., STS, clustering, classification, etc.), there is no guarantee of its performance in other applications. In this place, we introduce a brief set of experiments, where Longtriever is compared with SimCSE on STS and SentEval. We may observe that Longretriever's performance is only marginally better than SimCSE. As said, such a result is expected because Longtriever focuses on one specific type of application scenario. \n| |Semantic Textual Similarity| | | | | | | |\n|--|----|---- |---- |---- |---- |--- |--- |--- |\n|Methods|STS12|STS13|STS14|STS15|STS16|STS-B|STS-R|Average|\n|SimCSE-BERT|75.30|84.67|80.19|85.40|80.82|84.25|80.39|81.57|\n|Longretriever|78.28|82.64|79.93|86.03|82.84|84.09|79.08|81.84|\n| |SentEval Transfer Tasks| | | | | | | |\n|Methods|MR|CR|SUBJ|MPQA|SST|TREC|MRPC|Average|\n|SimCSE-BERT|82.69|89.25|94.81|89.59|87.31|88.40|73.51|86.51|\n|Longtriever|81.92|89.19|94.73|90.17|86.33|90.60|75.94|86.98|\n\n- \"Where the numbers of PROP and B-PROP come from? I’ve checked the original papers and cannot find the numbers reported in Table 1.\"\n\nIn fact, both results are collected from COSTA [1], which is the most recent work from the same group of authors of PROP/B-PROP. Such a difference is probably because the reported numbers in PROP/B-PROP were results from re-ranking, where the performances of first-stage retrieval cannot be directly observed. Therefore, our cited results of PROP/B-PROP present a fair comparison with other methods (all based on first-stage retrieval).\n\n[1] Pre-train a Discriminative Text Encoder for Dense Retrieval via Contrastive Span Prediction, Ma et. al., SIGIR'22\n\nWe will update our paper accordingly based on your suggestions and our newly presented analysis. Thanks again for your precious comments!", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The proposed Longtriever architecture is intuitive and the paper is well written and easy to follow.\n2. The fine-tuned Longtriever outperforms other existing retrievers on MS MARCO document ranking dataset.", "reasons_to_reject": "It seems to me that the main contribution of the paper comes from LMAE pre-training, which is a revised version of MAE pre-training specific for modelling long document rather than the proposed architecture. The second issue comes from the evaluation. \n1. From the experiments, it seems the main factor for Longtriever to outperform other models comes from LMAE pre-training, which the other models do not experience, rather than the proposed Longtriever architecture (by comparing III in Table 4 with other long-document model in Table 2). It is possible that if we apply the same LMAE pre-training strategy to other models, we can get similar ranking effectiveness.\n2. The evaluation is only conducted on MS MARCO document ranking dataset. However, there are many other document ranking datasets, such as Robust04, ClueWeb or Gov2.", "questions_to_authors": "1. Have you evaluated Longtriever’s effectiveness on other NLP tasks of long document used in Longformer, BigBird, Hi-Transformer? It would be nice to see if Longtriever not only outperform other models in retrieval tasks but also other NLP tasks.\n2. Where the numbers of PROP and B-PROP come from? I’ve checked the original papers and cannot find the numbers reported in Table 1.", "paper_topic_and_main_contributions": "This paper proposes Longtriever to efficiently model long text through intra-block and inter-block encoders. The main idea is to split text into several sub blocks; then model (or summarize) the information from each sub block using intra-block encoder and use inter-block encoder to further summarize the information from all sub blocks. In addition, the authors propose LMAE pre-training, which is a revised version of MAE pre-training to model long documents, which I think is the main contribution of the paper.", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Missing_References": "No reference of where the numbers of PROP and B-PROP come from."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper presents the Longtriever to efficiently model long text through intra-block and inter-block encoders, and then proposes a pre-trained LMAE task. This paper is well-presented, and the idea is interesting. Longtriever achieves superior performance on two dense retrieval datasets, MS MARCO and TREC. All reviewers were positive about this work, seeing it as solid work falling in an important area for the NLP community.
Decision: Accept-Main