Multi-Vector Retrieval as Sparse Alignment
Authors: Yujie Qian
Jinhyuk Lee
Karthik Duddu
Zhuyun Dai
Tao Lei
Siddhartha Brahma
Iftekhar Naim
Vincent Y Zhao
Venue: Submitted to ICLR 2023 (2023)
Keywords: natural language processing
document retrieval
information retrieval

Abstract: Multi-vector retrieval models improve over single-vector dual encoders on many information retrieval tasks. In this paper, we cast the multi-vector retrieval problem as sparse alignment between query and document tokens. We propose ALIGNER, a novel multi-vector retrieval model that learns sparsified pairwise alignments between query and document tokens (e.g. `dog' vs. `puppy') and per-token unary saliences reflecting their relative importance for retrieval. We show that controlling the sparsity of pairwise token alignments often brings significant performance gains. While most factoid questions focusing on a specific part of a document require a smaller number of alignments, others requiring a broader understanding of a document favor a larger number of alignments. Unary saliences, on the other hand, decide whether a token ever needs to be aligned with others for retrieval (e.g. `kind' from `what kind of currency is used in new zealand'). With sparsified unary saliences, we are able to prune a large number of query and document token vectors and improve the efficiency of multi-vector retrieval. We learn the sparse unary saliences with entropy-regularized linear programming, which outperforms other methods to achieve sparsity. In a zero-shot setting, ALIGNER scores 51.1 nDCG@10, achieving a new retriever-only state-of-the-art on 13 tasks in the BEIR benchmark. In addition, adapting pairwise alignments with a few examples (<= 8) further improves the performance up to 15.7 points nDCG@10 for argument retrieval tasks. The unary saliences of ALIGNER helps us to keep only 20% of the document token representations with minimal performance loss. We further show that our model often produces interpretable alignments and significantly improves its performance when initialized from larger language models.

Under review as a conference paper at ICLR 2023
MULTI-VECTOR RETRIEVAL AS SPARSE ALIGNMENT
Anonymous authors Paper under double-blind review
ABSTRACT
Multi-vector retrieval models improve over single-vector dual encoders on many information retrieval tasks. In this paper, we cast the multi-vector retrieval prob- lem as sparse alignment between query and document tokens. We propose ALIGNER, a novel multi-vector retrieval model that learns sparsified pairwise alignments between query and document tokens (e.g. 'dog' vs. 'puppy') and per-token unary saliences reflecting their relative importance for retrieval. We show that controlling the sparsity of pairwise token alignments often brings sig- nificant performance gains. While most factoid questions focusing on a specific part of document require smaller number of alignments, others requiring broader understanding of a document favor a larger number of alignments. Unary saliences, the other hand, decide whether token ever needs be aligned with others for retrieval (e.g. 'kind' from 'what kind of currency is used in new zealand"). With sparsified unary saliences, are able prune large number of query and document token vectors and improve the efficiency of multi-vector retrieval. We learn the sparse unary saliences with entropy entropy-reeullrrzze linear pro- gramming, which outperforms other methods to achieve sparsity. In a zero-shot setting, ALIGNER scores 51.1 points nDCG@ 10, achieving new retriever-only state-of-the-arr on 13 tasks in the BEIR benchmark. In addition, adapting pair- wise alignments with a few examples (S 8) further improves the performance up to 15.7 points nDCG@ @ 10 for argument retrieval tasks. The unary saliences of ALIGNER helps us to keep only 20% of the document token representations with minimal performance loss. We further show that our model often produces inter- pretable alignments and significantly improves its performance when initialized from larger language models.
INTRODUCTION
Neural information retrieval (IR) has become a promising research direction for improving tradi- tional IR systems. The most-commonly adopted approach called the dual encoder operates by rep- resenting every query and document as single dense vector. Given sufficient annotations, dual encoders directly learn task-driven similarity between vectors, and often surpass traditional IR sys- tems on complex tasks such question answering (Lee et al. 2019 Karpukhin et al.) 2020 Ni et al., 2021 However, these models can struggle to generalize over out-of-domain datasets (Thakur et al, 202 and/or entity-centric questions (Sciavolino et al. 202I) due to the limited representational
capacity of single vectors. As remedy, multi-vector retrieval models (Khattab & Zaharia 2020)
Luan et al. 2021 Gao al., (02T)) instead use multiple vectors, typically the contextualized token vectors, to represent the text. These models largely improve the model expressiveness, and exhibit much stronger performance and robustness compared their single-vector counterparts.
Existing multi- retrieval models such as ColBERT (Khattab & Zaharia 2020) computes query- document similarity by selecting the highest scoring document token for each query token and ag- gregating the scores. This sum-of-max method has two major limitations. First, restricting the selection to single document token can highly sub-optimal for some retrieval tasks. As we will show in our experiments, the retrieval performance can be improved by more than 16 points nDCG@ 10 by relaxing this constraint. Second, the method also leads to a large search index and expensive computation. Specifically, the retrieval and storage cost scales linearly with the query and document length, making multi-vector retrieval models an inferior choice for applications. We directly tackle these challenges to build faster and more accurate models.

---

Under review as a conference paper at ICLR 2023
similarity
alignment
[CLS]
BUS
sim(Q. D))
Σ 1J
Z
DPR
ME-BERT
ColBERT
COIL
(a) Our formulation
(b) Alignment in carlier work
Figure I: (a) We formulate multi-vector retrieval as token-level sparse alignment; (b) Earlier models can be covered by formulation using different alignments.
The representation learning problem of multi-vector retrieval can be formulated optimizing token- level alignment. Specifically, we use sparse aligument matrix to aggregate token-level similarities, where each element indicates the alignment of pair of tokens. From this point of view, we are able to formulate different retrieval models in unified manner (Figure and discern the drawbacks of existing models.
Based on our formulation, we propose ALIGNER, novel multi-vector retrieval model that consists of pairwise alignment and unary salience. Pairwise alignments form the basis of ALIGNER, where pairs of query and document tokens are sparsely aligned based their contextual representations. is discovered that changing the sparsity of alignment can significantly impact the performance on retrieval tasks. For instance, factoid questions often favor small number of alignments since they often focus on a small part a document. However, other queries for different tasks (e.g.. argument retrieval and fact checking) require larger number of alignments for a broader understanding of document. Our findings also support the claim of Dai et al. (2022b) that retrieval tasks with different intents should be modeled differently.
ALIGNER also learns unary saliences, which decides whether each token ever needs to be aligned with any other token for retrieval. This corresponds masking entire row or column of the alignment matrix, rather than individual token alignments. To sparsify entire rows or columns, we introduce an algorithm that produces sparse token salience and end-to-end differentiable based on novel formulation of entropy ceroopy-regulriiin regularized linear programming. Sparsified unary saliences allow us to prune large number of document and query token representations, making multi-vector retrieval more efficient and affordable solution.
We evaluate ALIGNER on the BEIR benchmark (Thakur al. 2021), which covers a diverse set of retrieval tasks in multiple domains In zero-shot setting, show that simply scaling model achieves the state-of-the-art performance, outperforming prior neural retrievers without contrastive pre-training, model-based hard negative mining, distillation. By adapting the pairwise alignments with a few examples from the target task similar to the setup of Dai et al. (2022b) ALIGNER be further improved by up 15.7 points nDCG@ @ 10 argument retrieval tasks. Meanwhile, pruning with our unary saliences can reduce 50% of query tokens for better run-time efficiency and 80% of document tokens for better storage footprint, with less than point decrease of nDCG@ 10. The pairwise alignments and unary saliences are also highly interpretable so that they often serve as concise rationales for retrieval.
MULTI-VECTOR RETRIEVAL AS SPARSE ALIGNMENT
Given query Q and a collection of N documents 8 = (D.)..,, how to represent these textual inputs in order to facilitate efficient search. To this end, one ap- proach is lexical retrieval using sparse bag-of-words representation of the text; the other approach is dense retrieval, which this work focuses on. Dense retrieval models learn a parameterized function that encodes the query and documents into query representation q and document representations (d ...., d' respectively. Typically, each representation is single d-dimensional vector. For retrieval, the similarity function is often defined as ( q d( and documents having high similarity scores to the query are retrieved.
key problem in retrieval
We will release our model checkpoints encourage future research.
2

---

Under review as a conference paper at ICLR 2023
2.1 MULTI- RETRIEVAL
Instead of representing each query and document as a single fixed-length vector, multi-vector re- trieval represents them with multiple token vectors, mainly to improve the limited capacity of fixed- length representations. Specifically, a query Q (91,-..,44 4n and a document D= {d],-..,dd ,m are encoded into of vectors {q........ and {di.... dm J. The similarity function between query and document is re-defined for multi-vector retrieval. For instance, ColBERT (Khattab & Zaharia, 2020) designs the similarity function as follows:
sim @
Σ max q d,
For retrieval, instead of indexing N document vectors, multi-vector retrieval pre-computes N x m document token vectors where is the average length of documents. Then, it retrieves K document token vectors for each query token vector with Maximum Inner-Product Search (MIPS), result- ing in n x K candidate document tokens. The retrieved tokens are used to trace back the original documents (Lee al., 2021a), often followed by final refinement stage that scores the similarity sim @ D) with all token representations of each document and the query (Khattab & Zaharia 2020 We adopt the same practice of ColBERT in our experiments.
2.2 SPARSE ALIGNMENT FORMULATION
A key design question for retrieval models is defining the similarity function in manner that bal- ances model expressiveness and inference cost. To facilitate our discussion, we formalize the sim- ilarities used in previous methods into class of sparse alignment functions. The formulation also leads to a principled extension over existing work, which we will describe in $3
We begin by defining similarity matrix S E _ computed from all pairs of query and docu- ment tokens, where S. = q; dj. Then, we use an alignment matrix A E [0, 1 to compute the similarity between Q and D as follows:
sim - ド=
ΣΣ
Σ
Aij
where Z is a normalization term defined as - Ai. The alignment matrix A can be directly derived from S or computed as function of Q and D.
We constrain the alignment matrix A to be sparsely activated: ||A||o S where Ilo is the number of non non-zero elements in matrix. Sparse activation assumes that only a few query-document token matches are critical for retrieval, inspired by traditional retrieval methods. Indeed, most existing dense retrieval models already enforce the sparse alignment with their own heuristics. Figure illustrates how different models can be described under our formulation:
single [CLS] vector to represent each query and document, This is equivalent to setting A = and otherwise, resulting in
Dense passage retriever (DPR; Karpukhin al. 2020
A lo = 1.
ME-BERT (Laan et al., (021) uses the first & document token vectors for multi-vector representations of documents but a single vector for query. The similarity function is max 9 dj. which is equivalent to setting A1, = 1 when Si,j is the maximum within S1,1 to S1,2, and 0 otherwise. The alignment sparsity is || 1oo =
ColBERT uses the sum-of-max similarity function Σ max St,, that is equivalent to setting an alignment matrix to select the maximum element from each row of S, i.e., Aij when SJ is the maximum within S... ||o in this case.
COIL (Gao et al. 2021). ) similar to ColBERT, also selects the maximum element from each row of S. but requires lexical exact match for selected pair, i.e. A., = 1 when St, is the maximum within (S.,y q: d }. ||o < n in this case.
The choice of similarity and sparsity can have a large impact on model capacity and efficiency. For instance, ColBERT is more expressive and robust than DPR (Thakur et al. 2021). but its retrieval and storage costs are much higher. Our work seeks to further advance expressiveness while retaining strong efficiency. We describe method in the next section.
3

---

Under review as a conference paper at ICLR 2023
similarity Sig q:
saliemce a
stllence
9:
DDO
9:
BBB
dg
alignment
pairwise alignment
usary salience a²* @ n
A
query
documennt
Figure 2: ALIGNER factorizes the alignment matrix into pairwise alignments and unary saliences. Pairwise alignment focuses on the alignment of individual token pairs. Unary saliences are deter- mined by per-token salience features.
ALIGNER
In this section, present ALIGNER built upon the sparse alignment formulation. ALIGNER fac- torizes the alignment matrix into pairwise alignment and unary salience:
A = - O (u & @2))
where @ is the Hadamard product and @ the outer product two vectors. Pairwise alignment Á E [R determines which pairs of query and document tokens should be aligned, with the sparsity constraints tailored for downstream tasks ($3.1). Unary salience 2 E IR" and ud E IR" are sparse token weights deciding whether a token ever needs to be aligned (:3.2).
The factorization is introduced based on two critical hypotheses. First, the optimal sparsity of align- ment can be task-dependent. Instead of imposing top-1 constraint in CoIBERT, activating more than one alignments for a query token can enhance retrieval performance for certain tasks. In our analyses for instance, we observe factoid questions that only concern specific part of a document require a small number of alignments, while some other queries (such as fact checking) require more alignments for broader understanding of the document. We explore different search spaces of the pairwise alignment matrix Á in order to achieve better retrieval performance for each downstream task. Second, alignment is only needed for very few tokens. For example, we analyzed 2000 most retrieved documents in our preliminary study, and found only 12.8% document tokens are retrieved by least one query Intuitively, tokens that are uninformative corresponding to sparse activation over entire row column of A. ALIGNER directly learns the row and column sparsity as unary salience, and utilizes them to enhance retrieval efficiency.
not need to be aligned and stored,
3.1 ADAPTING PAIRWISE ALIGNMENT
Queries and documents can have varied distributions. For example, a query can be single entity, a natural question, or few sentences, and document can range from a short paragraph to a long article. The search intent also changes from task task (Dai al. 20226). These changes can lead to different optimal alignment strategies. We explore the following sparse alignment variants that go beyond the top-1 strategy commonly adopted in existing work:
Top-k. Each query token is aligned with document tokens with highest similarity scores. Precisely, Á. = when the j-th token is within top-k of the row S. When & = 1, it is equivalent COlBERT
Top-p. This strategy is similar to top-k, but instead of aligning each query token with exactly & tokens, makes the number alignments proportional to the document length, i.c., each query token aligns with max( p1ppmmm ml,1) tokens where is the document length and E (0. 1] is the alignment ratio.
Despite their simplicity, these variants can indeed enhance retrieval accuracy significantly on tasks such as argument retrieval. More importantly, while is possible to train separate models for dif- ferent alignment variants, we are interested in fast test-time adaptation using a single shared model as many important retrieval tasks lack sufficient training data (Thakur et al. 2021). Specifically,
Thh analysis was performed on MS MARCO Nguyen et al. 2016 using our implementation of ColBERT.
4

---

Under review as a conference paper at ICLR 2023
first train ALIGNER using fixed alignment strategy such top-1 1 in source domain, and adapt the aligmment strategy to each target task without changing the model parameters the following few-shot alignment adaptation method. Given a corpus !D!!)....,D!!! velevance-annotated query-document pairs from the target task {(e', . )D.).... (e*, D we first retrieve candidate documents with the learned token representations, and decide the pairwise align- ment strategy based on the ranking performance on the annotated data. This adaptation can be per- formed efficiently because the aligmment only concerns the computation of similarity score (Eq. D in the refinement stage. In practice, for some tasks, we are able to find a well-suited alignment strategy and improve the retrieval performance with few 8 annotated examples.
3 We use D(N), }. and a few
3.2 LEARNING UNARY SALIENCE
ALIGNER predicts token saliences from their token representations. For brevity, we only present the formulation for document salience, and query salience is defined similarly. Specifically, the salience of the i-th document token uz is defined as:
uf = 2, f((Wdd d
(3)
where Wd and b' are learnable parameters. f is a non-linear activation function and we use ReLU such that salience is always non-negative. X² (2 are gating variables to control the overall sparsity of ud which we will elaborate next.
For the document salience to be meaningful, we enforce salience sparsity as an inductive bias. ALIGNER jointly optimizes sparse salience with other parts of the model. Since tokens with zero salience do not contribute to computing similarity, our model will be encouraged to identify more important tokens in order to retain good retrieval performance. Note that during training we do not have any explicit annotation on which tokens are important. Instead, u" (and similarly w²) are di- rectly optimized to minimize the training loss, under the sparsity constraint that || = a m) where a is a constant sparsity ratio and m is the document length.
A key question is how we can optimize the unary salience component given the controlled sparsity. We leverage a novel technique called entropy-regularized linear programming to enable end-to-end optimization. Specifically, let k [o m) denotes the desired sparsity, s; f(W" d, + ba) denotes the token score before the sparse gate 2, is applied, and 8, Aa E R. be the vectors {si} and {A
respectively. Ad is computed by solving the following optimization problem:
max T X (*+E((A)
s.t. 1 2 E [0, Vi I,......
where ([)) is the elementwise entropy function and & > O is a small constant. The optimization can be seen as a relaxed top-k operation. Without the entropy term EII()). it becomes an instance of linear programming where the solution Ad is binary mask indicating the top-k values of i.e., 2d =1 I if and only if s; is one of top-k values in 8. This top-k optimization is smoothed by adding the small entropy term (H()) and by relaxing 2 from exact binary [0. Given small E, this still produce a sparse solution X and can be solved using simple vector operations. Specifically, let E i and b; E R for i = 1,-.. . be auxiliary variables that are initialized to zero. We iteratively update these variables using the following equations:
}.
d' = =llnkkk)-Ell -Enn Σ exp
{I
Exp
Si + bi
b/.
-d,0)
(5)
In practice, it is sufficient to run only a few iterations and the final solution is given by 2, exp( s--bita). These vector operations are differentiable so À can be end-to-end trained with other parts of our model. The full derivation of this iterative algorithm is given in Appendix A.
3 We have also explored differentiable alignment with sparsity constraints (Appendix A.2 but alignment adaptation is still necessary to achieve gond performance on target tasks. *H()== = L. -2, log 44. À is not probability distribution. H()) is extension of the entropy function that is applied any positive vector X.
5

---

Under review as a conference paper at ICLR 2023
Supervision
Hard Negatives
Distillation
Retriever
Per-domain
# Param.
Splade,1
MS MS
model-bbaed model. haaeel.aeeed
lexical
CalBERT,2
multi-vector
GTR hnl
Pre train + MS MARCO
lixed
single-vector vector
110M 6B
PROMPTAGATOR
Few (S 8)
single-vector
V
137B + 1 110M
ALIGNERNes//st MARCO How" ($ 8)
lixed
multi-vector
110M 68
Table I: Comparison of different retrieval models.
optionally used for alignment adaptation.
Pruning Multi-vector Retrieval Despite good retrieval performance, multi vector retrieval mod- els are notorious for its expensive teken-level retrieval (Santhanam et al.] (022). This prevents multi-vector retrievers from being widely adopted in practice. With the learned unary salience, we can prune redudent tokens for multi-vector retrieval. Pruning document tokens reduces the number of vectors in search index, and pruning query tokens reduces the number of searches. Compared to the sparse salience method using Li-norm regularization (Hofstätter et al.) 2022), our proposed method offers a direct control over the sparsity. In our experiments, we prune query and document tokens using two pruning ratios 8 and Bd respectively. For each document, we obtain the token salience using B and only store the top B percent of tokens in the index. Similarly we select the top fa percent query tokens to perform max inner-product search. Note that we vary these two ratios to control retrieval efficiency, and these ratios can be smaller than the sparsity ratio a9 and ad which we use as constraints at training time. In the refinement stage, we still use the full model with all token vectors for scoring. With token embedding caching, the computation cost of refinement is relatively small compared to that of retrieval.
4
EXPERIMENTS
4.1
EXPERIMENTAL SETUP
ALIGNER uses shared transformer encoder initialized from T5 version 1.1 (Raffel et al.) 2020). We project token embeddings to 128 dimension and apply L2 normalization. Following GTR (Ni al. 2021) we finetune ALIGNER on MS MARCO with hard negatives released by RocketQA (Qu et al., 2021 The models are trained with a batch size of 256 for 25k steps, using query sequence length of 64 and document sequence length of 256. We train ALIGNER with top-1 pairwise alignment
For retrieval, we pre-compute the token encodings of all the documents the corpus, and use ScaNN (Guo et al., 2020) to index and perform max inner-produce search (MIPS). We retrieve 4,000 nearest neighbors for each query token and return the top-1,000 after the refinement stage. We evaluate ALIGNER
the BEIR benchmark (Thakur al. 2021) and compare with state-of-the- retrieval models shown in Table []] Note that ALIGNER does not rely on contrastive model pre- training (Izacard et al. 20222 Ni et al., 2021), model-basee hard negative mining (Santhanam al.) 202T], or distillation (Santhanam et al. (021). We intentionally decide this simple recipe and focus on studying the impact of pairwise alignment and unary salience.
For few-shot alignment adaptation of ALIGNER (..Ip. we split the test data into multiple folds such that each fold contains examples. Then we find the best alignment strategy that maximizes nDCG@ @ 10 on each fold with & E 21,2,4, 6,8 8} for top-k and p E {0.5%, 1 1%, .55%2%% for top-p. Based on the best alignment strategy from each fold, we measure the retrieval performance on the remaining test examples with the best strategy. We report the average (# std.) of these test scores where the number of test scores equals the number of folds. The average of few-shot adaptation indicates the expected performance of using few examples to choose the best alignment strategy.
4.2 RETRIEVAL ACCURACY
Table 2 shows the document retrieval performance of ALIGNER on both MS MARCO and the BEIR benchmark. For this experiment, we do not prune any query or document tokens with unary saliences. but show their effects in 44.3 instead. ALIGNERIas outperforms state-of-the-an sparse and dense retrievers on MS MARCO. performs slightly worse than ColBERT,,, given that we do not use distillation or model-based hard negatives to optimize the in-domain performance.
5 'ee have trained models with other top-k and top-p pairwise alignments, but the MS MARCO training data favors top-1 alignment (see Appendix A.4 for details).
Unlike ColBERT, ALIGNER does not pad token embeddings for retrieval. Hence, retrieving 4,000 neighbors per query token results in similar number of retrieved candidates to ColBERT

---

Under review as a conference paper at ICLR 2023
Zeto shot
Few shot
CAlBERT
ColBERTe
Pronguapato
AUKavE Rove
ALKNER.
BM225
Spicsk
DPR
OTR
OTR
MARCU
18.7
31.1
38.8
39.7
38.7
38.8
35-
ArguAna
31.5
414
46.3
28.8
59.4
46,2230
Trach6-2030
36.7
27.2
20.5
25.6
26.3
34.8
34.5
34.5
40.9=12
50.25
FEVER
75.3
78.6
56.9
66.0
74.0
78.5
72.4
74.2
77.0
71,2429 75 go4s
Ciiutt-FEVER
17.6
21,8
Seilact
66.5
69.3
47.8
00.0
56.2
69.3
70.4
73.1
65.0
68.8 68.8-21
21,4-22
TREC-COVID
65.6
71.0
56.1
53.9
50.1
73.8
68.3
75,8
75.6
73,3-14 79
NFCorpus
32.5
33.8
33.4
32,9420 33.4
39.5
49,0
Horpor(JA
61.3
68.4
37.1
53.5
99.9
66.7
61.7
65.2
61.4
59.5
63.2-13
SCIDOCS
FQJ
23.6
33.6
27.5
34.9
46.7
35.6
15.4
33.4
43.5
18.4
46.2
29,8 39.9445
14,4*04 16.311
ALIONER
DBPedia
31.3
23.6
44.6
40.6
Queen
78.9
83.5
84.2
88.1
89.2
85.2
82,3
36.0
82. 85.3221
40
Averago
44.0
498
45.1
40.1
49.9
47.1
51.1
49,3230 52,6251
Parama
NQ/QQQQ QuQr
46,2
Table 2: Results on MS MARCO (top; MRR @ 10) and the BEIR benchmark (bottom; nDCG@I1 10). Best zero-shot scores are de- noted in boldface. *: trained with distillation. 1; uses few exam- ples (E8) for task-specific adaptation.
Figure 3: Avg. nDCG@ @ 10 BEIR with different sized mod- cls. AA: alignment adaptation.
Top-k
Top-p
6
16
0.5%
1%
1.5%
2%
5%
ArguAna
28.8
24.4
18.3
14.5
11.4
5.1
33.3
45.5
48.1
46.9
32.6
DDCG@
To0ch6-2020
34.8
50.0
49.3
33.7
24.2
20.3
15.8
5.2
SeiFact
70.4
68.8
65.0
60.9
55.6
38.6
71.1
69.4
67.2
62.7
39.0
TREC-COVID
68.3
74.0
73.2
67.
61.7
41.8
66.8
64.4
56.7
46.7
30.9
FIQA
21.6
ALIGNER +AA ALIGNER
SCIDOCS
14.1
14.3
13.1
11.4
9.7
4.82
14.4
14.9
14,9
14.8
8.1
DRPedia
41.6
39.4
29.6
20.2
14.2
3.94
41.6
41.7
39.9
36.6
17.0
Average
43.1
36.5
19.5
38.8
34.6
0.5% 1% 1.5%
2%
4%
Table 3: nDCG @ 10 on the BEIR benchmark with different k and p in ALIGNERease (trained with top-k = 1).
Figure 4: Avg. nDCG@ 10 BEIR of ALIGNER trained with different alignments (top-k 1,2, 4,8 and top- p= 0.5%, 1% 1%)) 44%
ALIGNERRN achieves the strongest results, showing multi multi-vector retrieval models benefit from large pretrained language models. It also outperforms GTRxx on 9 out of 13 BEIR datasets and advances the retriever-only state-of-the-ard (ColBERT,>) by 1.2 points nDCG @ 10 on average. Figure shows that our
retriever model scales better than single-vector dual encoder GTR.
Alignment Adaptation In the rightmost column of Table 2, we show the effect of adapting pair- wise alignment with ALIGNER the BEIR benchmark. With only 8 examples for finding the proper alignment sparsity, its expected performance reaches 52.6 nDCG@ 10 on average. Alignment- adapted ALIGNER also benefits from scaling up, and consistently outperforms its non-adapted coun- terparts, as shown in Figure 3, The gains are further explained in Table where we show individual task's performance under various alignment strategies. Although ALIGNER trained with top-1 alignment, top-1 is not always the best strategy at inference time. Specifically, for ArguAna, we ob- serve 16 points improvement by adjusting the number of alignments proportional the document length with p = 1.5%. In general, keeping the sparsity low enough is preferable and supports our hypothesis that pairwise alignments should be sparse.
Figure compares ALIGNER variants trained with other pairwise alignment strategies, including top-k = 1, 2, 4,8 and top-p = 0.5%...., 44% We evaluate their performance with training-time align- ment strategy (default) and the optimal alignment strategy selected per dataset (oracle). Among these variants, top-1 and top-0.5% work the best, and increasing & or p hurts performance. Alignment adaptation improves all models, showing that retrieval tasks require different pairwise alignment.
Figure 5 shows the effectiveness of few-shot alignment adaptation-dyyamiice dynamically selecting task- specific alignment strategy based a few examples. When the default alignment (top-k=1) is not optimal, we can identify a good alignment strategy using only 8 examples, which significantly improves model performance argument retrieval tasks. Using 16 examples further improves the
7

---

Under review as a conference paper at ICLR 2023
Zeto sbot
S-shot
16-shot
Ocacie
Promgtogato
80
aDCG@10
60-
40
20-
ArguAaa
Touché-
FEVER
ScaPact
NPCorpus
NQ
HotporQA
HQA
DBPedia
Quera
Figure 5: ALIGNER xxl with few-shot alignment adaptation. We report nDCG 10 on BEIR.
Touch6-2020
TREC-COVID
FEVER
ClinateFEVEE
MR30610
10
NQ
HoqgotA
FIQA
DBBedda
40%
30%
201
105
nilo p
ALJENER (fall)
GTR
38
ALIONER, 6 50%
ReL.U. 8 50%
Hard, Lt. 5 50%6
225 105
431 30%
45% 375 205
ALIGNER, B 10%
Decument token ratio
Attrmoi (TAl)
Ar HUN
GTR
Figure 6: ALIGNER with unary salience on MS MARCO. B and Bd are ratios to prune query and document tokens, respectively.
Figure 7: ALIGNER with unary salience BEIR. We set query pruning ratio B 50% and vary document pruning ratio B We omit datasets with small corpora. We also report the performance of ALIGNER with alignment adaptation (AA).
average score and reduces the variance. However, when the default alignment is already optimal (top-k=1 is optimal for QA tasks), few-shot alignment adaptation hurts performance due to the variance of our few-shot method. Nevertheless, ALIGNER outperforms Promptagator (Dai et al. 2022b), another few-shot retrieval baseline, in 6 out of 11 datasets.
4.3 RETRIEVAL EFFICIENCY
Next show how ALIGNER's 's unary salience improves retrieval efficiency. We train ALIGNER with salience sparsity ratios (t' 50%, ad = 40%) and E = 0.002 based on empirical performance. At retrieval time, we prune query and document tokens with ratios B" and Bd (3.2).
Figure 6 shows the ALIGNER performance MS MARCO with various pruning ratios. When pruned at the same ratio as training (Ba =B"=500 50%, B 40%), the model performs similarly to a full model (MRR @1 10 38.1 vs. 38.8). We can further prune tokens by adjusting 8d and B The model achieves 37.3 MRR @ 10 with Bd 10%, i.e. it remains accurate with only 10% of the original index size. Decreasing the query pruning ratio 8 30% does not sacrifice performance although deceasing B to 10% leads to worse performance. Figure 6 also compares ALIGNER's s entropy-regularize linear program (Eg, 4 with alternative methods. With just ReLU gate and no sparsity constraints ('RLLU' in Figure 6). the model retains a good performance when Bd= 40%, but degrades for smaller Bd Removing the entropy regularization in Eq. leads simply selecting the hard top-k tokens with the highest predicted salience ('Hard" in Figure ( The hard top-k solution performs worse for all Bd. Another method to sparsify salience is adding an Ll-norm regularization ('L1' in Figure 6). With proper coefficient, it performs comparably to our method and slightly better when B 5%. Note that our method has the advantage of explicitly controlling the sparsity ratios, without tuning the coefficient of the L1 term.
much,
ALIGNER's s salience estimation also generalizes to other retrieval datasets. As shown in Figure 7 pruning with 8d 20% with B 30% causes minimal performance decrease for majority of BEIR datasets, which leads to an 80% reduction in the search index size and up to 94% reduction in the computation cost"] We even observe performance increase for Touché-2020, as the model can only retrieve salient tokens after pruning. Besides, we show that alignment adaptation can be combined with pruning, resulting in an effective yet efficient retrieval model.
A Asume brute force search.
8

---

Under review as a conference paper at ICLR 2023
Query
what happens when stop drinking alcohol
alcohol symptoms of alcohol withdrawal may begin from 4 to 12 hours after you cut down stop drinking-
long several days after the last drink, and can last few days they range from mild life- threatening mild withdrawal symptoms may include intense worry .3 nausea or vomiting hak 5 sweat ing..
Doc.
Query
where the heart in the human body
heart the heart is muscular organ in most animals which pumps blood through the hlood vessels of the circul atory system.. [1] blood provides the body with oxygen and nutrients as well as ussists in the removal of metabolic waste s [ in humans the heart is located between the lungs in the middle
Doc.
compartment of the chest. . [3]
Table 4: Examples of the pairwise alignment and unary salience learned by ALIGNER. Three most salient query tokens and their top-1 sairwise-allgneee document tokens are indicated with the same color. We highlight top 50% query tokens and 20% document tokens according to their salience.
4.4
INTERPRETABILITY
Table shows examples of the pairwise alignment and unary salience learned by ALIGNER. The model aligns query tokens to contextually similar document tokens, but not necessarily identical tokens. The salience features also highlighted where important noun phrases and verbs have higher salience, consistent with human intuition. We show more examples of alignments in the Ap- pendix A.3 In general, observe question answering tasks usually require fewer alignments, while other tasks that require broad understanding of the document favor larger number of alignments.
5 RELATED WORK
Recent research on information retrieval often improves the retrieval accuracy with contrastive pre- training (Ni et |021 Izaaard et al. 2022 Oguz et al. 2022), model-based hard negative min- ing (Xiong et al. 2020: Lu cl al. [021] Qu et a. 2021) and knowledge distillation (Santhanam et al.,
2 et al.) 2022 Reddi et al. 2021). Retrieval efficiency improved via quantization (San-
thanam et al. 2021) or lower-dimensional vectors (Hofstätter et al.) (022).
Term importance and salience have long history in information retrieval: from term frequency (tf) and inverse document frequency (idf), to recent BERT-based importance measures (Dai & Callan 2020 Zhao et al., 2021] Formal et al. 2021b]a). These works mostly focus on sparse lexical re- trieval and learn term weights for sparse bag-of- representations. Our work is most related to ColBERTer (Hofstätter et al. 2022), which proposed to fuse single- vector retrieval and multi-vector refinement. While ColBERTer prunes multi-vector word embeddings for refinement and tests in- domain retrieval task, we propose to prune multi-vector embeddings for retrieval and mainly study the generalization of retrieval oft-of-domain retrieval tasks. Zhou & Devlin (20TT) proposed multi-vector attention model for reranking, we have similar formulation but focus on the retrieval.
Recently, Promptagator DDai et al. 2022b) points out the importance of using few annotated amples to adapt to a new retrieval task. Promptagator achieves few-shot task adaptation via query generation (Ma et al., 202 Lee al. [b021b] Dai et al., 2022a) using large language models (Sanh) et al. 2022 Brown et al. 2020 Wei et al. (022). which has high inference cost. ALIGNER is more versatile and can be fast adapted to a new task via few-shot alignment adaptation.
CONCLUSION
In this paper, we introduce ALIGNER, novel sparse alignment method for multi- document retrieval. We first formulate different retrieval models with token-level sparse aligmments and pro- pose ALIGNER to tackle the limitations of existing models. Specifically, ALIGNNER uses pairwise alignments and unary saliences that allow us to adapt to different tasks and prune unimportant to- kens, respectively. As a result, we achieve strong performance on both zero-shot and few-shot doeu- ment retrieval tasks while drastically improving the run-time and storage complexity of multi-vector retrieval. With its interpretable alignments and better performance with large language models, we envision that multi-vector retrieval model can serve strong standalone retriever in the future.

---

Under review as a conference paper at ICLR 2023
REFERENCES
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranay Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. Language models are few-shot learners. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin (eds.), Advances in Neural Information Processing Systems, volume 33, pp. 1877-1901. Curran Associates, Inc., 2020.
Marco Cuturi. Sinkhorn distances: Lightspeed computation of optimal transport. Advances in neural information processing systems, 26, 2013.
Zhuyun Dai and Jamie Callan. Context-aware term weighting for first-stage passage retrieval. In Proceedings of the 42nd International ACM SIGIR Conference on Research and Development in Information Retrieval, 2020.
Zhuyun Dai, Arun Tejasvi Chaganty, Vincent Y. Zhao, Aida Amini, Qazi Mamunur Rashid, Mike Green, and Kelvin Guu. Dialog inpainting: Turning documents into dialogs. In Kamalika Chaud- huri, Stefamie Jegelka, Le Song. Csaba Szepesvári, Gang Niu, and Sivan Sabato (eds.). Inter- national Conference on Machine Learning, ICML. 2022, 17-23 July 2022, Baltimore, Maryland, USA, volume 162 of Proceedings of Machine Learning Research, pp. 4558 4588.458..00 PMLR, 2022a.
Zhuyun Dai, Vincent Y. Zhao, Ji Ma, Yi Luan, Jianmo Ni. Jing Lu, Anton Bakalov, Kelvin Guu, Keith B. Hall, and Ming-Wes Wei Chang. Promptagator: Few-shot dense retrieval from 8 examples. arXiv preprint arX9v:2209. 1755, 2022b.
Thibault Formal, Carlos Lassance, Benjamin Piwowarski, and Stéphane Clinchant. SPLADE v2: Sparse lexical and expansion model for information retrieval. CoRR, abs/2109. 2021a.
Thibault Formal, Benjamin Piwowarski, and Stéphane Clinchant. SPLADE: sparse lexical and expansion model for first stage ranking. In Fernando Diaz, Chirag Shah, Torsten Suel, Pablo Castells, Rosie Jones, and Tetsuya Sakai (eds.), SIGIR '21: The 44th International ACM SIGIR Conference on Research and Development in Information Retrieval, Virtual Event, Canada, July 11-15, 2021, pp. 2288 2292. ACM, 2021b.
doi: 10.1 10.114553404835.4463333
Luyu Gao, Zhuyun Dai, and Jamie Callan. Coil: Revisit exact lexical match in information retrieval with contextualized inverted list. In Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. 2021.
Ruiqi Guo, Philip Sun, Erik Lindgren, Quan Geng, David Simcha, Felix Chern, and Sanjiv Ku- mar. Accelerating large-scale inference with anisotropic vector quantization. In International Conference on Machine Learning. pp. 3887-3896. PMLR, 2020.
Sebastian Hofstätter, Omar Khattab, Sophia Althammer, Mete Sertkan, and Allan Hanbury. Intro- ducing neural bag of whole-words with colberter: Contextualized late interactions using enhanced reduction. arXiv preprint arXiv:2203.1 2022.
Gautier Izacard, Mathilde Caron, Lucas Hosseini, Sebastian Riedel, Piotr Bojanowski, Armand Joulin, and Edouard Grave Unsupervised dense information retrieval with contrastive learning. Transactions on Machine Learning Research, 2022.
Vladimir Karpukhin, Barlas Oguz, Sewon Min, Patrick Lewis, Ledell Wu, Sergey Edunov, Danqi Chen, and Wen-tau Yih. Dense passage retrieval for open-domain question answering. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 6769-6781, 2020.
Omar Khattab and Matei Zaharia. Colbert: Efficient and effective passage search via contextualized late interaction over bert. In Proceedings of the 43rd International ACM SIGIR conference on research and development in Information Retrieval, pp. 39-48, 2020.
Harold W Kuhn and Albert W Tucker. Nonlinear programming. In Traces and emergence of non- linear programming, pp. 247-258. Springer, 2014.
10

---

Under review as a conference paper at ICLR 2023
Jinhyuk Lee, Alexander Wettig, and Danqi Chen. Phrase retrieval learns passage retrieval, too. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pp- 3661-3672, 2021a.
Kenton Lee, Ming- Wei Chang, and Kristina Toutanova. Latent retrieval for weakly supervised open domain question answering. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pp. 6086-6096, 2019.
Kenton Lee, Kelvin Guu, Luheng He, Tim Dozat, and Hyung Won Chung. Neural data augmentation via example extrapolation. CoRR, abs/2102.01332 2021b.
Jing Lu, Gustavo Hernandez Abrego. Ji Ma, Jianmo Ni, and Yinfei Yang. Multi-stage training with improved negative contrast for neural passage retrieval. In Proceedings of the 2021 Conference on Empirical Methods Natural Language Processing, pp. 6091-6103, Online and Punta Cana, Dominican Republic, November 2021, Association for Computational Linguistics. doi: 10.18653/ v1/2021. wwwnpmmmii.n....
Yi Luan, Jacob Eisenstein, Kristina Toutanova, and Michael Collins. Sparse, dense, and attentional representations for text retrieval. Transactions of the Association for Computational Linguistics, 9:329-345, 2021.
Ma, Ivan Korotkov, Yinfei Yang, Keith B. Hall, and Ryan T. McDonald. Zero-shot neural passage retrieval via domain-targetedn synthetic question generation. In Paola Merlo, Jörg Tiedemann, and Reut Tsarfaty (eds.), Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume, EACL 2021, Online, April 19 23, 2021, pp. 1075-1088. Association for Computational Linguistics, 2021. doi: 10. 00.18633v1/2021. eacl-main.99
Tri Nguyen, Mir Rosenberg, Xia Song, Jianfeng Gao, Saurabh Tiwary, Rangan Majumder, and Li Deng. MS MARCO: A human generated machine reading comprehension dataset. In Tarek Richard Besold, Antoine Bordes, Artur S. d' Avila Garcez, and Greg Wayne (eds.), Proceed- ings of the Workshop on Cognitive Computation: Integrating neural and symbolic approaches 2016 co-located with the 30th Annual Conference on Neural Information Processing Systems (NIPS 2016), Barcelona, Spain, December 9, 2016, volume 1773 of CEUR Workshop Proceed- ings. CEUR-WS.org worg 2016. URL cewrr//eeeeeeeeee ws wwg/Vollloooo...oooooooooooo CC11333//OCNNPSS111111 paper9.pdf
Jianmo Ni, Chen Qu, Jing La, Zhuyun Dai, Gustavo Hernández Ábrego, Ji Ma, Vincent Y Zhao, Yi Luan, Keith B Hall, Ming- Wei Chang, et al. Large dual encoders are generalizable retrievers. arXiv preprint arXiv:2112.0783 2021.
Barlas Oguz, Kushal Lakhotia, Anchit Gupta, Patrick Lewis, Vladimir Karpukhin, Aleksandra Pik- tus, Xilun Chen, Sebastian Riedel, Scott Yih, Sonal Gupta, and Yashar Mehdad. mottain-matched pre-training tasks for dense retrieval. In Findings of the Association for Computational Linguis- tics: NAACL 2022, pp. 1524-1534, Seattle, United States, July 2022. Association for Computa- tional Linguistics. doi: 10. 18653v 1/2022 fingii.iinningn-naall. 114.
Yingqi Qu, Yuchen Ding, Jing Liu, Kai Liu, Ruiyang Ren, Wayne Xin Zhao, Daxiang Dong, Hua Wu, and Haifeng Wang. Rocketqa: An optimized training approach to dense passage retrieval for open-domain question answering. In Proceedings of the 202/ Conference of the North American Chapter of the Association for Computational Linguisties: Human Language Technologies, pp- 5835-5847, 2021.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of Machine Learning Research. 21:1-67, 2020.
Sashank J. Reddi, Rama Kumar Pasumarthi, Aditya Krishna Menon, Ankit Singh Rawat, Felix X. Yu, Seungyeon Kim, Andreas Veit, and Sanjiv Kumar. Rankdistil: Knowledge distillation for ranking. In A/STATS, pp. 2668-3376, 2021. URL Iattp://proceedingg .mr. .mlr. press/ v130/redd 2 .html

---

Under review as a conference paper at ICLR 2023
Victor Sanh, Albert Webson, Colin Raffel, Stephen Bach, Lintang Sutawika, Zaid Alyafeai, Antoine Chaffin, Arnaud Stiegler, Arun Raja, Manan Dey, M Saiful Bari, Canwen Xu, Urmish Thakker, Shanya Sharma Sharma, Eliza Szczechla, Taewoon Kim, Gunjan Chhablani, Nihal V. Nayak, Dehajyoti Datta, Jonathan Chang, Mike Tian-Jian Jiang. Han Wang, Matteo Manica, Sheng Shen, Zheng Xin Yong, Harshit Pandey, Rachel Bawden, Thomas Wang. Trishala Neeraj, Jos Rozen, Abheesht Sharma, Andrea Santilli, Thibault Févry, Jason Alan Fries, Ryan Techan, Teven Le Scao, Stella Biderman, Leo Gao, Thomas Wolf, and Alexander M. Rush. Multitask prompted training enables zero-shot task generalization. In The Tenth International Conference on Learning Representations, ICLR 2022, Virtual Event, April 25-29, 2022. OpenReview.net, 2022. URL httts:://ppenrevit net nett/Forummio---vr000000
Keshav Santhanam, Omar Khattab, Jon Saad-Falcon, Christopher Potts, and Matei Zaharia. Colbertv2: Effective and efficient retrieval via lightweight late interaction. arXiv preprint arXiv:2112.01188, 2021.
Keshav Santhanam, Omar Khattab, Christopher Potts, and Matei Zaharia. PLAID: an efficient engine for late interaction retrieval. In Mohammad Al Hasan and Li Xiong (eds.), Proceedings of the 31st ACM International Conference on Information & Knowledge Management, Atlanta, GA, USA, October 17-21, 2022, pp. 1747-1756. ACM, 2022. doi: 10. 111144/551888.35573252 URL 3557325
/ttpp:://doi.or//00..o
Christopher Sciavolino, Zexuan Zhong, Jinhyuk Lee, and Danqi Chen. Simple entity-centric ques- tions challenge dense retrievers. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pp. 6138 6138-6148, 2021.
Nandan Thakur, Nils Reimers, Andreas Rücklé, Abhishek Srivastava, and Iryna Gurevych. Beir: A heterogeneous benchmark for zero-shot evaluation of information retrieval models. In Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 2), 2021.
Jason Wei, Maarten Paul Bosma, Vincent Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew Mingbo Dai, and Quoc V. Le. Finetuned language models are zero-shot learners. In International Conference on Learning Representations, 2022.
Stephen J Wright. Coordinate descent algorithms. Mathematical Programming, 151(1):3-34, 2015.
Lee Xiong, Chenyan Xiong, Ye Li, Kwok-Fung Tang, Jialin Liu, Paul N Bennett, Junaid Ahmed, and Arnold Overwijk. Approximate nearest neighbor negative contrastive learning for dense text retrieval. In International Conference on Learning Representations, 2020.
Hang Zhang, Yeyun Gong, Yelong Shen, Jiancheng Lv, Nan Duan, and Weizhu Chen. Adversarial retriever-ranker for dense text retrieval. International Conference on Learning Representations, 2022.
Tiancheng Zhao, Xiaopeng Lu, and Kyusong Lee. SPARTA: efficient open-domain question swering via sparse transformer matching retrieval. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguisties: Human Language Technologies, NAACL-HLT, pp. 10..565-35 565-575, 2021.
Giulio Zhou and Jacob Devlin. Multi-vector attention models for deep re-ranking. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pp. 5452-5456, 2021.
12

---

Under review as a conference paper at ICLR 2023
A APPENDIX
A.1 DERIVATION OF THE ITERATIVE UPDATES
We present the derivation of Eq 5 for solving optimization problem (4) Section 3.2 The maxi- mization problem can be written as equivalent minimization problem:
max À + (A)
min - A EH())
min
eH( - -81 -11// T>
s.t. 1 X = k, 2 (0, 1), i= i=
Note the term E1 T À will be constant e x k, but we include it in the minimization object to make derivation simpler later.
Now, 1et a R and b E R= be the Lagrangian variables corresponding to the linear constraints 1 X = and 2 S Vi [ The minimization problem is equivalent to its Lagrangian expression:
min 2
max aE3,b50
- =H( e1 X A+a(k
(1
The objective function 66) is strongly convex and the solution space of Á is convex set. As a result, strong duality holds and we can instead solve the dual problem that exchanges the min and max operators in
min -8' etI(A) - -111-11aal 1') aER AER"
max
The optimal solution (a, A) must have the Marush-Kuhn-Tunn (KKT) conditions hold (Kuhn & Tucker 2014), namely
(-s ( +El X+ takk 1acakk11111 - (1 A))
=0
(sta+b)
(si s b; Vi l........
A exp
=xp
E
8
Substituting À using the above equation in 8). the dual problem has simple form:
8 s
max k- 1-++1'b 1 exp ae REO
We can solve this problem using coordinate descent (Wright 2015) by successively maximizing the
function with either fixed. This leads the iterative updates (Eq described in Section 3.2
3.2
s; + bi
a' ll((k) Eln exp
b/- min - d',0)
Discussion In short, we solve the dual problem of optimization 4 by performing coordinate de- cent of the dual variables and b. That is, find the optimal that maximizes the dual objective given a fixed b, and vice versa.
This iterative algorithm also closely related to the Sinkhorn algorithm of Optimal Transport (OT). In fact, Sinkhorn algorithm solves the 2013). However, our work concerns an different optimization instance. While OT solves trans- portation problem where the solution space is defined with the marginal constraints over the rows and columns of a transportation matrix, our optimization problem is constrained with total budget ( 2, k) and upper bounds (44 < 1 Vi). This leads to different iterative updates.
version of Optimal Transport (Cuturi
2, 0.4 already implied the entropy term (()) in the objective.
13

---

Under review as a conference paper at ICLR 2023
A.2 DIFFERENTIABLE ALIGNMENT WITH SPARSITY CONSTRAINTS
Besides the Top-k and Top-p alignments in $3.1 we also explore differentiable pairwise alignment with sparsity contraints (DA). Both Top-k adn Top-p are doing hard selection of alignments, i.e., A. is either or O. We relax it by introducing soft sparsity constraints. Similar to our formulation for unary salience ($3.2), we determine the alignment Á by the following optimization problem:
max (S, + +H((A)
s.t. Σ A. i= I,...,7
A. [0, 1], i= 1, n, j 1....,m
where H()) the elementwise entropy function and > 0 is a small constant. We constrain the 0, the solution of Eq. 9 is the same Top-k. When E > O, the entropy term makes the optimization problem strongly concave, which can be solved by the same algorithm in Appendix A. The solution is differentiable, thus be trained end-to-end in our model.
sum of each row of Á to equal k. When
A.3 QUALITATIVE ANALYSIS
Dataset
Query
Gold Document
Quora
what is best birth- day gift for a friend?
what ig(2) a(4) good!!) birthday gift for a friend?
MS MARCO
a fathom (1) is unit of length in the imperial and the u.s. customary systems equal to 6 feet (1.8288 ml)), used especially for measur- ing the depth of water. there are two yards (6 feet) in an imperial fathom should animals be used for scientific commercial testing?
when would you use a fathom measurement
(dev)
Touché-
animal testing should not he allowed. brtoredd..//runcaaattt [fruncated)... albeit the non- precocious mistakes of scientisus also. [trumcuted].. skepti of the scientist(3) in question's abilities "..fruncatedl...conninno if animals for clinical(4) and basic research." " ... ...ttnncated]... EiruncatedI...majortty majority of the scientific community thinks on this issue.
2020
Table 5: Examples of pairwise alignment with the top-k value up to 4 for the Quora, MS MARCO, and Touché-2020 datasets. Query tokens being aligned are shown blue, and corresponding aligned document tokens are shown in red. The superscript on the document token (k) indicates top-k align- ment. We notice that the top-1 alignment quality is generally good across all three tasks. However, larger value of k results in spurious irrelevant alignments for Quora and MS MARCO, while remains fairly useful for Touché-2020.
Table shows examples of top-k pairwise alignments of query token (highlighted in blue) to the corresponding document tokens for several different tasks. For question-answering (e.g., MS MARCO) and duplicate question retrieval (Quora), fewer alignments seem to be preferable, and as k increases, we start to see spurious alignments to unrelated documents tokens. For argument retrieval tasks such as Touché-2020, on the other hand, larger value of & tends to provide useful semantically relevant alignments (e.g., scientificc vs clinical). These qualitative examples provide intuitive insights regarding why different alignment strategies are helpful for different tasks, and why alignment adaptation is necessary.
A.4 RESULTS ON MS MARCO
Table 6 shows the retrieval performance of ALIGNER and previous models on the MS MARCO dev set. We deliberately kept the training configuration of ALIGNER relatively simpler (e.g., no distillation or model-based hard negatives). However, ALIGNER still achieves the best MRR@ 10 simply because of scaling to larger pretrained language models. We have also trained ALIGNER with other alignment strategies, such as top-4, top-1%, and DA (Appendix A.2). However, the results suggest top-1 is favorable in MS MARCO.
14

---

Under review as a conference paper at ICLR 2023
Model
MRR @1
Recall 1000
BM25
18.7
83.7
SPLADE
36.8
97.9
DPR
31.1
95.2
Ruse
36.6
98.3
GTR large
37.9
99.1
GTRAI
38.5
98.9
GTR
38.8
99.0
ColBERT
36.0
96.8
39.7
98.4
COIL
35.5
96.3
ME-BERT
33.4
ALIGNER
38.8
97.8
ALIGNER large
39.4
98.3
ALIGNERA
39.9
98.4
ALIGNER xxl
40.3
98.7
ALIGNER base (top-4)
37.1
97.5
ALIGNER base (top- 1%)
38.8
97.6
ALIGNERRase (DA)
37.8
97.4
Table Retrieval performance on MS MARCO. The top half shows baselines from previous work. The botton half shows different ALIGNER models. DA: differential alignment. See Appendix A.2
A.5 FULL RESULT TABLES ON BEIR
Tables to 10 presents complete results of ALIGNER'S performance on the BEIR datasets initialized from T5 base, large, XL, and XXL checkpoints. We set k = I during training, and show results across different inference-time alignment strategies (both top-k and top-p). As expected, model scale larger models. Moreover, observe similar benefits of alignment adaptation across all the different model sizes.
accuracy improves
Top-k
Top-p
ALIGNERbssc
1*
2
6
0.5%
1%
1.5%
ArguAna
28.8
24.4
18.3
14.5
11.4
33.3
A.R.
Touché-2020
34.8
50.0
51.1
49.3
46.0
31.3
24.2
20.3
15.8
FEVER
72.4
75.0
68.3
57.6
49.0
72.5
55,0
44.9
29.9
ECC
Elimate-FEVEr
18.1
20.8
23.0
23.0
22.7
18.2
13.7
13.8
10.8
SciFact
70.4
68.8
65.0
60.9
55.6
71.1
69.4
67.2
62.7
NQ
52.2
48.3
36.3
26.6
19.9
52.2
49.2
43.5
36.3
HotpotQA
61.7
58.6
36.0
21.9
13.9
61.7
60.1
54.3
47.3
Q.A.
FiQA
33.4
30.8
23.8
19.5
15.5
33.4
28.5
24.6
19.0
BioASQ
49.6
45.8
37.4
30.8
24.5
47.4
37.7
31.9
25.0
NFCorpus
34.0
33.2
32.0
29.9
27.8
33.6
30.5
28.6
TREC-COVID
68.3
74.0
73.2
67.2
61.7
66.8
64.4
56.7
46.7
SCIDOCS
14.1
14.3
13.1
11.4
9.7
14.9
14.9
MISC.
DBPedia
41.6
39.4
29.6
20.2
14.2
41.6
41.7
39.9
36.6
Quora
82.3
64.9
30.6
13.3
6.3
82.3
82.3
82.3
82.3
Average
47.3
46.3
38.4
31.9
27.0
47.1
44.2
40.9
35.9
Table 7: nDCG@ 10 on the BEIR benchmark with different k and p in ALIGNER * alignment strategy during training (k 1).
15

---

Under review as a conference paper at ICLR 2023
Top-
Top-p
ALIGNERIage
1"
2
4
6
0.5%
1%
1.5%
2%
ArguAna
29.5
25.2
19.5
16.0
13.6
33.2
47.9
A.R.
Touch6-2020
36.7
47.5
53.0
53.4
52.0
32.5
25.4
20.7
16.3
FEVER
72.9
75.2
69.6
61.8
53.6
72.8
52.8
43.0
29.3
EC.
Climate-FEVER
18.6
20.7
23.3
23.4
23.5
18.6
13.1
14.2
11.4
SciFact
71.5
70.7
69.5
67.7
64.1
72.2
70.6
69.6
66.8
NQ
57.2
52.8
43.4
36.3
31.2
57.2
53.7
47.5
41.0
HotpotQA
63.6
62.6
44.8
32,1
24.4
63.6
61.8
55.6
48.5
Q.A.
FIQA
39.4
35.6
30.4
26.2
23.4
39.4
33.3
28.4
22.8
BioASQ
53.3
50.7
43.1
36.8
31.6
49.2
38.2
33.2
27.4
NFCorpus
35.5
33.9
32.5
31.2
29.6
34.6
30.8
TREC-COVID
71.9
79.4
77.3
74.4
69.3
70.0
66.1
59.7
52.4
SCIDOCS
15.3
15.5
14.9
13.6
12.5
15.4
15.8
16.0
MISC.
DBPedia
43.5
41.9
34.7
29.0
24.3
43.5
43.5
41.5
37.4
Quora
84.5
75.5
46.4
20.9
8.4
84.5
84.5
84.5
84.5
Average
49.5
49.1
43.0
37.3
33.0
49.0
45.4
42.3
37.9
Table 8: nDCG@ 10 on the BEIR benchmark with different & and p in ALIGNERRarge ": alignment strategy during training (k = l).
Top-
Top-p
ALIGNER
1*
2
4
6
0.5%
1
1.5%
ArguAna
32.4
28.3
22.5
18.9
16.3
35.3
47.2
A.R.
Touché-2020
36.2
46.4
53.2
51.0
50.5
33.0
26.1
21.9
18.2
FEVER
72.9
75.6
70.9
63.2
56.0
72.9
56.9
48.2
34.1
ECC
Climate-FEVER
18.7
21.2
23.2
23.9
24.2
18.8
15.2
16.6
14.3
SciFact
71.5
70.8
69.7
67.6
65.6
73.0
71.6
69.9
69.5
NQ
58.8
54.9
46.0
39.5
34.1
58.8
55.5
50.1
43.8
HotpotQA
63.9
62.6
45.5
32.9
25.0
63.9
62.5
57.5
51.4
Q.A.
FiQA
40.8
37.4
32.2
28.9
25.2
40.8
35.0
31.2
25.7
BioASQ
53.6
50.4
43.0
36.6
32.2
50.2
39.7
34.4
28.9
NFCorpus
35.4
34.3
33.0
31.5
29.7
35.0
31.6
30.0
TREC-COVID
75.1
80.6
80.5
78.1
73.5
72.5
69.4
62.0
54.0
SCIDOCS
15.4
16.0
15.5
14.3
13.3
15.6
16.2
16.5
16.4
MISC.
DBPedia
43.6
42,2
36.2
30.6
26.5
43.6
43.6
42.4
39.4
Quora
85.3
76.5
50.1
26.9
13.1
85.3
85.3
85.3
85.3
Average
50.3
49.8
44.4
38,9
34.7
49.9
46.8
43.9
39.8
Table 9; nDCG@ 10 the BEIR benchmark with different k and p in ALIGNER, *: alignment strategy during training (k 1).
16

---

Under review as a conference paper at ICLR 2023
Top-
Top-p
ALIGNER
1"
2
4
6
0.5%
1%
1.5%
2%
ArguAna
33.8
29.6
24.1
20.4
18.2
36.6
49.8
A.R.
Touch6-2020
34.5
47.4
49.8
51.1
50.6
30.1
21.8
16.3
11.7
FEVER
74.2
77.0
73.9
67.9
62.2
75.2
47.1
36.2
EC.
Climate-FEVER
19.7
21.6
23.7
23.2
24.3
19.7
12.0
12.3
9.3
SciFact
73.1
71.2
71.3
69.1
67.0
74.4
71.5
69.9
69.5
NQ
60.5
56.0
49.1
44.0
40.1
60.4
54.5
45.8
37.9
HotpotQA
65.2
63.4
48.8
37.7
30.1
65.2
63.0
56.0
48.1
Q.A.
FIQA
43.5
40.3
36.8
33.8
31.4
43.5
35.9
30.4
24.0
BioASQ
54.8
51.1
43.3
38.2
34.4
49.6
35.8
29.4
24.9
NFCorpus
35.2
34.0
32.6
31.5
30.3
34.1
28.0
TREC-COVID
75.8
81.4
80.1
75.6
71.9
75.1
65.4
54.0
45.9
SCIDOCS
17.1
16.8
16.3
15.6
14.2
17.1
17.0
16.6
MISC.
DBPedia
45.0
43.2
38.3
33.9
30.6
45.0
44.9
42.6
35.7
Quora
86.0
79.0
58.6
38.2
22.8
86.0
86.0
86.0
86.0
Average
51.3
50.9
46.2
41.4
37.7
50.8
45.1
40.9
36.4
Table 10: nDCG@ 10 the BEIR benchmark with different and ALIGNER alignment strategy during training (k I).
Top-k
Top-p
ALIGNERpav
2
6
0.5%
1
1.5%
ArguAna
26.5
23.6
18.8
15.9
13.6
30.7
45.7
A.R.
Touché-2020
25.2
32.4
43.9
48.2
49.5
19.7
14.6
11.9
9.5
FEVER
59.0
67.0
72.8
73.6
73.0
59.0
34.3
25.4
18.4
ECC
Climate-FEVER
14.7
17.5
20.9
22.8
23.7
14.7
7.8
7.1
5.1
SciFact
70.1
70.6
70.4
69.2
68.0
69.6
66.7
65.6
64.9
NQ
42.3
49.4
52.4
51.2
49.1
42.3
37.3
30.9
25.3
HotpotQA
57.6
61.0
60.4
57.4
53.0
57.6
55.3
49.4
43.4
Q.A.
FiQA
29.8
32.2
32.0
30.1
27.8
29.9
22.2
17.7
14.0
BioASQ
48.3
50.9
49.0
48.0
45.9
41.1
26.4
21.1
17.4
NFCorpus
31.8
33.5
33.9
33.6
33.0
29.9
26.3
25.7
TREC-COVID
54.8
63.5
72.9
75.4
74.5
53.1
48.7
42.5
38.1
SCIDOCS
13.2
14.1
14.7
14.6
14.0
13.2
12.2
11.7
MISC.
DBPedia
31.5
39.1
42.1
39.9
36.9
31.4
31.4
29.7
25.1
Quora
82.7
80.5
72.1
57.4
41.9
82.7
82.7
82.7
82.7
Average
42.0
45.4
46.9
45.5
43.1
41.1
36.3
33.4
30.5
Table 11: nDCG@ 10 the BEIR benchmark with different & and p in ALIGNER * alignment strategy during training (k =). 4).
17

---

Under review as a conference paper at ICLR 2023
Top-k
Top-p
ALIGNERtese base
1
2
4
6
8
0.5%
1%
1.5%
2%
ArguAna
28.4
23.7
17.9
14.2
11.1
32,9
45.3
48.0
47.7
A.R.
Touché-2020
38.1
51.3
53.0
51.3
50.0
35.0
26.6
22.6
17.1
FEVER
72.1
74.6
67.7
57.5
49.2
72.2
56.1
46.2
31.2
E.C.
Climate-FEVER
17.8
20.3
22.0
22.1
21.8
17.8
13.5
14.4
11.3
SciFact
69.0
67.9
64.8
61.5
57.2
70.5
70.3
68.1
63.9
NQ
52.6
48.2
36.4
26.4
20.2
52.5
50.4
44.7
38.2
HotporQA
60.6
57.7
35.1
21.1
13.1
60.7
59.6
53.9
47.6
Q.A.
FIQA
33.7
29.4
23.5
18.8
15.8
33.6
29.8
26.2
20.6
BioASQ
40.3
48.4
39.9
33.1
26.4
50.0
35.3
NFCorpus
34.7
33.7
32.1
30.2
27.8
34.3
32.7
31.6
29.6
TREC-COVID
67.6
73.4
73.1
67.7
62.5
67.2
60.6
SCIDOCS
13.9
14.2
13.1
11.5
9.9
14.1
14,6
14.8
14,9
MISC.
DBPedia
41.3
40.3
29.5
21.2
14.0
41.2
41.3
40.3
37.1
Quora
83.6
60.8
22.8
8.9
4.3
83.6
83.6
83.6
83.6
Average
43.6
42.9
35.4
29.7
25.5
44.4
42.0
39.3
34.9
Table 12: nDCG@ 10 the BEIR benchmark with different & and p ALIGNER alignment strategy during training (p 1%).
k
ALIGNER
1*
2
4
6
8
0.01
0.02
0.04
0.06
0.1
ArguAna
30.8
26.3
20.1
15.88
12.4
30.8
33.2
38.8
43.5
50.1
A.R.
Touché-2020
37.4
50.1
52.0
51.1
49.0
37.4
35.0
29.8
26.9
19.4
FEVER
68.8
71.4
63.7
46.1
44.3
68.8
68.0
66.4
64.5
56.2
EC.
Climate- FEVER
16.7
19.4
21.2
21.6
21.3
16.7
16.2
15.6
15.6
15.4
Scifacct
69.8
68.2
64.7
61.8
57.0
69.8
69.5
70.3
71.0
70.1
NQ
52.0
47.4
35.0
25.8
18.5
$2.0
$1.6
50.8
50.0
46.0
HotporQA
59.6
56.1
33.4
20.6
12.6
59.6
59.5
60.0
61.0
$9.8
Q.A.
FiQA
32.9
29.6
23.0
18.6
14.5
32.9
32.7
32.9
32.4
30.9
BioASQ
50.4
47.2
38.0
31.4
24.5
50.4
50.4
50.8
50.4
43.7
NFCorpus
34.1
33.6
31.6
29.6
27.4
34.1
34.4
34.2
34.2
33.5
TREC-COVID
63.9
73.2
72.5
67.7
62.8
63.9
65.5
63.6
63.8
54.2
SCIDOCS
13.4
11.6
9.6
14.0
15.4
MISC.
DBPedia
40.9
38.7
28.7
19.9
13.2
40.9
40.5
39.9
39.7
38.6
Quora
82.8
61.2
21.5
7.4
3.5
82.8
83.1
83.5
84.0
85.0
Average
43.6
42.4
34.6
28.6
24.7
43.6
43.6
43.4
43.4
41.2
Table 13: nDCG@ 10 on the BEIR benchmark with ALIGNER base and differentiable alignment (Ap-
pendix A.2 alignment strategy during training (k = 1).
18


Reviews:
[{"review_id": "whKTBhKPc1", "reviewer": "Reviewer_4BA2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "Multi-vector retrieval models can fetch improved results on certain retrieval tasks, while at the same time the approach leads to large search indexes and expensive computation. Thus, this paper proposes ALIGNER, a novel multi-vector retrieval model that learns sparse pairwise alignments between query and document tokens, as well as the unary saliences of each token reflecting their relative importance to the retrieval. The authors show experimentally that controlling the sparsity of pairwise token alignments usually leads to significant performance gains. On the other hand, unary saliences determine whether a token needs to be aligned with other tokens for retrieval. With sparsified unary saliences, it is possible to prune a large number of query and document token vectors and improve the efficiency of multi-vector retrieval. In a zero-sample setup, ALIGNER scores 51.1 points nDCG@10 implements the latest state of the new retrieval-only retriever on 13 tasks in the BEIR benchmark test.", "summary_of_the_review": "This paper proposes a sparse aligned multi-vector retrieval model, a novel model structure, which achieves good retrieval results and alleviates the shortcomings of previous work to some extent. Despite there is lack of corresponding ablation experiments, I think it is an overall good paper.", "strength_and_weaknesses": "Generally speaking, this paper puts forward a universal retrieval scheme, and achieves good results, the specific advantages are as follows:\n\n(1) The sparse alignment of multi-vector retrieval is a good solution to solve the retrieval effect and efficiency.\n\n(2) The new proposed model in this paper has achieved better results than the previous work in many tasks on MSMARCO.\n\n(3) The proposed model in this paper uses sparse alignment matrix to aggregate token-level similarity, where each element represents the alignment of a pair of tokens, which can develop different retrieval models in a unified way and identify the shortcomings of existing models.\n\nHowever, there is a lack of unified experimental standards and ablation experiments in this paper. So I also have a few doubts about this article:\n\n(1) This paper uses 6B T5-V1.1, but the previous baseline work only GTRxxl has the same size, while ColBERTv2 using multi-vector retrieval model has only 110m model size. However, GTR is a single vector retrieval model, so there is no unified standard to show that the effect of the proposed model in this paper is better than the previous model.\n\n(2) In this paper, similarity and alignment structures are proposed, but no ablation experiments have been carried out to prove the effectiveness of the improved model.\n\n(3) This model still needs to calculate the similarity matrix for queries and documents, so the computational efficiency is not improved compared with other multi-vector retrieval models in the training stage.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}, {"review_id": "XVPa2Cl8yW", "reviewer": "Reviewer_sjAr", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "In IR, this paper generalizes some neural interaction models like e.g. ColBERT, ME-BERT or COIL, by \"learning\" the alignement, as well as pruning the multi-vector representations. The model is shown to have a few-sample adaptability on the BEIR dataset.\n", "summary_of_the_review": "This paper proposes to (1) use different alignment patterns for few-shot learning (2) sparsify the set of vectors used to compute a document. There are some unclear parts and a lack of motivation (in particular for (2)), and experimental results lack some rigor in the comparison. The alignment few-shot learning could have been more interesting and innovative by looking at various patterns (and motivating them better).\n", "strength_and_weaknesses": "\nStrengths:\n- Learning the alignement allows to improve results of already successful models such as ColBERT.\n- Few-shot learning based on selecting the right alignment improve the results\n- Interesting sparsity regularizer with a target ratio (or number? This is not really clear) of tokens kept for the index\n\nWeaknesses:\n- The alignement is not learned\n- The sparsity is not really learned since during indexing the ratio is different ($\\beta$ vs $\\alpha$) and more importantly it is fixed - why this discrepancy?\n- Novelty is low (pruning has been proposed in ColBERTer)\n- The experimental comparisons lack rigor (compared models have 60x times less parameters, ColBERTer is not included in the experiments, some models could have been tested with the alignement adaptation)\n- No complexity analysis (comparing it to ColBERTv2 or ColBERTer)\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "OHOcSUaECJ", "reviewer": "Reviewer_5rDb", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper leads a discussion on multi-vector document retrieval by introducing a sparse alignment perspective that can include serval previous document retrieval scheme. On top of that, the paper proposes an entropy-regularized sparse relaxation of the novel alignment scheme by separating the query-document token-wise similarity scores to a similarity times unary encoding of the query and the document . The advantage of that is to improve efficiency, avoiding costly computation on n*m matrix of the similarity. It is worth mentioning that the sparsity controls can be different between training and test. During training, the paper uses top-1 and during test, the paper find optimal sparsity using cross validation. ", "summary_of_the_review": "The paper provides a novel perspective on unifying various multi-vector document retrieval scheme under a same framework and proposed a sparse alignment on top of the framework. The idea is interesting and novel however is still with some under-explained concerns of mine", "strength_and_weaknesses": "Strengths: (1) novel perspective on unifying previous multi-vector document retrieval and build the sparsified end-to-end scheme on top of them. \n(2) Good efficiency trade off on the sparsity side\n(3) Good results on the  BEIR benchmark\n\n\nWeakness: (1) entropy-regularized linear programming? The paper does not explain why  entropy-regularization on the linear programming is a desired choice. For enabling end-to-end training, a quadratic term would also work? \n(2) The paper does not explain why end-to-end training in the  entropy-regularization is necessary. During training, isn't the  model uses top-1 alignment where the only one document token is activated for each query token.\n(3) As the sparsity of the training and test can be different, it is good to investigating optimal training sparsity.\n(4) When training with top-1 alignment, the model looks similar to ColBERT except for the unary masking. It is good to analyze how and why the  unary masking is beneficial", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "1ziDHVa6PY", "reviewer": "Reviewer_waGp", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The authors cast the multi-vector retrieval problem as sparse alignment between query and document tokens. They propose ALIGNER, a novel multi-vector retrieval model that learns sparsified pairwise alignments between query and document tokens (e.g. ‘dog’ vs. ‘puppy’) and per-token unary saliences reflecting their relative importance for retrieval. They show that controlling the sparsity of pairwise token alignments often brings significant performance gains. In a zero-shot setting, ALIGNER scores 51.1 points nDCG@10, achieving a new retriever-only state-of-the-art on 13 tasks in the BEIR benchmark.", "summary_of_the_review": "I think this is an interesting paper by introducing a new method of sparsified pairwise alignments, and achieve solid performance on benchmark IR dataset. It would be better if the author can discuss more details about the cost on speed/space and further compare with the setting in dense passage retrieval work.", "strength_and_weaknesses": "Strength\n1. The authors proposed a new way to learn sparsified pairwise alignments between query and document tokens.\n2. The unary saliences can significantly reduce the document token representations minimal performance loss.\n3. The method can achieve good performance on BEIR benchmark under the zero-shot setting.\n\nWeakness:\n1. No analysis on retrieval speed and index space.\n2. The size of the datasets seem not large enough. Most of the datasets have been pre-processed by some existing IR tools. It would be great if the authors could build index from scratch and compare to the method of \"Dense Passage Retrieval for Open-Domain Question Answering\".\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject