Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The authors cast the multi-vector retrieval problem as sparse alignment between query and document tokens. They propose ALIGNER, a novel multi-vector retrieval model that learns sparsified pairwise alignments between query and document tokens (e.g. ‘dog’ vs. ‘puppy’) and per-token unary saliences reflecting their relative importance for retrieval. They show that controlling the sparsity of pairwise token alignments often brings significant performance gains. In a zero-shot setting, ALIGNER scores 51.1 points nDCG@10, achieving a new retriever-only state-of-the-art on 13 tasks in the BEIR benchmark.
strength_and_weaknesses: Strength
1. The authors proposed a new way to learn sparsified pairwise alignments between query and document tokens.
2. The unary saliences can significantly reduce the document token representations minimal performance loss.
3. The method can achieve good performance on BEIR benchmark under the zero-shot setting.

Weakness:
1. No analysis on retrieval speed and index space.
2. The size of the datasets seem not large enough. Most of the datasets have been pre-processed by some existing IR tools. It would be great if the authors could build index from scratch and compare to the method of "Dense Passage Retrieval for Open-Domain Question Answering".

clarity,_quality,_novelty_and_reproducibility: What's the retrieval speed and index space? How the hyper parameters influence the speed and space?
summary_of_the_review: I think this is an interesting paper by introducing a new method of sparsified pairwise alignments, and achieve solid performance on benchmark IR dataset. It would be better if the author can discuss more details about the cost on speed/space and further compare with the setting in dense passage retrieval work.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: In IR, this paper generalizes some neural interaction models like e.g. ColBERT, ME-BERT or COIL, by "learning" the alignement, as well as pruning the multi-vector representations. The model is shown to have a few-sample adaptability on the BEIR dataset.

strength_and_weaknesses: 
Strengths:
- Learning the alignement allows to improve results of already successful models such as ColBERT.
- Few-shot learning based on selecting the right alignment improve the results
- Interesting sparsity regularizer with a target ratio (or number? This is not really clear) of tokens kept for the index

Weaknesses:
- The alignement is not learned
- The sparsity is not really learned since during indexing the ratio is different ($\beta$ vs $\alpha$) and more importantly it is fixed - why this discrepancy?
- Novelty is low (pruning has been proposed in ColBERTer)
- The experimental comparisons lack rigor (compared models have 60x times less parameters, ColBERTer is not included in the experiments, some models could have been tested with the alignement adaptation)
- No complexity analysis (comparing it to ColBERTv2 or ColBERTer)

clarity,_quality,_novelty_and_reproducibility: 
The paper is mostly clear and well organized. There is some novelty although sparsifying is largely studied in IR.

Section 3.2 (sparsity regularizer) is not well motivated - why not using a L1 loss (like in ColBERTer)? The authors should discuss and motivate much more this aspect since it could interesting for other works (but like it is, the approach is not really convincing). Also, why using two pruning ratios (for training and inference)? How were those ratios determined?

In the experiments,

1) ColBERTer and/or ColBERTv2 should be compared with the approach (or at least, your entropy-based regularization should be compared with a simpler L1 one). Some related works cited in the ColBERTer paper are purely ignored, but quite related to what you are doing (e.g. Multi-Vector Attention Models for Deep Re-ranking, )
2) An even simpler pruning strategy, random pruning, has not been experimented with
3) Some extra-care should be taken to compare comparable models (in terms of parameters at least, a 60x ratio is not acceptable)

Other issues:
- Foonote 4: $H$ is not a point-wise entropy if defined that way - looks like the standard entropy (but $\lambda_i$ does not define a probability distribution)
- "We further check" (p. 7): this paragraph is not self-contained and cannot be understood without reading the appendix
summary_of_the_review: This paper proposes to (1) use different alignment patterns for few-shot learning (2) sparsify the set of vectors used to compute a document. There are some unclear parts and a lack of motivation (in particular for (2)), and experimental results lack some rigor in the comparison. The alignment few-shot learning could have been more interesting and innovative by looking at various patterns (and motivating them better).
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: Multi-vector retrieval models can fetch improved results on certain retrieval tasks, while at the same time the approach leads to large search indexes and expensive computation. Thus, this paper proposes ALIGNER, a novel multi-vector retrieval model that learns sparse pairwise alignments between query and document tokens, as well as the unary saliences of each token reflecting their relative importance to the retrieval. The authors show experimentally that controlling the sparsity of pairwise token alignments usually leads to significant performance gains. On the other hand, unary saliences determine whether a token needs to be aligned with other tokens for retrieval. With sparsified unary saliences, it is possible to prune a large number of query and document token vectors and improve the efficiency of multi-vector retrieval. In a zero-sample setup, ALIGNER scores 51.1 points nDCG@10 implements the latest state of the new retrieval-only retriever on 13 tasks in the BEIR benchmark test.
strength_and_weaknesses: Generally speaking, this paper puts forward a universal retrieval scheme, and achieves good results, the specific advantages are as follows:

(1) The sparse alignment of multi-vector retrieval is a good solution to solve the retrieval effect and efficiency.

(2) The new proposed model in this paper has achieved better results than the previous work in many tasks on MSMARCO.

(3) The proposed model in this paper uses sparse alignment matrix to aggregate token-level similarity, where each element represents the alignment of a pair of tokens, which can develop different retrieval models in a unified way and identify the shortcomings of existing models.

However, there is a lack of unified experimental standards and ablation experiments in this paper. So I also have a few doubts about this article:

(1) This paper uses 6B T5-V1.1, but the previous baseline work only GTRxxl has the same size, while ColBERTv2 using multi-vector retrieval model has only 110m model size. However, GTR is a single vector retrieval model, so there is no unified standard to show that the effect of the proposed model in this paper is better than the previous model.

(2) In this paper, similarity and alignment structures are proposed, but no ablation experiments have been carried out to prove the effectiveness of the improved model.

(3) This model still needs to calculate the similarity matrix for queries and documents, so the computational efficiency is not improved compared with other multi-vector retrieval models in the training stage.

clarity,_quality,_novelty_and_reproducibility: Clarity: This paper is well-written and the motivation is very clear.
Quality: The quality of the article is good.
Novelty: In general, compared with the previous multi-vector retrieval model, this paper introduces sparse alignment, which is a good attempt.
Reproducibility: It is can be implemented following the description in the paper.

summary_of_the_review: This paper proposes a sparse aligned multi-vector retrieval model, a novel model structure, which achieves good retrieval results and alleviates the shortcomings of previous work to some extent. Despite there is lack of corresponding ablation experiments, I think it is an overall good paper.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper leads a discussion on multi-vector document retrieval by introducing a sparse alignment perspective that can include serval previous document retrieval scheme. On top of that, the paper proposes an entropy-regularized sparse relaxation of the novel alignment scheme by separating the query-document token-wise similarity scores to a similarity times unary encoding of the query and the document . The advantage of that is to improve efficiency, avoiding costly computation on n*m matrix of the similarity. It is worth mentioning that the sparsity controls can be different between training and test. During training, the paper uses top-1 and during test, the paper find optimal sparsity using cross validation. 
strength_and_weaknesses: Strengths: (1) novel perspective on unifying previous multi-vector document retrieval and build the sparsified end-to-end scheme on top of them. 
(2) Good efficiency trade off on the sparsity side
(3) Good results on the  BEIR benchmark


Weakness: (1) entropy-regularized linear programming? The paper does not explain why  entropy-regularization on the linear programming is a desired choice. For enabling end-to-end training, a quadratic term would also work? 
(2) The paper does not explain why end-to-end training in the  entropy-regularization is necessary. During training, isn't the  model uses top-1 alignment where the only one document token is activated for each query token.
(3) As the sparsity of the training and test can be different, it is good to investigating optimal training sparsity.
(4) When training with top-1 alignment, the model looks similar to ColBERT except for the unary masking. It is good to analyze how and why the  unary masking is beneficial
clarity,_quality,_novelty_and_reproducibility: The paper is clear and of good shape with reasonable reproducibility. It will be good if the author publish training and test code.
summary_of_the_review: The paper provides a novel perspective on unifying various multi-vector document retrieval scheme under a same framework and proposed a sparse alignment on top of the framework. The idea is interesting and novel however is still with some under-explained concerns of mine
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: This paper discusses multi-vector document retrieval based on embeddings as a way of performing Information Retrieval. The paper introduces a sparse alignment perspective that offers a unifying view of previous dense retrieval schemes. The paper proposes Aligner, a model that uses sparse alignment between terms combined with unary weights for term salience.

Experimental evaluation shows that the model outperforms other dense retrieval techniques. Further, the paper shows that using the unary salience score, the number of terms can be pruned to improve storage and runtime requirements with minimal losses of accuracy.

The paper has several weaknesses:
- There is no analysis of runtime complexity. The paper contains study showing that terms can be pruned and runtime improved, but it is unclear what is the starting point and how it compares to other algorithms. In fact, it is not completely clear from the paper how fast sparse methods can be used to limit the number of candidates for which the full alignment is computed.
- The experimental comparisons lack rigor -- models of different sizes and using different training data are compared. Since models from previous work are categorized in the same framework, it should be easy to compare models by changing a single dimension to establish the correct baselines and understand what makes the method work.
- It is hard to understand the paper and the different technical details. This is epitomized by all of the reviewers having different fundamental misunderstandings about the paper that the authors had to correct in the response period.
justification_for_why_not_higher_score: There are three weaknesses: no complexity analysis, low rigor in experimentation and the paper being difficult to understand (see writeup for details).

Additionally, while all reviewers agreed that the paper is marginally above the acceptance threshold, none of the reviewers is particularly excited about the paper.
Decision: Reject