Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper presents a pre-training method specifically for large-scale retrieval in the text domain. Specifically, it uses an auto-encoder architecture, with a bottleneck on the pooled lexicon-based representations in between the encoder and the decoder. After pre-training, the model is adapted to the task of large-scale text retrieval (with a technique called lexicon-weighting retriever). In terms of empirical studies, the paper benchmarked on several text retrieval tasks, either with full fine-tuning, or with zero-shot transfer. Also a lot of analyses are done, including a very extensive one for efficiency. According to the tables and figures, the proposed approach (LexMAE) achieves impressive improvement over methods compared.
strength_and_weaknesses: Strengths:
- While I am less familiar with the detailed metrics used for text retrieval, the results presented in the paper is the most impressive part to me. dIt seems LexMAE can significantly improve prior-arts, and also maintain a very healthy speed-accuracy trade-off. The evaluation is also not just done on one dataset, but on multiple datasets. 
- I also think the technical contributions of the paper is of good value. For example in how to construct a reasonable bottleneck to make the task meaningful enough to pre-train representations, while also get aligned with the downstream task (text retrieval with lexicon-weighting).

Weaknesses:
- One high-level weakness is that pre-training is really supposed to work with not just one task, but with multiple tasks. Right now the pre-training is adapted in way that works particularly well for one task. I am not saying this is bad, but doing pre-training for just one task sounds a bit less scalable -- for 1000 tasks it has to design 1000 pre-training models and actually run them.
- The writing of the paper is also a bit unfriendly to folks who are directly working on text retrieval. It mentioned about different ways of doing retrieval (dense or lexicon-weighting) at the beginning of the paper, and only explain such terms in more detail in later parts. It could be made better to just consolidate all the background knowledge into one section, for easier reference and make the paper more self-contained.
clarity,_quality,_novelty_and_reproducibility: Clarity: As I mentioned above. I think the paper should be quite clear to researchers who are familiar with the terminologies and background to text retrieval. However, it takes some efforts for others who are not directly working on the task to catch up.

Novelty: I do believe the paper has some nice technical contributions, especially on how to design the bottleneck and make the pre-training more friendly to lexicon-weighting based retrieval (though it is a bit too specific a task with the cost of pre-training).

Reproducibility: I don't seem to find a section that's dedicated to technical details or talking about code or model release. So I think it takes some efforts if one wants to reproduce the results presented in the paper. 
summary_of_the_review: Overall, I think the strengths outweighs the weaknesses for the paper. The paper does some task-specific pre-training, and shows some good results. The writing is probably okay for researchers within the domain but a bit effort-taking for ones outside of the domain. The results are impressive and the empirical validations are extensive, but there are concerns about the reproducibility of the paper.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The lexicon-weighting paradigm in large-scale retrieval has achieved certain results. However, the language modeling prefers certain or low-entropy words whereas the lexicon-weighting retrieval favors pivot or high-entropy words. In view of the above problems, a lexicon-bottlenecked masked autoencoder (LexMAE) is proposed to learn the vocabulary representation of importance perception. The lexicon-importance distribution is learned unsupervised by constructing a continuous word package bottleneck. The experimental results show the effectiveness of the proposed method.
strength_and_weaknesses: Strength:
（1）The innovation of this paper is novel. This work proposes a lexicon-bottlenecked masked autoencoder. LexMAE encoder assigns higher importance scores to essential vocabulary lexicons of the masked document but lowers to trivial ones. It is used to solve the gap between language modeling and lexicon-weighting retrieval in the vocabulary-weighted paradigm.
（2）The method proposed in this paper has a good effect and has achieved improvement on multiple datasets.
（3）The experimental analysis of this paper is sufficient. In addition to verifying the effectiveness of the proposed method on multiple datasets, this paper also conducts efficiency analysis, Zero-shot Retrieval, Multi-stage Retrieval Performance, and other experiments to comprehensively verify the effectiveness of the method in this paper.

Weaknesses:
(1) The sparsity defined in this paper is lack description, so it is hard to understand the experiments in Figure 2 and Table 4.  The efficiency of the proposed model is represented by increasing the sparsity, so it is necessary to make it clear in section 3.4.
clarity,_quality,_novelty_and_reproducibility: The structure of this paper is clear, the writing quality is good, and the method flow and experimental results are clearly displayed through a number of charts, which is convenient for readers to understand. The overall quality of this paper is good, and the method proposed in this paper has good novelty.
summary_of_the_review: The writing structure of this paper is clear and the innovation is relatively new. The experiment has fully verified the performance of the method from many aspects, which is a good article on the whole.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper proposed a new training framework for a language model that represents text in a lexicon space. The main motivation is that the large-scale retrieval (first stage retrieval) requires a high retrieval performance and a low latency. The dense representation approach could not work smoothly with the indexing technique developed in the search engine (inverted index etc.) Therefore, a method that maps the text in a sparse latent space is required. To learn such model, the paper proposed an encoder-decoder style self-supervised training framework. The encoder first encodes the text as a sparse embedding. However as the sparse embedding spans over the whole vocabulary. It is trivial to decode the full text from the sparse embedding. To add the difficulties, the author proposes the encode the sparse embedding into a continuous bag-of-word vector. The bag-of-word vector is concatenated with the masked text to be the input for the decoder. The proposed approach achieves a better performance than other state-of-the-art methods. It also achieves faster inference speed (in terms of the query per second metrics.)
strength_and_weaknesses: The paper is well written and the motivation is very clear. I love the idea of mapping the text into lexicon space and uses the inverted index as retrieval method to achieve better performance and better efficiency. 
It is also great to see the improved performance for passage retrieval on MSMarco and better efficiency comparing with the dense embedding.

I have some quick questions:
1. Conceptually speaking, why would the sparse embedding (lexicon representation) approach achieves better performance than dense representation model?
2. For Figure 2, how to calculate the query per second for the dense embedding approach? I just curious whether the sparse embedding approach achieves better efficiency than the dense representation, especially given the fact that there also exists retrieval system for dense representation (e.g. faiss[1] etc.) Are you arguing that the sparse representation is more efficient than the dense one on the CPU?
3. For appendix D, is it sparsify instead of specify?
4. For appendix D, I wonder what is the top-K referring to? Given a text, we can encode it into a sparse embedding. Does the top-K mean the top-K activations on the sparse embedding?
5. I just curious what is the performance if not using the continuous bag-of-word technique?


[1]Johnson, Jeff, Matthijs Douze, and Hervé Jégou. "Billion-scale similarity search with gpus." IEEE Transactions on Big Data 7.3 (2019): 535-547.
clarity,_quality,_novelty_and_reproducibility: I think the paper is well written and the motivation is very good.
The paper is with high quality.
The implementation of the approach should be pretty straightforward, given a researcher with background in MAE and large-scale retrieval system.
summary_of_the_review: I think the paper is novel and interesting to read. The observation of efficiency-efficacy is pretty interesting and important for real-world product. I think this paper is a good one and should be accepted.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper aimed to learn sparse representation for large-scale retrieval, which often enjoys low latency and smaller index size. The author proposed lexicon bottlenecked masked autoencoder (LexMAE). The encoder outputs an importance-aware lexicon representation and the decoder aims to reconstruct the missing tokens via the help of the lexicon representation. After three-staged fine-tuning, LexMAE not only achieved new SOTA results on MS-MARCO and TREC-DL benchmarks, but also enjoyed smaller index size and lower latency.  
strength_and_weaknesses: *Strength*
- Strong empirical results
- Controllable index size and low latency
- Predictions are explainable (advantage of sparse retrieval in general)

*Weakness*
- The sparse lexical presentation from encoder has a different form in pre-training (Eq. 3) and fine-tuning (Eq. 8). Why such disparity? Does performance drop if using the same?
- Many hyper-parameters to tune. For example, three-stage fine-tuning, alpha, and lambda, etc.

clarity,_quality,_novelty_and_reproducibility: Clarity: Writing is clear and easy to follow

Novelty: Proposed pre-training framework is novel

Reproducibility: No code provided, hence no data point to verify the reproducibility

summary_of_the_review: This is a novel work with strong empirical performance, which may have a large impact on the large-scale retrieval community. I would like to learn more details of LexMAE and its ablation.

Specifically,
- How scalable is LexMAE to a larger vocab space? In some applications (e.g., Product Search),  to index a wide range of model serial numbers or code-name of products, we may want the vocab space of size millions or more. Also, real-world text data can also be multilingual, hence the #sub-words can also be more than hundred of thousands.
- Does the pre-trained only (un-supervised) LexMAE always perform better than BM-25? For Table 1&2, can you include the Recall@k numbers of pre-trained only LexMAE, without fine-tuning on any supervised data (in-domain as well for BEIR)?
- How important is FLOP() term of Eq(10) in fine-tuning and how much it affect the performance?
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: This paper introduces a lexicon-bottlenecked masked auto encoder (LexMAE) that allows to train text representations. The motivation behind this work is to obtain high throughput encoders that allow to index large text databases for document retrieval. The proposed method is well motivated and described, the experiments presented support the claims. It has been noted that while obtaining good results, the proposed approach offers a healthy speed accuracy tradeoff. As weaknesses, as noted by reviewer Bg1k, some of the evaluations, especially comparing with BM25 or contriver are a bit misleading because of the amount of data available to fine-tune the model. The authors have provided zero-shot evaluations, which should be included in the main paper. The results are less overwhelming that in the case of finetuning, but do not invalidate the merit of this work. Given all that is stated above, I recommend accepting this paper as a poster.
justification_for_why_not_higher_score: As stated in the meta review, when the proposed method is evaluated in pure zero-shot, the performance compared to BM25 is much less impressive. However, the comparison in Table 2 includes methods which do not use as much data for finetuning. I would like the authors to include this result and discuss that in the main paper. 
justification_for_why_not_lower_score: The paper is well motivated and the experiments support the claims. Reviewers have judged this work as well executed.
Decision: Accept: poster