Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper proposed multiple pre-training tasks to be used with the bottlenecked masked autoencoder architecture, designed specifically for dense retrieval. The three types of tasks are corrupted passages recovering, related passages recovering, and pre-trained language model (PLM) outputs recovering. The tasks are integrated by formulating each tasks in a unified text-to-text format. Each pre-training task has its own task-specific shallow decoder. The claimed benefits is that the Transformer encoder is forced to compress the information into a dense vector. The empirical results show the proposed method outperforms competitive baselines.
strength_and_weaknesses: Strengths
1. The paper proposed integrating multiple pre-training tasks via a unified text-to-text format. The bottlenecked masked autoencoder also allows task-specific decoders.
2. The results are competitive.

clarity,_quality,_novelty_and_reproducibility: The paper is clear and easy to follow.

Minor suggestions:
1. Section 4, "Approach". "an approach to pre-training an effective ..." -> "pre-train"?

summary_of_the_review: The paper proposed carefully designed pre-training tasks, a way to integrate multiple pre-training tasks and show competitive results empirically.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The paper proposes a multi-task pretraining objective for dense retrieval models. By leveraging the bottlenecked masked autoencoder architecture, the encoder produces a single-vector embedding, which is used to condition the decoder models on three different types of pretraining tasks, which are all trained in parallel: recovery of corrupted (input) passages, recovery of related passages, and recovery of the outputs from an external pretrained LM.
The model is tested on standard web search and open domain QA datasets, and achieves state-of-the-art performance.
strength_and_weaknesses: Strengths:

* simple model that works well
* paper is easy to understand

Weaknesses (**UPDATED AFTER REBUTTAL**):
* the ablation results are not convincing:
    * the improvements from each pretraining objective are relatively small
    * ~could it be that the results are simply better because you effectively train on more examples the more pretraining objectives you have? Figure 2 suggests that convergence might not have been reached, and hence more pretraining objectives would mean that you actually do more training steps (in the sense of training examples) than what the x-axis shows. When removing each of the 5 pretraining objectives, you could control for this by instead training for 1/5 = 20% longer.~
    * there is no qualitative analysis that shows us that the different pretraining objectives actually teach the model distinct "skills". This could be done by training a model with only a single pretraining objective and then doing an error analysis for each.
* ~in your comparisons to SimLM, how do you make sure that the small differences in performance are not actually due to factors of variation other than the pretraining objective? Are all the model details the same between the two models? How do you make sure that both models receive equal treatment in terms of hyperparameter tuning?~
* the model is of limited novelty; neither the bottlenecked model architecture, nor the multi-task pretraining idea are novel. There is some novelty in the concrete pretraining tasks, but their effect is unclear.
clarity,_quality,_novelty_and_reproducibility: The paper is well written, easy to understand and covers a lot of related work. The proposed pretraining tasks are somewhat novel, but are not sufficiently analyzed. The paper puts a lot of emphasis on its method surpassing the previous state-of-the-art. ~Reproducibility is limited, as most model hyperparameters are not explicitly provided, and no code is available.~
summary_of_the_review: The paper proposes a simple method that reaches state-of-the-art in dense retrieval. However, the ablations are not sufficient to convincingly show that the modeling contributions are actually responsible for this success. Hence, the paper would need a major revision before it can be accepted.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper proposes to pre-train a dense retriever with multiple tasks (five in total), which are all related to masked language modelling (multi-decoder). Results on MS-Marco show moderate improvement with respect to baselines.

strength_and_weaknesses: Strengths:

- Unifying/studying the pre-training of information retrieval 
- Improved performances with respect to the state-of-the-art.
- The proposed pre-training is simple to set up, and might be a basis for future pre-training

Weaknesses:

- it is not obvious why the fact that using a more homogeneous task is better: There is a bold statement p.2: "due to the divergences of input formats and learning objectives among different tasks, an arbitrary integration of these tasks is inappropriate, which may even cause detrimental gradient interference, leading to performance degradation" - how is your setting more appropriate? Why having multiple tasks be detrimental - I checked the references and they do not provide any reason (but it is true that the magnitude of the respective losses has to be set as in ref. Kendall et al. 2018).
- no comparison on the BEIR dataset (which would show how robust the pre-training + training procedure is)
- apart from an ablation, there is no further analysis of the 5 proposed losses, although they are combined directly with no discussion on why they should have the same importance.
- No code is provided
- Using DocT5 (POR loss) shows very moderate improvement in comparison to the cost: it relies on a query generation model (trained on MS Marco which is used for the experiments here...) and document generation (which is costly). I fail to see why it justifies the 0.4 drop in MRR@10 (especially since the variance of the results is not really known)

clarity,_quality,_novelty_and_reproducibility: The paper is clear although some sentences are quite obscure (especially in section 4.2 where the loss is described). The paper seems to be reproducible (open source code would be a plus, but is not provided). Novelty-wise, the losses are (somehow) novel to the extent of my knowledge, being somehow inspired by Electra as acknowledged by the authors.


Some points:
- Results reported in table 5 show that losses are quite redundant: it would have been good to do ablations with just one loss to see the effect of each (and also to run experiments several time to get an idea of the variance of the results).

Minor points:
- p.5: "and remains the masked ones unchanged"? "that can deduce its neighbouring relations with other passages."? 
- Figure 1: the "complementary mask prediction" figure part is wrong
- Table 7: which dataset?
- Figure 2: what is a step is not defined


summary_of_the_review: This paper proposes a new pre-training for IR - even if the authors frame it as "multi-task", it is still heavily based on a masked language modelling. Results are good but not so impressive when looking at the SimLM baseline (which shares this idea of corruption loss) and especially at the quite expensive pre-training procedure.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: The authors propose a new pre-training method for dense retrievers, based on a bottlenecked masked auto-encoder. This allows to unify multiple pre-training tasks, using a text-to-text framework, where a different decoder is used for each task. The proposed method is evaluated on multiple benchmarks, such as MS-MARCO, TREC and BEIR.

Overall, the reviews for this paper are borderline. The reviewers like the fact that the proposed method is a simple way to unify different pre-training schemes for retrieval, using a text-to-text framework. On the other hand, the reviewers also believed that the method is quite similar to the SimLM baseline. Moreover, the improvements over baselines are quite small, and some reviewers believe that the ablations are not sufficient to motivate what technical contributions are responsible for the improvements. Given that other works also improve the SimLM baseline, the reviewers feel that this paper needs major revision before acceptance. I tend to share this conclusion, and thus recommend to reject the paper.
justification_for_why_not_higher_score: I think that the limited technical novelty compared to the baseline, and the limited improvements obtained by the method (esp. on the BEIR benchmark), do not justify the acceptance of this paper.
Decision: Reject