Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This paper proposes an approach based on MuZero to learn strategies to enhance the search query. It borrows the idea of pseudo relevance feedback of Rocchio, selects terms from the top-5 retrieved results, and incorporate a term using a predefined grammar.
The expanded query is used by BM25 to find documents.
The approach is tested on NQ dataset. It is shown to achieve a performance comparable to that of DPR.
The key contribution of the exploration of MuZero for query enhancement, combined with the idea of Rochio. This exploration is new.
main_review: The paper proposes an interesting idea to formulate an enhanced query using a grammar, in which 3 types of operators are allowed. The grammar corresponds to those used in Lucene (and other search engines). So it is quite general.
The key strength of the paper is its exploration of this idea in query reformulation context. RL is used to learn to reformulate the query.
The paper has several weaknesses:
- While the grammar is commonly used in search engines, it is not as refined as one could expect. In pseudo relevance feedback, adding a good term into the query is only part of the task. Another part of to assign an appropriate weight to it. This second part is not taken into account in this paper, or only partially taken into account. It is assumed that a term is added with a few fixed weights. This may be suboptimal.
- The proposed method is very expensive to train. I would assume that it is also expensive in inference (this is not reported in the paper). The high cost is not rewarded at the end: it performance is (at best) similar to that of DPR. So the question is whether it is beneficial to use the expensive solution instead of a simpler one.
- The experiments show that the strategy G1 works the best, although the grammar allows to add other operators. It would be interesting to analyze why this happened. Especially, G4 contains all the operators, and a good model would be able to choose the best one among them. The fact that G4 is much worse than G1 indicates that the model is unable to make a good choice.
- The retrieval using BM25 is a strong limitation to the approach. Although BM25 is a good and efficient IR method for retrieving full documents, for question answering where one deals with much shorter documents or passages, neural models are often better.
- Reading the example provided in the paper, it does not seem that T5 or BERT has a good capability to predict the expansion term to add. Some of the added terms are not very meaningful (e.g. "3"). This may hint that the way new terms are chosen may not work well in practice. This could stem from T5 or BERT used, or from the general learning framework based on Rocchio's pseudo relevance feedback. More analysis is needed to understand the problem.
- In IR, it is often the case that pseudo relevance feedback beyond a few times (usually once) is not beneficial and may easily lead to query drift. The process of expansion contains many steps of pseudo relevance feedback. Would there a high risk of query drift?
- Finally, experiments on more datasets is desired.
summary_of_the_review: + The idea to built a structured query following a grammar is interesting. This follows the work of MuZero.
+ The use of pseudo relevance feedback to generate candidate terms to add in this context is also interesting.
- Unfortunately, the performance is not at the level expected.
- The complexity of the method is very high, making it difficult to use in practice.
- The presented grammar is used only partially in the best performing case. It is unclear if the grammar defined in this way is reasonable.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper investigates a vital direction: How can RL agents learn to use search engines to find information. The authors propose a method that leverages pre-trained models conditioned on machine reading results to guide the selection of query refinement from aggregated search documents. The method performs comparably to neural retrievers on OpenQA-NQ but operates in a more interpretable way.
main_review: Strengths

- Contribution in the important direction of training RL agents to find information via iterative query refinement.
- The propose of generating synthetic search sessions via Rocchi query expansions.
-  Achieving comparable performance in a more interpretable way with pretrained LMs and RL training.
- The paper is well written and easy to follow

Weaknesses

- The results of the paper might be hard to reproduce though the authors have provided enough details. I hope that the author can provide a complete training recipe so the results will be reproducible for future work.
summary_of_the_review: This paper contributes to an interesting direction - training RL agents to find information via iterative query refinement. The proposed method achieves descent performance in a more interpretable way. But one concern is the model reproductivity. It would be very helpful to have the model full training recipe for reproductivity and benchmarking.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This work designs an RL (customized BERT MuZero) agent for query reformulation, where the underlying observation is defined as an aggregation of windows of answer spans found with a machine reading module, document titles, query tokens, and refinements. The action space involves adding keywords to the query or form search terms with three operators that constrain the search space. Due to the lack of training data, synthetic search sessions are generated with a pretrained transformer model and imitation learning. The reward is a linear combination of NDCG (well-established IR ranking for non-binary relevance), a revised NDCG-like metric for exact matches (NDCEM) and a top-k Passage Score. Results on an OpenQA dataset are *close* to neural passage retrieval methods. 


main_review: **Strengths:**
- This work is generally well-written with good coverage of the literature. 
- Most components are motivated and explained throughout the text. 
- There is also extensive qualitative analysis of the query expansion results.

**Weaknesses:**
- Unfortunately, despite the careful fine-tuning and modifications on the compilation of several components, results seem inferior to a neural passage retrieval model. There is a discussion on why this might be the case, and how this work provides the foundations for work on policy synthesis in search and retrieval tasks. But when comparing a distributed MuZero agent trained on 1.6 million steps (approx. 10 days) and a T5 agent with 11 billion parameters over DPR, it makes sense to also report on the number of parameters (model capacity comparison) and the total training times and the number of training examples (time/sample complexity comparison).
- A comparison with related work on RL for query reformulation, e.g., Nogueira & Cho (2017), Buck et al. (2018) and [1]. Albeit these models have been tested in one-step episodes, there is no explanation on why these cannot be trained in a multistep refinement setting.
- Please also include SOTA passage retrieval models, such as [2].

[1] Nogueira, Rodrigo, Jannis Bulian, and Massimiliano Ciaramita. "Learning to coordinate multiple reinforcement learning agents for diverse query reformulation." arXiv preprint arXiv:1809.10658 (2018).
[2] Qu, Yingqi, et al. "RocketQA: An optimized training approach to dense passage retrieval for open-domain question answering." Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. 2021.

**Questions:**
- Just to confirm, for the Rocchio feedback, the method operates in vector space, i.e., w_t and q_t are vector representations?
- Are results reported over multiple trials?
- A proper ablation study over the rewards would have been useful. To my understanding from Figure A.2, if one sets $\lambda_1 = 1$ and $\lambda_2=0$, better performance can be achieved in terms of NDCG and Top-5, rather than the reverse (i.e.,  $\lambda_1 = 0$ and  $\lambda_2 = 1$). Extracting insights for EM is not so clear. Does this mean that the proposed NDCEM does not really benefit the retrieval performance? Please provide further explanations.

**Minor comments:**
- It appears there is a typo after Eq.(1): "This may add a keyword .... $\sum_{idx}$, the search index vocabulary," ==> "$\sum_{idx}$, where $\sum_{idx}$ is the search index vocabulary"
- Please further explain the part after Eq.(3): "This means it is possible to upgrade accessible terms to exact matches, or weight boosting, if they also occur in the *ideal result set* and to exclude accessible terms if they are not present in the *ideal results*"

**Update after rebuttal:**
Thank you to the authors for their response that clarified some of the paper details. I have read the reviews and responses and my score remains unchanged. The main motivation of transparent and interpretable agent-based query reformulation could be a significant contribution but would be better highlighted with experiments on additional datasets, further discussion on the complexity of the method and qualitative results that show the enhanced interpretability over prior "pure RL" works.
(Minor clarification: Albeit a "contemporaneous" work, surprisingly RocketQA has been on arXiv since October 2020.)
summary_of_the_review: While this is, as the authors explain, an endeavor that required customization of several components to reach comparable performance with DPR, the method proposed appears to be computationally expensive without significant improvements over neural retrieval. Alongside the lack of comparison over prior work on RL for query reformulation and the lack of discussion over other dimensions apart from performance (and maybe some qualitative analysis), the paper will benefit from further extensions and revisions.
review_title: null

Meta-review: This paper presents a method to improve search engines; the method is designed based on the BM25 retrieval method and is evaluated on NQ open dataset. The reviewers agree that the motivation is interesting and implementation is reasonable, but the authors have only showed the impact of their approach over one retrieval method and one dataset, which is limited and does not show if the method is general enough or not.
Decision: Reject