Learning to Coordinate Multiple Reinforcement Learning Agents for Diverse Query Reformulation

Abstract: We propose a method to efficiently learn diverse strategies in reinforcement learning for query reformulation in the tasks of document retrieval and question answering. In the proposed framework an agent consists of multiple specialized sub-agents and a meta-agent that learns to aggregate the answers from sub-agents to produce a final answer. Sub-agents are trained on disjoint partitions of the training data, while the meta-agent is trained on the full training set. Our method makes learning faster, because it is highly parallelizable, and has better generalization performance than strong baselines, such as an ensemble of agents trained on the full data. We show that the improved performance is due to the increased diversity of reformulation strategies. 

Reviews and author responses:

Thanks for your comments and questions!

Question 1: It is counter-intuitive, e.g., why sub-agents trained on full training dataset obtain worse results than on its subset. 

Answer: This question is similar to questions 1 and 2 from AnonReviewer1, and we hope to have answered it there.


Question 2: Regarding diversity, one may use different random seeds or different dropout rates instead of sample a subset of training data. 

Answer: We did try other techniques such as random seeds and different, but we observed no or small gains. We also found that tuning the hyperparameters need to be done carefully to avoid unstable training or very low-performing sub-agents. Randomly partitioning the training set is easy to be done and we found that training is stable for all sub-agents as well as they end up having similar individual performance but different policies.


Question 3: The baseline is much lower than the current SOTA systems. Such as the best result on SearchQA in this paper is 50.5 in terms of F1 score. However, R3 and Re-Ranker obtain 55.3 and 60.6 respectively. Could the proposed approach be adapted to those models? Note that those SOTA systems are released.

Answer: We have new results using a SOTA system as an environment (Devlin et al., 2018, “BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding”), and will add them to the paper.


Question 4: The proposed system is quite similar to Nogueira & Cho 2017 and Buck et al. 2018. I'm not very sure the contribution of this work and its novelty.  

Answer: Although in their papers they show improvements using a single agent, scaling up to multiple agents by simple averaging their final predictions does not work, as shown by the performance of RL-10-Ensemble (table 1, 4-5th rows). In this work, we provide a simple method to combine the multiple agents that led to much higher performance.


Question 5: Why the authors didn't use beam search during the sub-agent training?

Answer: Using greedy and beam search decodings during training do not work because they do not allow for the exploration of new, unknown, rewrite strategies, which are fundamental to the agent to learn new, and possibly better, reformulation strategies. In other words, greedy and beam search decodings produce the best rewrite that has been successful so far, and they do not try new rewrites. Additionally, beam search is more expensive because for each original question there will be N calls to the environment (where N is the beam size). Since those calls are the slowest part of our pipeline, training time will be roughly N times slower. On the other hand, sampling from the decoder gives the agent the ability to explore with sometimes random rewrites while requiring only one environment call per original question.

Question 6: It seems that the proposed framework is a pipeline model: firstly it trains a bunch of sub-agents; and then trains meta-agent. Is it possible to fine-tune the model jointly?

Answer: We tried to fine-tune the model jointly but did not notice any improvement in performance. One advantage of the 2-step training we used is that we can preprocess and store all the reformulations and answers obtained by reformulators so that the aggregator training becomes much faster because it does not require any environment call, which is the slowest part in our pipeline.

Question 7: What is Extra Budget in Table 1?

Answer: It is the RL-10-Full trained for the same amount of time as an RL-10-Ensemble. This experiment was to show that our method keeps improving even when we use 10x more training time.


Thanks for your questions and careful review.

Question 1: It’s not clear why a random partition should be better than a semantically-motivated partition.
Answer: Besides the analysis provided in the appendix, we conjecture that the agents need to be unbiased so they can reformulate well off-topic questions that they will see at inference time. Any form of training that biases the agents, including training on semantically-motivated query clusters, will lead to a poor test/generalization performance. We will include this comment in the paper.

Question 2: It’s also not clear why training the reformulating agents individually on these partitions would do better than an ensemble.
Answer: When trained on the full dataset, each agent in the ensemble needs to find a single policy that is good for the whole dataset. If we partition, then the policy of each agent needs to be good only for a fraction of the training examples. This increases the diversity of policies because the agents see different training examples. With higher diversity, we are able to achieve better ensemble performance. For a similar result, please see Parascandolo et al., 2018. “Learning Independent Causal Mechanisms,” section 5.3, paragraph “A simple single-net baseline.”

Question 3: Why does the function for z_j in equation 2 need to be so complicated?
Answer:  Although equation 2 could be a simple concatenation of query and doc embeddings (table 8, 2nd row), we found that the more complicated version gave us a higher performance with minimal extra processing time. 

Question 4: Why are the CNN features of the query concatenated twice in the first part.
Answer: The query vector is concatenated with the element-wise multiplication of the query and document vectors, and the element-wise subtraction of the query and document vectors. We will change the notation to make this clear.

Question 5: What does the dot operator in the second part of the equation correspond to?
Answer: It is the element-wise multiplication of two vectors. We will add this description to the text.


Thanks for your review. We answered your questions below:

Q1: Thanks for the suggestion, we will move the diversity analysis to the main part of the paper.

Q2: Although each agent in the RL-10-Ensemble is trained in parallel, each is trained on the full training set. Thus, they take ten times more to reach the same aggregated performance of the ten agents trained on partitions. If we train each agent in the RL-10-Ensemble for the same time we use to train RL-10-Sub, then the RL-10-Ensemble performance is 3-5% lower than the RL-10-sub.

Q3: We agree that the gains come from both reformulators and aggregator and we will rephrase the sentence in the appendix to emphasize that.

Q4: We have new experiments with a state-of-art Q&A system (Devlin et al., 2018, “BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding”) as an environment and will add the results to the paper.

Q5: We presented that failure case to demonstrate that there is still room for improvement. On average, our method performs better than the others.

As for the originality, although ensemble methods are well studied, we found that the common practice of averaging the agents' probabilities do not work in our reinforcement learning setup. Thus, we provide one simple method to train multiple agents and another simple method to aggregate their results that, when combined, work much better than standard ensembling methods.


After replacing the StandardAnalyzer in Lucene with the EnglishStemmerAnalyzer (with Porter stemmer) from https://github.com/castorini/Anserini we noticed an improvement of 20-40% in all metrics in all three datasets. 

Thanks a lot for suggesting this modification!


The authors proposed a variant of ensemble method in reinforcement learning for query reformulation. They train multiple specialized sub-agents on disjoint partitions of the training data, and use a meta-agent, which can see all the training data, to decide the final answer. This can speed up the training thanks to parallelization. They observed that this can improve the diversity of learnt reformulations and the overall performance in some cases. 


Strengths
1. The paper is clear and easy to follow.
2. Multiple evaluation metrics and baseline models are considered

Weaknesses
1. The proposed method is simple and lacks novelty.
2. The performance improvement is marginal and some empirical results are not carefully analyzed. 


Significance
Exploring a diverse set of strategies is beneficial in reinforcement learning. This paper focuses on two aspects of this problem. One is how to learn diverse agents and the other one is how to efficiently learn these agents efficiently, which are important concerns in practice. 


Originality
The model learning approach they proposed is merely a simple variant of ensemble learning. The main difference is they train sub-agents on disjoint partitions of the training data, which seems a trivial modification although this shows to improve the overall model performance.


Technical Quality
Overall, the experiments are well-thought, but the following questions need to be explained:
1.	In the Introduction, the authors claim three contributions they made in this paper. My question is, if the third one is really an important contribution, why didn’t the authors demonstrate it in detail in the main text? Attaching it to the appendix could make the reader confused about its significance.
2.	In Table-1, the authors claim that their proposed architectures can outperform the baseline RL-10-Ensemble with only 1/10 time. The sub-agents are trained on a partition of the training set. My question is, are these sub-agents trained in parallel on different machine? If so, why cannot the RL-10-Ensemble be trained in parallel through some multithread or distributed computation? The implementation of the proposed model and the baseline seems not that fair.
3.	The main architecture is described in section 3.3 and 3.4, including the Sub-agents and the Aggregator. However, in Appendix C.1, the authors claim that the gains the proposed method comes mostly from the pool of diverse reformulators, and not from the simple use of a re-ranking function (Aggregator). This is confusing because if it is true, the proposed method is really reduced to an ensemble of the baseline model.
4.	In Table-2, some of the results are worse than the baseline methods like Re-Ranker. Although the authors claim the re-ranking is a post-processing, Re-Ranker performs significantly better than the proposed model. If the authors want to better demonstrate the advantages of the proposed model, a comparison between the proposed model with re-ranking and the Re-Ranker is required.
5.	In Table 10, why the proposed method fails to produce the right answer whereas the other methods perform well?


Summary:
The authors propose to train multiple distinct agents, each over a different subset of the training set. A meta-agent, known as the aggregator, groups and scores answers from the sub-agents for any given input. 

Each agent produces a unique reformulation that is applied to the environment, producing an answer for the reformulated query. The aggregator receives the original query and the answers provided by the environment and produces a relevance score for each answer with respect to the original query that is a function of both components.

The final answer is select using this relevance score, as well as an aggregate ranking score for over the space of reformulations for each answer.

The aggregator is trained to minimize the cross-entropy of the relevance score. Each reformulation agent is trained using Recall@40 as a reward for retrieving the correct answer from the environment given their reformulation. 

The authors argue that learning multiple specialized sub-agents is easier than learning a generalist agent responsible for being able to model the entire training data. Authors shows that this strategy is even more generalizable than training an ensemble for the same number of agents over the entire training set. Authors apply the approach to query reformulation for document retrieval and QA.

Review:

Pros:
-The paper provides convincing empirical evidence that training multiple distinct agents on different partitions of a dataset to learn to reformulate queries for environment feedback is a more efficient and accurate approach than training single or ensemble model on the whole dataset. Empirical result show that both the addition of the aggregator and the exclusivity of the agents contributes to this effect. Baselines are considerable and in-depth (though it seems like the Hui et al., 2017 model that is SOTA on TREC-CAR could be shown in Table 1 as well)
-The paper is well written and easy to understand in the approach.

Cons:
-The authors could do a better job explaining a couple of unclear points. First, how did the authors come up with equation 2 for computing the relevance score? While the empirical investigation in Table 8 indicates it does better than other simpler formulations, it’s not clear why the authors were motivated to try this one.
-I don’t come away with an idea of WHY the author’s proposed approach works better. While the empirical investigation is a contribution in it of itself, the results seem slightly counterintuitive. It’s not clear why a random partition should be better than a semantically-motivated partition. It’s also not clear why training the reformulating agents individually on these partitions would do better than an ensemble. I find the paper interesting, but the analysis of these results is missing.

Questions:
Why does the function for z_j in equation 2 need to be so complicated? Why are the CNN features of the query concatenated twice in the first part. What does the dot operator in the second part of the equation correspond to?

In this paper, authors proposed an ensemble approach for query reformulation (QR).  The basic idea is that 1) train a bunch of models/sub-agents on subsets, e.g., randomly partitioned, of the training data; 2) and then train an additional meta model/meta-agent to aggregate the results from the step 1).  They conduct experiments on document retrieval and question answering tasks to show the effectiveness of the proposed model.

This paper is well written and easy to follow.  
However there are several my concerns. 

1. It is counter intuitive, e.g., why sub-agents trained on full training dataset obtain worse results than on its subset. Regarding diversity, one may use different random seeds or different dropout rates instead of sample a subset of training data. 

2. The baseline is much lower than the current SOTA systems. Such as the best result on SearchQA in this paper is 50.5 in terms of F1 score. However R3 and Re-Ranker obtains 55.3 and 60.6 respectively. Could the proposed approach be adapted on those models? Note that those SOTA systems are released.

3. The proposed system is quite similar to Nogueira& Cho 2017 and Buck et al. 2018. I'm not very sure the contribution of this work and its novelty.  

Questions:
1. Why the authors didn't use beam search during the sub-agent training? 
2. It seems that the proposed framework is a pipeline model: firstly it trains a bunch of sub-agents; and then trains meta-agent. Is it possible to fine-tune the model jointly?
3. What is Extra Budget in Table 1?    

Thanks for your comments!

1. As noticed by the TREC-CAR organizers (https://groups.google.com/forum/#!topic/trec-car/0rpzY_6dP5c), the numbers in the TREC-CAR overview paper are inflated by ~15%. For example, the report paper from 2017's top submission system shows ~2.3 points lower MAP scores than the ones published in the overview paper (https://trec.nist.gov/pubs/trec26/papers/MPIID5-CAR.pdf, table 3).

The numbers we report in the paper are the non-inflated ones (i.e., computed with the "-c" option in the trec_eval tool) hence the lower numbers.

2. We tried to remove stop words and stemming (using NLTK), but we did not notice any improvement in performance.



Thanks for your submission!  I had a few comments on the experimentation.

1. Your results for Lucene, PRF, and RM3 in Table 1 and 4 seem quite low compared to results presented in the TREC CAR overview paper (https://trec.nist.gov/pubs/trec26/papers/Overview-CAR.pdf, Figure 4).  The UNH.bm25 achieves a MAP of 12.2, RPrecision of 9.7, MRR of 18.3, and NDCG of 19.6.  UNH.expan achieves a MAP of 14.4, RPrecision of 11.4, MRR of 21.2, and NDCG of 22.6.  These can be found here: https://trec.nist.gov/pubs/trec26/appendices/car.html

2. Did you stop and stem for your baselines (Lucene, PRF, and RM3)?  This may explain the low performance.  



Meta-review: 
pros:
- The paper is clear and easy to read
- Both Reviewer 1 and Reviewer 2 found the empirical evaluation to be good

cons:
- Some of the reviewers felt that the proposed approach lacked novelty (e.g. with respect to Nogueira and Cho)
- Some of the architecture choices seem complicated and it was not fully clear to the reviewers (even after the rebuttal) how and why things were working better in this approach than in other similar ones.

I think this is a good paper but it doesn't quite meet the bar for acceptance at this time. 



Decision: rejected-papers