Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The authors proposed to use multiple user simulators to train a dialogue policy and showed that using multiple user simulators rather than a single user simulator improves the policy performance.

The authors developed a sampling strategy to sample user simulators during training. The sampling ratios are inversely proportional to the policy's performance on each user simulator. The policy's performance is calculated by the success rate's upper confidence bound.
strength_and_weaknesses: Strength:

1. The idea of adapting multiple user simulators to train a dialogue policy is interesting. The adaptive phase is closely related to curriculum learning.
2. Instead of using the mean success rate to calculate sampling ratio, the authors proposed to use the UCB of success rate, which hopefully makes the algorithm more robust. Although there is no ablation study to show if this is true.

Weaknesses:

1. The authors only used 4 user simulators to train MUST_uniform and MUST_adaptive, which makes the work less impressive. In general, we wouldn't expect many rule-based user simulators exist, therefore, we would rely on data-driven user simulators. I think the work is more valuable if the authors extend the current work and learn a large set of (and possibly latent/complementary) user simulators from data, and then apply the proposed methods to see if the policy can benefit from the large pool of user simulators.

2. Follow up on the previous point, the authors didn't answer the question of how many user simulators are enough, and how different should they behave?

3. The related works should include curriculum learning literatures. A lot of works use the loss/rewards to adjust task/example weights. For example, on top of my head, focal loss[1] is one method. There should also be an ablation study to show that the UCB term is useful.

4. The authors proposed to use the inverse of UCB as the weights of user simulators. I am a little bit confused. The z_j in Section 4.2 Equation (3) is the *inverse* of success rate, reflecting the task difficulty. Therefore, we should use the UCB of z_j to calculate sampling ratios. In that case, should we pick lower confidence bound in equation (2)? For example, let's say we have two user simulators, the first one has a CI [0.2, 0.8], the second one has a CI [0.4, 0.6]. The proposed algorithm in the paper will assign lower weight to the first user simulator. However, the first user simulator has a lower bound of 0.2, which means that the user simulator could be very hard to train, and should be assigned a higher weight. 

5. Based on the results in Table 2 and Table 3, The performance of MUST_uniform and MUST_adaptive are super close. The performance of MUST_adaptive and SYS_AgenR are also very close. I don't see a clear benefit of the proposed algorithm.

6. The experiments only use MultiWOZ restaurant domain dataset, and no SoTA methods are compared.

[1] Focal Loss for Dense Object Detection https://arxiv.org/abs/1708.02002
clarity,_quality,_novelty_and_reproducibility: Overall, I think the novelty is limited, given the marginal performance improvement over baseline and the limited number of user simulators used for training.
summary_of_the_review: The authors proposed to use multiple user simulators to train a dialogue policy. While the idea is interesting, it is only trained on a small set of user simulators. The authors didn't justify the usefulness of the proposed UCB-based performance expectation. Finally, the performance improvements are marginal and are not compared with SoTA methods.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The authors introduce a multi-armed bandit approach to training task-oriented dialog (TOD) systems to leverage multiple user simulators. This approach is demonstrated to outperform ablative variants that suffer from catastrophic forgetting and over-fitting to specific "easy" simulators.
strength_and_weaknesses: Strengths:
- MUST as an adaptive approach seems well-justified
- Discussion of learning curves motivates further study of MUST as a possible adaptive approach for low data regimes
- Experiments explicitly target enumerated challenges with existing user simulators and mixtures (catastrophic forgetting, over-fitting to specific simulators)

Weaknesses:
- It is important to see empirically if MUST generalizes to a larger or smaller pool of user simulators and is not over-tuned for the current set chosen in the paper. For example, ablation studies with different subsets of the user simulators would be helpful
- Needs more discussion of the limitations on user behavior diversity etc. given that only the Restaurant domain is evaluated here. It is unclear from the experiments here how well the approach can succeed across multiple domains where the pool of user simulators potentially explodes
- Human evaluation details are relatively scarce - would like to see justification for having human raters apparently rate the success rate (the same as the automatic evaluation metric), rather than other metrics of quality (even subjective quality rating) that leverages human annotators more effectively
clarity,_quality,_novelty_and_reproducibility: Clarity: No problems with clarity.

Quality: The experiments are set up well but the theses would be better served by more comprehensive experiments.

Novelty: The approach is founded in bandit literature but advances beyond prior usages of multiple user simulators in this field.

Reproducibility: Code and data are provided in supplement, but I have not verified their accuracy or usability.
summary_of_the_review: The authors propose a well-justified bandit based method for learning from mixtures of different user simulators to train a TOD model. Automatic evaluation is encouraging but the central assertion of adaptibility probably requires a more comprehensive evaluation setting (models, domains)
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper introduces the MUST framework (multiple user simulators), for training Task-oriented Dialogue systems with Reinforcement Learning and incorporating multiple user simulators.

The usual way to train these models is with a single system and user simulators but, as the authors correctly point out, a single user simulator might fail to exhaustively capture all possible human behaviors. This paper contributes a stepping stone towards this objective by casting the problem of combining different user simulators as a multi-armed bandit problem. The authors show that this method provides better results than using simpler combination baselines or single user simulators during training.

strength_and_weaknesses: # Sumary of strengths

* To the best of my knowledge, the idea of combining multiple user simulators by casting this problem as a multi-armed bandit is novel.
* The way the idea is executed is not optimal (experiments on limited dataset), but the authors acknowledge this shortcoming. Despite the limitations of the paper's empirical grounding, its ideas and the evidence it does could provide open interesting avenues for future research on the interaction between user and agent simulators.
* The paper provides directional evidence showing that combining several user simulators is preferable to not doing so. This is reflected both in terms of overall success rate as measured automatically and by humans, and by showing that models trained through this method are less sensitive to unseen user simulators.
* It also evidences that adapting the distribution guiding the choice of user simulator during training results in faster training convergence.

# Summary of weaknesses

* There are some paragraphs that are poorly written, and make some key concepts difficult to understand.
* There are some conceptual gaps that make me doubt about the overall rigor of the paper.
* The more complex $MUST_{adaptive}$ model is only slightly better than the simpler $MUST_{uniform}$ baseline as measured by the in-domain automatic and human evaluations.


# Suggestions and questions for the authors.

* Good work! Do you intend to release the code in case the paper is accepted?
* In the second paragraph of page 2 you mention that "Extensive experimental results [...] show that the dialogue system trained by our proposed MUST achieves a better performance than those trained by any single user simulator", however I feel like the amount of experimentation described is not extensive, so I would suggest toning down this statement.
* In page 4 you introduce $MUST_{CRL}$ but you don't include that model in any of your results. In the paragraph "Challenges to leverage multiple user simulators" you mention some challenges related to this model. Are these challenges the reason why you did not analyze it? If that is the case then I suggest not giving it its own item (II) above, and instead give an overview of the "adaptive" variant of your framework. If this is not the case, then why was it not included?
* In the same "Challenges to leverage multiple user simulators." paragraph you mention: "unnecessary efforts will be costed for easily-adapted user simulators". I suggest rephrasing this statement as I don't think it's understandable in its current form.
* In the first paragraph of page 5 you link the "uniform adaptation" to reducing the catastrophic forgetting issue. However there is no experiment providing evidence that without "uniform forgetting" there's catastrophic forgetting. I suggest rephrasing this, or providing evidence that this is actually the case.
* In the first paragraph of section 5.2.2 you mention: "$SyS-MUST_{merging}$ is trained by $GPT_{IL}$ for implementing $MUST_{merging}$ strategy". First of all the statement is redundant by mentioning the "merging" strategy twice, but more importantly, I had understood that the merging strategy implied sampling dialogues from a set of user simulators and use these to train the final model. Does this statement mean that you generated dialogues with $GPT_{IL}$, which was trained on dialogues produced by 3 agenda-based simulators and MultiWoZ restaurant data, and define this method as sampling dialogues from several simulators?
* In the "Automatic Evaluation" paragraph of section 5.3 you say that the reason for the "merging" strategy not performing as well as the "uniform" and "adaptive" could be "because the merging strategy cannot effectively leverage multiple user simulators" which is a cyclical explanation. I suggest removing it or providing a more plausible explanation.
* In the third paragraph of section 5.3 you mention that the "uniform" and "adaptive" strategies achieve 2.4 absolute value improvements", but improvement over what?
* Finally, I suggest providing more details about the human evaluation. In Appendix B.2 you give some more details and mention that you "tell them how to judge if the generated dialogue is successful", "them" being the evaluators. But what was your definition of a successful dialogue? Every slot filled, for example? or something else?



# Typos and minor corrections

* Page 1, paragraph 4, line 3: best-performed -> best-performing
* P. 2, p. 1, l. 10: You reference challenges $\textit{i}$ and $\textit{ii}$ but you used 1 and 2 in the same paragraph for describing challenges. In the abstract you use $\textit{i}$ and $\textit{ii}$ for referring to the two types of adaptation rather than the challenges they address. I suggest being consistent with numbering.
* P. 2, p. 2, l. 5: Here you mention "$MUST_{adaptive}$ is indeed more efficient for leveraging multiple user simulators by our visualization analysis", which makes me wonder: more efficient than what? and what visualization analysis do you mean? I suggest clarifying this in the paper.
* P. 2, p. 3, l. 8: convergences -> converges
* P. 2, p. 6, l. 4: if accomplishing -> if it is accomplishing
* P. 3, p. 1, l. 1: Once the database result -> When the database result
* P. 4, p. 2, l. 1: first sample -> first samples
* P. 4, p. 2, l. 4: by RL algorithms -> with RL
* P. 4, p. 4, l. 11: Not sure what is meant by "unnecessary efforts will be costed for easily-adapted user simulators". I suggest rephrasing this.
* P. 4, p. 6, l. 1: recalls us a similar thought -> reminds us of a similar concept
* P. 4, p. 6, l. 3: weakly-performed -> weakly-performing; well-performed -> well-performing
* P. 4, p. 6, l. 4: "should reduce the interaction with user simulators that dialogue system has performed well and allocate more interactions with those user simulators that dialogue system has not performed well" -> "should reduce the interaction with user simulators with which the dialogue system has performed well and increase interactions in the opposite case."
* P. 5, p. 2, l. 5: masker's -> maker's
* P. 5, p. 4, l. 7: "p is expected to assign lower weights to user simulators that the system agent S already performs well and higher weights to those user simulators that S performs not well" -> "p is expected to assign lower weights to user simulators with which the system agent S already performs well and higher weights to those user simulators with which S performs poorly".
* P. 5, p. 7, l. 2: what do you mean with "latter" and "former" terms? Do you mean $\underbrace{\bar{x}_j}_\{\text\{exploitation\}\} + \underbrace{\sqrt{\frac{2\ln t}{T_\{j,t\}}}}_\{\text\{exploration\}\}$ in equation 2?
* P. 5, p. 8, l. 2: has been interacted so far -> has been interacted with so far
* P. 5, footnote, l. 2: "Then the index of the arm will be played from t = K + 1 to T is the sum of two terms: ..." makes no grammatical sense. Please correct it.
* P. 6, Algorithm 1: I suggest changing the verbs from -ing form to imperative, so initializing -> initialize, synthesizing -> synthesize, using -> use, evaluating -> evaluate, updating -> update. I also suggest clarifying what the lowercase s mentioned in the input is used for.
* P. 6, p. 1, l. 2-3: I think here you used both $\tau$ and $s$ to refer to the smoothing factor for distribution $\boldsymbol{p}$
* P. 6, p. 3, l. 3: "that the dialogue system has performed well" -> "with which the dialogue system has performed well"
* P. 7, p. 4, l. 4: "test the systems by them" -> "test the systems with them"
* P. 7, p. 4, l. 5: "there usually has a gap" -> "there usually is a gap"
* P. 9, Figure 2: There's no need to write "Tested by" for each subfigure. The name of the user simulator is enough. Also the label for (a) seems to be wrong.
* P. 14: There's mention to U-GPT here, but nowhere in the main text.

clarity,_quality,_novelty_and_reproducibility: The idea is novel, but clarity and quality could be improved (see comments above for more details) 
summary_of_the_review: I think the idea is interesting, and that it could inspire interesting follow ups, however the execution leaves a bit to be desired. I will set my score to "marginally above the acceptance threshold" due to the lacking points I mentioned in detail above.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: The authors consider the problem of training a dialogue policy with a set of multiple user simulators, developing a multi-armed bandit motivated strategy for selecting between the user simulators at each turn. Specifically, they develop a sampling policy where the sampling distribution is such that the sampling rate for a user simulator is inversely proportional to the UCB of the estimated success rate. The resulting approach mitigates catastrophic forgetting and avoid overfitting through the proposed sampling strategy. Using both agenda-based and learned user simulators recently developed for other work [Shi, et al., EMNLP19] and the restaurant subset of MultiWoZ, they show improved performance over any single user simulator and that their adaptive sampling rate method empirically has the best convergence properties.

The consensus strengths of this work include:
- Adapting multiple user simulators to learn a policy is interesting and MAB-based approaches are sensible and conceptually appealing.
- The authors consider multiple sampling strategies that are used as baselines for the proposed adaptive setting (that is closely related to curriculum learning)
- Using the UCB is well-motivated/established within the MAB formulation for increasing robustness of the resulting MAB system.
- The experiments include a good 'dive deep' element, especially when considering the appendices (which address some of the issues in the initial reviews)

The weaknesses discussed in the reviews (and my own reading) of this work include:
- The work is more of a proof-of-concept since they use a small number of existing simulators and don't characterize the relationship between the user simulators. Furthermore, this is difficult to address in future work as creating multiple good user simulators is difficult for a given setting (maybe personas?) -- which calls into question if this is a viable method in the general case. Even after partially addressing in the rebuttal, it really isn't clear how to scale this based on the submitted work, especially in academic research settings on benchmark datasets.
- This work is specific to dialogue systems and conceptually is a relatively straightforward application of MAB; while novel in this specific setting, it is somewhat straightforward if one assumes the existence of multiple simulators. In the other direction, there aren't any findings that contribute to general MAB literature and it isn't even clear what theoretical properties hold once the MAB modifications are made for this specific domain.
- In this vein, there is limited theoretical analysis to determine how practical this approach is; how many user simulators are needed? how different do they have to be? how does one measure diversity in the simulators? There are many such open questions that aren't sufficiently addressed empirically and this gap isn't mitigated with any analytical examination (which MAB literature has a rich foundation for this type of analysis).
- MUST_uniform and MUST_adaptive are very close in final performance (although the adaptive version has better convergence properties), the MultiWoZ restaurants subset is a very specific dataset and it isn't clear if the overall results are near SoTA. Thus, there are some practical empirical questions that must be addressed -- including the general question regarding how this changes with more or fewer simulators of different characteristics.
- While less critical, there are still many typos and a general lack of narrative continuity in the writing. 

Overall, the core idea is novel, the proposed approaches seem to work well empirically over single and more naive combination strategies, and the results appear promising overall. However, there remain open questions regarding if this is really a viable method as creating many sufficiently diverse user simulators isn't straightforward (and is not proven in this work). This could be addressed empirically or supported with analytical statements, but as it stands, this work has promising early results that need more practical or theoretical impact to qualify as a strong dialogue systems or MAB contribution.

Additional note: This paper was discussed with the SAC during final decisions (as there was also a 'late' change in scores). While I understand that reviewer 44kw felt their concerns were addressed, my assessment is more in line with reviewer H1xa -- that there are fundamental practical limitations of the proposed method that may limit its applicability in the general case that should be addressed for this to be a clear accept. The SAC, program chair, and myself agreed upon this decision after reading all of the reviews, rebuttals, and discussion -- and then discussing amongst ourselves.
justification_for_why_not_higher_score: The core concern with this work is that they basically combine a small number of user simulators to work on a relatively small dataset -- demonstrating interesting, but thus preliminary findings. Without additional empirical and/or theoretical work, it isn't clear that this is a viable method that be broadly applied in building dialogue systems.
Decision: Reject