Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper:  (To the authors, apologies for the delays - this was an emergency review taken on right before an unexpectedly busy week(s))

 In this work the authors explore ways in which to learn a reward function for task-oriented dialogue systems, proposing two new objectives phrased within the context of a learning-to-rank objective.  Given a set of scored (and hence ranked) TOD trajectories, the aim is to learn local reward functions whose accumulated scores reflect the original ranking.  They propose two methods, one doing just that (RewardMLE), the other minimizing the cross entropy between the two scores (RewardNet).  The learned reward function is incorporated into the training objective using Gumbel-softmax to reduce variance from the vanilla REINFORCE estimator.  On one benchmark dataset, variations of the proposed approach improve on standard TOD evaluation metrics, and show strong improvements in artificial data sparsity and human evaluations.
strength_and_weaknesses: Strengths:
The empirical results of this paper make a compelling case for its acceptance -- the method achieves a new state-of-the-art on the MultiWOZ 2.0 dataset, improving over the closely related CASPI method by ~1-1.5 pts in success and inform metrics.  In low data settings, the margins are quite large.  The analyses are generally good, and show advantages of this method in many settings.

Weaknesses:
Working against the paper is perhaps the similarity between previously best method, and concerns about the strengths of the results.  The relatively small margins of improvements in main evaluation are gained only in certain hyperparameter settings.  If one were to choose an architecture with a given N or \Phi in advance, there appears to still be a relatively good chance that there will be very small improvements (< 0.5, or even < 0.1) on some measures, to an extent that it would probably be insignificant.  Rather than tune these on some sort of dev set to mimic a realistic scenario, different runs get different rows in the table, and are presented in a manner of, ~"Look, one of our methods will always win by a good margin", only that which system that is is unfortunately not consistent.  I find this pretty disappointing.  It is saved primarily by the fact that usually all of these systems outperform (by microns) the previous SOTA, but it is not beyond reason to wonder if the case would be true for even more values of N, or on a different dataset or task.  This is an obvious place where the paper could be improved.

This relates to another weakness -- main results are averaged, and authors don't report the variance of their method.  The authors state "overall performance significantly improves over CASPI", but we are not given access to statistical significance tests / confidence margins over multiple runs.  These would be good to have in general, but the former is necessary to make claims of significance, and if that is not the authors' intent then it should be reworded.

The (automatic evaluation) results on MultiWOZ are promising, but the above issues make it difficult to have great faith in the results, given they are applied just to this dataset.  It would have been better to apply the method on a couple of other popular TOD datasets.  It seems KVRET and SGD benchmark are similar in spirit to MultiWoz.  The results presented here are maybe just about/above the threshold of publication, but there's certainly value in asking for additional experiments and metrics.

clarity,_quality,_novelty_and_reproducibility: The method is quite complicated and makes use of many parameterized tricks.  Although the novel contribution is a small deviation from existing work, getting it all working seems very involved and I think there will be significant reproducibility concerns if the code is not shared.  Are there plans to release the code?  I would have to consider lowering my score if the code was not going to be released or at least distributed by the authors upon request.

Some questions regarding clarity and soundness:
What is going on with BLEU as it relates to Inform/Success?  Especially in a case like the Fig 2 10%.

Maybe I missed this, but the description addresses the case where trajectory lengths are equal.  How does this work in practice?

I thought it was less than ideal that PAR and AEST pop up in Fig 1, and are basically not integrated with the written discussion.

Typos / Grammar:

(b): Does different -> Do difference

protocal -> protocol
summary_of_the_review: I think the existing experiments are probably sufficient for publication if I had to come down one way or another, but they do leave a lot to be desired.  The method seems to be a useful generalization of existing reward learning for TOD and one that could be applied in many situations.  If the authors can show that the performance gains are consistent, and that high performance model configurations are somewhat identifiable in advance, then there is value in the method.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper investigates the question of how to efficiently learn and utilize a reward function for training end-to-end task-oriented dialogue agents. To be specific, the authors introduce two generalized objectives (RewardNet and RewardMLE) for reward-function learning, motivated by the classical learning-to-rank literature. Further, they also propose a stable policy-gradient method to guide the training of the end-to-end TOD agents. The proposed method achieves competitive results on the end-to-end response-generation task on the widely-used dialogue benchmark MultiWOZ 2.0.
strength_and_weaknesses: Strengths:
1. The paper is well-organized and easy to understand. 
2. How to efficiently learn and utilize a reward function for training end-to-end task-oriented dialogue agents is fundamental and deserving problem.

Weaknesses:
1. The paper only conducts experiments on one benchmark MultiWOZ 2.0. As a methodological contribution, it will be more convincing to verify the effectiveness on more benchmarks.
2. The novelty is limited. How to efficiently learn the reward function in RL-based dialogue agents is a classic problem. Various existing literature[1][2][3] has discussed this problem and proposed the corresponding solution.
3. More implementation details are needed in the main paper, and I did not see the code. 

References:
1. Curriculum Learning Based on Reward Sparseness for Deep Reinforcement Learning of Task Completion Dialogue Management
2. Playing 20 question game with policy-based reinforcement learning
3. Guided Dialog Policy Learning without Adversarial Learning in the Loop
clarity,_quality,_novelty_and_reproducibility: Clarity: Most things are explained clearly either in the main paper or the appendix.
Quality: well-organized paper.
Novelty: The novelty is limited, details please refer to Weaknesses
Reproducibility: RL training is unstable. More experiment hyper-param setup is needed. The authors are better to provide source code to ensure reproducibility.
summary_of_the_review: I like the problem of efficiently learning the reward function in end-to-end TOD dialogue systems. My main concern is the limited technical novelty of the proposed RL method for a machine learning conference like ICLR. Moreover, considering the unclear reproducibility, I tend to give a weak rejection.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper aims to design reward functions for training TOD agents through reinforcement learning, inspired by Learning-to-Rank (LTR) methods. The authors demonstrate that their RewardNet and RewardMLE methods achieve strong improved performance on MultiWOZ 2.0 compared to SOTA methods for training TOD agents.
strength_and_weaknesses: Strengths:
- RewardNet and RewardMLE are both demonstrated to out-perform SOTA algorithms
- The authors demonstrate that existing work (CASPI) can be framed as a specific case of their proposed reward algorithms (RewardNet to be precise), showing that their contributions are more general
- The ablation questions (5.2) are well-posed and answered in a logical fashion
- Human evaluation 

Weaknesses:
- The number of trajectories seems to have a nontrivial effect on total score; in a real-world setting how well do e.g. the choices of N=3 for RewardNet and N=5 for RewardMLE perform? It would be helpful to see a discussion of parameter sensitivity.
- Would have liked to see more discussion in terms of a breakdown between multi-domain and dialogs from each separate domain in MultiWOZ, to investigate if there is a disparate effect of learning a TOD agent via RewardNet/RewardMLE depending on the complexity of the conversation (e.g. length, number of slots/actions) and the domain complexity (e.g. possible slot values, uniformity of distribution of slot values).
clarity,_quality,_novelty_and_reproducibility: Clarity: The paper is organized well, especially the ablation section (5.2).

Quality: The experiments and analysis seem of high quality.

Novelty: The approach builds upon prior work on learning TOD agents via rewards in RL, especially CASPI. The authors generalize the idea and demonstrate the effectiveness of the more general formulation(s)

Reproducibility: Could not identify whether code is to be released or if experiments are reproducible.
summary_of_the_review: This paper justifies its choice of approach to training TOD agents (via generalized rweard function designs) and effectively demonstrates their effectiveness in a multi-domain setting. I would have liked to see a more thorough discussion but the paper itself reads well.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper introduces two variants of reward learning functions in RL-based task-oriented dialogue (TOD) to address the issues of sparse and delayed rewards. RewardNet optimizes the rewards by computing each trajectory's total automatic evaluation scores and then sorting all trajectories by the scores. The RewardMLE uses only the ranking order instead of the automated evaluation scores. The ranked N dialogue trajectories are inputs into a neural network to learn the new rewards. 
The RewardNet is updated by optimizing the cross entropy between accumulated predicted rewards and automatic evaluation scores. The negative log-likelihood is used for the RewardMLE.
The policy gradient-based agent predicts the action and updates its parameters with the learned rewards. Experiments  are performed on the MultiWOZ 2.0 dataset.
strength_and_weaknesses: ## Strengths
- The goal and structure of this paper are clear and easy to follow
- This paper shows strong experiment results. Their methods achieve the best results on the MultiWOZ 2.0 dataset. 
- reasonable combination of well-known RL methods, new research direction in TOD domain

## Weaknesses
-  The novelty of reward techniques is limited
- no implementation available, reproducibility unclear
clarity,_quality,_novelty_and_reproducibility: ## clarity
The goal and structure of this paper are clear and easy to follow. In section 5, the author discusses details of their methods and analysis them with toy examples. Section 5.2 (a) discusses the number of trajectories; I do not understand this sentence ‘We hypothesize that the optimal trajectory number depends on the scoring quality.’  Does there have theoretical reasons to prove that more than two trajectories can bring more advantages than the classical pairwise preference? I wonder if this N-trajectories idea can apply to other scenarios since most previous works use pairwise trajectories.

## quality
Quality overall is good except for the mentioned issues with respect to reproducibility below.

## novelty
it is a combination of existing techniques, but the combination itself is novel

## reproducibility
Code is not provided as far as I can see. Regarding the reproduction of CASPI, it is not clear which version from the official CASPI codebase did the author run to get the results in Table 1. It should be possible to show the Median score of the RewardNet and RewardMLE experiments and compare them to CASPI original paper. It is not mentioned how the authors choose the length of the trajectories to update the reward model. If all of the dialogues have an equal length of trajectory, could it happen that the agent arrives at the goal earlier than the set length or need more steps to achieve it? (e.g., Simple dialogue vs. Complex dialogue)
summary_of_the_review: This paper applied the learning-to-rank-based rewards functions to TOD tasks. The basic idea behind RewardNet is similar to the method proposed by the CASPI paper. Additionally, this paper runs more experiments with different numbers of trajectories, introduces a new variant, RewardMLE, and discusses the policy gradient-based agent with the Gumbel-softmax trick. The novelty of their main reward techniques is limited. Still, this paper proposes a reasonable combination of well-known RL methods and a meaningful research direction with strong experiment results for the TOD domain. It’s useful for future work on the MultiWOZ 2.0 dataset to compare.  The author didn’t provide their implementations; I’m unsure about the reproducibility.



Minor suggestions: be sure to provide the full term when introducing an acronym; for example, section 3.2 (GS) occurs before its acronym Gumbel-softmax(GS).  It would be evident if Figure 1describe more details about the framework.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: The case for acceptance of this paper defining and evaluating new reward functions for RL-based task-oriented dialogue is fairly clear. Reviewer consensus is uniformly on the side of acceptance, ranging from sympathetic to excited. The authors responded the the majority of reviewer concerns, and I would ideally have liked to see some further engagement from reviewers during this period to determine if they were willing to raise their scores. The main concern expressed during review seems to be surrounding the novelty of the method, which is typically a highly subjective and noisy line of criticism. From reading the author rebuttals, I don't think there should be any outstanding doubts regarding the proposition that this work is sufficiently novel to warrant publication, and thus I am happy to write in support thereof.
justification_for_why_not_higher_score: With the majority of the support being sympathetic and supportive, but slightly lukewarm, I hesitate to recommend a higher rating than Accept (poster). That said, I think the author rebuttal addresses a number of the issues raised by the reviewers, which the reviewers have elected not to respond to further or revise their scores based on. If one paper in my stack were to be bumped up, I would suggest this one.
justification_for_why_not_lower_score: Reviewers unanimously support publication.
Decision: Accept: poster