Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper proposes an offline RL method applied to an end-to-end task-oriented dialogue model, where the proposed GPT-Critic is built on GPT-2 and fine-tuned on the self-generated sentences for policy updating. 
The paper claims that it is free from the issue of diverging from human language (a common issue in standard RL), because it learns from the sentences directly sampled from the pre-trained language model.
The conducted experiments show that the proposed model achieves better performance compared to other task-oriented end-to-end dialogue models in both offline and online settings (MultiWOZ and ConvLab respectively).
main_review: This paper focuses on improving the dialogue policy together with the responses by utilizing a pre-trained language model and offline RL.
The claimed contributions include:
1) The proposed method is free from the common issue of diverging from human language, because it learns from the sentences sampled from the pre-trained LM.
2) The proposed method outperforms other SOTA models in offline and interactive online settings, MultiWOZ and ConvLab respectively.

The proposed method is reasonable and moderately novel. The experimental results are promising for both settings.
However, there are unclear parts to be addressed or clarified.

- Because the policy learning procedure utilizes the additionally generated dialogue acts and corresponding responses, it is easy to think that naively fine-tuning the GPT-2 model on the additional generated data may also improve the dialogue model performance in terms of its policy and responses (similar to a data augmentation method). Did the authors try this as another compared baseline? This method should be included in the experiments in order to justify the proposed RL approach is necessary.

- The paper mentioned that the standard RL methods easily fail and generate responses diverging from human language, even when fine-tuning a pre-trained LM. Hence, it will be better to additionally include the results of other standard RL algorithms for better justifying this claim. (Current experiments only include the results of models that are free from this issue.)

- In the experiments in MultiWOZ, this paper only evaluates the response generation results. However, evaluating dialogue policy is also important to justify the learned policy is suitable. It is unclear why the authors only show the response generation results.

- The experiments contain two setups, one is offline response evaluation via MultiWOZ, and another is interactive simulation via ConvLab. Both settings show the better performance of the proposed method. The paper can be better if adding the real-user interactions, because the performance may be different between the simulation environment and the real-user interactions reported by prior results (DSTC in ConvLab). Conducting real-human interactions can better justify the effectiveness of the proposed RL method in practical scenarios.

In sum, the proposed method is relatively novel and the idea is reasonable. The performance seems promising in both settings.
However, the paper does not include detailed descriptions about the proposed method, making readers not easy to understand.
Also, some additional experiments need to be added in order to better justify its claims.
summary_of_the_review: The idea is reasonable and moderately novel. The conducted experiments demonstrate the better performance of the proposed model in two different settings.
The paper misses some details when describing the proposed method, making readers difficult to fully understand its idea.
Although the experiments already include a lot of baselines, there are still some results to be included for fair comparison (simple data augmentation method).
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper presents a reinforcement learning-based approach to building a task-oriented dialogue agent. Given a dialogue dataset annotated with rewards, the state-action value function is first trained by minimizing temporal differences. Then, a new training dataset is created by using the best actions selected among the candidates generated by the current policy (i.e., the language model). The policy is then updated by behavior cloning using the created dataset, and the whole process is repeated. The authors have conducted experiments using MultiWOZ and ConvLab and shown that this iterative process improves the performance of the agent.
main_review: The proposed approach is very simple and should be easy to implement. The experimental results seem promising. However, I do have a couple of concerns.

Firstly, I think that some important details are missing in the paper. For example, what is the action space of the agent? Do you treat the conjunction of a dialogue act and a system response as a single action? If so, how exactly are the candidate actions generated? Is some kind of beam search employed? I think some actual examples of actions (and rewards) would help the reader understand the proposed method more clearly.

The paper does not really describe how the rewards are computed, either. In particular, I am wondering how the reward for the newly selected action is computed. Is it given to the agent by an external program?  Then, it seems to me that the whole training procedure is more like (a somewhat restricted version of) actor-critic-based reinforcement learning than offline reinforcement learning (in which the agent cannot interact with the environment). If that is the case, what is the novelty of the proposed method?

The authors claim in Table 3 that their proposed approach gives much better results than UBAR, but the original paper of UBAR (Yang et al., 2021) reports much better results (e.g., Inform score of 95.4). Why is there such a big difference?

Algorithm 1 states that the policy is updated by behavior cloning until "convergence". I am wondering if it causes any overfitting problem. Is overfitting the reason why the whole training process is stopped at the fourth iteration (Table 2)?

Minor comments:

p. 1: outperforms the state-of-the-art -> outperforms the state of the art;
p. 1: not trained to for -> not trained for?
p. 2: fine-turning the GPT-2 -> fine-turning GPT-2? fine-turning the GPT-2 model?
p. 2: generates strategically -> generates a strategically;
p. 2: Pr(O_{t+1} ... ) -> should not be italic?
p. 3: by training action-value -> by training the action-value;
p. 3: of critic network -> of the critic network;
p. 4: for $i$-th -> for the $i$-th;
p. 4: in task-oriented -> in the task-oriented?
p. 4: generated system response -> generated system responses?
p. 4: from (Zhao et al., 2019) -> from Zhao et al. (2019);
p. 4: prohibitory -> prohibitively?
p. 4: over response candidates -> over the response candidates?
p. 4: updated policy by above -> the updated policy by the above?
p. 5: on MultiWOZ domain -> on the MultiWOZ domain;
p. 5: ConvLab framework -> the ConvLab framework;
p. 5: HuggingFace Transforms library -> the HuggingFace …;
p. 7: prices -> priced?
p. 8: user goal -> the user goal?
p. 8: with following -> with the following;
p. 8: the all -> all the;
p. 8: whereas original -> whereas the original?
p. 9: straightforward -> straightforwardly?
p. 8: large scale -> large-scale?

summary_of_the_review: The proposed algorithm is simple and seems effective. However, some important details are missing and the novelty is not very clear.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: To tackle the response generation diverging issue from human language, this paper introduces a critic on the top of the pretrained GPT-2 task-oriented dialogue agent, and demonstrates promising empirical results on two datasets (MultiWoz and ConvLab).  
main_review: Strengths:
This paper introduces a critic value function on the top of the pretrained GPT-2 task-oriented dialogue agent, to guide the response generation. 

Weakness:
1. The novelty may be limited by only adding a critic value function on the top of the existing work.

2. Is it possible to add human evaluation for at least of the datasets? (fixed in the rebuttal)

summary_of_the_review: The motivation (tackling the response generation diverging issue from human language) of this work is clear, and the solution is also good, by adding a critic on the top of existing pre-trained GPT-2 dialogue agent, and shows promising empirical results on the automatic evaluation of two datasets. However, the main concern for this work, is is quite incremental with limited novelty, and also lacks human evaluation to verify its effectiveness, may recommend for a workshop paper.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper works on offline reinforcement learning for natural language action space setting, particularly for task-oriented dialogue management. The paper nicely incorporate the policy network (to sample agent response) and the q network (to evaluate the agent response) into a single GPT-2 network and propose a policy interation algorithm to optimize both the q and policy network.
During policy evaluation, the q network is updated with sampled system actions and responses. During policy improvement, the sampled system actions and responses with maximum q values are used as labels to update the policy network. The model achieves SoTA performance on MultiWOZ dataset.
main_review: The paper proposes a nice way to handle large scale natural language action space for RL. An action in this model is a system action plus response (a sequence of tokens) rather than pre-specified variables in discrete latent models for dialogue (LaRL, etc.), so the proposed model can be easily incorporated into a transformer-based language model (e.g. GPT-2).

My biggest concern is the q-value estimation. Estimating the Q value for an offline dataset is hard due to the problem of overestimation, which is already pointed out by the authors in the paper. However, the authors didn't describe in detail how they mitigate this problem. The only sentence in the paper about this is "However, we avoid this OOD problem by xxx ... revised by generated system response and evaluated reward using offline automatic evaluation". More details are needed, and ablation study about this is needed. Otherwise, I don't know if the proposed method can generalize to other natural language action space applications where the ground-truth reward function is unknown for offline automatic evaluation.

My second concern is the exploration perspective of self-generation. The paper mentioned that {a_k}^N is a set of N response candidates generated from the current policy \pi. What number is N set to? What is the impact of N to the final performance? Moreover, is{a_k}^N sampled by beam search or vanilla sampling? I'd assume that we want to sample {a_k}^N that is diverse enough, rather than similar system responses with only one or two different words. More discussion about the generation is needed.


summary_of_the_review: 1. Good method to incorporate policy network and q network into a GPT-2 model to handle large scale natural language action space.
2. Have concern on offline q-value estimation and sampling of response candidates.
review_title: null

Meta-review: The reviewers are all weakly positive. The author response clarified important aspects of the paper. The new human evaluation was critical. However, the human evaluation result presentation is flawed: presenting Likert scores as means does not reflect them well. The authors should use something similar to a Gantt chart to fully reflect the distribution across Likert categories. Another detail in the human evaluation that are troubling: it does not reflect interaction with the system, but judgements through observation. Therefore, the human evaluation does not reflect the ability of the learned dialogue system to interact with users. Overall, the paper makes a nice, original contribution, but despite author improvement there are evaluation flaws (even if they are common in papers using these benchmarks).
Decision: Accept (Poster)