Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 8: Accept
review_content: The paper looks at the problem of knowledge selection for open-domain dialogue. The motivation is that selecting relevant knowledge is critical for downstream response generation.
The paper highlights the one-to-many relations when selecting knowledge which makes the problem even more challenging. It tries to address this by taking into account the history of knowledge selected at previous turns.
The paper proposes a Sequential Latent Model which represents the knowledge history as some latent representation. From this methodology they select a piece of knowledge at the current turn and use it to decode an utterance. The model is trained in a joint fashion to learn which knowledge to select and on generating the response. As the two are strongly correlated. Additionally there is an auxiliary loss to help correctly identify if the knowledge was correctly selected. Additionally a copy mechanism is introduced to try to copy words from the knowledge during decoding.
The experiments are run on the Wizard of Wikipedia dataset where there are annotations for which knowledge sentence is selected and on Holl-E, where they transform the dataset to have a single sentence tied to a response.
For automatic metrics there is significant improvement over baselines for correctly selecting a piece of knowledge and generating a response. Additionally there is human evaluation that also shows significant improvement.  Their model also seems to generalize well to domains that were not seen during training time over baselines models.

The contribution of the paper is the novel approach to selecting knowledge for open-domain dialogue. This work is significant in that by improving knowledge selection we see a subsequent improvement in response generation quality which is the overall downstream task within this problem space.
I believe this paper should be accepted because of the significant and novel approach of modeling previous knowledge sentences selected. The linking of this knowledge selection model to topic tracking as stated in the paper is of clear importance, as ensuring topical depth and topical transition are two key aspects for open-domain dialog.
 
Feedback on the paper
In Figure 3, please provide the knowledge sentence that was selected.
Please provide the inter-annotator agreement for human evaluation.
I think it would be interesting to see what is the copy mechanism actually adding in terms of integration of knowledge vs the WoW MemNet approach. Are those two truely comparable because one does not have copy?
For Related Work, also cite Topical-Chat: Towards Knowledge-Grounded Open-Domain Conversations

Small grammatical errors
"Recently, Dinan et al. (2019) propose to tackle" -> "Recently, Dinan et al. (2019) proposed to tackle"
"which subsequently improves the knowledge-grounded chit-chat." -> "which subsequently improves knowledge-grounded chit-chat."


Some questions for the authors in terms of future direction
How is the performance of the model impacted with longer dialog context vs shorter?

The Holl-E dataset was transformed from spans of knowledge to a single knowledge sentence. It would be interesting to see what happens when the knowledge selected is over multiple sentences.

The knowledge pool currently consists of 67.57 sentences on average. How will this method scale as the amount of knowledge sentences grows?





review_title: Official Blind Review #4

final_score: rating: 8: Accept
review_content: Post author response edit: The authors did a good job of addressing many of the concerns of reviewers. I believe with these new results (esp to reviewer 4), they will have a stronger version for the camera ready. I'm bumping up my recommendation for this reason.



The authors propose a novel architecture for selecting knowledge in knowledge-grounded multi-turn dialogue. Their knowledge selection module uses a sequential latent variable scheme, and is claimed to be able to both handle diversity of knowledge selection in conversation as well as leverage the information from the response. The proposed model yields state of the art on two relevant benchmark datasets in terms of perplexity and F1, and scores higher in human evaluations as well.

The paper is relatively well-written, and the authors offer extensive insight into their approach, providing relevant equations and diagrams where necessary. The approach is well-motivated, and the experiments indicate that the model indeed helps on all evaluation fronts. A variety of baselines are considered and are shown to be inferior, in nearly every metric. I did not spend a lot of effort to try to understand their factorization, but the intuition makes sense, and their use of gumbel softmax provides a clear avenue to fix some of the hard-backprop issues apparent in the original Dinan et al. paper. I also appreciate the addition of the knowledge loss to the PostKS baseline: it’s a good effort to make the baseline as good as possible.

A few things bother me with the paper. The primary one is it concerns me a bit that the BERT pretraining does not improve significantly over the E2E transformer memnet (with just bert vocabulary). Unless I’m missing something, that model contained NO pretraining, so I would expect massive improvements. A sanity check there would be checking ppl with gold knowledge: if that doesn’t significantly improve, then I suspect the authors have something really weird about the pretraining or fine tuning. However, It also appears to me that replacing the GRU with a transformer in PostKS might be unfair: Transformers are way more data hungry than RNNs, and so both variants should be tried (though I would be okay with the loser being relegated to a footnote or appendix).

The human evaluations are not as convincing as the authors propose them to be, especially the difference in the “Test Seen” case. It is unclear to me why the authors believe that their “model’s merit would be more salient” in a multi-turn setting, and I think such an experiment would be good to show - or, at the very least, an indication that such an experiment was tried but results were not considered due to reasons X,Y, Z, etc. Overall, the rough improvement that is being provided in the first-stage of the two stage setting seems rather minor (23% -> 26% accuracy; 2.21 -> 2.35 human eval), and that the task remains extremely difficult

Questions
* I know the Dinan et al. models, at human evaluation time, hardcoded to not pick the same knowledge twice. Do you have a similar restriction? If not, maybe you can at least say that you manage to get rid of the need for that!
* As mentioned earlier, I would be curious to see multi-turn human evaluations. I understand this is expensive and a large ask.
* How are the examples in figure 3 chosen? Are they generally indicative of what is seen throughout the human evaluation?
* It would be useful to see a qualitative example of the model’s knowledge selection process when comparing to other models, rather than just the utterance generation (which is not the novel contribution of the paper).

Nits
* Small grammatical errors dealing with subject-verb agreement (plurals mostly).
* Using “-” instead of n/a in tables would make it mildly easier to see digest and see where metrics don’t make sense.

review_title: Official Blind Review #1

final_score: rating: 6: Weak Accept
review_content: This paper presents a sequential latent variable model for knowledge selection in dialogue generation. More specifically, the authors extended the posterior attention model (Shankar and Sarawagi, 2019) to the latent knowledge selection problem. The proposed model achieved higher performances than previous state-of-the-art knowledge-grounded dialogue models on Wizard of Wikipedia and Holl-E datasets.

This work presents a reasonable ideas with new state-of-the-art results in both quantitative and qualitative evaluations.
And overall the paper reads well.

But I think it could be further improved with the following points:
- Could you describe the updates from the previous sequential latent variable models more clearly? It would help to further highlight the contribution of this work. Now it might not be very clear enough for those who are not familiar with the previous work.
- In the introduction, the authors claim the following three advantages of the proposed method: reduced scope of knowledge candidates, better utilization of response information, and weakly-supervised inference with no labels.
But I'm not very convinced whether the experimental results indicate the aspects clearly enough. More detailed analysis should be added to support the contributions.
- The current experiments mainly focus on end-to-end dialogue generation performances. But it would be also interesting to see more detailed aspects of knowledge-selection itself in both quantitative and qualitative manners. I guess this analysis can be done based on the sampled or selected knowledge from the attention distribution.
- Could you possibly add some ablation studies to show the effectiveness of each component? Especially, I'm curious about the results of the proposed model without knowledge loss.
review_title: Official Blind Review #2

Meta-review: This paper proposes a sequential latent variable model for the knowledge selection task for knowledge grounded dialogues. Experimental results demonstrate improvements over the previous SOTA in the WoW, knowledge grounded dialogue dataset, through both automated and human evaluation. All reviewers scored the paper highly, but they also made several suggestions for improving the presentation. Authors responded positively to all these suggestions and provided updated results and other stats. The paper will be a good contribution to ICLR.
Decision: Accept (Spotlight)