Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 6: Weak Accept
review_content: ### Summary
​
This paper provides a framework to augment dialogue generation with external data sources using K-Nearest Neighbors in the embedding space. The idea seems simple and intuitive, and the results show improvements over prior work in dialogue generation and retrieval.
​
​
### Strengths
- The paper shows quantitative improvement over some prior works on dialogue agents (however this needs to be correctly validated).
- The reviewer appreciates the human study and conversation example provided in the paper to qualitatively evaluate their model.
- The paper provides ablation studies of the various training tricks for each dataset they have proposed.
​
### Weaknesses
- There are certain changes in the training pipeline that makes the comparison with prior work difficult (Tables 1 and 2), and find the real contribution of data-augmentation caused by KNN-based information fetching (KIF). e.g., In Wizard of Wikipedia experiment, most recent dialogue utterance and turn number are used as salient features. Similarly, the paper utilizes a "personality" feature in ImageChat dataset. It seems the results taken from prior papers have different training settings. Could the authors verify that the extra assumptions made for their model are equivalently applied to other generative baselines? If not, the reviewer recommends to provide some experiments with same conditions.
- Could the authors provide the embedding dimensions and other training details, to assess the contribution of different components of the input's embeddings.
​
​
#### Minor:
- Is there a sound reason behind using concatenation of input representation and fetched representations? This design choice makes the architecture inflexible to the number of data sources. Another way to combine embeddings - e.g. addition or inner product, can also be tried to see if they provide performance improvements.
- It is mentioned that attention based modules scale poorly with large sized datasets. If the authors conducted a quantitative evaluation to test this, it would be valuable to add it to the paper.
- The title and abstract should be limited to generative dialogue modeling, instead of transformers, since the contributions proposed are more suited for this particular application and are quite engineered. Hence, it is not correct to make this claim for transformers in general.
​
​
### Score
Weak Reject (Leaning towards Accept if appropriate experiements can be provided)
review_title: Official Blind Review #2

final_score: rating: 3: Weak Reject
review_content: This paper uses KNN-based retrieval method for extracting relevant information from different sources for the task of generating response in a dialogue task. They utilized different sources of information from Wikipedia, YFCC image set and dialogue utterances. The method is very straight forward and using nearest neighbors from external information sets as auxiliary information and encoding that information via same neural net to produce the encoded external information. The concatenation of encoded input and auxiliary information is concatenated to produce the output in the dialogue via decoder network. The novelty of the proposed method is incremental over Dinan et al. (2018). The KNN-based retrieval module can produce some drift in dialogue from the actual context of the input which can result in irrelevant response in dialogue. Some failure cases can show the quality changes with those drifts in dialogues.

review_title: Official Blind Review #1

final_score: rating: 6: Weak Accept
review_content: Summary:
- The paper proposes augmenting transformer neural networks with KNN-based information fetching modules that can access relevant external knowledge, combine knowledge from different sources, and integrate the information into seq-to-seq architectures. The authors apply their proposal to generative dialog modeling, and apply it to two dialog datasets.

Strengths:
- The paper is well-written and well-motivated.
- The authors evaluate their approach on 2 publicly available datasets and compare it to existing approaches, showing improvements in terms of F1.
The authors conduct a human evaluation to compare their approach against other approaches.

Weaknesses:
- There are some details that are missing from the paper, for example details about the mapping operator, and the specific representations of E_i for the datasets used.
- Parts of the analysis are rushed (e.g., sections 6.2 and 6.3).

Questions/Comments:
- In the human evaluation, is there a difference between the ratings for seen and unseen topics for the Wizard of Wikipedia dataset?
- Section 6.3 (especially the part on the effect of gating) can be improved with additional information/analysis.
review_title: Official Blind Review #3

Meta-review: This paper augments transformer encoder-decoder networks architecture with k nearest neighbors to fetch knowledge or information related to the previous conversation, and demonstrates improvements through manual and automated evaluation. Reviewers note the fact that the approach is simple and clean and results in significant improvements, however, the approach is incremental over the previous work (including https://arxiv.org/pdf/1708.07863.pdf). Furthermore, although the authors improved the article in the light of reviewer suggestions (i.e., rushed analysis, not so clear descriptions) and some reviewers increased their scores, none of them actually marked the paper as an accept or a strong accept.
Decision: Reject