Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 3: Weak Reject
review_content: 1. Summary: The authors proposed to alleviate the generic response problem in open-domain dialog generation by generating a response to a prompt from a semantic latent vector. This vector needs to be located close to the latent vector of the corresponding prompt. To this end, they employ canonical correlation analysis and auto-encoder to learn the mapping from a sequence of text to a semantic latent vector. To model the variations and topic shifts that may happen in responses, they also use a separate intermediate vector in the auto-encoder of generating responses.
2. Overall assessment: While this paper is quite fun to read, it is not innovative enough and it lacks some critical experiments and error analysis to be accepted this time. I'll elaborate on these problems in my comments below. 
3. Strengths:
3.1 This paper is well written. The motivation and the main idea are well explained and pretty easy to understand. Although there are some details missing, it doesn't prevent readers from understanding and enjoying this paper.
3.2 The use of human evaluation is a great plus. Subtle characteristics of text, such as readability, coherence, and specificity cannot be well justified by automatic evaluation. Human evaluation is a must for these aspects. It's great to see the authors include this in this paper.
4. Weakness and questions:
4.1 It seems the model in this paper can be connected to adversarial learning from some angles. It would be great to see such analysis in this paper.
4.2 The experiments in this paper do not look thorough enough. There are many more things should be included, such as error analysis, comparisons over different variations of the proposed model and so on. What if we remove $Y_u$ in both training and inference but keep the overall model the same? How important is $Y_u$ and what's its effect? Many such questions haven't been answered in this paper.
4.3 Embedding average cosine similarity seems to be too simple to be used for evaluation as it omits the contextual information and semantic meaning of a sentence.
4.4 It would be interesting to see the Dist-1 and Dist-2 scores of the gold standard responses. It's a good reference to let us know ho diversified the real responses are.
4.5 I'd like to see some explanation of how the raters rate specificity and coherence. What are the standards they use? What are the instructions the author give to them?
4.6 Are improvements significant in Table 1? It seems the proposed model is not performing very stably over different metrics. Some anlysis on this would be great.
review_title: Official Blind Review #2

final_score: rating: 3: Weak Reject
review_content: [Summary]
This paper proposes a dialogue generation model that learns a semantic latent space. In this paper, the authors tackle the generic response issue. The proposed model consists of three (prompt/correlated response/uncorrelated response) encoders and a decoder. The authors add an objective that maximizes the correlation between prompt and response features to the existing autoencoder structure, which prevents the generation of generic responses. 

[Comments]
The paper is well-organized and clearly written but has some weakness:
- There seems to be a need for more detail on comparisons of papers that previously tackle the generic response problem.
- In addition to the proposed main architecture, there are so many things combined (ex. denoising, attention). It seems like it's doing well with additional things rather than the model structure. In fact, the results of the model without denoising in the ablation study show poor performance.
- The test pool for human-evaluation is too small (only 3 workers). It seems necessary to perform the human-evaluation on a larger number of participants. I think that it is a very important issue since the generic response addressed as the main problem in this paper can only be evaluated by human-evaluation.

[Questions]
- What are the criteria for specificity and coherence used in human-evaluation? If there are some criteria provided to the participants, it would be better to include a description together.
- Can you provide an intuitive or theoretical explanation of denoising (to add a <unk>)?
review_title: Official Blind Review #3

final_score: rating: 3: Weak Reject
review_content: This paper proposes a method for generating dialogue responses that are not generic. The authors propose maximizing correlation between latent representations of responses and prompts, such that the response is encouraged to contain information relevant to the prompt. The method consists of two parts, the first part predicts a latent representation of the response while the second part decodes a response from the latent representation.

The idea is conceptually intuitive, however I have some questions regarding the details:
- It's surprising that this works at all, since the inference representations seem to come from an entirely different distribution than in training (e.g. Figure 2)
- Given that this works, I'd like to see an ablation in which Y is removed (e.g. we go from X and Yu to Gy), so as to evaluate the utility of the "correlated response encoder". This is effectively what's happening during inference anyway.
- I am skeptical about the early stopping criteria, which involve human evaluation. I also don't find the justification convincing ("there is no obvious automatic metric") --- why don't we just use the automatic metrics listed in 4.3?
- Can the authors please comment on the choice of evaluation? It seems like the evaluation metrics are different from what's typically used (e.g. see http://convai.io for PersonaChat). Why don't the authors use these metrics instead? This, combined with the early stopping criterion, make it difficult to evaluate the effectiveness of the proposed method in light of prior work.

In terms of the writing, I think the authors could do with more polish:
- there are many generic claims which can be more detailed (e.g. abstract: enables our model to view semantically related responses collectively; achieving [...] better coherence)
- some terms are not clearly defined (e.g. semantics, one-to-many task vs. one-to-one task, canonical correlated feature extractor)
- the introduction is overly verbose, much of it should be in the methodology section. there should be an overview of experimental results and analysis in the intro.
- In 3.3, it doesn't seem like you "enforce" a normal distribution. If I understand correctly, you use KL to encourage a normal distribution. 
review_title: Official Blind Review #1

Meta-review: This paper proposes a response generation approach that aims to tackle the generic response problem. The approach is learning a latent semantic space by maximizing the correlation between features extracted from prompts and responses. The reviewers were concerned about the lack of comparison with previous papers tackling the same problem, and did not change their decision (i.e., were not convinced) even after the rebuttal. Hence, I suggest a reject for this paper.
Decision: Reject