Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This paper studies the model-agnostic explanations of dialogue response generation and proposes a new called local explanation of response generation (LERG). LERG extracts the sorted importance scores of every input-output segment pair from a dialogue response generation model. It views the sequence prediction as uncertainty estimation of a human response and creates explanations by perturbing the input and calculating the certainty change over the human response. The authors show the proposed LERG adhere to three properties of  an ideal explanation of text generation: (1) unbiased approximation, (2) intra-response consistency and (3) causal cause identification. The experiments on a popular benchmark DailyDialog empirically verify the effectiveness of the proposed approach.
main_review: **Strengths:**

The black box nature of current end2end neural dialogue models hinders us from understanding the underlying reasoning process of these models. This paper introduces a novel method to explore explanation for dialog response generation models for the first time. Overall, I like this idea.

They mathematically formulate the task and generalizes local explanation methods towards sequence generation and show that our method adheres to the desired properties for explaining conditional text generation.

They introduce the corresponding evaluation methods to assess the explanations of dialogue response generation and empirical study demonstrate the effectiveness of proposed LERG.


**Weaknesses:**

Experiment conducted on only one dataset is somewhat a little weak. Have you tried the other popular benchmarks in the dialogue generation task, such as Reddit Corpus, Persona-Chat (Liu et al., 2020) and Topical-Chat (Gopalakrishnan et al., 2019)?

Section 4.2 and 4.3 are lack of the insightful analysis for the reported results. I suggest the authors to share more insights to readers here, rather than just reporting the plain experimental results.

It seems there is no clarification about the code release in the paper. Will you make your code publicly available to the community for the reproduction later?

Minor comments:

missing reference on controllable dialogue generation in the related work:

[1] Zhang et al. Learning to control the specificity in neural response generation. ACL 2018.

[2] Xu et al. Neural response generation with meta-words. ACL 2019.

[3] See et al. What makes a good conversation? how controllable attributes affect human judgments. arXiv 2019.

[4] Smith et al. Controlling Style in Generated Dialogue. arXiv 2020.

[5] Wu et al. A controllable model of grounded response generation. AAAI 2020.

[6] Madotto et al. The Adapter-Bot: All-In-One Controllable Conversational Model. arXiv 2020.

[7] Yang et al. Progressive Open-Domain Response Generation with Multiple Controllable Attributes. IJCAI 2021.

[8] Gupta et al. Controlling Dialogue Generation with Semantic Exemplars. NAACL 2021.










limitations_and_societal_impact: Yes, the authors have addressed this.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This work proposes a method to locally explain the dialog response generation process. This method is motivated by the intuition that, such method should answer the question "which parts of the response are influenced the most by parts of the prompt". So the importance scores of every input-output segment pairs are calculated. They also proposed two metrics to evaluate explanations: "necessity" and "sufficiency". The former is defined as the perplexity change after removing top "salient" input features, and the latter is defined based on the intuition a complete explanation should recover model's prediction without the original input.
main_review: this work focuses on an important problem: the explanation of generative models. This work proposes a model-agnostic tool to provide explanations, and such tools should be very helpful to the process to debug, understand and trust the model. Their pairwise importance scores also seem a reasonable method.

However, this reviewer believe this work can be improved if the following concerns can be resolved: The examples shown in Figure 4 are some token-to-token relation, this together with the error analysis seems to indicate that ngram-level segment (not token-level) can be sometimes important and not well captured by the model. Could the authors comment on this? is it easy to show ngram-to-ngram relation? does it perform well?
limitations_and_societal_impact: See above concern regarding ngram-level explanation.
This reviewer doesn't think this work has negative societal impact. Actually, if reasonable explanations can be provided for machine learning generative models, it should have positive societal impact for people to understand the models.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This paper proposes a model-agnostic explanations model, local explanation of response generation (LERG), for dialogue response generation task. Due to the sequence-to-sequence natrual, previous works, which normal produces a single label as output, are no longer suitable for this task. This paper regards the
explanations as the mutual interaction of segments in input and output sentences. Specifically, it views the sequence prediction as uncertainty estimation of a human response and creates explanations by perturbing the input and calculating the certainty change over the human response. The experiment shows that the proposed approach extract both explicit and implicit relations between input and output segments.
main_review: Originality: 
model-agnostic explanation models for sequence-to-sequence task are not a new task. [1] proposes methods to explain the black-box sequence-to-sequence model, including machine translaton and dialogue task. This paper first review popular model-agnostic explanation models, LIME and Shapley value, then reformulate these two methods into a novel and unified framework. In order to model input and output segment interaction, this paper proposes to use a two-dimensional matrix to measure the influence of of input and output. This is different from the above two methods. However, I am concerned that this is not a novel approach because [1] proposes a similar idea to use a bipartite graph whose nodes are tokens in the input and output seuqences and input tokens are only connected with the output tokens for explaining machine translation and dialogue. Furthermore, the algorithm proposed here only consider human annotated output sequence. Most of the equation derivations in this paper (i.e., Eq 8 - 10) are based on this strict assumption. Although this brings some theorical benefits (shown in Section 3.3, somewhat similar to the analysis in [2]), explaining human annotated output could be redundant because we can directly ask the annotators for the explaination. Finally, this paper follows optimization in LIME and Shapley value method to solve the proposed two-dimensional explainable models. This is the main novelty that I am aware of after reading this submission. To improve this submission, I believe that expanding the method to model generated response will be very helpful. In addition, although the authors claim that dialogue task is "challenging with its open-ended sentences and multiple acceptable responses", the whole algorithm design does not consider any spefical properties for dialogue task. It is also very interesting to see if this algorithm can be applied to other sequence-to-sequence tasks, such as machine translation and text summarization.

Quality & Clarity:
This paper presents a fairly completed work with various experiments, including automatic metrics and human evaluation. However, the authors do not provide weaknesses (e.g., what happens if we apply it to model generated response) of their proposed model. The paper includes significant amount of equation derivations in the main text, and some of them could be moved to supplementary materials (i.e., Eq 8). The author provides source code with this submission which should be helpful for reproducibility.

Significance:
Model-agnostic explanations model for seqence-to-sequence tasks should be interesting to most of people in text generation community. However, the algorithms proposed in this paper are similar to [1]. They both use two-demensional matrix to represent the interaction between input and output sequence. In addition, in Sec 5, the author compare to [1] by saying "these tasks are less complex than explaining general text generation models". I cannot agree on this because [1] is proposed to explain black-box sequence-to-sequence models. 
limitations_and_societal_impact: n/a
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: The authors propose a method for “explaining” dialogue responses from sequence generation models; that is, determining what parts of the input influence or “explain” certain elements of the output. The method, LERG, is “learned” in one of two ways to assign a score for each input/output token pair indicating the measure of relevance, and ultimately to link the intent between the two. The authors propose that their method must satisfy three properties: unbiased approximation (LERG should explain benefits of picking response given the context); consistency (LERG should explain each generation step); and cause identification (LERG should sort input features by importance). The authors measure the effectiveness of the proposed method in both automatic evaluations and user studies. The automatic evaluations measure the **necessity** and **sufficiency** of the explained segments of the input text by capturing the model perplexity difference when stripping the segments and only including the segments, respectively, essentially measuring how accurate the method’s explanations are at capturing the relevant segments of the input context. Human evaluations display the segments to humans and ask if they can predict the gold response. Both evaluations show their explanation method significantly outperform both random baselines and other methods from the literature.
main_review: *Originality*: The authors claim to be the first to apply local explanation methods to dialogue response generation. It is indeed clear how the work differs from prior contributions; specifically, the authors note how their method can be seen as a combination of prior work in the area of controllable generation and explanation of sentiment classifiers.

*Quality*: The submission is very technically sound, with the equations derived in the text itself. The claims are very well supported, and the experimental results clearly outline the benefit of using the authors proposed method. I especially thought the method of human evaluation was a very appropriate way of measuring the effectiveness of the explanation method.

*Clarity*: The authors structure the paper appropriately, and presentation of results is done well and highlights the effectiveness of the method. However, The middle of the paper describing the methods involve several mathematical equations that are slightly cumbersome (I think some more details could be left to the appendix).

*Significance*: The authors propose a unique and quantifiable method for explanation of response generation models; this is very significant in the space of open-domain dialogue in any capacity where one would like to know why a model said what it said. Implications range from safety applications to control applications. 
limitations_and_societal_impact: See comments above
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: The paper addresses the local explanation of sequence generation models, specifically under the dialogue response generation setting, and proposes local explanation of response generation (LERG) that computes the mutual interactions between input and output sequences under perturbations as the explanations. Automatic and human evaluations show that LERG is more effective than other baselines.

Contributions: 
1. A new local explanation method for dialogue response generation
2. Such an explanation method could be generalized to other generation models as well
3.  A systematic framework to evaluate the dialogue model explanation quality involving both automatic evaluations and human studies.
main_review: Originality: The method is a novel and unified extension of LIME and Shapley value tailored for the dialogue generation setting. The adaption for the dialogue setting is clear with mostly reasonable assumptions. 

Quality: The experimental design is reasonable and complete. One drawback is that the experiments are done on only one social dialogue dataset while there are other types of dialogues such as task-oriented dialogues as well. 

Clarity: The paper is written clearly and easy to follow. I learned a lot from it. One suggestion is that some important information is left in the appendix and if space permits in the final version, more details on the human experiment can be added to the main paper.

Significance: It's not clear if statistical tests were performed from the tables and figures. If the results are statistically significant, I think this work could be a good addition to the current local explanation framework.






limitations_and_societal_impact: Instead of a rigid assumption that  P(y|x)=1 under a dataset, is it possible to model the probability of y given x for the dataset with finetuned language models?

For the human experiments in line 261, I think the setting might be easy for humans since half of the candidates are randomly selected from other dialogues, and it would be easy for humans to get the appropriate next response given the context, even without the provided explanation. 


Typos and grammar:
line 202: grammar
line 254: give -> gives
review_title: null

Meta-review: The paper studies the problem of generating model-agnostic explanations of dialog responses. The proposed method (LERG) estimates importance scores between every input-output segment pair by exploring perturbations of the input. LERG uses two optimization variations (Shapley value and LIME) originally proposed for classification and newly extended to sequence-to-sequence problems. The paper also comes with theoretical justifications, showing that LERG has the following desirable properties: unbiased approximation, consistency, cause identification (proofs are provided in the appendix). While the paper is evaluated only on dialogue, the methods of this paper seem applicable to other tasks such as machine translation.

The paper addresses a very important task, as the black-box nature of most current conversational AI systems is likely hindering our understanding of what goes wrong and what could be improved. Reviewers found the approach to be well motivated, technically sound, and the presentation of the work is quite clear. The only two concerns are:

* Alvarez-Melis and Jaakkola (2017) [1] presents a somewhat similar model-agnostic explanation generation method for sequence-to-sequence models. That said, there are some technical differences, and the paper offers technical contributions relative to [1]: the adaption of established model-agnostic explanations methods to seq2seq problems (LIME and Shapley value), and theoretical justifications (unbiased approximation, etc.). Furthermore, [1] is mainly focused on translation and its application to dialog seems a bit preliminary (the authors of [1] seem frank about that as they call their system “mediocre”). The few examples of dialog responses in [1] seem to suffer from a lack of diversity (“I don’t know”, etc.).

* The paper performs all experiments on gold responses, which is a bit unrealistic. The authors provide some good justifications (i.e., “we are explaining a reasonable response”, mitigating the effects of the sampling process typically used during inference, evaluating on the same gold outputs makes results more comparable), but I think evaluating both on gold and generated responses would have made the paper stronger.

Compared to previous work (e.g., [1]), the paper offers some empirical contributions: evaluation of "necessity" and "sufficiency" (“How is the model influenced after removing explanations?” and “How does the model perform when only the explanations are given?”, respectively). The paper also offers a user study to evaluate the effectiveness of LERG with real users, and the improvement over several baselines seems quite significant. 
 
Decision: Accept (Poster)