Local Explanation of Dialogue Response Generation
Authors: Yi-Lin Tuan
Connor Pryor
Wenhu Chen
Lise Getoor
William Yang Wang
Venue: NeurIPS 2021 Poster (2021)
Keywords: explainable artificial intelligence
dialogue response generation
text generation

Abstract: In comparison to the interpretation of classification models, the explanation of sequence generation models is also an important problem, however it has seen little attention. In this work, we study model-agnostic explanations of a representative text generation task -- dialogue response generation. Dialog response generation is challenging with its open-ended sentences and multiple acceptable responses. To gain insights into the reasoning process of a generation model, we propose a new method, local explanation of response generation (LERG) that regards the explanations as the mutual interaction of segments in input and output sentences. LERG views the sequence prediction as uncertainty estimation of a human response and then creates explanations by perturbing the input and calculating the certainty change over the human response. We show that LERG adheres to desired properties of explanations for text generation including unbiased approximation, consistency and cause identification. Empirically, our results show that our method consistently improves other widely used methods on proposed automatic- and human- evaluation metrics for this new task by $4.4$-$12.8$\%. Our analysis demonstrates that LERG can extract both explicit and implicit relations between input and output segments.

Local Explanation of Dialogue Response Generation
Yi-Lin Tuan', Connor Pryor?, Wenhu Chen', Lise Getoor", William Yang Wang'
University of Califormia, Santa Barbara
University of California, Santa Cruz
Cytuan, wenhuchen, william}@es www.bucsbbee.o. edu {cipryor, getoor)@ucsc. edu
Abstract
In comparison to the interpretation of classification models, the explanation of sequence generation models also important problem, however it has seen little attention. In this work, we study model-agnostic explanations of a representative text generation task dialogue response generation. Dialog response generation is challenging with its open-ended sentences and multiple acceptable responses. To gain insights into the reasoning process of generation model, we propose a new method, local explanation of response generation (LERG), that regards the explanations as the mutual interaction segments in input and output sentences. LERG views the sequence prediction uncertainty estimation of a human response and then creates explanations by perturbing the input and calculating the certainty change over the human response. We show that LERG adheres desired properties explanation for text generation, including unbiased approximation, consistency, and cause identification. Empirically, our results show that our method consistently improves other widely used methods on proposed automatic- and human- evaluation metrics for this new task by 4.4-12.81 Our analysis demonstrates that LERG extract both explicit and implicit relations between input and output segments.
Introduction
As we use machine learning models daily tasks, such as medical diagnostics [6, 19], speech assistants [31] etc., being able trust the predictions being made has become increasingly important. To understand the underlying reasoning process of complex machine learning models a sub-field of explainable artificial intelligence (XAI) [2, 17, 36] called local explanations, has promising results [35]. Local explanation methods [27, 39] often approximate an underlying black box model by fitting an interpretable proxy, such linear model tree, around the neighborhood of individual predictions. These methods have the advantage of being model-agnostic and locally interpretable.
Traditionally, off-the-shelf local explanation frameworks, such the Shapley value in game the- ory [38] and the learning- based Local Interpretable Model-agnostic Explanation (LIME) [35] have been shown to work well on classification tasks with small number of classes. In particular, there has been work on image classification [35], sentiment analysis [8], and evidence selection for question an- swering [32]. However, the best of our knowledge, there has been less work studying explanations over models with sequential output and large class sizes at each time step. An attempt by [1] aims at explaining machine translation by aligning the sentences in source and target languages. Nonetheless, unlike translation, where it is possible to find almost all word alignments of the input and output sentences, many text generation tasks are not dlignment-based. We further explore explanations over sequences that contain implicit and indirect relations between the input and output utterances.
Our code is available at htttts::/gittai http:sss//ggttbb..mmm/assslllll com//aascalson/EEGG
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

In this paper, we study explanations over set of representative conditional text generation models dialogue response generation models [45, 55]. These models typically aim to produce an engaging and informative [3, 24] response to an input message. The open-ended sentences and multiple acceptable responses in dialogues pose two major challenges: (1) an exponentially large output space and (2) the implicit relations between the input and output texts. For example, the open-ended prompt "How are you today?" could lead to multiple responses depending on the users' emotion, situation, social skills, expressions, etc. A simple answer such "Good. Thank you for asking." does not have an explicit alignment to words in the input prompt. Even though this alignment does not exist, is clear that "good" is the key response to "how are you". To find such crucial corresponding parts in a dialogue, we propose to extract explanations that can answer the question: "Which parts of the response are influenced the most by parts of the prompt?"
To obtain such explanations, we introduce LERG, novel yet simple method that extracts the sorted importance scores of every input-output segment pair from dialogue response generation model. We view this sequence prediction as the uncertainty estimation of one human response and find a linear proxy that simulates the certainty caused from one input segment to an output segment. We further derive two optimization variations of LERG. One is learning-basee [35] and another is the derived optimal similar to Shapley value [38]. To theoretically verify LERG, we propose that an ideal explanation of text generation should adhere to three properties: unbiased approximation, intra-responae consistency, and causal cause identification. To the best of our knowledge, our work is the first to explore explanation over dialog response generation while maintaining all three properties.
To verify if the explanations are both faithful (the explanation is fully dependent on the model being explained) [2] and interpretable (the explanation understandable by humans) [14), we conduct comprehensive automatic evaluations and user study. We evaluate the necessity and sufficiency of the extracted explanation to the generation model by evaluating the perplexity change of removing salient input segments (necessity) and evaluating the perplexity of only salient segments remaining (sufficiency). In our user study, we present annotators with only the most salient parts in an input and ask them to select the most appropriate response from set of candidates. Empirically, our proposed method consistently outperforms baselines on both automatic metrics and human evaluation.
Our key contributions are:
We propose a novel local explanation method for dialogue response generation (LERG).
We propose a unified formulation that generalizes local explanation methods towards se- quence generation and show that our method adheres to the desired properties for explaining conditional text generation.
We build a systematic framework to evaluate explanations of response generation including automatic metrics and user study.
2 Local Explanation
Local explanation methods aim to explain predictions of an arbitrary model by interpreting the neighborhood of individual predictions [35]. It can be viewed training a proxy that adds the contributions of input features to a model's predictions [27]. More formally, given an example with
input features x {xx J the corresponding prediction y with probability fx) Po y/ (the classifier is parameterized by 0), we denote the contribution from each input feature x, as
6; and denote the concatenation of all contributions as c [MM] dMI" RM Two popular
local explanation methods are the barning-based Local Interpretable Model-agnostic Explanations (LIME) [35] and the game theory-based Shapley value [38].
LIME interprets complex classifier f based locally approximating linear classifier around given prediction f(x). The optimization of the explanation model that LIME uses adheres to:
{(x) = arg min [[,, 4,, ) + 2(+)),
where we sample perturbed input ã from n.. (ā) erp(-D(xx 2)2 /0²) taking D(x, .) as distance function and as the width. Ω is the model complexity of the proxy sp The objective of {(x) is to find the simplest yp that can approximate the behavior of f around x. When using a linear classifier
2

---

dallgg
OUEUEE
inpur text
positive
dialng ipput sea
negative
(a) Controllable dialogue models
(b) Explanation of classifier
(c) Our concept
Figure 1: The motivation of local explanation for dialogue response generation. =c)
= (a)+(b).
Φ as the yp minimize ()p)) [35], we can formulate the objective function as:
arg min ES (Pa(Y = -y(²) Φ z)²
(2)
where E {0, 1JM is simplified feature vector of E by a mapping function h such that h(,, = {1 x)} The optimization means to minimize the classification error in the neighborhood of x sampled from Therefore, using LIME, we can find an interpretable linear model that approximates any complex classifier's behavior around an example I.
Shapley value takes the input features | M independent players who cooperate to achieve benefit in a game [38]. The Shapley value computes how much each player x; contributes to the total received benefit:
[ x
1)! PP U P&(Y y|ā)]
(i(x)
Σ BC=
To reduce the computational cost, instead of computing all combinations, proportional to sp; and rewrite the above equation as am expectation over æ sampled from P(t):
find surrogates Φ;
Φ
Pi = EEMP(E) y P&(Y = y|))| Vi,
(4)
where P(ž)
is the perturb function. 2 We can also transform the above formulation
([0) (1) [8)
into argmin:
Φ mig (PPo(Y ylūu P&(Y y( -
(5)
d
3
Local Explanation for Dialogue Response Generation
We aim to explain model's response prediction to a dialogue history one time and call it the local explanation of dialogue response generation. We focus on the local explanation for a more fine-grat fine-grained understanding of the model's behavior.
3.1 Task Definition
As depicted in Figure 1, we draw inspiration from the notions of controllable dialogue generation models (Figure la) and local explanation in sentiment analysis (Figure lb). The first one uses concept predefined classes as the relation between input text and the response; the latter finds the features that correspond positive negative sentiment. We propose to find parts within the input and output texts that are related by an underlying intent (Figure Ic).
We first define the notations for dialogue response generation, which aims to predict a response y = y182...yNv given an input message = .1122...MM I; is the i-th token in sentence with length M and yj is the j-th token in sentence with length N. To solve this task, typical sequence-tor -o-encee----seeeeeee model ff parameterized by produces a sequence of probability masses <Po(vi Jær,, (o(u2lu xr, vI),... ..., Po(yN] (2. MeN) >> [45]. The probability of given can then be com- puted as the product of the sequence Po(y)a) = Pa(yn Pa (y2 (r. y/ y1)... ... Po(yN a H<N).
"ECH((E)) P(t) Thj-T Escar(.) 1/( 1/(""=") = Σ ("T ( (لع-)) =1 This
affirms that the P(#) is a valid probability mass function.

---

To explain the prediction, we then define new explanation model O E RMMNN where each column 1 RM linearly approximates single sequential prediction the j-th time step in text generation. To learn the optimal O, we sample perturbed inputs . from a distribution centered on the original inputs z through probability density function E = (x:). Finally, we optimize & by ensuring u ( 2) z g(ă) whenever is a simplified embedding of ă by a mapping function = = hxx,, .), where we define g as the gain function of the target generative model f. u as a transform function of o and and L as the loss function. Note that can be vector or a matrix and g(+), u(-) can return scalar or a vector depending on the used method. Therefore, we unify the local explanations (LIME and Shapley value) under dialogue response generation as:
Definition 1: A Unified Formulation of Local Explanation for Dialogue Response Generation
1 = arg min L[g(y, Ueح,), u(0f for = 1.2,..,,N.
4;
The proofs of unification into Equation 6 can be found in Appendix A. However, direct adaptation of LIME and Shapley value to dialogue response generation fails to consider the complexity of text generation and the diversity of generated examples. We develop disciplines to alleviate these problems.
3.2 Proposed Method
Our proposed method is designed to (1) address the exponential output space and diverse responses built within the dialogue response generation task and (2) compare the importance of segments within both input and output text.
First, considering the exponential output space and diverse responses, recent work often generates responses using sampling, such as the dominant beam search with top-k sampling [11]. The generated response is therefore only a sample from the estimated probability mass distribution over the output space. Further, the samples drawn from the distribution will inherently have built-in errors that accumulate along generation steps [34]. To avoid these errors we instead explain the estimated probability of the ground truth human responses. In this way, we are considering that the dialogue response generation model is estimating the certainty to predict the human response by Po (y)r)) Meanwhile, given the nature of the collected dialogue dataset, we observe only one response per sentence, and thus the mapping is deterministic. We denote the data distribution by P and the probability of observing a response y given input x in the dataset by P(y|z). Since the mapping of x and y is deterministic in the dataset, assume P(y[z) 1.
Second, if we directly apply prior explanation methods of classifiers on sequential generative models, it turns into a One-vs-Rest classification situation for every generation step. This can cause an unfair comparison among generation steps. For example, the impact from perturbed input on yj could end up being the largest just because the absolute certainty Po [x, yzz) was large. However, the impact from a perturbed input on each part in the output should be how much the certainty has changed after perturbution and how much the change is compured to other parts.
Therefore we propose find explanation in an input-response pair (x,y) by comparing the inter- actions between segments in (z.y), To identify the most salient interaction pair (z,, 2/)) (the i-th segment x and the j-th segment in y), we anticipate that a perturbation ž impacts the j-th part most in y if it causes
D ye v,,)) D(Po Já, veý (lPo |z, br, Vj' #
where D represents a distance function measuring the difference between two probability masses. After finding the different part a;; in x and T. we then define an existing salient interaction in (Ir is V.).
In this work, we replace the distance function D in Equation 7 with Kullback-Leibler divergence (Del) [20]. However, since we reduce the complexity by considering (e(()) as the certainty estimation of y, we are limited to obtaining only one point in the distribution. We transfer the equation by modeling the estimated joint probability by of and y. We reconsider the joint distributions as Po UsJ) such that Στ, Po Vzs) = 1 and q v) = Po. (ž. U53) Po(a,, (I. y) such that
Σx., (65, v) = -Ea, Po Usi) Ea.,, Po. (7. Bes) = 1 with Tine being the inverse function

---

of π. Therefore,
Po Nz,))
D(Po(E. = Dr.((P((,,ss)lllldd. = Σ Σ
Po (T, UsS) log
Po Usi)
Moreover, since are estimating the certainty of response y drawn from data distribution, know that the random variables т is independently drawn from the perturbation model я. Their independent conditional probabilities are (wy(x) = 1 and (T]2)) We approximate the multiplier Po(ū, ~ P(ž.ys,la) = P = x(](x). The divergence can be simplified to
ग (( log Po Mes) = (a log Σ F
Po( MS:) Po No()
D(Po(E. ~
Σ
(9)
To meet the inequality for all j and j' + j. we estimate each value D z in the explanation model & being proportional to the divergence term, where h(x, = {1(r, E =) It turns out to be re-estimating the distinct of the chosen segment yj by normalizing over its original predicted probability.
Po(T,y5) Eg,aCel(e,) log (a Usi))
0'2 z x
(10)
We propose variations optimize @ following the unified formulation defined in Equation 6. First, since logarithm is strietly increasing, so to get the same order of Dij. we can drop off the logarithmic term in Equation 10. After reducing the non-linear factor, we use mean square error as (ā(5))))
the optimization equation becomes the loss function, With the gain function 9 = Polanes))
D = arg min E(() Po(T, (o yıy)
D 22
(11)
We call this variation as LERG_L Algorithm 1, since this optimization is similar to LIME but differs by the gain function being a ratio.
To derive the second variation, we suppose an optimized @ exists and is denoted by Q', we can write that for every è and its correspondent h(x, R).
0 log Po(r, v<)
(12)
We can then find the formal representation of 0iy by
D -21-
0j (4) Ojz, Vi E x\ {2+) and z = h(x
(13)
EsEn((m.) 00 + e:) - 032] (n.))(a;) [log Po | (,.),,e)) - log Po (U [., vej)]
We call this variation LERG_S S in Algorithm 1, since this optimization is similar to Shapley value but differs by the gain function being the difference of logarithm. To further reduce computations, we use Monte Carlo sampling with examples sampling version of Shapley value [41].
3.3 Properties
We propose that an explanation of dialogue response generation should adhere to three properties to prove itself faithful to the generative model and understandable to humans.
unbiased approximation To ensure the explanation model O explains the benefits of picking the sentence y, the summation of all elements in OD should approximate the difference between the certainty of y given x and without x (the language modeling of y).
Property 1:
ΣΣ Dij so log P(y]z) log P(y)
(14)
5

---

Algorithm 1: LOCAL EXPLANATION OF RESPONSE GENERATION
Input: input message 31 R. ground-truth response y y1y2.-UN
Input: a response generation model to be explained
Input: local explanation model parameterized by ①
I/ 1st variation LERG_L for each iteration do sample a batch of ā perturbed from T()) map
{0, 1
compute gold probability Po(Mr (t, yeÿ)
compute perturbed probability Po Polyy 22, MKj)
optimize ① to minimize loss function
L = Σ3 Ez(
Polu, (2,8e)) Polw)
0 z)2
I/ 2nd variation LERG_S
for each i do
sample a batch of # perturbed from T(2) xx(){x;))
① 1 Σn log Po(y, {zt). - log Po Me,), for Vj
return 1yj, for Vi,j
Property 2: consistency ensure the explanation model 1 consistently explains different genera- tion steps given a distance function if
D(Poly,le. ves), Polu, Bol
D(Poluyl.. Poluyl Bey)) Vj', Va E x (x.)
(15)
then Qij > Diye
Property cause identification To ensure that the explanation model sorts different input features by their importance to the results, if
u ulul (24)) Y8 2x)
(16)
then Diy Dej
We prove that our proposed method adheres to all three Properties in Appendix B. Meanwhile Shapley value follows Properties 2 and 3, while LIME follows Property 3 when optimized solution exists. These properties also demonstrate that our method approximates the text generation process while sorting the important segments in both the input and output texts. This could be the reason to serve as explanations to any sequential generative model.
4 Experiments
Explanation is notoriously hard evaluate even for digits and sentiment classification which generally more intuitive than explaining response generation. For digit classification (MNIST), explanations often mark the key curves in figures that identify digit numbers. For sentiment analysis, explanations often mark the positive and negative words in text. Unlike them, we focus on identifying the key parts in both input messages and their responses. Our move requires an explanation include the interactions of the input and output features.
To evaluate the defined explanation, we quantify the necessity and sufficiency of explanations towards a model's 's uncertainty of a response. We evaluate these aspects by answering the following questions.
necessity: How the model influenced after removing explanations?
sufficiency: How does the model perform when only the explanations are given?
Furthermore, we conduct a user study to judge human understandings of the explanations gauge how trustworthy the dialog agents are.

---

24
LME
gradion
rardom
dlant on
UME
166
1.8
Stupley W
Shupley
LERIO 5
200
14
Shap my
Shap Ryva
20
12
1.8
LEnGLS
0310 a.5 0.20
0.5
0.35
1986
0.75 0.00 6:45 050
PPLCR.
1.4
14.5
15.0
USE
Stapley Stapley W
12
12
13.5
140
13.5
1310
1950
(a) PPLCR-
(b) PPLA-
(b) PPLA.
Figure The explanation results of GPT model fine-tuned on DailyDialog.
Figure 3: The explanation re- sults of fine-tuned DialoGPT.
4.1 Dataset, Models, Methods
We evaluate our method over chit-chat dialogues for their more complex and realistic conversations. We specifically select and study popular conversational dataset called DailyDialog [25] because its dialogues are based on daily topics and have less uninformative responses. Due the large variation of topics, open-ended nature of conversations and informative responses within this dataset, explaining dialogue response generation models trained on DailyDialog challenging but accessible."
We fine-tune a GPT-based language model [33, 47) and a DialoGPT [55] on DailyDialog by minimiz- ing the following loss function:
L EE log Po(Ur1æ, (e.5) Vej),
(17)
where f is the model's parameter. We train until the loss converges both models and achieve fairly low test perplexities compared to [25]: 12.35 and 11.83 respectively. The low perplexities demonstrate that the models are more likely to be rationale and therefore, evaluating explanations over these models will be more meaningful and interpretable.
We compare explanations LERG and LERG_S S with attention [46], gradient [43], LIME [35] and Shapley value [42]. We use sample mean for Shapley value to avoid massive computations (Shapley for short), and drop the weights in Shapley value (Shapley-w for short) due to the intuition that not all permutations should exist in natural language [12, 21]. Our comparison is fair since all methods requiring permutation samples utilize the same amount of samples."
4.2 Necessity: How is the model influenced after removing explanations?
Assessing the correctness of estimated important feature relevance requires labeled features for each model and example pair, which rarely accessible. Inspired by [2. 4] who removes the estimated salient features and observe how the performance changes, we introduce the notion necessity that extends their idea. We quantify the necessity of the estimated salient input features to the uncertainty estimation of response generation by perplexity change of removal (PPLCR). defined as:
PPLCR == exp i 1- Σ, log Polyg(RR.W.5)++ Σ, log (o
(18)
where x R is the remaining sequence after removing top-k% salient input features.
P We include our experiments on personalized dialogues and abstractive summarization Appendix Moor experiment details are in Appendix C
7

---

As shown in Figure 2a and Figure 3a², removing larger number of input features consistently causes the monotonically increasing PPLCR- Therefore, reduce the factor that the PPLCR caused by, the removal ratio, we compare all methods with an additional baseline that randomly removes features. LERG_S S and LERG both outperform their counterparts Shapley-w and LIME by 12.8% and 2.2% respectively. We further observe that Shapley-w outperforms the LERG_L. We hypothesize that this because LERG_ L and LIME do not reach an optimal state.
4.3 Sufficiency: How does the model perform when only the explanations are given?
Even though necessity can test whether the selected features are crucial to the model's prediction, it lacks to validate how possible the explanation itself can determine response. complete explanation is able to recover model's prediction without the original input. We name this notion sufficiency testing and formalize the idea as:
PPLA == exp th Eg log Po(vylea)1s))
(19)
where F is the sequential concatenation of the top-k% salient input features.
As shown Figure 2b and Figure 3b, removing larger number of input features gets the PPL.A closer to the perplexity of using all input features 12.35 and 11. 1.88. We again adopt random baseline to compare. LERG S and LERG L again outperform their counterparts Shapley-w and LIME by 5.1% and 3.4% respectively. Furthermore, 12.35 and 11.83 perplexities. This result indicates that LERG able to identify the most relevant features while avoiding features that cause more uncertainty during prediction.
found that LERG S is able to go lower than the original
4.4 User Study
To ensure the explanation is easy-to-undeds understaerdd by non machine learning experts and gives users insights into the model, we resort to user study to answer the question: "If an explanation can be understood by users to respond?"
Method
Acc
Conf
Random
36.15
3.00
Attention
34.75
2.81
We ask human judges to compare explanation methods. Instead of asking judges to annotate their explanation for each dialogue, to increase their agreements we present only the explanations (Top 20% features) and ask them to choose from four response candidates, where one is the ground-truti two are randomly sampled from other dialogues, and the last one is randomly sampled from other turns in the same dialogue. Therefore the questionnaire requires human to interpret the explanations but not guess response that has word overlap with the explanation. The higher accuracy indicates the higher quality of explanations. To conduct more valid human evaluation, we randomly sample 200 conversations with sufficiently long input prompt (length> 10). This way it filters out possibly non-explainable dialogues that can cause ambiguities to annotators and make human evaluation less reliable.
Gradient
42.52
2.97
LIME
46.37
3.26
LERG
47.97
3.24
Shapley-w LERG_S
53.65
3.20
56.03
3.35
Table 1: Confidence (1-5) with denotes not confident and 5 denotes highly confident.
We employ three workers on Amazon Mechanical Turk [7] 6 for each method of each conversation, resulting total 600 annotations. Besides the multiple choice questions, we also ask judges to claim their confidences of their choices. The details can be seen in Appendix D. The results are listed in Table We observe that LERG_L performs slightly better than LIME accuracy while maintaining similar annotator's confidence. LERG S significantly outperforms Shapley-w in both accuracy and annotators' confidence. Moreover, these results indicates that when presenting users with only 20% of the tokens they are able to achieve 56% accuracy while random selection is around 25%.

---

90
you
ary
10T
201atc
cerranor
taey
(b) Sociability: find "No" for the "question mark" and "thanks" for the "would like", the polite way to say "want".
(a) Implication: find the "hot potato" might indicate "gasoline".
(c) Error analysis: related but not the best
Figure 4: Two major categories of local explanation except word alignment and one typical error. The horizontal text is the input prompt and the vertical text is the response.
4.5 Qualitative Analysis
We further analyzed the extracted explanation for each dialogue. We found that these fine- ane-grrined level explanations can be split into three major categories: implication/ / meaning, sociability, and one- to-one word mapping. As shown in Figure 4a, the "hot potato" in response implies the phenomenon of "reduce the price of gasoline". On the other hand, Figure 4b demonstrates that response with sociability can sense the politeness and responds with "thanks". We ignore word-to- word mapping here since it is intuitive and can already be successfully detected by attention models. Figure 4c shows a typical error that our explanation methods can produce. As depicted, the word "carry" related to "bags", "suitcases"" and "luggage". Nonetheless a complete explanation should cluster "carry-on luggages". The error of explanations can result from (1) the target model or (2) the explanation method. When taking the first view, in future work, might explanations method for dialogue generation models where the correct evaluation metrics are still in debates. When taking the second view, need to understand that these methods are trying to explain the model and are not absolutely correct. Hence, we should carefully analyze the explanations and use them as reference and should not fully rely on them.
evaluation
5 Related Work and Discussion
Explaining dialogue generation models is of high interest to understand if a generated response is reasonably produced rather than being a random guess. For example, among works about controllable dialogue generation [15, 26, 37, 40, 48, 50, 51, 53J. Xu et al. [49] takes the dialog act in a controllable response generation model as the explanation. On the other hand, some propose to make dialogue response generation models more interpretable through walking on knowledge graphs [18, 28, 44]. Nonetheless, these works still rely on models with complex architecture and thus are not fully interpretable. We observe the lack of model-agnostic method to analyze the explainability of dialogue response generation models, thus proposing LERG.
Recently, there are applications and advances of local explanation methods [27, 35, 38). For instance in NLP, some analyze the contributions of segments in documents positive and negative senti- ments (4, 8, 9, 29]. Some move forwards to finding segments towards text similarity [10], retrieving text span towards question-answering [32], and making local explanation alignment model in machine tramslation [1]. These tasks could be less complex than explaining general text generation models, such as dialogue generation models, since the the output space is either limited to few classes or able to find one-to-one mapping with the input text. Hence, we need to define how local explanations on text generation should work. However, we would like to note that LERG serves as a general formulation for explaining text generation models with flexible setups. Therefore, the distinct of prior work can also be used extend LERG, such as making the explanations hierarchical. To move forward with the development of explanation methods, LERG can also be extended to dealing
'e did zetest and t-test [22] with the null hypothesis between LERG L and LIME (and LERG and Shapley). For both settings the p-value was less than 0.001, indicating that the proposed methods significantly outperform the baselines.
htttps /wwww mturk. com
9

---

with off- /on- data manifold problem of Shapley value introduced in [13], integrating causal structures separate direct / in-direct relations [12, 16], and fusing concept- / feature- level explanations [5].
Conclusion
Beyond the recent advances interpreting classification models, we explore the possibility to understand sequence generation models in depth. We focus on dialogue response generation and find that its challenges lead to complex and less transparent models. We propose local explanation of response generation (LERG), which aims at explaining dialogue response generation models through the mutual interactions between input and output features. LERG views the dialogue generation models as a certainty estimation of a human response so that it avoids dealing with the diverse output space. To facilitate future research, we further propose unification and three properties of explanations for text generation. The experiments demonstrate that LERG can find explanations that can both recover model's prediction and be interpreted by humans. Next steps can be taking models' explainability as evaluation metrics. integrating concept-level explanations, and proposing new methods for text generation models while still adhering to the properties.
7 Acknowledgement
We thank all the reviewers precious comments in revising this paper. This material is based on work that is partially funded by an unrestricted gift from Google.
References
[1] David Alvarez-Melis and Tommi Jaakkola. A causal framework for explaining the predictions of black-box sqquence-to-sequence models. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 412-421, 2017.
[2] David Alvarez-Melis and Tommi S Jaakkola. Towards robust interpretability with self-explaining neural networks. In NeurlPS, 2018.
[3] Nabiha Asghar, Pascal Poupart, Xin Jiang, and Hang Li. Deep active learning for dialogue generation. In Proceedings of the 6th Joint Conference on Lexical and Computational Semantics (* SEM 20/7), pages 78-83, 2017.
[4] Pepa Atanasova, Jakob Grue Simonsen, Christina Lioma, and Isabelle Augenstein. diagnostic study of explainability techniques for text classification. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language P Processing (EMNLP), pages 3256-3274, 2020.
[5] Mohammad Taha Bahadori and David Heckerman. Debiasing concept- explanations with causal analysis. In International Conference on Learning Representations, 2021. URL http:::/ pet/fformmiid--puuooArra
[6] Mihalj Bakator and Dragica Radosav. Deep learning and medical diagnosis: A review of literature. Multimodal Technologies and Interaction, 2(3):47, 2018.
[7] Michael Buhrmester, Tracy Kwang, and Samuel D Gosling. Amazon's 's mechanical turk: A new source of inexpensive, yet high-quality data? 2016.
[8] Hanjie Chen and Yangfeng Ji. Learning variational word masks to improve the interpretability of neural text classifiers. In Proceedings of the 2020 Conference Empirical Methods Natural Language Processing (EMNLP), pages 4236-425 4251, 2020.
[9] Hanjic Chen, Guangtao Zheng, and Yangfeng Generating hierarchical explanations on text classification via feature interaction detection. Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 5578-5593, 2020.
[10] Hanjie Chen, Song Feng, Jatin Ganhotra, Hui Wan, Chulaka Gunasekara, Sachindra Joshi, and Yangfeng
Ji. Explaining neural network predictions on sentence pairs via learning word-group masks. NAACL, 2021.
[11] Angela Fan, Mike Lewis, and Yann Dauphin. Hierarchical neural story generation. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume I: Long Papers), pages 889-898, 2018
10

---

[12] Christopher Frye, Colin Rowat, and Ilya Feige. Asymmetric shapley values: incorporating causal knowl- edge into model-agnostic explainability. Advances in Neural Information Processing Systems, 33, 2020.
[13] Christopher Frye, Damien de Mijola.. Tom Begley, Laurence Cowton, Megan Stanley, and Ilya Feige. Shapley explainability the data manifold. In International Conference Learning Representations. 2021. URL
net/forum? oer.
[14] Leilani H Gilpin, David Bau, Ben Z Yuan, Ayesha Bajwa, Michael Specter, and Lalana Kagal. Explaining explanations: An overview of interpretability of machine learning. In 2018 IEEE 5th International Conference on data science and advanced analytics (DSAA), pages 80-89. IEEE, 2018.
[15] Prakhar Gupta, Jeffrey P Bigham, Yulia Tsvetkov, and Amy Pavel. Controlling dialogue generation with semantic exemplars. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 3018-3029, 2021.
[16] Tom Heskes, Evi Sijben, Ioan Gabriel Bucur, and Tom Claassen. Causal shapley values: Exploiting causal knowledge to explain individual predictions of complex models. Advances in Neural Information Processing Systems, 33, 2020.
[17] Jeya Vikranth Jeyakumar, Joseph Noor, Yu-Hsi Cheng. Luis Garcia, and Mani Srivastava. How can i explain this to you? an empirical study of deep neural network explanation methods. Advances in Neural Information Processing Systems, 2020.
[18] Rishabh Joshi, Vidhisha Balachandran, Shikhar Vashishth, Alan Black, and Yulia Tsvetkov. Dialograph: Incorporating interpretable strategygraph networks into negotiation dialogues. In International Conference on Learning Representations, 2021.
[19] Igor Kononeako. Machine learning for medical diagnosis: history, state of the art and perspective. Artificial Intelligence in medicine, 23(1):89-1099 2001.
[20] Solomon Kullback and Richard A Leibler. On information and sufficiency. The annals of mathematical statistics, 22(1):79-86, 1951.
[21] Elizabeth Kumar, Suresh Venkatasubramaniin, Carlos Seheidegger, and Sorelle Friedler. Problems with shapley-value-basee explanations as feature importance measures. In International Conference on Machine Learning, pages 5491-5500. PMLR, 2020.
[22] Erich Leo Lehmann and Joseph P Romano. Testing statistical hypotheses, volume 3. Springer.
[23] Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Ahdelrahman Mohamed, Omer Levy, Ves Stoyanov, and Luke Zettlemoyer. Bart: Denoising squeence-o-sequeee pre- traiirain for natural language generation, translation, and comprehension. arXiv preprint 19t.:1190.13611111 2019.
[24] Jiwei Li, Will Monroe, Alan Ritter, Dan Jurafsky, Michel Galley, and Jianfeng Gao. Deep reinforcement learning for dialogue generation. In Proceedings the 2016 Conference on Empirical Methods in Natural Language Processing, pages 192-1202, 2016.
[25] Yanran Li, Hui Su, Xiaoyu Shen, Wenjie Li, Ziqiang Cao, and Shuzi Niu. DailyDialog: A manually labelled multi-turn dialogue dataset. In Proceedings of the Eighth International Joins Conference Natural Language Processing (Volume 1: Long Papers), pages 986-995, Taipei. Taiwan, November 2017. Asian Federation of Natural Language Processing. URL https: /////////ww aclweb. org/ http//nttoloogg111-199999
[26] Zhaojiang Lin, Andrea Madotto, Yejin Bang, and Pascale Fung. The adapter-bot: All-in-one controllable conversational model. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pages 16081-16083, 2021.
[27] Scott M Lundberg and Su-In Lee. A unified approach to interpreting model predictions. In Advances in neural information processing systems, pages 4765-4774, 2017.
[28] Scungwhan Moon, Pararth Shah, Anuj Kumar, and Rajen Subba. Opendialkg: Explainable conversational reasoning with attention-based walks over knowledge graphs. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 845-854, 2019.
[29] W James Murdoch, Peter J Liu, and Bin Yu. Beyond word importance: Contextual decomposition to extract interactions from Istms. In International Conference on Learning Representations. 2018
[30] Shashi Narayan. Shay B Cohen, and Mirella Lapata. Don't give me the details, just the summary! topic- aware convolutional neural networks for extreme summarization. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 1797-1807, 2018
11

---

[31] Daniel Povey, Arnab Ghoshal, Gilles Boulianne. Lukas Burget, Ondrej Glembek, Nagendra Goel, Mirko Hannemann, Peur Motlicek, Yanmin Qian, Petr Schwarz, al. The kaldi speech recognition toolkit. In IEEE 2011 workshop on automatic speech recognition and understanding. number CONE. IEEE Signal Processing Society, 2011.
[32] Danish Pruthi, Bhuwan Dhingra, Livio Baldini Soares, Michael Collins, Zachary C Lipton, Graham Neubig, and William W Cohen. Evaluating explanations: How much do explanations from teacher aid students? arXiv preprint arXiv:2012.0089) 2020.
[33] Alec Radford, Karthik Narasinhan, Tim Salimans, and llya Sutskever. Improving language understanding by generative pre-training. 2018.
[34] Marc' Aurelio Ranzato, Sumit Chopra, Michael Auli, and Wojciech Zaremba. Sequence level training with recurrent neural networks. ICIR, 2016.
[35] Marco Tulio Ribeiro, Sameer Singh, and Carlos Guestrin. why should i trust you?" explaining the predictions of any classifier. In Proceedings of the 22nd ACM SIGKDD international conference on knowledge discovery and data mining, pages 1135-1144, 2016.
[36] Cynthia Rudin. Stop explaining black box machine learning models for high stakes decisions and use interpretable models instead. Nature Machine Intelligence, 1 215, 2019.
[37] Abigail See, Stephen Roller, Douwe Kiela, and Jason Weston. What makes a good conversation? how controllable attributes affect human judgments. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguisties: Human Language Technologies. Volume (Long and Short Papers), pages 1702-1723, 2019.
[38] Lloyd S Shapley. A value for n-person games.
[39] Avanti Shrikumar, Peyton Greenside, and Anshul Kundaje. Learning important features through propagat- ing activation diffcrences. In International Conference on Machine Learning, pages 3145-3125 PMLR, 2017.
[40] Eric Michael Smith, Diana Gonzalez-Rica Rico, Emily Dinan, and Y-Lan Boureau. Controlling style in generated dialogue. arXiv preprint wrXi:2200911885: 2020.
[41] Erik Strumbelj and Igor Kononenko. An efficient explanation of individual classifications using game theory. The Journal of Machine Learning Research, 11:1-18. 2010.
[42] Erik Strumbelj and Igor Kononenko. Explaining prediction models and individual predictions with feature contributions. Knowledge and information systems, 41(3):647-666 665, 2014.
[43] Mukund Sundararajan, Ankur Taly, and Qiqi Yan. Axiomatic attribution for deep networks. In International Conference on Machine Learning, pages 3319-3328. PMLR, 2017.
[44] Yi-Lin Tuan, Yun-Nung Chen, and Hung-yi Lee. Dykgehat: Benchmarking dialogue generation grounding dynamic knowledge graphs. In Proceedings of the 2019 Conference on Empirical Methods Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP- IJCNLP), pages 1855-1865, 2019.
[45] Orial Vinyals and Quoc T.e. A neural conversational model. arXiv preprint | 2015
[46] Sarah Wiegreffe and Yuval Pinter. Attention is not not explanation. In Proceedings of the 2019 Conference Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNIPP), pages 1 11-20, 2019.
[47] Thomas Wolf, Victor Sanh, Julien Chaumond, and Clement Delangue. Transfertransfo: A transfer learning approach for neural network based conversational agents. NeurIPS 2018 CAI Workshop, 2019.
[48] Zeqiu Wu. Michel Galley, Chris Brockett, Yizhe Zhang, Xiang Gao, Chris Quirk, Rik Koncel- Kedziorski, Jianfeng Gao, Hannanch Hajishirzi, Mari Ostendorf, et al. A controllable model of grounded response generation. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pages 14085- 14093, 2021.
[49] Can Xu, Wei Wu, and Yu Wu. Towards explainable and controllable open domain dialogue generation with dialogue acts. arXiv preprint arXiv: 1807.07255 2018
[50] Can Xu, Wei Wu, Chongyang Tao, Huang Hu, Matt Schuerman, and Ying Wang. Neural response genera- tion with nords. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 54 5416-5426, 2019.
12

---

[51] Haiqin Yang, Xiaoyuan Yao, Yiqun Duan, Jianping Shen, Jie Zhong, and Kun Zhang, Progressive open-domain response generation with multiple controllable attributes. 2021.
[52] H Peyton Young. Monotonic solutions of cooperative games. International Journal of Game Theary, 14 (2):65-722 1985.
[53] Ruqing Zhang. Jiafeng Guo, Yixing Fan. Yanyan Lan, Jun Xu, and Xueqi Cheng. Learning to control the specificity in neural response generation. Proceedings the 56th Annual Meeting of the Association for Computational Linguistics (Volume Long Pupers), pages 1108 1117,111111111 2018.
[54] Saizheng Zhang. Emily Dinan, Jack Urbanek. Arthur Szlam, Douwe Kiela, and Jason Weston. Personalizing dialogue agents: have dog. do you have pets too? Proceedings of 56th Anmual Meeting of Association for Computational Linguistics (Volume 1: Long Pupers), pages 2204-2213. 2018.
[55] Yizhe Zhang, Siqi Sun, Michel Galley, Yen- Ven-Chun Chen, Chris Brockett, Xiang Gao, Jianfeng Gao, Jingjing
Liu, and William B Dolan. Dialogpt: Large-scale generative pre-training training for conversational response generation. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics: System Demonstrations, pages 270-276 2020.
13


Reviews:
[{"review_id": "srtdEwHnTKN", "reviewer": "Reviewer_BdaH", "summary": "This work proposes a method to locally explain the dialog response generation process. This method is motivated by the intuition that, such method should answer the question \"which parts of the response are influenced the most by parts of the prompt\". So the importance scores of every input-output segment pairs are calculated. They also proposed two metrics to evaluate explanations: \"necessity\" and \"sufficiency\". The former is defined as the perplexity change after removing top \"salient\" input features, and the latter is defined based on the intuition a complete explanation should recover model's prediction without the original input.", "questions": "", "limitations": "", "rating": 6, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "this work focuses on an important problem: the explanation of generative models. This work proposes a model-agnostic tool to provide explanations, and such tools should be very helpful to the process to debug, understand and trust the model. Their pairwise importance scores also seem a reasonable method.\n\nHowever, this reviewer believe this work can be improved if the following concerns can be resolved: The examples shown in Figure 4 are some token-to-token relation, this together with the error analysis seems to indicate that ngram-level segment (not token-level) can be sometimes important and not well captured by the model. Could the authors comment on this? is it easy to show ngram-to-ngram relation? does it perform well?", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "See above concern regarding ngram-level explanation.\nThis reviewer doesn't think this work has negative societal impact. Actually, if reasonable explanations can be provided for machine learning generative models, it should have positive societal impact for people to understand the models.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "24"}, "extra_text": {}}, {"review_id": "Wdcs4FZGGB", "reviewer": "Reviewer_R5ns", "summary": "This paper studies the model-agnostic explanations of dialogue response generation and proposes a new called local explanation of response generation (LERG). LERG extracts the sorted importance scores of every input-output segment pair from a dialogue response generation model. It views the sequence prediction as uncertainty estimation of a human response and creates explanations by perturbing the input and calculating the certainty change over the human response. The authors show the proposed LERG adhere to three properties of  an ideal explanation of text generation: (1) unbiased approximation, (2) intra-response consistency and (3) causal cause identification. The experiments on a popular benchmark DailyDialog empirically verify the effectiveness of the proposed approach.", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "**Strengths:**\n\nThe black box nature of current end2end neural dialogue models hinders us from understanding the underlying reasoning process of these models. This paper introduces a novel method to explore explanation for dialog response generation models for the first time. Overall, I like this idea.\n\nThey mathematically formulate the task and generalizes local explanation methods towards sequence generation and show that our method adheres to the desired properties for explaining conditional text generation.\n\nThey introduce the corresponding evaluation methods to assess the explanations of dialogue response generation and empirical study demonstrate the effectiveness of proposed LERG.\n\n\n**Weaknesses:**\n\nExperiment conducted on only one dataset is somewhat a little weak. Have you tried the other popular benchmarks in the dialogue generation task, such as Reddit Corpus, Persona-Chat (Liu et al., 2020) and Topical-Chat (Gopalakrishnan et al., 2019)?\n\nSection 4.2 and 4.3 are lack of the insightful analysis for the reported results. I suggest the authors to share more insights to readers here, rather than just reporting the plain experimental results.\n\nIt seems there is no clarification about the code release in the paper. Will you make your code publicly available to the community for the reproduction later?\n\nMinor comments:\n\nmissing reference on controllable dialogue generation in the related work:\n\n[1] Zhang et al. Learning to control the specificity in neural response generation. ACL 2018.\n\n[2] Xu et al. Neural response generation with meta-words. ACL 2019.\n\n[3] See et al. What makes a good conversation? how controllable attributes affect human judgments. arXiv 2019.\n\n[4] Smith et al. Controlling Style in Generated Dialogue. arXiv 2020.\n\n[5] Wu et al. A controllable model of grounded response generation. AAAI 2020.\n\n[6] Madotto et al. The Adapter-Bot: All-In-One Controllable Conversational Model. arXiv 2020.\n\n[7] Yang et al. Progressive Open-Domain Response Generation with Multiple Controllable Attributes. IJCAI 2021.\n\n[8] Gupta et al. Controlling Dialogue Generation with Semantic Exemplars. NAACL 2021.\n\n\n\n\n\n\n\n\n\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes, the authors have addressed this.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "6"}, "extra_text": {}}, {"review_id": "HqbnqsimwxQ", "reviewer": "Reviewer_CScg", "summary": "The paper addresses the local explanation of sequence generation models, specifically under the dialogue response generation setting, and proposes local explanation of response generation (LERG) that computes the mutual interactions between input and output sequences under perturbations as the explanations. Automatic and human evaluations show that LERG is more effective than other baselines.\n\nContributions: \n1. A new local explanation method for dialogue response generation\n2. Such an explanation method could be generalized to other generation models as well\n3.  A systematic framework to evaluate the dialogue model explanation quality involving both automatic evaluations and human studies.", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Originality: The method is a novel and unified extension of LIME and Shapley value tailored for the dialogue generation setting. The adaption for the dialogue setting is clear with mostly reasonable assumptions. \n\nQuality: The experimental design is reasonable and complete. One drawback is that the experiments are done on only one social dialogue dataset while there are other types of dialogues such as task-oriented dialogues as well. \n\nClarity: The paper is written clearly and easy to follow. I learned a lot from it. One suggestion is that some important information is left in the appendix and if space permits in the final version, more details on the human experiment can be added to the main paper.\n\nSignificance: It's not clear if statistical tests were performed from the tables and figures. If the results are statistically significant, I think this work could be a good addition to the current local explanation framework.\n\n\n\n\n\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Instead of a rigid assumption that  P(y|x)=1 under a dataset, is it possible to model the probability of y given x for the dataset with finetuned language models?\n\nFor the human experiments in line 261, I think the setting might be easy for humans since half of the candidates are randomly selected from other dialogues, and it would be easy for humans to get the appropriate next response given the context, even without the provided explanation. \n\n\nTypos and grammar:\nline 202: grammar\nline 254: give -> gives\n\n\n", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "1 hour"}, "extra_text": {}}, {"review_id": "EV7ZdtwzlrP", "reviewer": "Reviewer_TNnK", "summary": "The authors propose a method for “explaining” dialogue responses from sequence generation models; that is, determining what parts of the input influence or “explain” certain elements of the output. The method, LERG, is “learned” in one of two ways to assign a score for each input/output token pair indicating the measure of relevance, and ultimately to link the intent between the two. The authors propose that their method must satisfy three properties: unbiased approximation (LERG should explain benefits of picking response given the context); consistency (LERG should explain each generation step); and cause identification (LERG should sort input features by importance). The authors measure the effectiveness of the proposed method in both automatic evaluations and user studies. The automatic evaluations measure the **necessity** and **sufficiency** of the explained segments of the input text by capturing the model perplexity difference when stripping the segments and only including the segments, respectively, essentially measuring how accurate the method’s explanations are at capturing the relevant segments of the input context. Human evaluations display the segments to humans and ask if they can predict the gold response. Both evaluations show their explanation method significantly outperform both random baselines and other methods from the literature.", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "*Originality*: The authors claim to be the first to apply local explanation methods to dialogue response generation. It is indeed clear how the work differs from prior contributions; specifically, the authors note how their method can be seen as a combination of prior work in the area of controllable generation and explanation of sentiment classifiers.\n\n*Quality*: The submission is very technically sound, with the equations derived in the text itself. The claims are very well supported, and the experimental results clearly outline the benefit of using the authors proposed method. I especially thought the method of human evaluation was a very appropriate way of measuring the effectiveness of the explanation method.\n\n*Clarity*: The authors structure the paper appropriately, and presentation of results is done well and highlights the effectiveness of the method. However, The middle of the paper describing the methods involve several mathematical equations that are slightly cumbersome (I think some more details could be left to the appendix).\n\n*Significance*: The authors propose a unique and quantifiable method for explanation of response generation models; this is very significant in the space of open-domain dialogue in any capacity where one would like to know why a model said what it said. Implications range from safety applications to control applications. ", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "See comments above", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "1.5 hours"}, "extra_text": {}}, {"review_id": "CfTevaWRQLS", "reviewer": "Reviewer_Dmqp", "summary": "This paper proposes a model-agnostic explanations model, local explanation of response generation (LERG), for dialogue response generation task. Due to the sequence-to-sequence natrual, previous works, which normal produces a single label as output, are no longer suitable for this task. This paper regards the\nexplanations as the mutual interaction of segments in input and output sentences. Specifically, it views the sequence prediction as uncertainty estimation of a human response and creates explanations by perturbing the input and calculating the certainty change over the human response. The experiment shows that the proposed approach extract both explicit and implicit relations between input and output segments.", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Originality: \nmodel-agnostic explanation models for sequence-to-sequence task are not a new task. [1] proposes methods to explain the black-box sequence-to-sequence model, including machine translaton and dialogue task. This paper first review popular model-agnostic explanation models, LIME and Shapley value, then reformulate these two methods into a novel and unified framework. In order to model input and output segment interaction, this paper proposes to use a two-dimensional matrix to measure the influence of of input and output. This is different from the above two methods. However, I am concerned that this is not a novel approach because [1] proposes a similar idea to use a bipartite graph whose nodes are tokens in the input and output seuqences and input tokens are only connected with the output tokens for explaining machine translation and dialogue. Furthermore, the algorithm proposed here only consider human annotated output sequence. Most of the equation derivations in this paper (i.e., Eq 8 - 10) are based on this strict assumption. Although this brings some theorical benefits (shown in Section 3.3, somewhat similar to the analysis in [2]), explaining human annotated output could be redundant because we can directly ask the annotators for the explaination. Finally, this paper follows optimization in LIME and Shapley value method to solve the proposed two-dimensional explainable models. This is the main novelty that I am aware of after reading this submission. To improve this submission, I believe that expanding the method to model generated response will be very helpful. In addition, although the authors claim that dialogue task is \"challenging with its open-ended sentences and multiple acceptable responses\", the whole algorithm design does not consider any spefical properties for dialogue task. It is also very interesting to see if this algorithm can be applied to other sequence-to-sequence tasks, such as machine translation and text summarization.\n\nQuality & Clarity:\nThis paper presents a fairly completed work with various experiments, including automatic metrics and human evaluation. However, the authors do not provide weaknesses (e.g., what happens if we apply it to model generated response) of their proposed model. The paper includes significant amount of equation derivations in the main text, and some of them could be moved to supplementary materials (i.e., Eq 8). The author provides source code with this submission which should be helpful for reproducibility.\n\nSignificance:\nModel-agnostic explanations model for seqence-to-sequence tasks should be interesting to most of people in text generation community. However, the algorithms proposed in this paper are similar to [1]. They both use two-demensional matrix to represent the interaction between input and output sequence. In addition, in Sec 5, the author compare to [1] by saying \"these tasks are less complex than explaining general text generation models\". I cannot agree on this because [1] is proposed to explain black-box sequence-to-sequence models. ", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "n/a", "review_text": "", "extra_scores": {"ethical_concerns": "n/a", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "6"}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper studies the problem of generating model-agnostic explanations of dialog responses. The proposed method (LERG) estimates importance scores between every input-output segment pair by exploring perturbations of the input. LERG uses two optimization variations (Shapley value and LIME) originally proposed for classification and newly extended to sequence-to-sequence problems. The paper also comes with theoretical justifications, showing that LERG has the following desirable properties: unbiased approximation, consistency, cause identification (proofs are provided in the appendix). While the paper is evaluated only on dialogue, the methods of this paper seem applicable to other tasks such as machine translation.

The paper addresses a very important task, as the black-box nature of most current conversational AI systems is likely hindering our understanding of what goes wrong and what could be improved. Reviewers found the approach to be well motivated, technically sound, and the presentation of the work is quite clear. The only two concerns are:

* Alvarez-Melis and Jaakkola (2017) [1] presents a somewhat similar model-agnostic explanation generation method for sequence-to-sequence models. That said, there are some technical differences, and the paper offers technical contributions relative to [1]: the adaption of established model-agnostic explanations methods to seq2seq problems (LIME and Shapley value), and theoretical justifications (unbiased approximation, etc.). Furthermore, [1] is mainly focused on translation and its application to dialog seems a bit preliminary (the authors of [1] seem frank about that as they call their system “mediocre”). The few examples of dialog responses in [1] seem to suffer from a lack of diversity (“I don’t know”, etc.).

* The paper performs all experiments on gold responses, which is a bit unrealistic. The authors provide some good justifications (i.e., “we are explaining a reasonable response”, mitigating the effects of the sampling process typically used during inference, evaluating on the same gold outputs makes results more comparable), but I think evaluating both on gold and generated responses would have made the paper stronger.

Compared to previous work (e.g., [1]), the paper offers some empirical contributions: evaluation of "necessity" and "sufficiency" (“How is the model influenced after removing explanations?” and “How does the model perform when only the explanations are given?”, respectively). The paper also offers a user study to evaluate the effectiveness of LERG with real users, and the improvement over several baselines seems quite significant. 
 
Decision: Accept (Poster)