Conclusion and Future Work	We have constructed a new dataset and simulator, e-QRAQ, designed to test a network's ability to explain its predictions in a set of multi-turn, challenging reasoning problems. In addition to providing supervision on the correct response at each turn, the simulator provides two types of explanation to the Agent: A natural language assessment of the Agent's prediction which includes language about whether the prediction was correct or not, and a description of what can be inferred in the current state – both about the possible answers and the relevant variables. We used the relevant variable and possible answer explanations to jointly train a modified E2E memory network to both predict and explain it's predictions. Our experiments show that the quality of the explanations strongly correlates with the quality of the predictions. Moreover, when the network has trouble predicting, as it does with queries, requiring it to generate good explanations slows its learning. For future work, we would like to investigate whether we can train the net to generate natural language explanations and how this might affect prediction performance.

Results	To evaluate the model's ability to jointly learn to predict and explain its predictions we performed two experiments. First, we investigate how the prediction accuracy is affected by jointly training the network to produce explanations. Second, we evaluate how well the model learns to generate explanations. To understand the role of the explanation content in the learning process we perform both of these experiments for each of the two types of explanation: relevant variables and possible answers. We do not perform hyperparameter optimization on the E2E Memory Network, since we are more interested in relative performance. While we only show a single experimental run in our Figures, results were nearly identical for over five experimental runs. The experimental results differ widely for the two kinds of explanation considered, where an explanation based on possible answers provides better scores for both experiments. As illustrated in Figure FIGREF52 , simultaneously learning possible-answer explanations does not affect prediction, while learning relevant-variable explanation learning severely impairs prediction performance, slowing the learning by roughly a factor of four. We can observe the same outcome for the quality of the explanations learned, shown in Figure FIGREF53 . Here again the performance on possible-answer explanations is significantly higher than for relevant-variable explanations.

Ambiguities arise because some of the entities in the story have been replaced by variables, some of which may need to be known to answer the challenge question. A successful Agent must reason about what the answers might be, given the ambiguity, and, if there is more than one possible answer, ask for the value of a relevant variable to reduce the possible answer set. In this paper we present a new dataset e-QRAQ constructed by augmenting the QRAQ simulator with the ability to provide detailed explanations about whether the Agent's response was correct and why. Using this dataset we perform some preliminary experiments, training an extended End-to-End Memory Network architecture  to jointly predict a response and a partial explanation of its reasoning. We consider two types of partial explanation in these experiments: the set of relevant variables, which the Agent must know to ask a relevant, reasoned question; and the set of possible answers, which the Agent must know to answer correctly. We demonstrate a strong correlation between the qualities of the prediction and explanation.

e-QRAQ: A Multi-turn Reasoning Dataset and Simulator with Explanations	In this paper we present a new dataset and user simulator e-QRAQ (explainable Query, Reason, and Answer Question) which tests an Agent's ability to read an ambiguous text; ask questions until it can answer a challenge question; and explain the reasoning behind its questions and answer. The User simulator provides the Agent with a short, ambiguous story and a challenge question about the story. The story is ambiguous because some of the entities have been replaced by variables. At each turn the Agent may ask for the value of a variable or try to answer the challenge question. In response the User simulator provides a natural language explanation of why the Agent's query or answer was useful in narrowing down the set of possible answers, or not. To demonstrate one potential application of the e-QRAQ dataset, we train a new neural architecture based on End-to-End Memory Networks to successfully generate both predictions and partial explanations of its current understanding of the problem. We observe a strong correlation between the quality of the prediction and explanation.

The “interaction flow”	The normal interaction flow between the User and the Agent during runtime of the simulator is shown in Figure FIGREF49 , and is - with the exception of the additional explanations - identical to the interaction flow for the original QRAQ proglems  . This means that the User acts as a scripted counterpart to the Agent in the simulated e-QRAQ environment. We show interaction flows for both supervised and reinforcement learning modes. Additionally, we want to point out that INLINEFORM0 in Figure FIGREF49 can be both U and U INLINEFORM1 , i.e. both the natural language explanation and the internal state explanations. Performance and accuracy are measured by the User, that compares the Agent's suggested actions and the Agent's suggested explanations with the ground truth known by the User.

Possible-answer explanations reach an F-Score of .9, while relevant-variable explanations one of .09 only, with precision and recall only slightly deviating from the F-Score in all experiments. We would expect that explanation performance should correlate with prediction performance. Since Possible-answer knowledge is primarily needed to decide if the net has enough information to answer the challenge question without guessing and relevant-variable knowledge is needed for the net to know what to query, we analyzed the network's performance on querying and answering separately. The memory network has particular difficulty learning to query relevant variables, reaching only about .5 accuracy when querying. At the same time, it learns to answer very well, reaching over .9 accuracy there. Since these two parts of the interaction are what we ask it to explain in the two modes, we find that the quality of the explanations strongly correlates with the quality of the algorithm executed by the network.

Experimental Setup	For the experiments, we use the User simulator explanations to train an extended memory network. As shown in Figure FIGREF50 , our network architecture extends the End-to-End Memory architecture of  , adding a two layer Multi-Layer Perceptron to a concatenation of all “hops” of the network. The explanation and response prediction are trained jointly. In these preliminary experiments we do not train directly with the natural language explanation from U, just the explanation of what can be inferred in the current state U INLINEFORM0 . In future experiments we will work with the U explanations directly. Specifically, for our experiments, we provide a classification label for the prediction output generating the Agent's actions, and a vector INLINEFORM0 of the following form to the explanation output (where INLINEFORM1 is an one-hot encoding of dimensionality (or vocabulary size) INLINEFORM2 of word INLINEFORM3 , and INLINEFORM4 is the explanation set: DISPLAYFORM0  For testing, we consider the network to predict a entity in the explanation if the output vector INLINEFORM0 surpasses a threshold for the index corresponding to that entity.

In Example UID13 , illustrating a successful interaction, the Agent asks for the value of $V0 and the User responds with the answer (Silvia) as well as an explanation indicating that it was correct (helpful) and why. Specifically, in this instance it was helpful because it enabled an inference which reduced the possible answer set (and reduced the set of relevant variables). On the other hand, in Example UID30 , we see an example of a bad query and corresponding critical explanation. In general, the e-QRAQ simulator offers the following explanations to the Agent: When answering, the User will provide feedback depending on whether or not the Agent has enough information to answer; that is, on whether the set of possible answers contains only one answer. If the Agent has enough information, the User will only provide feedback on whether or not the answer was correct and on the correct answer if the answer was false. If the agent does not have enough information, and is hence guessing, the User will say so and list all still relevant variables and the resulting possible answers. When querying, the User will provide several kinds of feedback, depending on how useful the query was.

The QRAQ Dataset	A QRAQ domain, as introduced in  , has two actors, the User and the Agent. The User provides a short story set in a domain similar to the HomeWorld domain of  ,  given as an initial context followed by a sequence of events, in temporal order, and a challenge question. The stories are semantically coherent but may contain hidden, sometimes ambiguous, entity references, which the Agent must potentially resolve to answer the question. To do so, the Agent can query the User for the value of variables which hide the identity of entities in the story. At each point in the interaction, the Agent must determine whether it knows the answer, and if so, provide it; otherwise it must determine a variable to query which will reduce the potential answer set (a “relevant” variable). In example SECREF1 the actors $v, $w, $x and $y are treated as variables whose value is unknown to the Agent. In the first event, for example, $v refers to either Hannah or Emma, but the Agent can't tell which.

Introduction	In recent years deep neural network models have been successfully applied in a variety of applications such as machine translation  , object recognition  ,  , game playing  , dialog  and more. However, their lack of interpretability makes them a less attractive choice when stakeholders must be able to understand and validate the inference process. Examples include medical diagnosis, business decision-making and reasoning, legal and safety compliance, etc. This opacity also presents a challenge simply for debugging and improving model performance. For neural systems to move into realms where more transparent, symbolic models are currently employed, we must find mechanisms to ground neural computation in meaningful human concepts, inferences, and explanations. One approach to this problem is to treat the explanation problem itself as a learning problem and train a network to explain the results of a neural computation. This can be done either with a single network learning jointly to explain its own predictions or with separate networks for prediction and explanation. Regardless, the availability of sufficient labelled training data is a key impediment. In previous work  we developed a synthetic conversational reasoning dataset in which the User presents the Agent with a simple, ambiguous story and a challenge question about that story.

The Dataset	This paper's main contribution is an extension to the original QRAQ simulator that provides extensive explanations of the reasoning process required to solve a QRAQ problem. These explanations are created dynamically at runtime, in response to the Agent's actions. The following two examples illustrate these explanations, for several different scenarios: The context (C), events (E), and question (Q) parts of the problem are identical to those in a QRAQ problem. In addition there is a trace of the interaction of a trained Agent (A) model with the User (U) simulator. The simulator provides two kinds of explanations in response to the Agent's query or answer. The first kind denoted “U” indicates whether the Agent's response is correct or not and why. The second kind of explanation, denoted “U INLINEFORM0 ” provides a full description of what can be inferred in the current state of the interaction. In this case the relevant information is the set of possible answers at different points in the interaction (Porch, Boudoir / Porch for Example UID13 ) and the set of relevant variables ($V0 / none for Example UID13 ).

A query on a variable not even occurring in the problem will trigger an explanation that says that the variable is not in the problem. A query on an irrelevant variable will result in an explanation showing that the story's protagonist cannot be the entity hidden by that variable. Finally, a useful (i.e. relevant) query will result in feedback showing the inference that is possible by knowing that variable's reference. This set of inference can also serve as the detailed explanation to obtain the correct answer above. The e-QRAQ simulator will be available upon publication of this paper at the same location as QRAQ  for researchers to test their interpretable learning algorithms.

Related Work	Current interpretable machine learning algorithms for deep learning can be divided into two approaches: one approach aims to explain black box models in a model-agnostic fashion  ,  ; another studies learning models, in particular deep neural networks, by visualizing for example the activations or gradients inside the networks  ,  ,  . Other work has studied the interpretability of traditional machine learning algorithms, such as decision trees  , graphical models  , and learned rule-based systems  . Notably, none of these algorithms produces natural language explanations, although the rule-based system is close to a human-understandable form if the features are interpretable. We believe one of the major impediments to getting NL explanations is the lack of datasets containing supervised explanations. Datasets have often accelerated the advance of machine learning in their perspective areas  , including computer vision  ,  ,  ,  ,  , natural language  ,  ,  , reasoning  ,  ,  , etc. Recently, natural language explanation was added to complement existing visual datasets via crowd-sourcing labeling  . However, we know of no question answering or reasoning datasets which offer NL explanations.

Obviously labeling a large number of examples with explanations is a difficult and tedious task – and not one which is easily delegated to an unskilled worker. To make progress until such a dataset is available or other techniques obviate its need, we follow the approach of existing work such as  ,  , and generate synthetic natural language explanations from a simulator.

We tried several thresholds, some adaptive (such as the average of the output vector's values), but found that a fixed threshold of .5 works best.

In a realistic text this entity obfuscation might occur due to spelling or transcription errors, unknown descriptive references such as “Emma's sibling”, or indefinite pronouns such as “somebody”. Several datasets with 100k problems each and of varying difficulty have been released to the research community and are available for download  .

Question: How do they measure correlation between the prediction and explanation quality?
Answer: They look at the performance accuracy of explanation and the prediction performance