In recent years deep neural network models have been successfully applied in a variety of applications such as machine translation BIBREF0 , object recognition BIBREF1 , BIBREF2 , game playing BIBREF3 , dialog BIBREF4 and more. However, their lack of interpretability makes them a less attractive choice when stakeholders must be able to understand and validate the inference process. Examples include medical diagnosis, business decision-making and reasoning, legal and safety compliance, etc. This opacity also presents a challenge simply for debugging and improving model performance. For neural systems to move into realms where more transparent, symbolic models are currently employed, we must find mechanisms to ground neural computation in meaningful human concepts, inferences, and explanations. One approach to this problem is to treat the explanation problem itself as a learning problem and train a network to explain the results of a neural computation. This can be done either with a single network learning jointly to explain its own predictions or with separate networks for prediction and explanation. Regardless, the availability of sufficient labelled training data is a key impediment. In previous work BIBREF5 we developed a synthetic conversational reasoning dataset in which the User presents the Agent with a simple, ambiguous story and a challenge question about that story. Ambiguities arise because some of the entities in the story have been replaced by variables, some of which may need to be known to answer the challenge question. A successful Agent must reason about what the answers might be, given the ambiguity, and, if there is more than one possible answer, ask for the value of a relevant variable to reduce the possible answer set. In this paper we present a new dataset e-QRAQ constructed by augmenting the QRAQ simulator with the ability to provide detailed explanations about whether the Agent's response was correct and why. Using this dataset we perform some preliminary experiments, training an extended End-to-End Memory Network architecture BIBREF6 to jointly predict a response and a partial explanation of its reasoning. We consider two types of partial explanation in these experiments: the set of relevant variables, which the Agent must know to ask a relevant, reasoned question; and the set of possible answers, which the Agent must know to answer correctly. We demonstrate a strong correlation between the qualities of the prediction and explanation.Current interpretable machine learning algorithms for deep learning can be divided into two approaches: one approach aims to explain black box models in a model-agnostic fashion BIBREF7 , BIBREF8 ; another studies learning models, in particular deep neural networks, by visualizing for example the activations or gradients inside the networks BIBREF9 , BIBREF10 , BIBREF11 . Other work has studied the interpretability of traditional machine learning algorithms, such as decision trees BIBREF12 , graphical models BIBREF13 , and learned rule-based systems BIBREF14 . Notably, none of these algorithms produces natural language explanations, although the rule-based system is close to a human-understandable form if the features are interpretable. We believe one of the major impediments to getting NL explanations is the lack of datasets containing supervised explanations.Datasets have often accelerated the advance of machine learning in their perspective areas BIBREF15 , including computer vision BIBREF16 , BIBREF17 , BIBREF18 , BIBREF19 , BIBREF20 , natural language BIBREF21 , BIBREF22 , BIBREF23 , reasoning BIBREF24 , BIBREF25 , BIBREF5 , etc. Recently, natural language explanation was added to complement existing visual datasets via crowd-sourcing labeling BIBREF26 . However, we know of no question answering or reasoning datasets which offer NL explanations. Obviously labeling a large number of examples with explanations is a difficult and tedious task – and not one which is easily delegated to an unskilled worker. To make progress until such a dataset is available or other techniques obviate its need, we follow the approach of existing work such as BIBREF24 , BIBREF4 , and generate synthetic natural language explanations from a simulator.A QRAQ domain, as introduced in BIBREF5 , has two actors, the User and the Agent. The User provides a short story set in a domain similar to the HomeWorld domain of BIBREF24 , BIBREF27 given as an initial context followed by a sequence of events, in temporal order, and a challenge question. The stories are semantically coherent but may contain hidden, sometimes ambiguous, entity references, which the Agent must potentially resolve to answer the question.To do so, the Agent can query the User for the value of variables which hide the identity of entities in the story. At each point in the interaction, the Agent must determine whether it knows the answer, and if so, provide it; otherwise it must determine a variable to query which will reduce the potential answer set (a “relevant” variable).In example SECREF1 the actors $v, $w, $x and $y are treated as variables whose value is unknown to the Agent. In the first event, for example, $v refers to either Hannah or Emma, but the Agent can't tell which. In a realistic text this entity obfuscation might occur due to spelling or transcription errors, unknown descriptive references such as “Emma's sibling”, or indefinite pronouns such as “somebody”. Several datasets with 100k problems each and of varying difficulty have been released to the research community and are available for download BIBREF28 .This paper's main contribution is an extension to the original QRAQ simulator that provides extensive explanations of the reasoning process required to solve a QRAQ problem. These explanations are created dynamically at runtime, in response to the Agent's actions. The following two examples illustrate these explanations, for several different scenarios:The context (C), events (E), and question (Q) parts of the problem are identical to those in a QRAQ problem. In addition there is a trace of the interaction of a trained Agent (A) model with the User (U) simulator. The simulator provides two kinds of explanations in response to the Agent's query or answer. The first kind denoted “U” indicates whether the Agent's response is correct or not and why. The second kind of explanation, denoted “U INLINEFORM0 ” provides a full description of what can be inferred in the current state of the interaction. In this case the relevant information is the set of possible answers at different points in the interaction (Porch, Boudoir / Porch for Example UID13 ) and the set of relevant variables ($V0 / none for Example UID13 ).In Example UID13 , illustrating a successful interaction, the Agent asks for the value of $V0 and the User responds with the answer (Silvia) as well as an explanation indicating that it was correct (helpful) and why. Specifically, in this instance it was helpful because it enabled an inference which reduced the possible answer set (and reduced the set of relevant variables). On the other hand, in Example UID30 , we see an example of a bad query and corresponding critical explanation.In general, the e-QRAQ simulator offers the following explanations to the Agent:When answering, the User will provide feedback depending on whether or not the Agent has enough information to answer; that is, on whether the set of possible answers contains only one answer. If the Agent has enough information, the User will only provide feedback on whether or not the answer was correct and on the correct answer if the answer was false. If the agent does not have enough information, and is hence guessing, the User will say so and list all still relevant variables and the resulting possible answers.When querying, the User will provide several kinds of feedback, depending on how useful the query was. A query on a variable not even occurring in the problem will trigger an explanation that says that the variable is not in the problem. A query on an irrelevant variable will result in an explanation showing that the story's protagonist cannot be the entity hidden by that variable. Finally, a useful (i.e. relevant) query will result in feedback showing the inference that is possible by knowing that variable's reference. This set of inference can also serve as the detailed explanation to obtain the correct answer above.The e-QRAQ simulator will be available upon publication of this paper at the same location as QRAQ BIBREF28 for researchers to test their interpretable learning algorithms.The normal interaction flow between the User and the Agent during runtime of the simulator is shown in Figure FIGREF49 , and is - with the exception of the additional explanations - identical to the interaction flow for the original QRAQ proglems BIBREF5 . This means that the User acts as a scripted counterpart to the Agent in the simulated e-QRAQ environment. We show interaction flows for both supervised and reinforcement learning modes. Additionally, we want to point out that INLINEFORM0 in Figure FIGREF49 can be both U and U INLINEFORM1 , i.e. both the natural language explanation and the internal state explanations. Performance and accuracy are measured by the User, that compares the Agent's suggested actions and the Agent's suggested explanations with the ground truth known by the User.For the experiments, we use the User simulator explanations to train an extended memory network. As shown in Figure FIGREF50 , our network architecture extends the End-to-End Memory architecture of BIBREF6 , adding a two layer Multi-Layer Perceptron to a concatenation of all “hops” of the network. The explanation and response prediction are trained jointly. In these preliminary experiments we do not train directly with the natural language explanation from U, just the explanation of what can be inferred in the current state U INLINEFORM0 . In future experiments we will work with the U explanations directly.Specifically, for our experiments, we provide a classification label for the prediction output generating the Agent's actions, and a vector INLINEFORM0 of the following form to the explanation output (where INLINEFORM1 is an one-hot encoding of dimensionality (or vocabulary size) INLINEFORM2 of word INLINEFORM3 , and INLINEFORM4 is the explanation set: DISPLAYFORM0 For testing, we consider the network to predict a entity in the explanation if the output vector INLINEFORM0 surpasses a threshold for the index corresponding to that entity. We tried several thresholds, some adaptive (such as the average of the output vector's values), but found that a fixed threshold of .5 works best.To evaluate the model's ability to jointly learn to predict and explain its predictions we performed two experiments. First, we investigate how the prediction accuracy is affected by jointly training the network to produce explanations. Second, we evaluate how well the model learns to generate explanations. To understand the role of the explanation content in the learning process we perform both of these experiments for each of the two types of explanation: relevant variables and possible answers. We do not perform hyperparameter optimization on the E2E Memory Network, since we are more interested in relative performance. While we only show a single experimental run in our Figures, results were nearly identical for over five experimental runs.The experimental results differ widely for the two kinds of explanation considered, where an explanation based on possible answers provides better scores for both experiments. As illustrated in Figure FIGREF52 , simultaneously learning possible-answer explanations does not affect prediction, while learning relevant-variable explanation learning severely impairs prediction performance, slowing the learning by roughly a factor of four. We can observe the same outcome for the quality of the explanations learned, shown in Figure FIGREF53 . Here again the performance on possible-answer explanations is significantly higher than for relevant-variable explanations. Possible-answer explanations reach an F-Score of .9, while relevant-variable explanations one of .09 only, with precision and recall only slightly deviating from the F-Score in all experiments.We would expect that explanation performance should correlate with prediction performance. Since Possible-answer knowledge is primarily needed to decide if the net has enough information to answer the challenge question without guessing and relevant-variable knowledge is needed for the net to know what to query, we analyzed the network's performance on querying and answering separately. The memory network has particular difficulty learning to query relevant variables, reaching only about .5 accuracy when querying. At the same time, it learns to answer very well, reaching over .9 accuracy there. Since these two parts of the interaction are what we ask it to explain in the two modes, we find that the quality of the explanations strongly correlates with the quality of the algorithm executed by the network.We have constructed a new dataset and simulator, e-QRAQ, designed to test a network's ability to explain its predictions in a set of multi-turn, challenging reasoning problems. In addition to providing supervision on the correct response at each turn, the simulator provides two types of explanation to the Agent: A natural language assessment of the Agent's prediction which includes language about whether the prediction was correct or not, and a description of what can be inferred in the current state – both about the possible answers and the relevant variables. We used the relevant variable and possible answer explanations to jointly train a modified E2E memory network to both predict and explain it's predictions. Our experiments show that the quality of the explanations strongly correlates with the quality of the predictions. Moreover, when the network has trouble predicting, as it does with queries, requiring it to generate good explanations slows its learning. For future work, we would like to investigate whether we can train the net to generate natural language explanations and how this might affect prediction performance.

Q: How do they measure correlation between the prediction and explanation quality?
A: They look at the performance accuracy of explanation and the prediction performance

Q: Does the Agent ask for a value of a variable using natural language generated text?