Introduction
Question Answering is a task that requires capabilities beyond simple NLP since it involves both linguistic techniques and inference abilities. Both the document sources and the questions are expressed in natural language, which is ambiguous and complex to understand. To perform such a task, a model needs in fact to understand the underlying meaning of text. Achieving this ability is quite challenging for a machine since it requires a reasoning phase (chaining facts, basic deductions, etc.) over knowledge extracted from the plain input data. In this article, we focus on two Question Answering tasks: a Reasoning Question Answering (RQA) and a Reading Comprehension (RC). These tasks are tested by submitting questions to be answered directly after reading a piece of text (e.g. a document or a paragraph).
Recent progress in the field has been possible thanks to machine learning algorithms which automatically learn from large collections of data. Deep Learning BIBREF0 algorithms achieve the current State-of-The-Art in our tasks of interest. A particularly promising approach is based on Memory Augmented Neural Networks. These networks are also known as Memory Networks BIBREF1 or Neural Turing Machines BIBREF2 . In the literature the RQA and RC tasks are typically solved by different models. However, the two tasks share a similar scope and structure. We propose to tackle both with a model called Question Dependent Recurrent Entity Network, which improves over the model called Recurrent Entity Network BIBREF3 .
Our major contributions are: 1) exploiting knowledge of the question for storing relevant facts in memory, 2) adding a tighter regularisation scheme, and 3) changing the activation functions. We test and compare our model on two datasets, bAbI BIBREF4 and BIBREF5 , which are standard benchmark for both tasks. The rest of the paper is organised as follows: section Related outlines the models used in QA tasks, while section Model the proposed QDREN model. Section Experiments and Results show training details and performance achieved by our model. The section Analysis reports a visualisation with the aim to explain the obtained results. Finally, section Conclusions summarise the work done.
Reasoning Question Answering
A set of synthetic tasks, called bAbI BIBREF4 , has been proposed for testing the ability of a machine in chaining facts, performing simple inductions or deductions. The dataset is available in two sizes, 1K and 10K training samples, and in two settings, i.e. with and without supporting facts. The latter allows knowing which facts in the input are needed for answering the question (i.e. a stronger supervision). Obviously, the 1K sample setting with no supporting facts is quite hard to handle, and it is still an open research problem. Memory Network BIBREF1 was one of the first models to provide the ability to explicitly store facts in memory, achieving good results on the bAbI dataset. An evolution of this model is the End to End Memory Network BIBREF6 , which allows for end-to-end training. This model represents the State-of-The-Art in the bAbI task with 1K training samples. Several other models have been tested in the bAbI tasks achieving competitive results, such as Neural Turing Machine BIBREF2 , Differentiable Neural Computer BIBREF7 and Dynamic Memory Network BIBREF8 , BIBREF9 . Several other baselines have also been proposed BIBREF4 , such as: an $n$ -gram BIBREF10 models, an LSTM reader and an SVM model. However, some of them still required strong supervision by means of the supporting facts.
Reading Comprehension
Reading Comprehension is defined as the ability to read some text, process it, and understand its meaning. A impending issue for tackling this task was to find suitably large datasets with human annotated samples. This shortcoming has been addressed by collecting documents which contain easy recognizable short summary, e.g. news articles, which contain a number of bullet points, summarizing aspects of the information contained in the article. Each of these short summaries is turned into a fill-in question template, by selecting an entity and replacing it with an anonymized placeholder.
Three datasets follows this style of annotation: Children’s Text Books BIBREF11 , CNN & Daily Mail news articles BIBREF5 , and Who did What BIBREF12 . It is also worth to mention Squad BIBREF13 , a human annotated dataset from Stanford NLP group. Memory Networks, described in the previous sub-section, has been tested BIBREF11 on both the CNN and CBT datasets, achieving good results. The Attentive and Impatient Reader BIBREF5 was the first model proposed for the CNN and Daily Mail dataset, and it is therefore often used as a baseline. While this model achieved good initial results, shortly later a small variation to such model, called Standford Attentive Reader BIBREF14 , increased its accuracy by 10%. Another group of models are based on an Artificial Neural Network architecture called Pointer Network BIBREF15 . Attentive Sum Reader BIBREF16 and Attention over Attention BIBREF17 use a similar idea for solving different reading comprehension tasks. EpiReader BIBREF18 and Dynamic Entity Representation BIBREF19 , partially follow the Pointer Network framework but they also achieve impressive results in the RC tasks. Also for this task several baselines, both learning and non-learning, have been proposed. The most commonly used are: Frame-Semantics, Word distance, and LSTM Reader BIBREF5 and its variation (windowing etc.).
Proposed Model
Our model is based on the Recurrent Entity Network (REN) BIBREF3 model. The latter is the only model able to pass all the 20 bAbI tasks using the 10K sample size and without any supporting facts. However, this model fails many tasks with the 1K setting, and it has not been tried on more challenging RC datasets, like the CNN news articles. Thus, we propose a variant to the original model called Question Dependent Recurrent Entity Network ( $QDREN$ ). This model tries to overcome the limitations of the previous approach. The model consists in three main components: Input Encoder, Dynamic Memory, and Output Module.
The training data consists of tuples $\lbrace (x_i,y_i)\rbrace _{i=1}^n$ , with $n$ equal to the sample size, where: $x_i$ is composed by a tuple $(T, q)$ , where $T$ is a set of sentences $\lbrace  s_{1},\dots ,s_{t}\rbrace $ , each of which has $m$ words, and $q$ a single sentence with $k$ words representing the question. Instead, $y_i$ is a single word that represents the answer.
The Input Encoder transforms the set of words of a sentence $s_{t}$ and the question $q$ into a single vector representation by using a multiplicative mask. Let's define $E\in \mathbb {R}^{|V|\times d}$ the embedding matrix, that is used to convert words to vectors, i.e. $E(w)=e \in \mathbb {R}^d$ . Hence, $\lbrace e_{1},\dots ,e_{m} \rbrace $ are the word embedding of each word in the sentence $s_{t}$ and $\lbrace e_{1},\dots ,e_{k}\rbrace $ the embedding of the question's words. The multiplicative masks for the sentences are defined as $f^{(s)} = \lbrace  f_1^{(s)},\dots ,f_m^{(s)}\rbrace $ and $f^{(q)} = \lbrace  f_1^{(q)},\dots ,f_m^{(q)}\rbrace $ for the question, where each $f_i \in \mathbb {R}^d$ . The encoded vector of a sentence is defined as:
$$s_{t} = \sum _{r=1}^m e_{r} \odot f_r^{(s)} \qquad \qquad q= \sum _{r=1}^k e_{r} \odot f_r^{(q)} \nonumber $$   (Eq. 4)
Dynamic Memory stores information of entities present in $T$ . This module is very similar to a Gated Recurrent Unit (GRU) BIBREF20 with a hidden state divided into blocks. Moreover, each block ideally represents an entity (i.e. person, location etc.), and it stores relevant facts about it. Each block $i$ is made of a hidden state $h_i\in \mathbb {R}^d$ and a key $k_i\in \mathbb {R}^d$ , where $d$ is the embedding size. The Dynamic Memory module is made of a set of blocks, which can be represent with a set of hidden states $\lbrace  h_1,\dots ,h_z \rbrace $ and their correspondent set of keys $\lbrace  k_1,\dots ,k_z \rbrace $ . The equation used to update a generic block $i$ are the following: $ g_i^{(t)} =& \sigma (s_t^T h_i^{(t-1)} + s_t^T k_i^{(t-1)} + s_t^T q ) &\text{(Gating Function)}&\\ \hat{h}_i^{(t)} =& \phi (U h_i^{(t-1)} + V k_i^{(t-1)} + W s_t ) &\text{(Candidate Memory)}&\\ h_i^{(t)} =& h_i^{(t-1)} + g_i^{(t)} \odot \hat{h}_i^{(t)} &\text{(New Memory)}&\\ h_i^{(t)} =& h_i^{(t)}/\Vert  h_i^{(t)} \Vert  &\text{(Reset Memory)}&\\ $
where $\sigma $ represents the sigmoid function, $\phi $ a generic activation function which can be chosen among a set (e.g. sigmoid, ReLU, etc.). $g_i^{(t)}$ is the gating function which determines how much the $i$ th memory should be updated, and $\hat{h}_i^{(t)}$ is the new candidate value of the memory to be combined with the existing one $h_i^{(t-1)}$ . The matrix $U \in \mathbb {R}^{d \times d}$ , $V \in \mathbb {R}^{d \times d}$ , $W \in \mathbb {R}^{d \times d}$ are shared among different blocks, and are trained together with the key vectors.
The addition of the $s_t^T q$ term in the gating function is our main contribution. We add such term with the assumption that the question can be useful to focus the attention of the model while analyzing the input sentences.
The Output Module creates a probability distribution over the memories' hidden states using the question $q$ . Thus, the hidden states are summed up, using the probability as weight, to obtain a single state representing all the input. Finally, the network output is obtained by combining the final state with the question. Let us define $R \in \mathbb {R}^{|V|\times d }$ , $H \in \mathbb {R}^{d \times d}$ , $\hat{y} \in \mathbb {R}^{|V|}$ , $z$ is the number of blocks, and $\phi $ can be chosen among different activation functions. Then, we have: $ p_i =& \text{Softmax}(q^T h_i)\\ u =& \sum _{j=1}^{z} p_j h_j \\ \hat{y} =& R \phi (q + H u) $
The model is trained using a cross entropy loss $H(\hat{y}, y)$ plus L2 regularisation term, where $y$ is the one hot encoding of the correct answer. The sigmoid function and the L2 term are two novelty added to the original REN. Overall, the trainable parameters are:
$$\Theta = [E,f^{(s)},f^{(q)}, U, V, W, k_1,\dots ,k_z, R, H ] \nonumber $$   (Eq. 6)
where $f^{(s)}$ refers to the sentence multiplicative masks, $f^{(q)}$ to the question multiplicative masks, and each $k_i$ to the key of a generic block $i$ . The number of parameters is dominated by $E$ and $R$ , since they depend on the vocabulary size. However, $R$ is normally is much smaller than $E$ like in the CNN dataset, in which the prediction is made on a restricted number of entities. All the parameters are learned using the Backpropagation Through Time (BPTT) algorithm. A schematic representation of the model is shown in Figure 1 .
Experiments and Results
Our model has been implemented using TensorFlow v1.1 BIBREF21 and the experiments have been run on a Linux server with 4 Nvidia P100 GPUs. As mentioned earlier, we tested our model in two datasets: the bAbI 1k sample and the CNN news articles. The first dataset have 20 separate tasks, each of which has 900/100/1000 training, validation, and test samples. Instead, the second one has 380298/3924/3198 training, validation and test samples. We kept the original splitting to compare our results with the existing ones.
Analysis
To better understand how our proposed model (i.e. QDREN) works and how it improves the accuracy of the existing REN, we studied the gating function behavior. Indeed, the output of this function decides how much and what we store in each memory cell, and it is also where our model differs from the original one. Moreover, we trained the QDREN and the original REN using the bAbI task number 1 (using 20 memory blocks). We pick up this task since both models pass it, and it is one of the simplest, which also allows to better understand and visualize the results. Indeed, we have tried to visualize other tasks but the result was difficult to understand since there were too many sentences in input and it was difficult to understand how the gate opened. The visualization result is shown in Figure 2 , where we plotted the activation matrix for both models, using a sample of the validation set. In these plots, we can notice how the two models learn which information to store.
In Figure 2 (a), we notice that the QDREN is opening the gates just when in the sentence appears the entity named Mary. This because such entity is also present in the question (i.e., "where is Mary?"). Even though the model is focusing on the right entity, its gates are opening all at the same times. In fact, we guess that a sparser activation would be better since it may have modeled which other entities are relevant for the final answer. Instead, the gaiting activation of the original REN is sparse, which is good if we would like to learn all the relevant facts in the text. Indeed, the model effectively assigns a block to each entity and it opens the gates just ones such entity appears in the input sentences. For example, in Figure 2 (b) the block cell number 13 supposedly represents the entity Sandra, since each sentence in which this name appears the gate function of the block fully opens (value almost 1). Futher, we can notice the same phenomenon with the entity John (cell 10), Daniel (cell 4), and Mary (cell 14). Other entities (e.g., kitchen, bathroom, etc.) are more difficult to recognize in the plot since their activation is less strong and probably distributes this information among blocks.
Conclusion
In this paper we presented the Question Dependent Recurrent Entity Network, used for reasoning and reading comprehension tasks. This model uses a particular RNN cell in order to store just relevant information about the given question. In this way, in combination with the original Recurrent Entity Network (keys and memory), we improved the State-of-The-Art in the bAbI 1k task and achieved promising results in the Reading comprehension task on the CNN & Daily news dataset. However, we believe that there are still margins for improving the behavior for the proposed cell. Indeed, the cell has not enough expressive power to make a selective activation among different memory blocks (notice in Figure 2 (a) the gates open for all the memories). This does not seem to be a serious problem since we actually outperform other models, but it could be the key to finally pass all the bAbI tasks.
Acknowledgments
This work has been supported in part by grant no. GA_2016_009 "Grandi Attrezzature 2016" by the University of Pisa.

Question: How does the model recognize entities and their relation to answers at inference time when answers are not accessible?
Answer: gating function, Dynamic Memory