Improving Slot Filling by Utilizing Contextual Information	Slot Filling is the task of extracting the semantic concept from a given natural language utterance. Recently it has been shown that using contextual information, either in work representations (e.g., BERT embedding) or in the computation graph of the model, could improve the performance of the model. However, recent work uses the contextual information in a restricted manner, e.g., by concatenating the word representation and its context feature vector, limiting the model from learning any direct association between the context and the label of word. We introduce a new deep model utilizing the contextual information for each work in the given sentence in a multi-task setting. Our model enforce consistency between the feature vectors of the context and the word while increasing the expressiveness of the context about the label of the word. Our empirical analysis on a slot filling dataset proves the superiority of the model over the baselines.

Conclusion & Future Work	In this work we introduce a new deep model for the task of Slot Filling. In a multi-task setting, our model increase the mutual information between word representations and its context, improve the label information in the context and predict which concepts are expressed in the given sentence. Our experiments on an image edit request corpus shows that our model achieves state-of-the-art results on this dataset.

However, the simple concatenation of the global context and the word embeddings would not separately ensure these two benefits of the global context. In order to address this problem, we introduce a multi-task setting to separately ensure the aforementioned benefits of utilizing contextual information. In particular, to ensure the better contextualized representations of the words, the model is encourage to learn representations for the word which are consistent with its context. This is achieved via increasing the mutual information between the word representation and its context. To ensure the usefulness of the contextual information for the final task, we introduce two novel sub-tasks. The first one aims to employ the context of the word instead of the word representation to predict the label of the word. In the second sub-task, we use the global representation of the sentence to predict which labels exist in the given sentence in a multi-label classification setting. These two sub-tasks would encourage the contextual representations to be informative for both word level classification and sentence level classification.

Model	Our model is trained in a multi-task setting in which the main task is slot filling to identify the best possible sequence of labels for the given sentence. In the first auxiliary task we aim to increase consistency between the word representation and its context. The second auxiliary task is to enhance task specific information in contextual information. In this section, we explain each of these tasks in more details.

Model ::: Prediction by Contextual Information ::: Predicting Sentence Labels	The word label prediction enforces the context of each word to contain information about its label but it would not ensure the contextual information to capture the sentence level patterns for expressing intent. In other words, the word level prediction lacks a general view about the entire sentence. In order to increase the general information about the sentence in the representation of the words, we aim to predict the labels existing in a sentence from the representations of its words. More specifically, we introduce a new sub-task to predict which labels exit in the given sentence (Note that sentences might have only a subset of the labels; e.g. only action and object). We formulate this task as a multi-class classification problem. Formally, given the sentence $X=x_1,x_2,...,x_N$ and label set $S=\lbrace action, attribute, object, value\rbrace $ our goal is to predict the vector $L^s=l^s_1,l^s_2,...,l^s_{|S|}$ where $l^s_i$ is one if the sentence $X$ contains $i$th label from the label set $S$ otherwise it is zero.

Introduction	Slot Filling (SF) is the task of identifying the semantic concept expressed in natural language utterance. For instance, consider a request to edit an image expressed in natural language: “Remove the blue ball on the table and change the color of the wall to brown”. Here, the user asks for an "Action" (i.e., removing) on one “Object” (blue ball on the table) in the image and changing an “Attribute” (i.e., color) of the image to new “Value” (i.e., brown). Our goal in SF is to provide a sequence of labels for the given sentence to identify the semantic concept expressed in the given sentence. Prior work have shown that contextual information could be useful for SF. They utilize contextual information either in word level representation (i.e., via contextualize embedding e.g., BERT ) or in the model computation graph (e.g., concatenating the context feature to the word feature ). However, such methods fail to capture the explicit dependence between the context of the word and its label.

Model ::: Consistency with Contextual Representation	In this sub-task we aim to increase the consistency of the word representation and its context. To obtain the context of each word we perform max pooling over the all words of the sentence excluding the word itself: where $h_i$ is the representation of the $i$th word from the Bi-LSTM. We aim to increase the consistency between vectors $h_i$ and $h^c_i$. One way to achieve this is by decreasing the distance between these two vectors. However, directly enforcing the word representation and its context to be close to each other would not be efficient as in long sentences the context might substantially differs from the word. So in order to make enough room for the model to represent the context of each word while it is consistent with the word representation, we employ an indirect method. We propose to maximize the mutual information (MI) between the word representation and its context in the loss function. In information theory, MI evaluates how much information we know about one random variable if the value of another variable is revealed.

Related Work	The task of Slot Filling is formulated as a sequence labeling problem. Deep learning has been extensively employed for this task (, , , , , , , , , ). The prior work has mainly utilized the recurrent neural network as the encoder to extract features per word and Conditional Random Field (CRF)  as the decoder to generate the labels per word. Recently the work  shows that the global context of the sentence could be useful to enhance the performance of neural sequence labeling. In their approach, they use a separate sequential model to extract word features. Afterwards, using max pooling over the representations of the words, they obtain the sentence representations and concatenate it to the word embedding as the input to the main task encoder (i.e. the RNN model to perform sequence labeling). The benefit of using the global context along the word representation is 2-fold: 1) it enhance the representations of the word by the semantics of the entire sentence thus the word representation are more contextualized 2) The global view of the sentence would increase the model performance as it contains information about the entire sentence and this information might not be encoded in word representations due to long decencies.

Model ::: Prediction by Contextual Information	In addition to increasing consistency between the word representation and its context representation, we aim to increase the task specific information in contextual representations. This is desirable as the main task is utilizing the word representation to predict its label. Since our model enforce the consistency between the word representation and its context, increasing the task specific information in contextual representations would help the model's final performance. In order to increase task-specific information in contextual representation, we train the model on two auxiliary tasks. The first one aims to use the context of each word to predict the label of that word and the goal of the second auxiliary task is to use the global context information to predict sentence level labels. We describe each of these tasks in more details in the following sections.

Model ::: Slot Filling	The input to the model is a sequence of words $x_1,x_2,...,x_N$. The goal is to assign each word one of the labels action, object, attribute, value or other. Following other methods for sequence labelling, we use the BIO encoding schema. In addition to the sequence of words, the part-of-speech (POS) tags and the dependency parse tree of the input are given to the model. The input word $x_i$ is represented by the concatenation of its pre-trained word embedding and its POS tag embedding, denoted by $e_i$. These representations are further abstracted using a 2-layer Bi-Directional Long Short-Term Memory (LSTM) to obtain feature vector $h_i$. We use the dependency tree of the sentence to utilize the syntactical information about the input text. This information could be useful to identify the important words and their dependents in the sentence. In order to model the syntactic tree, we utilize Graph Convolutional Network (GCN)  over the dependency tree. This model learns the contextualized representations of the words such that the representation of each word is contextualized by its neighbors.

Experiments	In our experiments, we use Onsei Intent Slot dataset. Table TABREF21 shows the statics of this dataset. We use the following hyper parameters in our model: We set the word embedding and POS embedding to 768 and 30 respectively; The pre-trained BERT  embedding are used to initialize word embeddings; The hidden dimension of the Bi-LSTM, GCN and feed forward networks are 200; the hyper parameters $\alpha $, $\beta $ and $\gamma $ are all set to 0.1; We use Adam optimizer with learning rate 0.003 to train the model. We use micro-averaged F1 score on all labels as the evaluation metric. We compare our method with the models trained using Adobe internal NLU tool, Pytext  and Rasa  NLU tools. Table TABREF22 shows the results on Test set. Our model improves the F1 score by almost 2%, which corresponds to a 12.3% error rate reduction. This improvements proves the effectiveness of using contextual information for the task of slot filling.

First, we find representation of the sentence from the word representations. To this end, we use max pooling over all words of the sentence to obtain vector $H$: Afterwards, the vector $H$ is further abstracted by a 2-layer feed forward neural net with a sigmoid function at the end: where $W_2$ and $W_1$ are trainable parameters. Note that since this tasks is a multi-class classification the number of neurons at the final layer is equal to $|S|$. We optimize the following binary cross entropy loss function: where $l_k$ is one if the sentence contains the $k$th label otherwise it is zero. Finally, to train the model we optimize the following loss function: where $\alpha $, $\beta $ and $\gamma $ are hyper parameters to be tuned using development set performance.

Moreover, such limited use of contextual information (i.e., concatenation of the feature vector and context vector) in the model cannot model the interaction between the word representation and its context. In order to alleviate these issues, in this work, we propose a novel model to explicitly increase the predictability of the word label using its context and increasing the interactivity between word representations and its context. More specifically, in our model we use the context of the word to predict its label and by doing so our model learns label-aware context for each word in the sentence. In order to improve the interactivity between the word representation and its context, we increase the mutual information between the word representations and its context. In addition to these contributions, we also propose an auxiliary task to predict which labels are expressed in a given sentence. Our model is trained in a mutli-tasking framework. Our experiments on a SF dataset for identifying semantic concepts from natural language request to edit an image show the superiority of our model compared to previous baselines. Our model achieves the state-of-the-art results on the benchmark dataset by improving the F1 score by almost 2%, which corresponds to a 12.3% error rate reduction.

Afterwards, we concatenate the structural features $\hat{h}_i$ and sequential features $h_i$ to represent $i$th word by feature vector $h^{\prime }_i$: Finally in order to label each word in the sentence we employ a task specific 2-layer feed forward neural net followed by a logistic regression model to generate class scores $S_i$ for each word: where $W_{LR}, W_1$ and $W_2$ are trainable parameters and $S_i$ is a vector of size number of classes in which each dimension of it is the score for the corresponding class. Since the main task is sequence labeling we exploit Conditional Random Field (CRF) as the final layer to predict the sequence of labels for the given sentence. More specifically, class scores $S_i$ are fed into the CRF layer as emission scores to obtain the final labeling score: where $T$ is the trainable transition matrix and $\theta $ is the parameters of the model to generate emission scores $S_i$. Viterbi loss $L_{VB}$ is used as the final loss function to be optimized during training.

Model ::: Prediction by Contextual Information ::: Predicting Word Label	In this sub-task we use the context representations of each word to predict its label. It will increase the information encoded in the context of the word about the label of the word. We use the same context vector $h^c_i$ for the $i$th word as described in the previous section. This vector is fed into a 2-layer feed forward neural network with a softmax layer at the end to output the probabilities for each class: Where $W_2$ and $W_1$ are trainable parameters. Biases are omitted for brevity. Finally we use the following cross-entropy loss function to be optimized during training: where $N$ is the length of the sentence and $l_i$ is the label of the $i$th word.

In order to analyze the contribution of the proposed sub-tasks we also evaluate the model when we remove one of the sub-task and retrain the model. The results are reported in Table TABREF23. This table shows that all sub-tasks are required for the model to have its best performance. Among all sub-tasks the word level prediction using the contextual information has the major contribution to the model performance. This fact shows that contextual information trained to be informative about the final sub-task is necessary to obtain the representations which could boost the final model performance.

In this work, we propose to address this issue by employing the mutual information neural estimation (MINE) in  that seeks to estimate the lower bound of the mutual information between the high dimensional vectors via adversarial training. To this goal, MINE attempts to compute the lower bound of the KL divergence between the joint and marginal distributions of the given high dimensional vectors/variables. In particular, MINE computes the lower bound of the Donsker-Varadhan representation of KL-Divergence: However, recently, it has been shown that other divergence metrics (i.e., the Jensen-Shannon divergence) could also be used for this purpose , , offering simpler methods to compute the lower bound for the MI. Consequently, following such methods, we apply the adversarial approach to obtain the MI lower bound via the binary cross entropy of a variable discriminator. This discriminator differentiates the variables that are sampled from the joint distribution from those that are sampled from product of the marginal distributions. In our case, the two variables are the word representation $h_i$ and context representation $h^c_i$.

In order to sample from joint distributions, we simply concatenate $h_i$ and $h^c_i$ (i.e., the positive example). To sample from the product of the marginal distributions, we concatenate the representation $h_i$ with $h^c_j$ where $i\ne j$ (i.e., the negative example). These samples are fed into a 2-layer feed forward neural network $D$ (i.e., the discriminator) to perform a binary classification (i.e., coming from the joint distribution or the product of the marginal distributions). Finally, we use the following binary cross entropy loss to estimate the mutual information between $h_i$ and $h^c_i$ to add into the overall loss function: where $N$ is the length of the sentence and $[h,h^c_i]$ is the concatenation of the two vectors $h$ and $h^c_i$. This loss is added to the final loss function of the model.

Formally, the mutual information between two random variable $X_1$ and $X_2$ is obtained by: Using this definition of MI, we can reformulate the MI equation as KL Divergence between the joint distribution $P_{X_1X_2}=P(X_1,X_2)$ and the product of marginal distributions $P_{X_1\bigotimes X_2}=P(X_1)P(X_2)$: Based on this understanding of MI, we can see that if the two random variables are dependent then the mutual information between them (i.e. the KL-Divergence in equation DISPLAY_FORM9) would be the highest. Consequently, if the representations $h_i$ and $h^c_i$ are encouraged to have large mutual information, we expect them to share more information. The mutual information would be introduced directly into the loss function for optimization. One issue with this approach is that the computation of the MI for such high dimensional continuous vectors as $h_i$ and $h^c_i$ is prohibitively expensive.

In the inference time, the Viterbi decoder is employed to find the sequence of labels with highest score.

We employ 2-layer GCN with $h_i$ as the initial representation for the node (i.e., word) $i$th. The representations of the $i$th node is an aggregation of the representations of its neighbors. Formally the hidden representations of the $i$th word in $l$th layer of GCN is obtained by: where $N(i)$ is the neighbors of the $i$th word in the dependency tree, $W_l$ is the weight matrix in $l$th layer and $deg(i)$ is the degree of the $i$th word in the dependency tree. The biases are omitted for brevity. The final representations of the GCN for $i$th word, $\hat{h}_i$, represent the structural features for that word.

Question: How does their model utilize contextual information for each work in the given sentence in a multi-task setting? setting?
Answer: we use the context of the word to predict its label and by doing so our model learns label-aware context for each word in the sentence