One weakness of self-attention model it that they cannot encode the position information efficiently. Some methods have been proposed to encode the relative or absolute position of tokens in the sequence as the additional input to the model.  used sine and cosine functions of different frequencies and added positional encodings to the input embeddings together. It used absolute position embedding to capture relative positional relation by the characteristic of sine and cosine functions. Moreover, several studies show that explicitly modeling relative position can further improve performance. For example,  proposed relative position encoding to explicitly model relative position by independent semantic parameter. It demonstrated significant improvements even when entirely replacing conventional absolute position encodings.  proposed to model localness for the self-attention network by a learnable Gaussian bias which enhanced the ability to model local relationship and demonstrated the effectiveness on the translation task. In our study, we design a local contextual attention model, which incorporates relative position information by a learnable Gaussian bias into original attention distribution. Different from , in our method, the distribution center is regulated around the corresponding utterance with a window, which indicates the context dependency preference, for capturing more local contextual dependency.

Local Contextual Attention with Hierarchical Structure for Dialogue Act Recognition	Dialogue act recognition is a fundamental task for an intelligent dialogue system. Previous work models the whole dialog to predict dialog acts, which may bring the noise from unrelated sentences. In this work, we design a hierarchical model based on self-attention to capture intra-sentence and inter-sentence information. We revise the attention distribution to focus on the local and contextual semantic information by incorporating the relative position information between utterances. Based on the found that the length of dialog affects the performance, we introduce a new dialog segmentation mechanism to analyze the effect of dialog length and context padding length under online and offline settings. The experiment shows that our method achieves promising performance on two datasets: Switchboard Dialogue Act and DailyDialog with the accuracy of 80.34\% and 85.81\% respectively. Visualization of the attention weights shows that our method can learn the context dependency between utterances explicitly.

By modeling context information, the performance of the hierarchical model is improved by at least 3%, even compared to BERT. In order to better analyze the semantic dependency learned by attention, in our experiments, we removed the CRF module. In terms of different hierarchical models, our LSTM+BLSTM achieved good result. The accuracy was 80.00% which is even a little better than Hierarchical BLSTM-CRF . Relying on attention mechanism and local contextual modeling, our model, LSTM+Attention and LSTM+Local Contextual Attention, achieved 80.12% and 80.34% accuracy respectively. Compared with the previous best approach Hierarchical BLSTM-CRF, we can obtain a relative accuracy gain with 1.1% by our best model. It indicated that self-attention model can capture context dependency better than the BLSTM model. With adding the local constraint, we can get an even better result. To further illustrate the effect of the context length, we also performed experiments with different sliding window $W$ and context padding $P$. Table TABREF22 shows the result.

Experiments ::: Results on SwDA	In this section, we evaluate the proposed approaches on SwDA dataset. Table TABREF20 shows our experimental results and the previous ones on SwDA dataset. It is worth noting that  combined GloVe and pre-trained ELMo representations as word embeddings. However, in our work, we only applied the pre-trained word embedding. To illustrate the importance of context information, we also evaluate several sentence classification methods (CNN, LSTM, BERT) as baselines. For baseline models, both CNN and LSTM, got similar accuracy (75.27% and 75.59% respectively). We also fine-tuned BERT  to do recognition based on single utterance. As seen, with the powerful unsupervised pre-trained language model, BERT (76.88% accuracy) outperformed LSTM and CNN models for single sentence classification. However, it was still much lower than the models based on context information. It indicates that context information is crucial in the DA recognition task. BERT can boost performance in a large margin. However, it costs too much time and resources. In this reason, we chose LSTM as our utterance encoder in further experiment.

In our method, for the given utterance $u_{i}$, the distribution center $c_{i}$ is regulated for capturing the not only local but also contextual dependency, which can be formally expressed as: $c_{i} \in (i-C,i+C)$. However, in their work, the distribution center can be anywhere in the sequence, and it is designed for capturing the phrasal patterns, which are essential for Neural Machine Translation task.

Conclusions and Future Work	In the paper, we propose our hierarchical model with local contextual attention to the Dialogue Act Recognition task. Our model can explicitly capture the semantic dependencies between utterances inside the dialogue. To enhance our model with local contextual information, we revise the attention distribution by a learnable Gaussian bias to make it focus on the local neighbors. Based on our dialog segmentation mechanism, we find that local contextual attention reduces the noises through relative position information, which is essential for dialogue act recognition. And this segmentation mechanism can be applied under online and offline settings. Our model achieves promising performance in two well-known datasets, which shows that modeling local contextual information is crucial for dialogue act recognition. There is a close relation between dialogue act recognition and discourse parsing . The most discourse parsing process is composed of two stages: structure construction and dependency labeling , . For future work, a promising direction is to apply our method to multi-task training with two stages jointly. Incorporating supervised information from dependency between utterances may enhance the self-attention and further improve the accuracy of dialogue act recognition.

In this example, utterance “Okay.” corresponds to two different DA labels within different semantic context. Many approaches have been proposed for DA recognition. Previous work relies heavily on handcrafted features which are domain-specific and difficult to scale up , , . Recently, with great ability to do feature extraction, deep learning has yielded state-of-the-art results for many NLP tasks, and also makes impressive advances in DA recognition. ,  built hierarchical CNN/RNN models to encode sentence and incorporate context information for DA recognition.  achieved promising performance by adding the CRF to enhance the dependency between labels.  applied the self-attention mechanism coupled with a hierarchical recurrent neural network. However, previous approaches cannot make full use of the relative position relationship between utterances. It is natural that utterances in the local context always have strong dependencies in our daily dialog. In this paper, we propose a hierarchical model based on self-attention  and revise the attention distribution to focus on a local and contextual semantic information by a learnable Gaussian bias which represents the relative position information between utterances, inspired by .

Further, to analyze the effect of dialog length quantitatively, we introduce a new dialog segmentation mechanism for the DA task and evaluate the performance of different dialogue length and context padding length under online and offline settings. Experiment and visualization show that our method can learn the local contextual dependency between utterances explicitly and achieve promising performance in two well-known datasets. The contributions of this paper are: We design a hierarchical model based on self-attention and revise the attention distribution to focus on a local and contextual semantic information by the relative position information between utterances. We introduce a new dialog segmentation mechaism for the DA task and analyze the effect of dialog length and context padding length. In addition to traditional offline prediction, we also analyze the accuracy and time complexity under the online setting.

It covers various topics about our daily life. Each utterance in the conversation is manually labeled as one out of 4 dialogue act classes. Table TABREF18 presents the statistics for both datasets. In our preprocessing, the text was lowercased before tokenized, and then sentences were tokenized by WordPiece tokenizer  with a 30,000 token vocabulary to alleviate the Out-of-Vocabulary problem. [1]The author claimed that they achieved 78.7%(81.3%) accuracy with pre-trained word embedding (fine-grained embedding). For a fair comparison, both previous and our work is simply based on pre-trained word embedding. [2]The author randomly selected two test sets which are different from previous and our work and achieved 77.15% and 79.74%, and we reimplemented in standard test sets.

applied the context-aware self-attention mechanism coupled with a hierarchical recurrent neural network and got a significant improvement over state-of-the-art results on SwDA datasets. On another aspect,  combined a recurrent neural network language model with a latent variable model over DAs.  proposed a Discrete Information Variational Autoencoders (DI-VAE) model to learn discrete latent actions to incorporate sentence-level distributional semantics for dialogue generation.

Methodology ::: Online and Offline Predictions	Previous work mainly focuses on the offline setting where we can access the whole utterances in the dialogue and predict all the DA labels simultaneously. However, the online setting is the natural demand in our real-time applications. For the online setting, we only care about the recognition result of the last utterance in the given context, as seen in the area with the red dashed line in Figure FIGREF6, our model is well compatible with online setting, we can calculate the attention between the last utterance and the other utterances directly where $K \in \mathbb {R}^{1\times d}, Q \in \mathbb {R}^{n\times d}, V \in \mathbb {R}^{n\times d}$. For LSTM, we still have to model the entire sequence, which is slower than attention based models. Table TABREF17 shows the time complexity comparison excluding the time cost of first layer encoding, and the dialogue length $n$ is smaller than the representation dimension $d$. Our model is easy to expand into the online setting, however, to have a fair comparison with previous work, in our experiments, we applied the models under the offline setting by default.

Experiments ::: Datasets	We evaluate the performance of our model on two high-quality datasets: Switchboard Dialogue Act Corpus (SwDA)  and DailyDialog . SwDA has been widely used in previous work for the DA recognition task. It is annotated on 1155 human to human telephonic conversations about the given topic. Each utterance in the conversation is manually labeled as one of 42 dialogue acts according to SWBD-DAMSL taxonomy . In , they used 43 categories of dialogue acts, which is different from us and previous work. The difference in the number of labels is mainly due to the special label “+”, which represents that the utterance is interrupted by the other speaker (and thus split into two or more parts). We used the same processing with , which concatenated the parts of an interrupted utterance together, giving the result the tag of the first part and putting it in its place in the conversation sequence. It is critical for fair comparison because there are nearly 8% data has the label “+”. Lacking standard splits, we followed the training/validation/test splits by . DailyDialog dataset contains 13118 multi-turn dialogues, which mainly reflect our daily communication style.

Methodology	Before we describe the proposed model in detail, we first define the mathematical notation for the DA recognition task in this paper. Given the dataset, $X = (D_1,D_2,... D_L)$ with corresponding DA labels $(Y_1,Y_2,...Y_L)$. Each dialogue is a sequence of $ N_l $ utterances $ D_l = (u_1,u_2,...u_{N_l})$ with $ Y_l = (y_1,y_2,...y_{N_l}) $. Each utterance is padded or truncated to the length of $ M $ words, $u_j = (w_1,w_2,...w_{M})$. Figure FIGREF6 shows our overall model structure. For the first layer, we encode each utterance $u_j$ into a vector representation. Each word $w_m$ of the utterance $u_j$ is converted into dense vector representations $e_m$ from one-hot token representation. And then, we apply LSTM , a powerful and effective structure for sequence modeling, to encode the word sequence.

Background ::: Related Work	DA recognition is aimed to assign a label to each utterance in a conversation. It can be formulated as a supervised classification problem. There are two trends to solve this problem: 1) as a sequence labeling problem, it will predict the labels for all utterances in the whole dialogue history , , ; 2) as a sentence classification problem, it will treat utterance independently without any context history , . Early studies rely heavily on handcrafted features such as lexical, syntactic, contextual, prosodic and speaker information and achieve good results , , . Recent studies have applied deep learning based model for DA recognition.  proposed a model based on RNNs and CNNs that incorporates preceding short texts to classify current DAs. ,  used hierarchical CNN and RNN to model the utterance sequence in the conversation, which can extract high-level sentence information to predict its label. They found that there is a small performance difference among different hierarchical CNN and RNN approaches.  added a CRF layer on the top of the hierarchical network to model the label transition dependency.

Formally, for the utterance $u_j$: where $embed$ represents the embedding layer which can be initialized by pre-trained embeddings. To make a fair comparison with previous work, we do not use the fine-grained embedding presented in . LSTM helps us get the context-aware sentence representation for the input sequence. There are several approaches to represent the sentence from the words. Following , we add a max-pooling layer after LSTM, which selects the maximum value in each dimension from the hidden units. In our experiment, LSTM with max-pooling does perform a little better than LSTM with last-pooling, which is used in . Afterwards, we get the utterances vector representations $ u = (u_1,...,u_{N_l}) $ of $N_l$ elements for the dialogue $D_l$ where $ u_j \in \mathbb {R}^{d_s}, d_s$ is the dimension of hidden units. As we discussed in section SECREF7, given the sequence $ s \in \mathbb {R}^{N_l*d_s}$, self-attention mechanism calculates the attention weights between each pair of utterances in the sequence and get the weighted sum as output.

It is worth noting that it is actually the same as single sentence classification when $P = 0$ (without any context provided). First, we set $W$ to 1 to discuss how the length of context padding will affect. As seen in the result, the accuracy increased when more context padding was used for both LSTM+BLSTM and LSTM+Attention approaches, so we did not evaluate the performance of LSTM+LC Attention when context padding is small. There was no further accuracy improvement when the length of context padding was beyond 5. Therefore, we fixed the context padding length $P$ to 5 and increased the size of the sliding window to see how it works. With sliding window size increasing, the more context was involved together with more unnecessary information. From the experiments, we can see that both LSTM+BLSTM and LSTM+Attention achieved the best performance when window size was 1 and context padding length was 5. When window size increased, the performances of these two models dropped.

Experiments ::: Visualization	In this section, we visualize the attention weights for analyzing how local contextual attention works in detail. Figure FIGREF24 shows the visualization of original attention and local contextual attention for the example dialogue shown in Table TABREF1. The attention matrix $M$ explicitly measures the dependency among utterances. Each row of grids is normalized by $softmax$, $M_{ij}$ represents for the dependency score between the utterance i and utterance j. As demonstrated in Figure FIGREF24, there are some wrong and uninterpretable attention weights annotated with red color, which is learned by the original attention. The original attention model gives the utterance “B: Hi” (position 0) and “A: Okay.” (position 7) a high dependency score. However, local contextual attention weakens its attention weights due to the long distance apart. Overall, the additional Gaussian bias trend to centralize the attention distribution to the diagonal of the matrix, which is in line with our linguistic intuition that utterances that are far apart usually don't have too strong dependencies.

Relying on modeling local contextual information, LSTM+LC Attention achieved the best accuracy (85.81%) when the window size was 5. For the longer sliding window, the performance of LSTM+LC Attention was still better and more robust than the other two models. For online prediction, we added 2 preceding utterances as context padding, and the experiment shows that LSTM+LC Attention outperformed the other two models under the online setting, although the performances of these three models dropped without subsequent utterances.

Experiments ::: Result on DailyDialog	The classification accuracy of DailyDialog dataset is summarized in Table TABREF23. As for sentence classification without context information, the fine-tuned BERT still outperformed LSTM and CNN based models. From table TABREF18 we can see that, the average dialogue length $|U|$ in DailyDialog is much shorter than the average length of SwDA. So, in our experiment, we set the maximum of the $W$ to 10, which almost covers the whole utterances in the dialogue. Using the same way as SwDA dataset, we, first, set W to 1 and increased the length of context padding. As seen, modeling local context information, hierarchical models yielded significant improvement than sentence classification. There was no further accuracy improvement when the length of context padding was beyond 2, so we fixed the context padding length P to 2 and increased the size of sliding window size W. From the experiments, we can see that LSTM+Attention always got a little better accuracy than LSTM+BLSTM. With window size increasing, the performances of these two models dropped.

However, our model (LSTM+LC Attention) can leverage the context information more efficiently, which achieved the best performance when window size was 10, and the model was more stable and robust to the different setting of window size. For online prediction, we only care about the recognition result of the last utterance in the given context. We added 5 preceding utterances as context padding for every predicted utterance because we cannot access subsequent utterances in the online setting. As seen in Table TABREF22, without subsequent utterances, the performances of these three models dropped. However, LSTM+LC Attention still outperformed the other two models.

Background ::: Self-Attention	Self-attention  achieves great success for its efficiently parallel computation and long-range dependency modeling. Given the input sequence $ s = \left( s_1,...,s_n \right) $ of n elements where $ s_i \in \mathbb {R}^{d_s} $. Each attention head holds three parameter matrices, $W_h^Q, W_h^K, W_h^V \in {\mathbb {R}}^{d_s \times d_z} $ where $ h $ present the index of head. For the head $h$, linear projection is applied to the sequence $s$ to obtain key (K), query (Q), and value (V) representations. the attention module gets the weight by computing dot-products between key/query pair and then $softmax$ normalizes the result. it is defined as: where $\sqrt{d_z}$ is the scaling factor to counteract this effect that the dot products may grow large in magnitude. For all the heads, where $W^O \in \mathbb {R}^{(d_z*h)\times d_s}$ is the output projection.

The $W$ and $P$ are both hyperparameters; in the experiment SECREF21, we will talk about the effect of the window size and the context padding length.

For example, the dialog $D$ is a sequence of utterances with length $n$, and we will get $\lceil x/w \rceil $ sub-dialogues, for the k-th sub-dialogues, the utterances sequence is $(u_{(k-1)*W+1},u_{(k-1)*W+2},...,u_{k*W})$. In order to avoid losing some context information caused by being separated, which will affect the context modeling for the utterances in the begin and end of the sub-dialog, we add the corresponding context with $P$ (stands for context padding) utterances at the begin and the end of each sliding window, so for the k-th sub-dialogues, the revised utterances sequence is $(u_{(k-1)*W-P+1},u_{(k-1)*W-P+2},...,u_{k*W+P})$. Moreover, we mask the loss for the context padding utterances, which can be formally expressed as: $M(i)=0$ if utterance $i$ is in the context padding otherwise 1, $L$ is the cross entropy.

The attention module explicitly models the context dependency between utterances. We employ a residual connection  around the attention module, which represents the dependency encoder between utterances, and the current utterance encoder $s$: Finally, we apply a two-layer fully connected network with a Rectified Linear Unit (ReLU) to get the final classification output for each utterance.

Methodology ::: Modeling Local Contextual Attention	The attention explicitly models the interaction between the utterances. However, for context modeling, original attention mechanism always considers all of the utterances in a dialogue which inhibits the relation among the local context and is prone to overfitting during training. It is natural that utterances in the local context always have strong dependencies in our daily dialog. Therefore, we add a learnable Gaussian bias with the local constraint to the weight normalized by $softmax$ to enhance the interaction between concerned utterances and its neighbors. The attention module formula is revised as: The first term is the original dot product self-attention model. $POS \in \mathbb {R}^{N\times N}$ is the bias matrix, where N is the length of dialogue. The element $POS_{i,j}$ is defined following by gaussian distribution: $POS_{i,j}$ measures the dependency between the utterance $u_j$ and the utterance $u_i$ in terms of the relative position prior. $w_{i}$ represents for the standard deviation, which controls the weight decaying.

Because of local constraint, $|c_{i} - i| <= C$, for each utterance $u_i$, the predicted center position $c_{i}$ and window size $ w_{i}$ is defined as followed: where $W_i^c,W_i^d \in \mathbb {R}^{1*N}$ are both learnable parameters. We initialized the parameter $W_i^c$ to 0, which leads to center position $ c_i = i $ by default. Furthermore, $c_{i}$ and $w_{i}$ are both related to the semantic context of the utterances, so we assign the mean of key $\overline{K}$ in attention mechanism to represent the context information. Moreover, the central position also indicates the dependency preference of the preceding utterances or subsequent utterances. It is worth noting that there is a little difference with , although we both revise the attention module by the Gaussian distribution.

As demonstrated in Figure FIGREF24, benefiting of the additional Gaussian bias, the revised attention mechanism weakens the attention weights between utterances which cross the long relative distance. For the grids near diagonal, it strengthens their dependency score and doesn't bring other useless dependencies for its learnable magnitude.

Methodology ::: Separate into Sub-dialogues	The length of different dialogues in the dataset varies a lot. It is worth noting that the length of dialog affects the model prediction. On the one hand, under the offline setting, we can access the whole utterances in the dialogue and predict all the DA labels simultaneously, so the more utterances, the more efficient. However, on the other hand, if we put too many utterances in once prediction, it will model too much unrelated dependency in the long utterances sequence for both LSTM and attention mechanism based model. The sub-dialogues with the same length also enable efficiently batch training. To study how the dialogue length and context padding length will affect the performance, so we defined a sliding window $W$ which is the sub-dialogue length. Then, we separate each long dialogue into several small sub-dialogues.

Introduction	Dialogue act (DA) characterizes the type of a speaker's intention in the course of producing an utterance and is approximately equivalent to the illocutionary act of  or the speech act of . The recognition of DA is essential for modeling and automatically detecting discourse structure, especially in developing a human-machine dialogue system. It is natural to predict the Answer acts following an utterance of type Question, and then match the Question utterance to each QA-pair in the knowledge base. The predicted DA can also guide the response generation process . For instance, system generates a Greeting type response to former Greeting type utterance. Moreover, DA is beneficial to other online dialogue strategies, such as conflict avoidance . In the offline system, DA also plays a significant role in summarizing and analyzing the collected utterances. For instance, recognizing DAs of a wholly online service record between customer and agent is beneficial to mine QA-pairs, which are selected and clustered then to expand the knowledge base. DA recognition is challenging due to the same utterance may have a different meaning in a different context. Table TABREF1 shows an example of some utterances together with their DAs from Switchboard dataset.

Question: What previous methods is the proposed method compared against?
Answer: BLSTM+Attention+BLSTM
Hierarchical BLSTM-CRF
CRF-ASN
Hierarchical CNN (window 4)
mLSTM-RNN
DRLM-Conditional
LSTM-Softmax
RCNN
CNN
CRF
LSTM
BERT