The experiments reveal some interesting points: The results show that our proposed model CRF-ASN obviously outperforms the state-of-the-art baselines on both SwDA and MRDA datasets. Numerically, Our model improves the DAR accuracy over Bi-LSTM-CRF by 2.1% and 0.8% on SwDA and MRDA respectively. It is remarkable that our CRF-ASN method is nearly close to the human annotators' performance on SwDA, which is very convincing to prove the superiority of our model. The deep neural networks outperform the other feature-based models. We can see the last three non-deep models obtain worse performance than the top five deep-based methods. This suggests that the performance of dialogue act recognition can be improved significantly with discriminative deep neural networks, either in convolutional neural network or the recurrent neural network. Apart from deep learning tactics, the problem formulations are also critical to the DAR problem. We see structured prediction approaches eg. CRF-ASN, Bi-LSTM-CRF obtain better results than multi-classification eg. LSTM-Softmax.

Dialogue Act Recognition via CRF-Attentive Structured Network	Dialogue Act Recognition (DAR) is a challenging problem in dialogue interpretation, which aims to attach semantic labels to utterances and characterize the speaker's intention. Currently, many existing approaches formulate the DAR problem ranging from multi-classification to structured prediction, which suffer from handcrafted feature extensions and attentive contextual structural dependencies. In this paper, we consider the problem of DAR from the viewpoint of extending richer Conditional Random Field (CRF) structural dependencies without abandoning end-to-end training. We incorporate hierarchical semantic inference with memory mechanism on the utterance modeling. We then extend structured attention network to the linear-chain conditional random field layer which takes into account both contextual utterances and corresponding dialogue acts. The extensive experiments on two major benchmark datasets Switchboard Dialogue Act (SWDA) and Meeting Recorder Dialogue Act (MRDA) datasets show that our method achieves better performance than other state-of-the-art solutions to the problem. It is a remarkable fact that our method is nearly close to the human annotator's performance on SWDA within 2% gap.

Conclusion	In this paper, we formulate the problem of dialogue act recognition from the viewpoint of capturing hierarchical rich utterance representations and generalize richer CRF attentive graphical structural dependencies without abandoning end-to-end training. We propose the CRF-Attentive Structured Network (CRF-ASN) for the problem. We implement the model in two steps. We first encode the rich semantic representation on the utterance level by incorporating hierarchical granularity and memory enhanced inference mechanism. The learned utterance representation can capture long term dependencies across the conversation. We next adopt the internal structured attention network to compute the dialogue act influence and specify structural dependencies in a soft manner. This approach enable the soft-selection attention on the structural CRF dependencies and take account of the contextual influence on the nearing utterances. We demonstrate the efficacy of our method using the well-known public datasets SwDA and MRDA. The extensive experiments demonstrate that our model can achieve better performance than several state-of-the-art solutions to the problem.

For symmetry, we use the softmax in a general sense, i.e. INLINEFORM18 , where INLINEFORM19 is the implied recognition function. Here INLINEFORM20 comes from the former memory enhanced deep model over utterances INLINEFORM21 and corresponding dialogue acts INLINEFORM22 . The INLINEFORM0 INLINEFORM1 over the utterances and dialogue acts is defined as expectation: INLINEFORM2   where we assume the annotation function INLINEFORM0 factors into INLINEFORM1 . The annotation function is defined to simply return the selected hidden state. The INLINEFORM2 INLINEFORM3 can be interpreted as an dialogue act aware attentive conversation as taking the expectation of the annotation function with respect to a latent variable INLINEFORM4 , where INLINEFORM5 is parameterized to be function of utterances INLINEFORM6 and dialogue acts INLINEFORM7 . The expectation is a linear combination of the input representation and represents how much attention will be focused on each utterance according to the dialogue act sequence. We can model the structural dependencies distribution over the latent INLINEFORM0 with a linear chain CRF with n states: INLINEFORM1   where INLINEFORM0 is the pairwise potential for INLINEFORM1 and INLINEFORM2 .

Performance Comparisons	We compare our propose method with other several state-of-the-art methods for the problem of dialogue act recognition as follows: Bi-LSTM-CRF  method builds a hierarchical bidirectional LSTM as a base unit and the conditional random field as the top layer to do the dialogue act recognition task. DRLM-Conditional  method combines postive aspects of neural network architectures with probabilistic graphical models. The model combines a recurrent neural network language model with a latent variable model over shallow discourse structure. LSTM-Softmax  method applies a deep LSTM structure to classify dialogue acts via softmax operation. The authors claim that the word embeddings, dropout, weight decay and number of LSTM layers all have large effect on the final performance. RCNN  method composes both sentence model and discourse model to extend beyond the single sentence. The authors propose hierarchical CNN on sentence model and RNN on the contextual discourses. CNN  method incorporates the preceding short texts to classify dialogue act. The authors demonstrate that adding sequential information improves the quality of the predictions. HMM  method treats the discourse structure of a conversation as a hidden Markov model and the individual dialogue acts as observations emanating from the model states.

Experiments	In this section, we conduct several experiments on two public DA datasets SwDA and MRDA, and show the effectiveness of our approach CRF-ASN for dialogue act recognition.

Recently, approaches based on deep learning methods improved many state-of-the-art techniques in NLP including DAR accuracy on open-domain conversations      . Kalchbrenner et al.  used a mixture of CNN and RNN. CNNs were used to extract local features from each utterance and RNNs were used to create a general view of the whole dialogue. Khanpour et al.  design a deep neural network model that benefits from pre-trained word embeddings combined with a variation of the RNN structure for the DA classification task. Ji et al.  also investigated the performance of using standard RNN and CNN on DA classification and got the cutting edge results on the MRDA corpus using CNN. Lee et al.  proposes a model based on CNNs and RNNs that incorporates preceding short texts as context to classify current DAs. Zhou et al.  combine heterogeneous information with conditional random fields for Chinese dialogue act recognition. Kumar et al.  build a hierarchical encoder with CRF to learn multiple levels of utterance and act dependencies. Unlike the previous studies, we formulate the problem from the viewpoint of integrating contextual dependencies in both utterance level and the act label level.

Data Preparation	We evaluate the performance of our method on two benchmark DA datasets: Switchboard Dialogue Act Corpus (SwDA) and The ICSI Meeting Recorder Dialogue Act Corpus (MRDA). These two datasets have been widely used to conduct the dialogue act recognition or the dialogue act classification tasks by several prior studies. SwDA: Switchboard Dialogue Act Corpus is a large hand-labeled dataset of 1155 conversations from the Switchboard corpus of spontaneous human-to-human telephone speech. Each conversation involved two randomly selected strangers who had been charged with talking informally about one of several, self-selected general interest topics. For each utterance, together with a variety of automatic and semiautomatic tools, the tag set distinguishes 42 mutually exclusive utterance types via DAMSL taxonomy. The top five frequent DA types include STATEMENT, BACKCHANNEL / ACKNOWLEDGE, OPINION, ABANDONED / UNINTERPRETABLE, AGREEMENT / ACCEPT. We list the top five percentages of utterance type in the overall corpus in table2. MRDA: The ICSI Meeting Recorder Dialogue Act Corpus consists of hand-annotated dialog act, adjacency pair, and hotspot labels for the 75 meetings in the ICSI meeting corpus.

Visualization	In Figure 3, we visualize of the output edge marginals produced by the CRF-ASN model for a conversation. In this instance, the actual dialogue act recognition procedure is displayed as INLINEFORM0 . In the testing step, the model is uncertain and select the most attentive path to maximize the true dialogue act recognition. Here we can see from the marginal edges the path INLINEFORM1 occupies more attentive weights than the path INLINEFORM2 in predicting the dialogue act label. Thus we ultimately select the right way to recognize the dialogue act. Figure 4 shows the confusion heatmap of our proposed CRF-ASN model for the SwDA dataset. Each element in the heatmap denotes the rate that the predicted label is the same to the true label. We can see from the diagonal, the <sd,sd> <b,b> pairs achieve the most satisfying matching score while <qyd, qyd> is much worse than other pairs. This can be explained that the sd (statement) and b(acknowledge) have clearly self-identification while qyd(Declarative Yes-No-Question) is more easier to be mistakenly recognized.

The MRDA scheme provides several class-maps and corresponding scripts for grouping related tags together into smaller number of DAs. In this work we use the most widely used class-map that groups all tags into 5 DAs, i.e., Disruption (D) indicates the current Dialogue Act is interrupted. BackChannel (B) are utterances which are not made directly by a speaker as a response and do not function in a way that elicits a response either. FloorGrabber (F) are dialogue acts for grabbing or maintaining the floor. Question (Q) is for eliciting listener feedback. And finally, unless an utterance is completely indecipherable or else can be further described by a general tag, then its default status is Statement (S). We respectively list the percentage of the five general dialogue acts in table 3. From the table 2 and table 3, we can see the datasets are highly imbalanced in terms of label distributions. The dialogue act type STATEMENT occupies the largest proportion in both two datasets. Following the second place is the BACKCHANNEL act type which somewhat reflect the speaker's speech style. We present the detailed data preparation procedure for obtaining the clear dataset.

CRF Simple baseline which applies the text encoding and CRF-based structure prediction on the DAR problem. SVM Simple baseline which applies the text encoding and multi-classification algorithm on the DAR problem. Among them, The former five approaches eg. Bi-LSTM-CRF, DRLM-Conditional, LSTM-Softmax, RCNN, CNN all adopt the deep neural network model in order to better capture the utterances semantic representations. The latter three methods (HMM, CRF, SVM) just employ the simple feature selection on the text processing. About half of the baselines including Bi-LSTM-CRF, DRLM-Conditional, HMM, CRF consider the graphical structured prediction while the others eg. RCNN, CNN, LSTM-Softmax, SVM just adopt the traditional multi-classification algorithms. Table 5 and Table 6 respectively show the experimental Accuracy results of the methods on the SwDA and MRDA datasets. The hyper-parameters and parameters which achieve the best performance on the validation set are chosen to conduct the testing evaluation.

For testing, we adopt Viterbi algorithm to obtain the optimal sequence by using dynamic programming techniques. The testing procedure can be written as: INLINEFORM0  [t] Viterbi algorithm for CRF-ASN [1]  The observation space INLINEFORM0  The state space INLINEFORM0  The observation sequence INLINEFORM0  The probabilities INLINEFORM0  The most likely hidden state sequence INLINEFORM0    Construct transition matrix INLINEFORM0 , each element stores the transition probability of transiting from state INLINEFORM1 to state INLINEFORM2 Construct emission matrix INLINEFORM3 , each element stores the probability of observing INLINEFORM4 from state INLINEFORM5 each state INLINEFORM6 INLINEFORM7 INLINEFORM8 each observation INLINEFORM9 each state INLINEFORM10 INLINEFORM11 INLINEFORM12 INLINEFORM13 INLINEFORM14 INLINEFORM15 INLINEFORM16 INLINEFORM17 X

What's more, under the same text encoding situation, the CRF-based model achieves much better results than the SVM-based method. Which can fully prove the superiority of the structured prediction formulation. We also notice that CRF is better than HMM when adopted to the DAR task. The major differences between our proposed model CRF-ASN and the strong baseline BI-LSTM-CRF lie in two aspects: First we adopt a more fine grained manner to encode the utterances and utilize the memory enhanced mechanism to compute the contextual dependencies. Second we employ an adapted structured attention network on the CRF layer, rather than directly apply the original CRF on the utterances. These two modifications are essential and improve the performance significantly.

Ablation Results	We respectively evaluate the individual contribution of the proposed module in our model. We conduct thorough ablation experiments on the SwDA dataset, which are recorded on the table 7. To make it fair, we only modify one module at a time and fix the other components to be in the same settings. We replace the proposed structured CRF-attention layer to simple CRF, the results show structured CRF-attention layer results in major improvement in the accuracy, approximately over 2.1% absolute points. We further replace the structure prediction formulation to multi-classification on SVM, the results drop dramatically, which illustrate the benefit of considering structural dependencies among utterances. We replace the fine-grained word INLINEFORM0 to the simple Glove vector. The results suggest that fine grained word embedding is useful to represent a text. We also adapt the context state INLINEFORM1 to only care its neighbor utterances. The result is not satisfying, which conveys us that the basic text understanding is critical in the semantic representations. We replace the memory network to directly apply CRF layer to the utterance layer. We also conduct a comparing experiment which plus the original utterance to memory enhanced output.

For dialogue act sequence labeling problem, greedily predicting the dialogue act at each time-step might not optimal the solution. Instead, it is better to look into the correlations in both utterance level and the dialogue act level in order to jointly decode the best chain of dialogue acts. Formally, let INLINEFORM0 represent a sequence of utterance inputs, let INLINEFORM1 be the corresponding dialogue act sequence. Variable INLINEFORM2 are discrete latent act variables INLINEFORM3 with sample space INLINEFORM4 that encodes the desired selection among these inputs. The aim of the structured attention is to produce a sequence aware INLINEFORM5 INLINEFORM6 based on the utterances INLINEFORM7 and the dialogue act sequence INLINEFORM8 . We assume the attentive distribution INLINEFORM9 , where we condition INLINEFORM10 on the input utterances INLINEFORM11 and the dialogue act sequence INLINEFORM12 . Here we assume the utterances in the conversation as an undirected graph structure with INLINEFORM13 vertices. The CRF is parameterized with clique potentials INLINEFORM14 , indicating the subset of INLINEFORM15 give by clique INLINEFORM16 . Under this definition, the attention probability is defined as INLINEFORM17 .

For two datasets, we performed pre-processing steps in order to filter out the noise and some informal nature of utterances. We first strip the exclamations and commas, and then we convert the characters into lower-case. Notice that for SwDA, we only get the training and testing datasets. In order to smooth the training step and tune the parameters, we depart the original training dataset into two parts, one for training and the other small part used to be the validation set. We list the detailed statistics of the two datasets in table 4.

Implemental Details	We preprocess each utterance using the library of nltk  and exploit the popular pretrained word embedding Glove with 100 dimensional vectors  . The size of char-level embedding is also set as 100-dimensional and is obtained by CNN filters under the instruction of Kim  . The Gated Recurrent Unit  which is variant from LSTM  is employed throughout our model. We adopt the AdaDelta  optimizer for training with an initial learning rate of 0.005. We also apply dropout  between layers with a dropout rate of 0.2. For the memory network enhanced reasoning, we set the number of hops as 1 to preliminary learn the contextual dependencies among utterances. We do not set too many hops as increasing the number of GRU layers reduced the accuracy of the model. Early stopping is also used on the validation set with a patience of 5 epochs. Conversations with the same number of utterances were grouped together into mini-batches, and each utterance in a mini-batch was padded to the maximum length for that batch. The maximum batch-size allowed was 48.

The problem	Before presenting the problem, we first introduce some basic mathematical notions and terminologies for dialogue act recognition. Formally, we assume the input is in the form of sequence pairs: INLINEFORM0 with INLINEFORM1 . INLINEFORM2 is the input of the INLINEFORM3 -th conversation in dataset INLINEFORM4 and INLINEFORM5 is the INLINEFORM6 -th targeted dialogue act type. Each conversation INLINEFORM7 is composed of a sequence of utterances which denoted as INLINEFORM8 with aligned act types INLINEFORM9 . We have each dialogue act type assigned to utterance INLINEFORM10 and each associated INLINEFORM11 denoted the possible dialogue act belongs to INLINEFORM12 act types. Again each utterance consists of a sequence of diverse words INLINEFORM13 . Most of the previous models do not leverage the implicit and intrinsic dependencies among dialogue act and utterances. They just consider a conversation as a flat structure with an extremely long chain of words. However, such a construction suffers vanishing gradient problem as the extremely long words become impractical in the neural network back-propagation training process.

Dialogue Act Recognition	The main task of dialogue act recognition is to assign an act label to each utterance in a conversation, which can be defined as a supervised problem due to the properties that each utterance has a corresponding act label. Most of the existing work for the problem of dialogue act recognition can be categorized as following two groups. Regarding the DAR as a multi-classification problem. Reithinger et al.  present deal with the dialogue act classification using a statistically based language model. Webb et al.  apply diverse intra-utterance features involving word n-gram cue phrases to understand the utterance and do the classification. Geertzen et al.  propose a multidimensional approach to distinguish and annotate units in dialogue act segmentation and classification. Grau et al.  focus on the dialogue act classification using a Bayesian approach. Serafin et al.  employ Latent Semantic Analysis (LSA) proper and augmented method to work for dialogue act classification. Chen et al.  had an empirical investigation of sparse log-linear models for improved dialogue act classification. Milajevs et al.  investigate a series of compositional distributional semantic models to dialogue act classification.

Structured CRF-Attention Network	Traditional attention networks have proven to be an effective approach for embedding categorical inference within a deep neural network. However, In DAR problem, we need to further explore the structural dependencies among utterances and dialogue acts. As we see, utterances in a conversation are not exist independently. The latter utterance may be the responding answer to the former question, or that the chunk of utterances are in the same act type. Here we consider generalizing selection to types of chunks selecting attention, and propose the structured attention to model richer dependencies by incorporating structural distributions within networks. Such a structured attention can be interpreted as using soft-selection that considers all possible structures over the utterance input. In our paper, we formulate the DAR as a sequence labeling problem. It is a natural choice to assign a label to each element in the sequence via linear chain CRF, which enable us to model dependencies among labels. Here we do not directly apply the original linear chain CRF to the learned utterance. Although the dependencies among utterances have been captured by the former hierarchical semantic networks, we still need to further explore the dialogue act dependencies in the label level.

The injected temporal signal can further explore the contextual influence on the current input utterance. We thus can make use of this obtained INLINEFORM1 to represent another INLINEFORM2 which cares more about the context influence. For the current input utterance INLINEFORM0 , in memory networks, the input is required to be in the same space as the input memory. Here we adopt the popular attention mechanism in the memory by measuring the relevance between current input utterance INLINEFORM1 and the contextual new representation INLINEFORM2 . The relevance is measured with a softmax function: INLINEFORM3  Once the attention weights have been computed, the output memory can be used to generate the final output of the memory layer in the form of a weighted sum over the attention and the input utterance: INLINEFORM0  The output allows the model to have unrestricted access to elements in previous steps as opposed to a single hidden state INLINEFORM0 in recurrent neural networks. Thereby we can effectively detect the long range dependencies among utterances in a conversation.

Regarding the DAR as a sequence labeling problem. Stolcke et al.  treat the discourse structure of a conversation as a hidden Markov model and the individual dialogue acts as observations emanating from the model states. Tavafi et al.  study the effectiveness of supervised learning algorithms SVM-HMM for DA modeling across a comprehensive set of conversations. Similar to the SVM-HMM, Surendran et al.  also use a combination of linear support vector machines and hidden markov models for dialog act tagging in the HCRC MapTask corpus. Lendvai et al.  explore two sequence learners with a memory-based tagger and conditional random fields into turn-internal DA chunks. Boyer et al.  also applied HMM to discover internal dialogue strategies inherent in the structure of the sequenced dialogue acts. Galley et al.  use skip-chain conditional random field to model non-local pragmatic dependencies between paired utterances. Zimmermann et al.  investigate the use of conditional random fields for joint segmentation and classification of dialog acts exploiting both word and prosodic features.

Introduction	Dialogue Act Recognition (DAR) is an essential problem in modeling and detecting discourse structure. The goal of DAR is to attach semantic labels to each utterance in a conversation and recognize the speaker's intention, which can be regarded as a sequence labeling task. Many applications have benefited from the use of automatic dialogue act recognition such as dialogue systems, machine translation, automatic speech recognition, topic identification and talking avatars    . One of the primary applications of DAR is to support task-oriented discourse agent system. Knowing the past utterances of DA can help ease the prediction of the current DA state, thus help to narrow the range of utterance generation topics for the current turn. For instance, the "Greeting" and "Farewell" acts are often followed with another same type utterances, the "Answer" act often responds to the former "Question" type utterance. Thus if we can correctly recognize the current dialogue act, we can easily predict the following utterance act and generate a corresponding response. Table 1 shows a snippet of the kind of discourse structure in which we are interested. The essential problem of DAR lies on predicting the utterance's act by referring to contextual utterances with act labels.

To alleviate this problem, we consider the conversation to be a hierarchical structure composed of three level encoders: first encode each word in a fine grained manner, and the second encoder operates at the utterance level, the last encoder encode each utterance in the conversation level. Each encoder is based on the previous one thus can make sure the output of the previous one can capture the dependencies across the conversation. Here we take an example to illustrate the sequence structure in Figure 1. Apart from hierarchical neural encoders, we also integrate external memory to allow the model to have unrestricted access to the whole sequence rather than localized features as in RNNs. Naturally the dialogue act recognition problem can be regarded as a sequence labeling task which can be assigned dialogue act through multi-classification method or the structured prediction algorithms. In our formulation, we adopt the linear chain conditional random field (CRF) along with hierarchical attentive encoders for the structured prediction. Instead of labeling each utterance in isolation, structured prediction models such as HMM, CRF can better capture the contextual dependencies among utterances.

In our model, we define the structured attention model as being an extended attention model which provides an alternative approach to incorporate the machinery of structural inference directly into our neural network.

We can see that <qyd,qy> which represents (Declarative Yes-No-Questio,Yes-No-Question) is indeed hard to recognize since their dialogue type are too similar with each other. For another reason, we notice that due to the bias of the ground truth, there are some cases that we predict the dialogue act correctly while the ground truth is wrong. To some reason, classifying so many fine-grained dialogue act labels is not easy for human annotators, besides the human-subjectivity occupies an important role in recognizing the dialogue act.

CRF-attentive Structured Network 	In this section, we study the problem of dialogue act recognition from the viewpoint of extending rich CRF-attentive structural dependencies. We first present the hierarchical semantic inference with memory mechanism from three levels: word level, utterance level and conversation level. We then develop graphical structured attention to the linear chain conditional random field to fully utilize the contextual dependencies.

The weights are determined by measuring the similarity between the input memory and the current utterance input. Fine Grained Embedding: For a given conversation, each utterance INLINEFORM0 is encoded by a fine grained embedding layer. We first try to utilize the rich lexical factors and linguistic properties to enhance the word representation. For each word token INLINEFORM1 in each utterance, we initialized the word embedding using pretrained embeddings such as Word2vec or Glove. Furthermore, in order to tackle the out-of-vocabulary (OOV) problem, we adopt the character-level word embedding via CNN to combine with pretrained word level embeddings. We also extend the lexical factors via POS tag and NER tag to enhance the utterance understanding. The obtained four factors are concatenated to form a rich lexical representation as: INLINEFORM2  Since we consider the bidirectional GRU to encode the representation of each utterance, we concatenate the outputs from the forward and backward GRU hidden representations at the time step.

It is notably that the whole process is differentiable thus can be trained in an end-to-end manner. The main contributions of this paper are as follows: The rest of this paper is organized as follows. In section 2, we introduce the problem of dialogue act recognition from the viewpoint of introducing CRF-structured attention, and propose the CRF-attentive structural network with hierarchical semantic inference and memory mechanism. A variety of experimental results are presented in Section 3. We have a comprehensive analysis on the experiment results and conduct the ablations to prove the availability of our model. We then provide a brief review of the related work about dialogue act recognition problem in Section 4. Finally, we provide some concluding remarks in Section 5.

A further limitation is that although these works claim they have considered the contextual correlations, in fact they view the whole conversation as a flat sequence and neglect the dual dependencies in the utterance level and act level    . Until now, the achieved performances in DAR field are still far behind human annotator's accuracy. In this paper, we present the problem of DAR from the viewpoint of extending richer CRF-attentive structural dependencies along with neural network without abandoning end-to-end training. For simplicity, we call the framework as CRF-ASN (CRF-Attentive Structured Network). Specifically, we propose the hierarchical semantic inference integrated with memory mechanism on the utterance modeling. The memory mechanism is adopted in order to enable the model to look beyond localized features and have access to the entire sequence. The hierarchical semantic modeling learns different levels of granularity including word level, utterance level and conversation level. We then develop internal structured attention network on the linear-chain conditional random field (CRF) to specify structural dependencies in a soft manner. This approach generalizes the soft-selection attention on the structural CRF dependencies and takes into account the contextual influence on the nearing utterances.

Attention Network	Attention mechanism has become an essential component in text understanding in recent years. Since the first work proposed by Bahdanau et al.  that adopt the attention mechanism in neural machine translation, attention mechanism based neural networks have become a major trend in diverse text researching field, such as in machine comprehension     , machine translation   , abstract summarization   , text classification    and so on. The principle of attention mechanism is to select the most pertinent piece of information, rather than using all available information, a large part of it being irrelevant to compute the neural response. In our work, we propose the CRF-attentive structured network in order to encode the internal utterance inference with dialogue acts. The structured attention is a more general attention mechanism which take account of the graphical dependencies and allow for extending attention beyond the standard soft-selection approach. The most similar work to our model is proposed by Kim et al.  . Kim et al. also experiment with two different classes of structured attention networks: subsequence selection and syntactic selection. However, the objectives of these two networks aims to segment the structure dependencies, which are quite different from our DAR task.

Hierarchical Semantic Network	Due to the hierarchical nature of conversations, our proposed model is constructed at multiple levels of granularity, e.g. word level, utterance level and conversation level. The representation of a conversation can be composed by each utterance INLINEFORM0 , and each INLINEFORM1 can be obtained by combining the representations of constituent words INLINEFORM2 . Taking inspiration from Memory Networks and incorporate so-called memory hops, we adopt the memory enhanced contextual representations in order to have unrestricted access to the whole sequence rather than localized features as former recurrent neural network. Here we include the memory enhanced hierarchical representation in Figure 2 to depict the conversation level representation. As illustrated in Figure 2, the hierarchical semantic network can be divided into two parts: (1) fine grained embedding layer (2) memory enhanced contextual representation layer. The second part can be further broken down into three main components: (a) the input memory INLINEFORM0 which takes in the output from the word embedding layer (b) the contextual attention which takes the consideration of the former utterance and the latter one. (c) the output memory INLINEFORM1 which is obtained from the input memory connected with the attention mechanism.

Notice that the utterance INLINEFORM3 and the dialogue act sequence INLINEFORM4 are both obtained from downstream learned representation. The marginal distribution INLINEFORM5 can be calculated efficiently in linear time via the forward-backward algorithm. These marginals further allow us to implicitly sum over the linear chain conditional random field. We refer to this type of attention layer as a INLINEFORM6 INLINEFORM7 INLINEFORM8 , where we can explicitly look into the undirected graphical CRF structure to find which utterances are in the same chunk or in isolation. Here we define the node potentials with a unary CRF setting: INLINEFORM0   where for each utterance we summarize the possible dialogue act to perform sequential reasoning. Given the potential, we compute the structural marginals INLINEFORM0 using the forward-backward algorithm, which is then used to compute the final probability of predicting the sequence of dialogue acts as: INLINEFORM1

For each utterance INLINEFORM0 which consists a sequence of words INLINEFORM1 , the original semantic representation is as follows: INLINEFORM2  Here we utilize INLINEFORM0 and INLINEFORM1 to represent the word level embedding function and utterance level encoder in our hierarchical model. After obtained the original semantic representations on each utterance, we later apply the memory enhanced contextual layer to further explore the correlations between utterances. Memory Enhanced Contextual Representation: Every utterance in a conversation is encoded with INLINEFORM0 , where INLINEFORM1 is the encoding function via Bi-GRU to map the input words into a vector INLINEFORM2 . The original sequence utterances are denoted as INLINEFORM3 . While this original semantic representation can be the input component in the context of memory network. In order to tackle the drawback of insensitivity to temporal information between memory cells, we adopt the approach in injecting temporal signal into the memory using a contextual recurrent encoding: INLINEFORM4   where INLINEFORM0 , INLINEFORM1 , INLINEFORM2 are learnable parameters. It is a remarkable fact that the new sequence INLINEFORM0 can be seen as the contextual integrated representations which take consider of the former utterances and the latter ones.

To further extend the complex reasoning over multiple supporting facts from memory, we adopt a stacking operation which stacks hops between the original utterance semantic representation INLINEFORM0 and the k-th output hop INLINEFORM1 to be the input to the INLINEFORM2 th hop: INLINEFORM3   where INLINEFORM0 encodes not only information at the current step ( INLINEFORM1 ), but also relevant knowledge from the contextual memory ( INLINEFORM2 ). Note that in the scope of this work, we limit the number of hops to 1 to ease the computational cost.

We not only consider the fine grained multi-level semantic representations, but also integrate the structured attention network to further capture the structure designpendencies in the CRF layer.

End-to-End Training	We adopt the maximum likelihood training estimation to learn the CRF-attentive structured parameters. Given the training set INLINEFORM0 with INLINEFORM1 conversation pairs, the log likelihood can be written as: INLINEFORM2   where we denote the INLINEFORM0 as the set of parameters within neural networks from hierarchical layers: word embedding layer, memory enhanced utterance modeling layer, CRF-attentive structured layer. We define the objective function in training process: DISPLAYFORM0   INLINEFORM0 is a hyper-parameter to trade-off the training loss and regularization. By using SGD optimization with the diagonal variant of AdaGrad, at time step t, the parameter INLINEFORM1 is updated as follows: DISPLAYFORM0  where INLINEFORM0 is the initial learning rate and INLINEFORM1 is the sub-gradient at time t. Notice that one of our contributions is to apply CRF structural attention as the final layer of deep models. The whole model can be trained in an end-to-end manner. Here we consider the standard Viterbi algorithm for computing the distribution INLINEFORM0 . The main procedure is summarized in Algorithm 1.

In DAR task we care more on the dialogue act influences on the overall conversation structure, thus the former structured attention may not be suitable for our problem.

Evaluation Criteria	We mainly evaluate the performance of our proposed CRF-ASN method based on the widely-used evaluation criteria for dialogue act recognition, Accuracy. The Accuracy is the normalized criteria of accessing the quality of the predicted dialogue acts based on the testing utterance set INLINEFORM0 . Given the testing conversation INLINEFORM1 with its ground-truth dialogue acts INLINEFORM2 , we denote the predicted dialogue acts from our CRF-ASN method by INLINEFORM3 . We now introduce the evaluation criteria below. INLINEFORM4

During training, we set the moving averages of all weights as the exponential decay rate of 0.999  . The whole training process takes approximately 14 hours on a single 1080Ti GPU. All the hyper-parameters were selected by tuning one hyper-parameter at a time while keeping the others fixed.

Related Work	In this section, we briefly review some related work on dialogue act recognition and attention network.

The two results show the designed hierarchical memory-enhanced components are helpful in the utterance understanding and modeling the contextual influence.

Most of existing models adopt handcrafted features and formulate the DAR as a multi-classification problem. However, these methods which adopt feature engineering process and multi-classification algorithms reveal deadly weakness from two aspects: First, they are labor intensive and can not scale up well across different datasets. Furthermore, they abandon the useful correlation information among contextual utterances. Typical multi-classification algorithms like SVM, Naive Bayes    can not account for the contextual dependencies and classify the DA label in isolation. It is evident that during a conversation, the speaker's intent is influenced by the former utterance such as the previous "Greeting" and "Farewell" examples. To tackle these two problems, some works have turn to structured prediction algorithm along with deep learning tactics such as DRLM-Conditional  , LSTM-Softmax  and RCNN  . However, most of them failed to utilize the empirical effectiveness of attention in the graphical structured network and relies completely on the hidden layers of the network, which may cause the structural bias.

Question: By how much do they outperform state-of-the-art solutions on SWDA and MRDA?
Answer: improves the DAR accuracy over Bi-LSTM-CRF by 2.1% and 0.8% on SwDA and MRDA respectively