# Research on Oral English Dialogue Understanding Based on Deep Learning

Authors: Chan Li; Rui Tian
Journal: Scientific Programming
 (2022)

---

## Abstract

Oral English dialogue is a crucial part of a dialogue system that enables a computer to “understand” the input language as a human does, so the performance of a dialogue system is closely related to the performance of oral English dialogue understanding. In task-based human-machine dialogue systems, external knowledge bases can provide the machine with valid information beyond the training data, helping the model to better perform the oral English dialogue comprehension task. In this paper, we propose a deep recurrent neural network based on feature fusion, which directly stacks multiple nodes at a single time node to deepen the complexity of nonlinear transformations. The feature fusion network structure is applied to the ATIS dataset for oral English dialogue comprehension experiments, and the experimental results demonstrate that the feature fusion RNN network can further improve the effectiveness of the RNN network and the GRU network structure unit can obtain better results among different RNN node units.


---

## Body

## 1. Introduction

With the rapid development of deep learning, the field of artificial intelligence has undergone a cross-generational change [1–3]. As one of the core technologies in the field of natural language processing, human-computer dialogue technology has now become a hot spot for research in this field. Human-computer dialogue systems are used in a wide range of applications, such as smart home appliances, navigation systems, human-machine customer service, and mobile phone assistants, to provide convenient and personalized services for staff and users [4, 5]. Therefore, human-computer dialogue technology is highly valued in both academia and industry, and the research on the human-computer dialogue has important academic significance and application value.The study of human-computer dialogue technology first originated in the 1960s, when the famous computer scientist Alan Turing proposed the Turing test [6] to test whether machines really have human intelligence. Since then, researchers have been working on human-computer dialogue. The authors of [7] simulate a psychotherapist to psychologically de-escalate and treat psychiatric patients. On the basis of the chat system, the authors of [8] developed the Parry [9] system.In recent years, with the spread of social networks and new media, the emergence of massive amounts of data has acted as an invisible hand to drive the development of human-computer dialogue technology. Unlike template- and rule-based dialogue systems, deep learning-based human-computer dialogue technologies, driven by large amounts of data, are able to use neural networks to derive effective features from large-scale training data to achieve a level of understanding of user conversations and learning language expressions. The rapid growth of deep learning techniques today is largely dependent on massive amounts of chart data and rising hardware computing performance [10]. Currently, the most representative human-computer dialogue systems include Apple’s Siri, Microsoft’s Cortana, Taobao’s Aliyun Xiaomi, Baidu’s in-car system Apollo, and Alibaba’s Tmall Genie [11–13]. These products not only use human-machine dialogue to efficiently help users complete specific tasks and save them time but also allow them to have an excellent service experience. The core systems of these products are called task-based dialogue systems, which can help users complete specific tasks, with the main application scenario being personal assistants.Users generally interact with the system via voice and text to enable various queries and agent functions such as car navigation, weather enquiries, e-mail sending and receiving, intelligent search, and schedule reminders. The system enables users to complete various daily tasks quickly and accurately [14].We focus on spoken language comprehension tasks under a modular task human-computer dialogue system. The modular human-computer dialogue system is a modulation task-based human-computer dialogue system, which is divided into the following modules as shown in Figure1, namely, speech recognition and speech synthesis module, oral English dialogue comprehension module, dialogue management module, and dialogue generation module. In the task-based dialogue system, the oral English dialogue understanding module, as the first part of the system, is the basis of the whole dialogue system, and its function is to understand the user’s questions, convert the user’s dialogue information into structured data that can be recognized by the computer, and serve as the data input for the dialogue management module. The oral English dialogue comprehension module, therefore, plays a key role in the overall task-based human-computer dialogue system [15]. Currently, the oral English dialogue understanding module is also being developed by many companies as a cloud service open to the public, for example, Microsoft's LUIS platform [16].Figure 1
The modular task-based human-computer oral English dialogue system.In summary, oral English dialogue comprehension as an important module in task-based dialogue systems is a fundamental part of current human-computer interaction technology and is a challenging and innovative research task in both academia and industry. With the rapid development of deep learning, the use of deep learning techniques can effectively improve the effectiveness of spoken language comprehension tasks, promote the development of HCI research, and better meet the needs of practical application scenarios.


## 2. Related Work

As one of the core technologies in the field of artificial intelligence, human-computer dialogue systems have always been a hot topic of interest to researchers [17]. As human-computer dialogue technology continues to mature and the range of applications in human-computer dialogue systems continues to expand, people's requirements for dialogue systems are becoming higher and higher. In task-based human-computer dialogue systems, spoken language comprehension plays an important and indispensable role in the whole system, and although the research work on spoken language comprehension tasks is relatively mature, there are still many difficulties and challenges in this field. In [18], a graph-and-rule-based approach was proposed to obtain intentional templates for the consumer intention recognition task, which has yielded good results in the consumer domain. The researchers of [19] showed that the difference in natural language expressions under a single domain can lead to a dramatic increase in the number of templates and rules, which requires a lot of human and material resources. Thus, although rule- and template-based approaches to oral English dialogue understanding do not require the support of huge amounts of training data and can guarantee the accuracy, the human and material resources consumed rise dramatically as the complexity of the task increases and cannot address the high costs associated with changing domains, intention templates, and rules [20].Reference [21] used DBN to solve the task of intention recognition in oral English dialogue understanding. RNNs, an important model in the field of NLP, were used by [22] to build and perform the task of intention recognition. The authors of [23] compared and investigated traditional recurrent neural networks with the above two variants of the structure on sentence classification tasks and demonstrated that long and short-term memory networks and recurrent networks of gate units were significantly better than ordinary recurrent neural networks. The authors of [10] used recurrent neural networks for the first time for the slot filling task and obtained good results. CRF, a paradigmatic model for solving slot filling tasks in machine learning, was used by [24] in combination with CNN for the slot filling task. Bidirectional recurrent neural networks (BiRNN) were used in [25] to solve the slot filling problem. LSTM with conditional random fields was proposed in [2] to solve the slot filling problem with good results. A joint model using recurrent neural networks and the Viterbi algorithm was first proposed in [12] to solve both the intention recognition and slot filling tasks. Their work, despite the improvement in model performance, still suffers from information loss caused by recurrent neural networks. The authors of [3] adopted the idea of a joint model for intent recognition and slot filling using GRU to perform a sequence annotation task based on word representation learned by GRU to predict slot labels and a pooling layer to capture global information about the user's utterance for the intent recognition task.Deep learning, supported by a large amount of data and reliable hardware, has gradually become an important technical support for human-computer dialogue research. Deep learning techniques have also been used because of the good effect as an important module in human-computer dialogue systems.


## 3. Joint Model of Oral Language Comprehension

### 3.1. Description of the Problem

The knowledge-driven multiround dialogue oral English dialogue comprehension task makes use of an external knowledge base to supplement the multiround dialogue oral English dialogue comprehension model with external information, thus improving the accuracy of the intention recognition and slot filling tasks. The task can be defined as given multiple rounds of user dialogueU=u1,u2,u3,…,ut, where ut is taken as the current round of dialogue and u1,u2,u3,…,ut−1 as the historical information for each round of dialogue ui, it is divided into ui=x1i,x2i,…,xki; and the external knowledge base is taken as K=k1,k2,…,kM. The goal of the joint model of oral English dialogue comprehension, etc., is to learn the current round of dialogue based on the historical information provided by H to be able to identify the intent I and slot of the current ut dialogue label S. The formula is(1)I,S=Fut,H,K.


### 3.2. Model Design

The addition of an external knowledge base enables the oral English dialogue comprehension model to obtain better inferences, and how to obtain good quality candidate knowlghedge is the main problem that needs to be solved.Firstly, we design a candidate knowledge recall rule for rouly extracting the candidate knowledge with higher relevance from the external knowledge base; secondly, we propose a candidate knowledge attention module to filter the candidate knowledge based on the implicit information of words so as to obtain a higher quality candidate knowledge vector; and finally, we use the joint model of oral English dialogue comprehension to complete the task of simultaneously completing intention recognition and semantic slot filling [19]. In addition, we build on the model by using the BRET model as a subcoding layer for candidate knowledge, historical information, and current sentences, resulting in a better-quality representation of word vectors.Based on this, this paper proposes a joint model for a multiround oral English dialogue comprehension task based on an external knowledge base and historical information. We selected ConceptNet [22] as a supplement to the external knowledge base and used its semantic network properties to recall the candidate knowledge related to each word in the user’s conversation from the knowledge base; meanwhile, we used the slot attention layer to weigh the average of the historical information and slot candidate knowledge, respectively; finally, we used the joint model to complete the task with the addition of external knowledge and historical information to the oral English dialogue comprehension task.The oral English dialogue comprehension task contains not only an intention recognition task for sentence-level classification but also a sequence annotation task for word-level classification. In the candidate knowledge selection phase, this paper focuses on the sequence annotation task, which looks for relevant candidate knowledge through the information of each word in the user’s dialogue.As shown in Table1, for the sample utterance “Please lead me to the MIT,” the word MIT refers to “Massachusetts Institute of Technology,” which stands for a university acronym and also the geographical location of the university “Cambridge, Massachusetts, USA.” This information is inaccessible to the human-computer dialogue system, and the introduction of an external knowledge base can solve this problem. This paper chooses to use “word” as the “query information” of the query knowledge base. For example, the ten most relevant information of “MIT” are queried, and the results contain most of the related knowledge of the word “MIT,” which can help the computer better understand the user's dialogue [13].Table 1
Sample candidate knowledge query.
Sample statement:Please lead me to the MITQuery words:MITQuery resultsMIT is a type of a universityMIT is in MassachusettsMIT is an institute of higher learningFor each dialogue textM=m1,m2,…,mn, the ConceptNet knowledge base was used to conduct an intrabase search for m and recall the 10 most relevant candidate knowledge (or all of them if less than 10), denoted as Ki=k1,k2,…,kp, where 0 < p < 11.At present, encoding user dialogue utterances into a word vector that the computer can recognize is the first step. We use the BERT pretraining model proposed by Goge in 2018 as the word encoding layer and define the input data as follows: for each dialogue textM=m1,m2,…,mn, BERT is used to transform it into an X = x1,x2,…,xn text vector. For the current word mi0<i<n+1, the recall knowledge Ki=k1,k2,…,kp is encoded using BERT, and the vector of CLS bits in BERT is taken as the sentence representation of each knowledge K to obtain the candidate knowledge encoding L=l1,l2,…,lp corresponding to the current word mi, where the value of p does not exceed 10.


### 3.3. Joint Model

The federated model has two main layers of BILSTM as the base model, including a historical information attention module and a candidate knowledge attention module attached to the first layer of BILSTM (current sentence encoding layer) and an auxiliary gate structure attached to the second layer of BILSTM (sequence annotator), as shown in Figure2.Figure 2
Structure of the oral comprehension model.The same encoding layer as designed in Chapter 3 is used in the encoding layer of the current utterance to encode the currentut=x1,x2,…,xn using BILSTM, which is calculated as shown in equation (2) to obtain the output hidden layer o1 and the final representation of the utterance c1, respectively. Here, o1=o11,o21,…,on1 represents the hidden state of each word of the user’s utterance after BiLSTMencoder encoding; c1 is the final state representation of BiLSTMencoder, which is used to represent the information of the whole user’s current utterance, as follows:(2)O1,C1=BiLSTMencoderut.Our model encapsulates the attention module in the same way for both historical information and candidate knowledge filtering. The focus of the attention module is to calculate the attention weights,αk representing the attention weight between the kth word of the current statement after the encoding layer and the corresponding hidden layer ok1 and the historical information H or candidate knowledge L (uniformly denoted by h in the formula), vk=v1k,v2k,…,vsk representing the set of information between the corresponding hidden layer ok1 and the historical information or candidate knowledge after the attention weight layer, and the attention weights are calculated as follows:(3)αk=expscoreok1,hi∑i=1nexpscoreok1,hi,vk=∑i=1nαk,ihi.After the attention module, both historical information and candidate knowledge were weighted to obtain the most relevant and important information about the word, where the final representation of the current wordxi was vni and the final representation of the candidate knowledge wasvli. A layer of a forward propagation neural network was used to fuse Vh,Vl, and ok, and a ReLU activation function was added for the nonlinear transformation. D=d11,d21,…,dn1 represents the set of vectors over which the historical information representation, the candidate knowledge representation, and the current hidden layer are fused, and dk1 is computed as follows:(4)dk1=reluFFVh,Vl,ok.The input of the decoder isD and c1 of the output of the attention module, and the output of the decoder is O2 and I, which are used for the semantic slot filling and intention recognition tasks, respectively.(5)O2,I=BiLSTMtagD,C1.Using the correlation between intention recognition and semantic slot filling in oral English dialogue comprehension, adding intention information to slot recognition can effectively improve the performance of the semantic slot filling model. The combination of the context vectorO2 and the intention vector I of the slot is used to obtain the weight vector g through the gate structure and use the g congregation as the weight for predicting the slot yiS, where w1 and wyS are trainable parameters and are calculated as follows:(6)g=∑w1.tanhoi2+w2.I,yiS=softmaxwySoi2.g+oi2.The intention recognition task uses the final cell state/prediction from the sequence annotation, where w is the trainable parameter and y is calculated as follows:(7)yil=softmaxwyl⋅I.


## 3.1. Description of the Problem

The knowledge-driven multiround dialogue oral English dialogue comprehension task makes use of an external knowledge base to supplement the multiround dialogue oral English dialogue comprehension model with external information, thus improving the accuracy of the intention recognition and slot filling tasks. The task can be defined as given multiple rounds of user dialogueU=u1,u2,u3,…,ut, where ut is taken as the current round of dialogue and u1,u2,u3,…,ut−1 as the historical information for each round of dialogue ui, it is divided into ui=x1i,x2i,…,xki; and the external knowledge base is taken as K=k1,k2,…,kM. The goal of the joint model of oral English dialogue comprehension, etc., is to learn the current round of dialogue based on the historical information provided by H to be able to identify the intent I and slot of the current ut dialogue label S. The formula is(1)I,S=Fut,H,K.


## 3.2. Model Design

The addition of an external knowledge base enables the oral English dialogue comprehension model to obtain better inferences, and how to obtain good quality candidate knowlghedge is the main problem that needs to be solved.Firstly, we design a candidate knowledge recall rule for rouly extracting the candidate knowledge with higher relevance from the external knowledge base; secondly, we propose a candidate knowledge attention module to filter the candidate knowledge based on the implicit information of words so as to obtain a higher quality candidate knowledge vector; and finally, we use the joint model of oral English dialogue comprehension to complete the task of simultaneously completing intention recognition and semantic slot filling [19]. In addition, we build on the model by using the BRET model as a subcoding layer for candidate knowledge, historical information, and current sentences, resulting in a better-quality representation of word vectors.Based on this, this paper proposes a joint model for a multiround oral English dialogue comprehension task based on an external knowledge base and historical information. We selected ConceptNet [22] as a supplement to the external knowledge base and used its semantic network properties to recall the candidate knowledge related to each word in the user’s conversation from the knowledge base; meanwhile, we used the slot attention layer to weigh the average of the historical information and slot candidate knowledge, respectively; finally, we used the joint model to complete the task with the addition of external knowledge and historical information to the oral English dialogue comprehension task.The oral English dialogue comprehension task contains not only an intention recognition task for sentence-level classification but also a sequence annotation task for word-level classification. In the candidate knowledge selection phase, this paper focuses on the sequence annotation task, which looks for relevant candidate knowledge through the information of each word in the user’s dialogue.As shown in Table1, for the sample utterance “Please lead me to the MIT,” the word MIT refers to “Massachusetts Institute of Technology,” which stands for a university acronym and also the geographical location of the university “Cambridge, Massachusetts, USA.” This information is inaccessible to the human-computer dialogue system, and the introduction of an external knowledge base can solve this problem. This paper chooses to use “word” as the “query information” of the query knowledge base. For example, the ten most relevant information of “MIT” are queried, and the results contain most of the related knowledge of the word “MIT,” which can help the computer better understand the user's dialogue [13].Table 1
Sample candidate knowledge query.
Sample statement:Please lead me to the MITQuery words:MITQuery resultsMIT is a type of a universityMIT is in MassachusettsMIT is an institute of higher learningFor each dialogue textM=m1,m2,…,mn, the ConceptNet knowledge base was used to conduct an intrabase search for m and recall the 10 most relevant candidate knowledge (or all of them if less than 10), denoted as Ki=k1,k2,…,kp, where 0 < p < 11.At present, encoding user dialogue utterances into a word vector that the computer can recognize is the first step. We use the BERT pretraining model proposed by Goge in 2018 as the word encoding layer and define the input data as follows: for each dialogue textM=m1,m2,…,mn, BERT is used to transform it into an X = x1,x2,…,xn text vector. For the current word mi0<i<n+1, the recall knowledge Ki=k1,k2,…,kp is encoded using BERT, and the vector of CLS bits in BERT is taken as the sentence representation of each knowledge K to obtain the candidate knowledge encoding L=l1,l2,…,lp corresponding to the current word mi, where the value of p does not exceed 10.


## 3.3. Joint Model

The federated model has two main layers of BILSTM as the base model, including a historical information attention module and a candidate knowledge attention module attached to the first layer of BILSTM (current sentence encoding layer) and an auxiliary gate structure attached to the second layer of BILSTM (sequence annotator), as shown in Figure2.Figure 2
Structure of the oral comprehension model.The same encoding layer as designed in Chapter 3 is used in the encoding layer of the current utterance to encode the currentut=x1,x2,…,xn using BILSTM, which is calculated as shown in equation (2) to obtain the output hidden layer o1 and the final representation of the utterance c1, respectively. Here, o1=o11,o21,…,on1 represents the hidden state of each word of the user’s utterance after BiLSTMencoder encoding; c1 is the final state representation of BiLSTMencoder, which is used to represent the information of the whole user’s current utterance, as follows:(2)O1,C1=BiLSTMencoderut.Our model encapsulates the attention module in the same way for both historical information and candidate knowledge filtering. The focus of the attention module is to calculate the attention weights,αk representing the attention weight between the kth word of the current statement after the encoding layer and the corresponding hidden layer ok1 and the historical information H or candidate knowledge L (uniformly denoted by h in the formula), vk=v1k,v2k,…,vsk representing the set of information between the corresponding hidden layer ok1 and the historical information or candidate knowledge after the attention weight layer, and the attention weights are calculated as follows:(3)αk=expscoreok1,hi∑i=1nexpscoreok1,hi,vk=∑i=1nαk,ihi.After the attention module, both historical information and candidate knowledge were weighted to obtain the most relevant and important information about the word, where the final representation of the current wordxi was vni and the final representation of the candidate knowledge wasvli. A layer of a forward propagation neural network was used to fuse Vh,Vl, and ok, and a ReLU activation function was added for the nonlinear transformation. D=d11,d21,…,dn1 represents the set of vectors over which the historical information representation, the candidate knowledge representation, and the current hidden layer are fused, and dk1 is computed as follows:(4)dk1=reluFFVh,Vl,ok.The input of the decoder isD and c1 of the output of the attention module, and the output of the decoder is O2 and I, which are used for the semantic slot filling and intention recognition tasks, respectively.(5)O2,I=BiLSTMtagD,C1.Using the correlation between intention recognition and semantic slot filling in oral English dialogue comprehension, adding intention information to slot recognition can effectively improve the performance of the semantic slot filling model. The combination of the context vectorO2 and the intention vector I of the slot is used to obtain the weight vector g through the gate structure and use the g congregation as the weight for predicting the slot yiS, where w1 and wyS are trainable parameters and are calculated as follows:(6)g=∑w1.tanhoi2+w2.I,yiS=softmaxwySoi2.g+oi2.The intention recognition task uses the final cell state/prediction from the sequence annotation, where w is the trainable parameter and y is calculated as follows:(7)yil=softmaxwyl⋅I.


## 4. Experiment

### 4.1. Experimental Environment

The experimental environment is a 64 bit Linux environment using Ubuntu 16.04.3. Development tools used are Python 2.7.14 in the anaconda environment, and the python deep learning libraries scene and eras are used. Numpy and Scipy packages can be installed directly from the Linux command line using the pipe command. To better handle deep neural network algorithms, think of a deep learning library that can perform fast operations on both CPUs and GPUs and is designed to handle large neural network algorithms. Eras is an application programming interface for high-level neural network applications using thumb, tomorrow, and CNTK as backends. It is written in python, is highly modular in nature, and runs seamlessly on both CPUs and GPUs. This paper experiments with deep neural networks are based on Python2.7 using thumb and keras deep learning libraries.


### 4.2. Data Preparation

This paper applies data information from the Airline Travel Information System (ATIS) domain. This database was developed in the early 1990s as a project of the US Defense Advanced Research Projects Agency (DARPA) in the area of airline travel information. The task consists of voice queries for flight-related information, where the input speech is converted into corresponding textual information by speech recognition, and the understanding of the text is reduced to the extraction of task-specific parameter problems such as destination and departure. The ATIS data include 4978 sentences in the training set from the ATIS-2 and ATIS-3 datasets of Class A data and 893 sentences in the test set from the Nov93 and Dec94 datasets of ATIS-3, where the words in each sentence can be found in a table of named corpora, including domain-specific entities such as city, airline, airport name, and date [23].The ATIS database uses a semantic framework to represent that each sentence has an intention and each word in the sentence has a corresponding semantic slot. The slot values shown in Table2 have not been normalized. There are 128 tags in the ATIS database.Table 2
An example of an ATIS dataset.
SentenceHow much is the cheapest flight from Boston to New York tomorrow morning?ObjectivesAirfareCost relativeCheapestDepart cityBostonArrival cityNew YorkDepart date. RelativeTomorrowDepart time periodMorning


### 4.3. Evaluation Indicators

A more common evaluation metric for sequence annotation units in spoken language comprehension is theF value, and the F value is the reconciled balance of recall and precision.


### 4.4. Experimental Results and Analysis

CRF is the most commonly used method to deal with sequence problems in oral English dialogue comprehension, and it has achieved good results in sequence annotation. The results of the experiments are shown in Table3.Table 3
Experimental results of CRF and RNN.
ModelPrecision (%)Recall (%)F1 (%)CRF93.1592.7392.24Jordan-RNN93.2593.0293.13Elman-RNN94.3694.1394.24LSTM94.6994.494.54GRU95.0394.694.81From Table3, we can see that the F1 values of Elman and Jordan are 0.19% and 1.3% higher in the normal RNN model compared to the CRF model. Therefore, ordinary RNN is more advantageous in solving the problem of sequence annotation for oral English dialogue understanding. The LSTM and GRU variants of the RNN improved by 1.41% and 1.68%, respectively, compared to the Jordan of the normal RNN, and by 0.3% and 0.51% compared to the Elman, with GRU the LSTM and GRU variants. The main reason for this is the inclusion of gate control units in the recurrent network nodes, which makes the network more convenient for controlling history and input information and solves the gradient disappearance problem of ordinary recurrent neural networks.Several variants of RNNs are proposed, and we also perform oral English dialogue comprehension experiments on the ATIS database. The purpose of this series of experiments is to verify which of the different LSTM variants achieves better results in this experiment and to analyze the results of this experiment.From the experimental results in Table4, it can be seen that the experimental results of different long and short-term memory networks on the ATIS dataset are different. From the experimental results, it seems that the LSTM model achieves a 0.48% improvement over the PeepholeLSTM experimental results and a 0.13% improvement relative to the CoupledLSTM model, and it can be seen through these experiments that, for many variants of the LSTM, the authors of [4] compared eight different structural variants of the LSTM model and concluded that none of the variants achieved results that exceeded those of ordinary long and short-term memory networks. In this paper, the two variants of PeepholeLSTM and CoupledLSTM were not introduced into the experimental comparison of the structure of the hidden layer units of deep neural networks for feature fusion. The main three types of structural units, namely, Basic RNN, LSTM, and GRU, are used for training and learning in the deep neural network structure of feature fusion.Table 4
LSTM, PeepholeLSTM, and CoupledLSTM structural model experiments.
LSTM variantPrecision (%)Recall (%)F1 (%)LSTM94.6994.494.54PeepholeLSTM94.1493.9794.06CoupledLSTM94.1894.0494.41In the deep recurrent neural network structure based on feature fusion, the data are processed by changing the recurrent neurons in the network structure, and the experimental results obtained are shown in Table5.Table 5
Deep feature fusion network with different recurrent neurons.
Circulating neuronPrecision (%)Recall (%)F1 (%)Basic95.194.3494.71LSTM95.4894.5795.02GRU95.594.8995.19In comparing the experimental results of the deep recurrent neural network based on feature fusion with the experimental results of the basic RNN and variants of the LSTM and GRU in Table3, the performance of the Elman with feature fusion improved by 0.47% over the Elman neural network structure for the same recurrent neurons, the LSTM with feature fusion improved by 0.51% compared to the LSTM neural network structure, and the GRU with feature fusion improved by 0.38% compared to the GRU. The experimental performance of the GRU with feature fusion was improved by 0.38% compared to the GRU. These experiments show that the deep recurrent neural network model with feature fusion can achieve better performance metrics in oral English dialogue understanding than the normal RNN and its variants. In contrast, among the different recurrent neurons in the feature recurrent neural network structure, the recurrent neurons of the normal RNN were compared with the LSTM and GRU, the recurrent neurons of the variants obtained better performance, and we can compare the performance plots of the normal RNN with those of the feature fusion recurrent neural network by means of a bar chart.The experiments in this paper are built using thumb and keras deep learning libraries in Python, and the number of training epochs for the neural network set in this paper is 50.Table6 shows that the neural network model is a normal RNN model with LSTM and GRU. The results obtained from the first training are higher for the variant than for the two models of the basic RNN. However, when compared with the variants, the LSTM and the gated recurrent network structures yielded better results mainly because these two structures mainly incorporate gating and cellular memory states on the neuron nodes, which can control the reception or rejection of historical information and input information, thus solving the gradient disappearance problem of ordinary RNNs. Due to the large number of parameters contained in the neurons of the LSTM and GRU, the training time is 2-3 times longer than that of a normal RNN model when training and learning, but the performance of the network is obtained. In order to visualize the training and learning process, the performance of the network is described by depicting the change in F1 value, and a line graph is drawn as shown in Figure 3.Table 6
F1 values during training of different network structures.
Epoch0919293949Elman78.1891.3592.6793.593.2994.24Jordan81.4491.2592.3392.7893.0493.13LSTM85.9590.7893.0994.0594.294.54GRU88.7591.8293.7894.494.7294.81Feature Elman80.9591.6392.793.7894.594.71Feature LSTM80.6990.292.5694.1394.6995.02Feature GRU90.392.593.6994.5495.0395.19Figure 3
Training process of different neural network structures.


## 4.1. Experimental Environment

The experimental environment is a 64 bit Linux environment using Ubuntu 16.04.3. Development tools used are Python 2.7.14 in the anaconda environment, and the python deep learning libraries scene and eras are used. Numpy and Scipy packages can be installed directly from the Linux command line using the pipe command. To better handle deep neural network algorithms, think of a deep learning library that can perform fast operations on both CPUs and GPUs and is designed to handle large neural network algorithms. Eras is an application programming interface for high-level neural network applications using thumb, tomorrow, and CNTK as backends. It is written in python, is highly modular in nature, and runs seamlessly on both CPUs and GPUs. This paper experiments with deep neural networks are based on Python2.7 using thumb and keras deep learning libraries.


## 4.2. Data Preparation

This paper applies data information from the Airline Travel Information System (ATIS) domain. This database was developed in the early 1990s as a project of the US Defense Advanced Research Projects Agency (DARPA) in the area of airline travel information. The task consists of voice queries for flight-related information, where the input speech is converted into corresponding textual information by speech recognition, and the understanding of the text is reduced to the extraction of task-specific parameter problems such as destination and departure. The ATIS data include 4978 sentences in the training set from the ATIS-2 and ATIS-3 datasets of Class A data and 893 sentences in the test set from the Nov93 and Dec94 datasets of ATIS-3, where the words in each sentence can be found in a table of named corpora, including domain-specific entities such as city, airline, airport name, and date [23].The ATIS database uses a semantic framework to represent that each sentence has an intention and each word in the sentence has a corresponding semantic slot. The slot values shown in Table2 have not been normalized. There are 128 tags in the ATIS database.Table 2
An example of an ATIS dataset.
SentenceHow much is the cheapest flight from Boston to New York tomorrow morning?ObjectivesAirfareCost relativeCheapestDepart cityBostonArrival cityNew YorkDepart date. RelativeTomorrowDepart time periodMorning


## 4.3. Evaluation Indicators

A more common evaluation metric for sequence annotation units in spoken language comprehension is theF value, and the F value is the reconciled balance of recall and precision.


## 4.4. Experimental Results and Analysis

CRF is the most commonly used method to deal with sequence problems in oral English dialogue comprehension, and it has achieved good results in sequence annotation. The results of the experiments are shown in Table3.Table 3
Experimental results of CRF and RNN.
ModelPrecision (%)Recall (%)F1 (%)CRF93.1592.7392.24Jordan-RNN93.2593.0293.13Elman-RNN94.3694.1394.24LSTM94.6994.494.54GRU95.0394.694.81From Table3, we can see that the F1 values of Elman and Jordan are 0.19% and 1.3% higher in the normal RNN model compared to the CRF model. Therefore, ordinary RNN is more advantageous in solving the problem of sequence annotation for oral English dialogue understanding. The LSTM and GRU variants of the RNN improved by 1.41% and 1.68%, respectively, compared to the Jordan of the normal RNN, and by 0.3% and 0.51% compared to the Elman, with GRU the LSTM and GRU variants. The main reason for this is the inclusion of gate control units in the recurrent network nodes, which makes the network more convenient for controlling history and input information and solves the gradient disappearance problem of ordinary recurrent neural networks.Several variants of RNNs are proposed, and we also perform oral English dialogue comprehension experiments on the ATIS database. The purpose of this series of experiments is to verify which of the different LSTM variants achieves better results in this experiment and to analyze the results of this experiment.From the experimental results in Table4, it can be seen that the experimental results of different long and short-term memory networks on the ATIS dataset are different. From the experimental results, it seems that the LSTM model achieves a 0.48% improvement over the PeepholeLSTM experimental results and a 0.13% improvement relative to the CoupledLSTM model, and it can be seen through these experiments that, for many variants of the LSTM, the authors of [4] compared eight different structural variants of the LSTM model and concluded that none of the variants achieved results that exceeded those of ordinary long and short-term memory networks. In this paper, the two variants of PeepholeLSTM and CoupledLSTM were not introduced into the experimental comparison of the structure of the hidden layer units of deep neural networks for feature fusion. The main three types of structural units, namely, Basic RNN, LSTM, and GRU, are used for training and learning in the deep neural network structure of feature fusion.Table 4
LSTM, PeepholeLSTM, and CoupledLSTM structural model experiments.
LSTM variantPrecision (%)Recall (%)F1 (%)LSTM94.6994.494.54PeepholeLSTM94.1493.9794.06CoupledLSTM94.1894.0494.41In the deep recurrent neural network structure based on feature fusion, the data are processed by changing the recurrent neurons in the network structure, and the experimental results obtained are shown in Table5.Table 5
Deep feature fusion network with different recurrent neurons.
Circulating neuronPrecision (%)Recall (%)F1 (%)Basic95.194.3494.71LSTM95.4894.5795.02GRU95.594.8995.19In comparing the experimental results of the deep recurrent neural network based on feature fusion with the experimental results of the basic RNN and variants of the LSTM and GRU in Table3, the performance of the Elman with feature fusion improved by 0.47% over the Elman neural network structure for the same recurrent neurons, the LSTM with feature fusion improved by 0.51% compared to the LSTM neural network structure, and the GRU with feature fusion improved by 0.38% compared to the GRU. The experimental performance of the GRU with feature fusion was improved by 0.38% compared to the GRU. These experiments show that the deep recurrent neural network model with feature fusion can achieve better performance metrics in oral English dialogue understanding than the normal RNN and its variants. In contrast, among the different recurrent neurons in the feature recurrent neural network structure, the recurrent neurons of the normal RNN were compared with the LSTM and GRU, the recurrent neurons of the variants obtained better performance, and we can compare the performance plots of the normal RNN with those of the feature fusion recurrent neural network by means of a bar chart.The experiments in this paper are built using thumb and keras deep learning libraries in Python, and the number of training epochs for the neural network set in this paper is 50.Table6 shows that the neural network model is a normal RNN model with LSTM and GRU. The results obtained from the first training are higher for the variant than for the two models of the basic RNN. However, when compared with the variants, the LSTM and the gated recurrent network structures yielded better results mainly because these two structures mainly incorporate gating and cellular memory states on the neuron nodes, which can control the reception or rejection of historical information and input information, thus solving the gradient disappearance problem of ordinary RNNs. Due to the large number of parameters contained in the neurons of the LSTM and GRU, the training time is 2-3 times longer than that of a normal RNN model when training and learning, but the performance of the network is obtained. In order to visualize the training and learning process, the performance of the network is described by depicting the change in F1 value, and a line graph is drawn as shown in Figure 3.Table 6
F1 values during training of different network structures.
Epoch0919293949Elman78.1891.3592.6793.593.2994.24Jordan81.4491.2592.3392.7893.0493.13LSTM85.9590.7893.0994.0594.294.54GRU88.7591.8293.7894.494.7294.81Feature Elman80.9591.6392.793.7894.594.71Feature LSTM80.6990.292.5694.1394.6995.02Feature GRU90.392.593.6994.5495.0395.19Figure 3
Training process of different neural network structures.


## 5. Conclusions

The current technique of combining deep learning models with external knowledge bases is widely admired by researchers in the field of NLP. In task-based human-machine dialogue systems, external knowledge bases can provide the machine with valid information beyond the training data to help the model better perform the oral English dialogue comprehension task. Firstly, we propose a candidate knowledge recall rule to extract highly relevant knowledge sets from external knowledge bases; secondly, we propose a “word” related knowledge attention module to assign relevance weights to the knowledge in candidate knowledge sets to obtain the most effective knowledge vectors; and finally, we introduce a pretrained language model based on the idea of transfer learning. Finally, a pretrained language model BERT is introduced to replace Word2Vec as the word vector support for the model based on the idea of transfer learning. Experiments show that the model proposed in this paper has better results on the restructured ATIS dataset.



---

*Source: 8005321-2022-03-14.xml*