# Construction of the Open Oral Evaluation Model Based on the Neural Network

Authors: Zhixin Chen; Xu Zhang; Zhiyuan Li; Anchu Li
Journal: Scientific Programming
 (2021)

---

## Abstract

According to the problem of low efficiency and low scoring accuracy of the traditional oral language scoring system, this study builds an open oral language evaluation model based on the basic principles of deep learning technology. Firstly, the basic methods of the convolutional neural network (CNN) and long short-term memory (LSTM) neural network are introduced. Then, we combine the convolutional neural network (CNN) and long short-term memory (LSTM) neural network to design an open oral scoring model based on CNN + LSTM, which divides the oral evaluation model into the speech scoring model and text scoring model and makes a specific implementation of two scoring models, respectively. An experimental environment is then built to preprocess the data, and finally, the model built in this study is trained and simulated. The experimental results show that the CNN + LSTM network evaluation model has a better comprehensive scoring performance, higher scoring efficiency, and higher accuracy and has feasibility and practicability.


---

## Body

## 1. Introduction

The rapid development of the internet information level has promoted the continuous progress of education. In traditional oral English learning, teachers mostly adopt the manual correction method to students’ oral test and training results, which is tedious, time-consuming, and inefficient and is not conducive to the long-term development of students’ oral English learning. In recent years, computer-assisted learning system has been widely used in the field of education, and more and more students and teachers in China begin to use computer-assisted oral learning. However, the traditional computer-assisted oral learning system has some shortcomings in the open oral evaluation, such as the lack of intelligence and low performance. It is an inevitable trend for the development of education to design and implement an intelligent and open oral scoring system.How to improve the performance and efficiency of the open oral evaluation model is still the focus of many scholars. According to the demand of diagnostic oral English proficiency assessment, this paper puts forward corresponding solutions and strategies for the demand of the oral English assessment model. Based on the GESE oral test method, a formative hierarchical evaluation model of oral English teaching in higher vocational colleges is constructed, and its positive “backwash effect” law is obtained. Many of the research tasks proposed in [1] focus on English, and it is difficult to explore a richer variety of languages within the framework of the SLU task. Minor languages need to be updated in the training process of the SLU model. This problem is explained from three aspects: the input of the neural network, the comparison between the French version and the best setting in different ways, and the comparison with the most advanced methods. Tsai et al. [2] aimed at the use of mobile phones, mailboxes, and other tools in daily life, which are realized by recurrent neural networks (RNNs). The character language model based on LSTM can not only capture the Boltzmann statistics of the system but also reproduce the dynamics. It can be applied to capture the time evolution of typical trajectories in chemistry and biophysics. An acoustic speech method for SLD proposed in [3] is a new method based on the original SLD. It uses an attention-based neural network model to capture the language and a Gaussian smoothing method to locate language changes. It is more effective in dealing with code-switching of monolingual segments, but its performance will be affected by the duration of monolingual segments, which needs further study. Peng [4] proposed that learning assessment can let students know whether they have achieved the ideal academic achievements and goals through computer English teaching, so as to further improve their shortcomings. Based on the neural network and artificial intelligence technology, this model can realize this evaluation. It is based on hearing, introduces wavelet entropy features, and applies it to the adaptive model. Combined with the control experiment, the performance of the model is analyzed. Mathematical statistics can directly show the effect. Research proves that the model can meet the expectations. With the deepening development of higher education, English teaching has been paid more and more attention. While improving the quality of education [5], learning evaluation is the most fundamental measure. By studying the existing problems and characteristics of the learning evaluation system, analyzing the defects of the traditional system, then summing up the shortcomings, and developing the advantages, this paper puts forward a college English teaching quality evaluation system based on information fusion and optimized RBF neural network decision-making algorithm. Jiang et al. [6] proposed an improved fuzzy RBF neural network model based on backpropagation learning, which combines a large number of problems in English teaching before, such as large quantity and complexity, and then optimizes with the development limitations and existing shortcomings of neural networks. This model is an objective reference to a university teaching case and puts forward an improved quality evaluation method, which is subjective and random, so that the evaluation results are more in line with the actual situation. Mohammed et al. [7] used Arabic, English, and French to evaluate the recognition performance of linear predictive coding (LPC) and/or mel-frequency cepstrum coefficient (MFCC) and artificial neural network (ANN) and also tested LPC and MFCC, hidden layer, different neurons in the hidden layer, and different transfer functions to illustrate their usability. Li [8] proposed a teaching evaluation model based on the improved BP neural network, which is helpful to improve English teaching quality. It summarizes and analyzes several teaching elements, designs the evaluation system, and improves the shortcomings in calculation by improving nonmonotone linear search and adaptive step change. Finally, through practical verification, the model can better evaluate English teaching and improve the effect. Hermanto et al. [9] used two models, namely, recurrent neural network (RNN) and statistical network, using the n-gram model. Compared with the statistical language model, the neural language model achieves better results in the field of machine translation. The latter two are deficient in accuracy and calculation speed, while the RNN has improved its evaluation scores in these aspects. Esan et al. [10] proposed a machine translation method based on the recurrent neural network model. By testing the model with manual and automatic evaluation techniques, combined with real manual evaluation, it shows that the model is smooth and usable, basically consistent with human judgment, and relevant and has high translation effect and accuracy.However, the above research has not solved the intelligent problem of open oral learning, and the research direction needs to be further studied. The above literature research also puts forward many oral evaluation methods and has achieved good results. However, many solutions are based on machine learning or algorithms, which have low effect, do not realize intelligent processing, and cannot meet the requirements of oral evaluation. Although there are some neural network solutions, the translation efficiency and accuracy are relatively low, and this paper combined with the CNN + LSTM model can solve such problems. Based on this, this study combines the learning characteristics of the neural network, according to the basic principles and structural characteristics of the convolution neural network (CNN) and long short-term memory (LSTM) neural network, and designs and implements an open oral scoring model based on CNN + LSTM. The final results show that this model can improve the intelligent level of open oral learning, and the evaluation model has relatively high scoring efficiency and accuracy, which further shows that the evaluation model is feasible and practical.This paper proposes that the neural network can solve the translation problem from the perspective of intelligent analysis of oral English, which can improve the accuracy of oral English and build a standard evaluation model. The CNN-LSTM model proposed in this paper has the highest evaluation accuracy, which can reach 82%, and the effect is the best. The key points will be to score the oral pronunciation and oral content separately, and finally, the total score after summation will be taken as the final score result of the examinee. When constructing the scoring model, this paper uses different neural networks to implement and test. Finally, the model with the highest relevance of the man-machine score will be applied to the actual correction task to help teachers reduce work pressure. At the same time, it can also be used as a tool for students’ oral self-evaluation.


## 2. Basic Method of the Neural Network

### 2.1. Convolution Neural Network (CNN)

#### 2.1.1. Fundamentals of the CNN

Convolutional neural network (CNN) is a representative algorithm of deep learning [11]. It is a feedforward neural network with convolution calculation and depth structure, which is very suitable for processing one-dimensional time-domain sequence data and image data. Compared with other neural networks, CNN can learn the original data efficiently and quickly, so as to extract the specific features of the data; that is, it has the ability of representation learning. CNN is widely used in computer vision, natural language processing, and other fields. The network has four remarkable characteristics, which are local perception, weight sharing (convolution operation), pooling processing, and multiconvolution kernel operation [12]. Among them, convolution operation is an operation that defines two integrable functions for convolution operation. It mainly includes two operation modes, namely, continuous convolution operation and discrete convolution operation.Let bothfx and gx belong to integrable functions in the real number field, and the new function Jx=f∗gx is the convolution of functions fx and gx, in which f∗gx=g∗fx and f∗gx belong to integrable functions.The continuous convolution operation is expressed as(1)Jx=f∗gx=∫_fagx−ada.The expression of the discrete convolution operation is(2)Jt=f∗gt=∑n=−∞∞fngt−n.Among them, the convolution operation mode in the CNN is discrete convolution, which can easily deal with discrete data problems in actual measurement.


#### 2.1.2. Central Neural Network Structure

As shown in Figure1, the network structure of the convolution neural network (CNN) is mainly divided into five network layers, namely, input layer, convolution layer, pooling layer, full connection layer, and output layer, belonging to multilayer perceptron (MLP) [13]. The following five layers are described in detail.Figure 1
Central neural network structure.(1) Input Layer. The input layer of the CNN can process multidimensional data. Like other neural network algorithms for deep learning, CNN’s learning method is calculated by the gradient descent algorithm, and the input data features of the CNN need to be standardized. Standardizing the features of the input data can greatly improve the learning efficiency of the CNN.(2) Convolution Layer. Convolution layer, pooling layer, and full connection layer are all hidden layers in the CNN. The convolution layer has the function of feature extraction from the input data. There are many convolution kernels in the convolution layer, and the composition of convolution kernels is similar to that of neurons in the feedforward neural network [14], such as(3)Zl+1i,j=Zl⊗ωl+1i,j+b=∑k=1Kl∑x=1f∑y=1fZkls0i+x,s0j+yωkl+1,i,j∈0,1,…,Ll+1Ll+1=Ll+2p−fs0+1.In formula (3), the summation part is equivalent to solving a cross-correlation, B represents the deviation, Zl and Zl+1 represent the input and output of the first layer, respectively, which can also be called the feature map, Ll+1 represents the size of Zl+1, if the length and width of the feature map are the same, Zi,j represents the pixels of the feature map, f,s0,p represents the number of channels of the feature map, and GG is the parameters of the convolution layer, which are convolution kernel size, convolution step size, and filling layer, respectively.Under special circumstances, if the convolution kernel sizef=1 and convolution step size S0=1 do not include filling, the cross-correlation calculation in the convolution layer is similar to matrix multiplication, thus constructing a fully connected network between convolution layers, as shown in the following:(4)Zl+1=∑k=1Kl∑i=1L∑j=1LZi,j,klωkl+1+b=ωl+1TZl+1+b,Ll+1=L.(3) Pooling Layer. After the convolution layer extracts the features, the pooling layer selects the features and filters the information. The pooled area selected by the pooled layer is similar to the step of scanning the feature map by the convolution kernel and is also determined by the pooled size, the pooled step size, and the number of filled layers [15]. Among them, the pooling layer is divided into Lp pooling and random pooling or mixed pooling. Lp pooling is mainly inspired by the hierarchical structure in the visual cortex, and its expression is as follows:(5)Akli,j=∑x=1f∑y=1fAklS0i+x,S0j+yp1/p.In equation (5), S0 is the pooling step size, i,j is the pixel, and p is the prespecified parameter. If p=1 and Lp pooling takes the average value in the pooling area, it is mean pooling; if p⟶∞ and Lp pooling takes the maximum value in the region, it is maximum pooling. These two pooling methods are the most commonly used methods in the CNN.Hybrid pooling and random pooling are mainly extended from the concept of Lp pooling. Random pooling is randomly selected according to a specific probability in the pooled area, which can ensure that a part of nonmaximum excitation signals enter the next construction. Hybrid pooling can represent a linear combination of mean pooling and maximum pooling, as shown in the following:(6)Akl=λL1Akl+L∞Akl;λ∈0,1.It is found that hybrid pooling and random pooling have a regularization function compared with mean pooling and maximum pooling, which can avoid overfitting of the CNN.(4) Fully Connected Layer. The fully connected layer belongs to the hidden layer in the traditional feedforward neural network. It is mainly located at the end of the hidden layer of the CNN, which only transmits signals to other fully connected layers. The main function of the full connection layer is to combine the extracted data features nonlinearly and output them.(5) Output Layer. The upper part of the output layer is mostly a fully connected layer, and its structure and working principle are the same as those of the output layer in the traditional feedforward neural network. The output layer only outputs the image classification label through the logic function or normalized exponential function. The output layer recognizes objects according to the center coordinates, size, and classification of output objects. The output layer directly outputs the classification results of each pixel in the image to complete the semantic segmentation of the image.


### 2.2. Long Short-Term Memory (LSTM) Neural Network

#### 2.2.1. Fundamentals of LSTM

Long short-term memory (LSTM) is a kind of time cycle neural network, which is designed to solve the long-term dependence of the RNN of the cycle neural network [16]. The main working mechanisms are mainly divided into the gating mechanism, forgetting mechanism, and circular memory mechanism. The three working mechanisms can effectively guarantee the normal operation of LSTM and make it play its long-term and circular memory functions, thus avoiding gradient disappearance and explosion. The following three mechanisms are analyzed in detail. LSTM output dimension setting is 300, return_sequences parameter is set to true, and the activation parameter is set to sigmoid.(1) Gating Mechanism. In the LSTM network model, the key element of the gating mechanism is the activation function, which is usually sigmoid function and tanh function. S function is a commonly used activation function in neural networks. The characteristic of this function is that it can control the output real value within the range of [0, 1], as shown in the following:(7)σx=11+e−x.Tanh function, that is, hyperbolic tangent function: this function can control the output real value in the range of [−1, 1], with 0 bit as the center, which is just complementary to the S function without 0 as the center. The function is shown in the following:(8)tanhx=ex−e−xex+e−x.The gating mechanism can control the storage, memory, and update of information, and the basic principle is shown in formulas (9) and (10):(9)fx=σWx+b,(10)σx=11+e−x.(2) Forgetting Mechanism. In the LSTM network model, forgetting mechanism must be realized based on the gating mechanism. By analyzing the influence of historical information on the memory unit, it is determined whether to retain or forget some information [17].(3) Cyclic Memory Mechanism. Among them, in the LSTM network model, a new state ct, namely, memory unit, is added in the LSTM network model, and the main function of this memory unit is to transmit information circularly.


#### 2.2.2. Remote Access Service Network Architecture

As shown in Figure2, LSTM network structure is mainly divided into four parts: input gate, forgetting gate, memory unit, and output gate. The basic principles and methods of the four parts are analyzed in detail.Figure 2
LSTM network architecture.(1) Input Gate. The input gate mainly controls the filtering input data through the sigmoid function, wherein the input is mainly divided into two inputs, namely, the data directly input are controlled as shown in equation (11), and the other control input is stored in the candidate memory unit as shown in equation (12):(11)Zi=σ∑jKi,jxji+∑jWi,jhji−1+bi,(12)Z=tanh∑jKi,jxji+∑jWi,jhji−1+b.In the above formula,xji represents the input data at time t, Ki,j represents the input weight, Wi,j represents the cycle weight of the forgetting gate, hji−1 represents the output value of LSTM cells at the previous time, bi represents the bias of the input gate, and b represents the bias of candidate memory cells.(2) Forgetting the Door. Forgetting gate Zf mainly uses the sigmoid function to map the input data xji at time t in the range of [0, 1]. When the value is 0, it means that all information cannot pass this gate; when the value is 1, it means that all information is allowed to pass, as in the following:(13)Zf=σ∑jKi,jfxfj+∑jWi,jfhji−1+bf.In equation (13), xji represents the data input at time t; Ki,jf represents the input weight; Wi,jf represents the cyclic weight of the forgetting gate; hji−1 represents the cell output value of LSTM at last time; bf stands for the offset of the forgetting gate [18].(3) Memory Unit. The memory cell Ci is cyclically updated mainly through the candidate memory cell Z and the memory cell Ci−1 at the previous time and is adjusted by the input gate Zi and the forgetting gate Zf, as shown in the following:(14)Ci=Zf·Ci−1+Zi·Z.(4) Output Gate. The output gate Zo mainly controls whether the memory unit Ci outputs through the sigmoid function. When the value is 0, it means that all information cannot pass this gate [19]; when the value is 1, it means that all information is allowed to pass, as shown in equation (15). The final output value Hi of LSTM cells is controlled by the tanh activation function, as shown in equation (16):(15)Zo=σ∑iKi,joxji+∑jWi,johji−1+bo,(16)hi=Zo·tanhCi.In the above formula,xji represents the data input at time t; Ki,jo represents the input weight; Wi,jo represents the cyclic weight of the output gate; hji−1 represents the cell output value of LSTM at last time; bo represents the offset of the output gate.


## 2.1. Convolution Neural Network (CNN)

### 2.1.1. Fundamentals of the CNN

Convolutional neural network (CNN) is a representative algorithm of deep learning [11]. It is a feedforward neural network with convolution calculation and depth structure, which is very suitable for processing one-dimensional time-domain sequence data and image data. Compared with other neural networks, CNN can learn the original data efficiently and quickly, so as to extract the specific features of the data; that is, it has the ability of representation learning. CNN is widely used in computer vision, natural language processing, and other fields. The network has four remarkable characteristics, which are local perception, weight sharing (convolution operation), pooling processing, and multiconvolution kernel operation [12]. Among them, convolution operation is an operation that defines two integrable functions for convolution operation. It mainly includes two operation modes, namely, continuous convolution operation and discrete convolution operation.Let bothfx and gx belong to integrable functions in the real number field, and the new function Jx=f∗gx is the convolution of functions fx and gx, in which f∗gx=g∗fx and f∗gx belong to integrable functions.The continuous convolution operation is expressed as(1)Jx=f∗gx=∫_fagx−ada.The expression of the discrete convolution operation is(2)Jt=f∗gt=∑n=−∞∞fngt−n.Among them, the convolution operation mode in the CNN is discrete convolution, which can easily deal with discrete data problems in actual measurement.


### 2.1.2. Central Neural Network Structure

As shown in Figure1, the network structure of the convolution neural network (CNN) is mainly divided into five network layers, namely, input layer, convolution layer, pooling layer, full connection layer, and output layer, belonging to multilayer perceptron (MLP) [13]. The following five layers are described in detail.Figure 1
Central neural network structure.(1) Input Layer. The input layer of the CNN can process multidimensional data. Like other neural network algorithms for deep learning, CNN’s learning method is calculated by the gradient descent algorithm, and the input data features of the CNN need to be standardized. Standardizing the features of the input data can greatly improve the learning efficiency of the CNN.(2) Convolution Layer. Convolution layer, pooling layer, and full connection layer are all hidden layers in the CNN. The convolution layer has the function of feature extraction from the input data. There are many convolution kernels in the convolution layer, and the composition of convolution kernels is similar to that of neurons in the feedforward neural network [14], such as(3)Zl+1i,j=Zl⊗ωl+1i,j+b=∑k=1Kl∑x=1f∑y=1fZkls0i+x,s0j+yωkl+1,i,j∈0,1,…,Ll+1Ll+1=Ll+2p−fs0+1.In formula (3), the summation part is equivalent to solving a cross-correlation, B represents the deviation, Zl and Zl+1 represent the input and output of the first layer, respectively, which can also be called the feature map, Ll+1 represents the size of Zl+1, if the length and width of the feature map are the same, Zi,j represents the pixels of the feature map, f,s0,p represents the number of channels of the feature map, and GG is the parameters of the convolution layer, which are convolution kernel size, convolution step size, and filling layer, respectively.Under special circumstances, if the convolution kernel sizef=1 and convolution step size S0=1 do not include filling, the cross-correlation calculation in the convolution layer is similar to matrix multiplication, thus constructing a fully connected network between convolution layers, as shown in the following:(4)Zl+1=∑k=1Kl∑i=1L∑j=1LZi,j,klωkl+1+b=ωl+1TZl+1+b,Ll+1=L.(3) Pooling Layer. After the convolution layer extracts the features, the pooling layer selects the features and filters the information. The pooled area selected by the pooled layer is similar to the step of scanning the feature map by the convolution kernel and is also determined by the pooled size, the pooled step size, and the number of filled layers [15]. Among them, the pooling layer is divided into Lp pooling and random pooling or mixed pooling. Lp pooling is mainly inspired by the hierarchical structure in the visual cortex, and its expression is as follows:(5)Akli,j=∑x=1f∑y=1fAklS0i+x,S0j+yp1/p.In equation (5), S0 is the pooling step size, i,j is the pixel, and p is the prespecified parameter. If p=1 and Lp pooling takes the average value in the pooling area, it is mean pooling; if p⟶∞ and Lp pooling takes the maximum value in the region, it is maximum pooling. These two pooling methods are the most commonly used methods in the CNN.Hybrid pooling and random pooling are mainly extended from the concept of Lp pooling. Random pooling is randomly selected according to a specific probability in the pooled area, which can ensure that a part of nonmaximum excitation signals enter the next construction. Hybrid pooling can represent a linear combination of mean pooling and maximum pooling, as shown in the following:(6)Akl=λL1Akl+L∞Akl;λ∈0,1.It is found that hybrid pooling and random pooling have a regularization function compared with mean pooling and maximum pooling, which can avoid overfitting of the CNN.(4) Fully Connected Layer. The fully connected layer belongs to the hidden layer in the traditional feedforward neural network. It is mainly located at the end of the hidden layer of the CNN, which only transmits signals to other fully connected layers. The main function of the full connection layer is to combine the extracted data features nonlinearly and output them.(5) Output Layer. The upper part of the output layer is mostly a fully connected layer, and its structure and working principle are the same as those of the output layer in the traditional feedforward neural network. The output layer only outputs the image classification label through the logic function or normalized exponential function. The output layer recognizes objects according to the center coordinates, size, and classification of output objects. The output layer directly outputs the classification results of each pixel in the image to complete the semantic segmentation of the image.


## 2.1.1. Fundamentals of the CNN

Convolutional neural network (CNN) is a representative algorithm of deep learning [11]. It is a feedforward neural network with convolution calculation and depth structure, which is very suitable for processing one-dimensional time-domain sequence data and image data. Compared with other neural networks, CNN can learn the original data efficiently and quickly, so as to extract the specific features of the data; that is, it has the ability of representation learning. CNN is widely used in computer vision, natural language processing, and other fields. The network has four remarkable characteristics, which are local perception, weight sharing (convolution operation), pooling processing, and multiconvolution kernel operation [12]. Among them, convolution operation is an operation that defines two integrable functions for convolution operation. It mainly includes two operation modes, namely, continuous convolution operation and discrete convolution operation.Let bothfx and gx belong to integrable functions in the real number field, and the new function Jx=f∗gx is the convolution of functions fx and gx, in which f∗gx=g∗fx and f∗gx belong to integrable functions.The continuous convolution operation is expressed as(1)Jx=f∗gx=∫_fagx−ada.The expression of the discrete convolution operation is(2)Jt=f∗gt=∑n=−∞∞fngt−n.Among them, the convolution operation mode in the CNN is discrete convolution, which can easily deal with discrete data problems in actual measurement.


## 2.1.2. Central Neural Network Structure

As shown in Figure1, the network structure of the convolution neural network (CNN) is mainly divided into five network layers, namely, input layer, convolution layer, pooling layer, full connection layer, and output layer, belonging to multilayer perceptron (MLP) [13]. The following five layers are described in detail.Figure 1
Central neural network structure.(1) Input Layer. The input layer of the CNN can process multidimensional data. Like other neural network algorithms for deep learning, CNN’s learning method is calculated by the gradient descent algorithm, and the input data features of the CNN need to be standardized. Standardizing the features of the input data can greatly improve the learning efficiency of the CNN.(2) Convolution Layer. Convolution layer, pooling layer, and full connection layer are all hidden layers in the CNN. The convolution layer has the function of feature extraction from the input data. There are many convolution kernels in the convolution layer, and the composition of convolution kernels is similar to that of neurons in the feedforward neural network [14], such as(3)Zl+1i,j=Zl⊗ωl+1i,j+b=∑k=1Kl∑x=1f∑y=1fZkls0i+x,s0j+yωkl+1,i,j∈0,1,…,Ll+1Ll+1=Ll+2p−fs0+1.In formula (3), the summation part is equivalent to solving a cross-correlation, B represents the deviation, Zl and Zl+1 represent the input and output of the first layer, respectively, which can also be called the feature map, Ll+1 represents the size of Zl+1, if the length and width of the feature map are the same, Zi,j represents the pixels of the feature map, f,s0,p represents the number of channels of the feature map, and GG is the parameters of the convolution layer, which are convolution kernel size, convolution step size, and filling layer, respectively.Under special circumstances, if the convolution kernel sizef=1 and convolution step size S0=1 do not include filling, the cross-correlation calculation in the convolution layer is similar to matrix multiplication, thus constructing a fully connected network between convolution layers, as shown in the following:(4)Zl+1=∑k=1Kl∑i=1L∑j=1LZi,j,klωkl+1+b=ωl+1TZl+1+b,Ll+1=L.(3) Pooling Layer. After the convolution layer extracts the features, the pooling layer selects the features and filters the information. The pooled area selected by the pooled layer is similar to the step of scanning the feature map by the convolution kernel and is also determined by the pooled size, the pooled step size, and the number of filled layers [15]. Among them, the pooling layer is divided into Lp pooling and random pooling or mixed pooling. Lp pooling is mainly inspired by the hierarchical structure in the visual cortex, and its expression is as follows:(5)Akli,j=∑x=1f∑y=1fAklS0i+x,S0j+yp1/p.In equation (5), S0 is the pooling step size, i,j is the pixel, and p is the prespecified parameter. If p=1 and Lp pooling takes the average value in the pooling area, it is mean pooling; if p⟶∞ and Lp pooling takes the maximum value in the region, it is maximum pooling. These two pooling methods are the most commonly used methods in the CNN.Hybrid pooling and random pooling are mainly extended from the concept of Lp pooling. Random pooling is randomly selected according to a specific probability in the pooled area, which can ensure that a part of nonmaximum excitation signals enter the next construction. Hybrid pooling can represent a linear combination of mean pooling and maximum pooling, as shown in the following:(6)Akl=λL1Akl+L∞Akl;λ∈0,1.It is found that hybrid pooling and random pooling have a regularization function compared with mean pooling and maximum pooling, which can avoid overfitting of the CNN.(4) Fully Connected Layer. The fully connected layer belongs to the hidden layer in the traditional feedforward neural network. It is mainly located at the end of the hidden layer of the CNN, which only transmits signals to other fully connected layers. The main function of the full connection layer is to combine the extracted data features nonlinearly and output them.(5) Output Layer. The upper part of the output layer is mostly a fully connected layer, and its structure and working principle are the same as those of the output layer in the traditional feedforward neural network. The output layer only outputs the image classification label through the logic function or normalized exponential function. The output layer recognizes objects according to the center coordinates, size, and classification of output objects. The output layer directly outputs the classification results of each pixel in the image to complete the semantic segmentation of the image.


## 2.2. Long Short-Term Memory (LSTM) Neural Network

### 2.2.1. Fundamentals of LSTM

Long short-term memory (LSTM) is a kind of time cycle neural network, which is designed to solve the long-term dependence of the RNN of the cycle neural network [16]. The main working mechanisms are mainly divided into the gating mechanism, forgetting mechanism, and circular memory mechanism. The three working mechanisms can effectively guarantee the normal operation of LSTM and make it play its long-term and circular memory functions, thus avoiding gradient disappearance and explosion. The following three mechanisms are analyzed in detail. LSTM output dimension setting is 300, return_sequences parameter is set to true, and the activation parameter is set to sigmoid.(1) Gating Mechanism. In the LSTM network model, the key element of the gating mechanism is the activation function, which is usually sigmoid function and tanh function. S function is a commonly used activation function in neural networks. The characteristic of this function is that it can control the output real value within the range of [0, 1], as shown in the following:(7)σx=11+e−x.Tanh function, that is, hyperbolic tangent function: this function can control the output real value in the range of [−1, 1], with 0 bit as the center, which is just complementary to the S function without 0 as the center. The function is shown in the following:(8)tanhx=ex−e−xex+e−x.The gating mechanism can control the storage, memory, and update of information, and the basic principle is shown in formulas (9) and (10):(9)fx=σWx+b,(10)σx=11+e−x.(2) Forgetting Mechanism. In the LSTM network model, forgetting mechanism must be realized based on the gating mechanism. By analyzing the influence of historical information on the memory unit, it is determined whether to retain or forget some information [17].(3) Cyclic Memory Mechanism. Among them, in the LSTM network model, a new state ct, namely, memory unit, is added in the LSTM network model, and the main function of this memory unit is to transmit information circularly.


### 2.2.2. Remote Access Service Network Architecture

As shown in Figure2, LSTM network structure is mainly divided into four parts: input gate, forgetting gate, memory unit, and output gate. The basic principles and methods of the four parts are analyzed in detail.Figure 2
LSTM network architecture.(1) Input Gate. The input gate mainly controls the filtering input data through the sigmoid function, wherein the input is mainly divided into two inputs, namely, the data directly input are controlled as shown in equation (11), and the other control input is stored in the candidate memory unit as shown in equation (12):(11)Zi=σ∑jKi,jxji+∑jWi,jhji−1+bi,(12)Z=tanh∑jKi,jxji+∑jWi,jhji−1+b.In the above formula,xji represents the input data at time t, Ki,j represents the input weight, Wi,j represents the cycle weight of the forgetting gate, hji−1 represents the output value of LSTM cells at the previous time, bi represents the bias of the input gate, and b represents the bias of candidate memory cells.(2) Forgetting the Door. Forgetting gate Zf mainly uses the sigmoid function to map the input data xji at time t in the range of [0, 1]. When the value is 0, it means that all information cannot pass this gate; when the value is 1, it means that all information is allowed to pass, as in the following:(13)Zf=σ∑jKi,jfxfj+∑jWi,jfhji−1+bf.In equation (13), xji represents the data input at time t; Ki,jf represents the input weight; Wi,jf represents the cyclic weight of the forgetting gate; hji−1 represents the cell output value of LSTM at last time; bf stands for the offset of the forgetting gate [18].(3) Memory Unit. The memory cell Ci is cyclically updated mainly through the candidate memory cell Z and the memory cell Ci−1 at the previous time and is adjusted by the input gate Zi and the forgetting gate Zf, as shown in the following:(14)Ci=Zf·Ci−1+Zi·Z.(4) Output Gate. The output gate Zo mainly controls whether the memory unit Ci outputs through the sigmoid function. When the value is 0, it means that all information cannot pass this gate [19]; when the value is 1, it means that all information is allowed to pass, as shown in equation (15). The final output value Hi of LSTM cells is controlled by the tanh activation function, as shown in equation (16):(15)Zo=σ∑iKi,joxji+∑jWi,johji−1+bo,(16)hi=Zo·tanhCi.In the above formula,xji represents the data input at time t; Ki,jo represents the input weight; Wi,jo represents the cyclic weight of the output gate; hji−1 represents the cell output value of LSTM at last time; bo represents the offset of the output gate.


## 2.2.1. Fundamentals of LSTM

Long short-term memory (LSTM) is a kind of time cycle neural network, which is designed to solve the long-term dependence of the RNN of the cycle neural network [16]. The main working mechanisms are mainly divided into the gating mechanism, forgetting mechanism, and circular memory mechanism. The three working mechanisms can effectively guarantee the normal operation of LSTM and make it play its long-term and circular memory functions, thus avoiding gradient disappearance and explosion. The following three mechanisms are analyzed in detail. LSTM output dimension setting is 300, return_sequences parameter is set to true, and the activation parameter is set to sigmoid.(1) Gating Mechanism. In the LSTM network model, the key element of the gating mechanism is the activation function, which is usually sigmoid function and tanh function. S function is a commonly used activation function in neural networks. The characteristic of this function is that it can control the output real value within the range of [0, 1], as shown in the following:(7)σx=11+e−x.Tanh function, that is, hyperbolic tangent function: this function can control the output real value in the range of [−1, 1], with 0 bit as the center, which is just complementary to the S function without 0 as the center. The function is shown in the following:(8)tanhx=ex−e−xex+e−x.The gating mechanism can control the storage, memory, and update of information, and the basic principle is shown in formulas (9) and (10):(9)fx=σWx+b,(10)σx=11+e−x.(2) Forgetting Mechanism. In the LSTM network model, forgetting mechanism must be realized based on the gating mechanism. By analyzing the influence of historical information on the memory unit, it is determined whether to retain or forget some information [17].(3) Cyclic Memory Mechanism. Among them, in the LSTM network model, a new state ct, namely, memory unit, is added in the LSTM network model, and the main function of this memory unit is to transmit information circularly.


## 2.2.2. Remote Access Service Network Architecture

As shown in Figure2, LSTM network structure is mainly divided into four parts: input gate, forgetting gate, memory unit, and output gate. The basic principles and methods of the four parts are analyzed in detail.Figure 2
LSTM network architecture.(1) Input Gate. The input gate mainly controls the filtering input data through the sigmoid function, wherein the input is mainly divided into two inputs, namely, the data directly input are controlled as shown in equation (11), and the other control input is stored in the candidate memory unit as shown in equation (12):(11)Zi=σ∑jKi,jxji+∑jWi,jhji−1+bi,(12)Z=tanh∑jKi,jxji+∑jWi,jhji−1+b.In the above formula,xji represents the input data at time t, Ki,j represents the input weight, Wi,j represents the cycle weight of the forgetting gate, hji−1 represents the output value of LSTM cells at the previous time, bi represents the bias of the input gate, and b represents the bias of candidate memory cells.(2) Forgetting the Door. Forgetting gate Zf mainly uses the sigmoid function to map the input data xji at time t in the range of [0, 1]. When the value is 0, it means that all information cannot pass this gate; when the value is 1, it means that all information is allowed to pass, as in the following:(13)Zf=σ∑jKi,jfxfj+∑jWi,jfhji−1+bf.In equation (13), xji represents the data input at time t; Ki,jf represents the input weight; Wi,jf represents the cyclic weight of the forgetting gate; hji−1 represents the cell output value of LSTM at last time; bf stands for the offset of the forgetting gate [18].(3) Memory Unit. The memory cell Ci is cyclically updated mainly through the candidate memory cell Z and the memory cell Ci−1 at the previous time and is adjusted by the input gate Zi and the forgetting gate Zf, as shown in the following:(14)Ci=Zf·Ci−1+Zi·Z.(4) Output Gate. The output gate Zo mainly controls whether the memory unit Ci outputs through the sigmoid function. When the value is 0, it means that all information cannot pass this gate [19]; when the value is 1, it means that all information is allowed to pass, as shown in equation (15). The final output value Hi of LSTM cells is controlled by the tanh activation function, as shown in equation (16):(15)Zo=σ∑iKi,joxji+∑jWi,johji−1+bo,(16)hi=Zo·tanhCi.In the above formula,xji represents the data input at time t; Ki,jo represents the input weight; Wi,jo represents the cyclic weight of the output gate; hji−1 represents the cell output value of LSTM at last time; bo represents the offset of the output gate.


## 3. Construction of the Open Oral Scoring Model Based on CNN + LSTM

Combined with the basic principles and network structure of the CNN and LSTM neural network models, this study proposes an open oral scoring model based on the CNN + LSTM neural network. Because the model data trained by the traditional machine learning model and BP neural network model have low accurate fitting degree, the correlation between the manual feature extraction and manual score is low. This study breaks through the limitations of artificial extraction through the one-dimensional convolution neural network and long short-term memory network to integrate and process data more effectively, combined with the CNN + LSTM neural network to build the speech scoring model and text scoring model. Among them, the cyclic neural model has a large computational workload when dealing with lengthy sequence data. In this study, before the LSTM neural network, the one-dimensional convolution neural network is used to preprocess the data to shorten the sequence data, so as to improve the computational efficiency, extract more accurate features, and input them to the LSTM layer for processing.As shown in Figure3, the designed speech scoring model is mainly composed of MFCC feature vectors; 2 convolution blocks; bidirectional LSTM layer, namely, BLSTM, MeanOverTime layer, and full connection layer. The specific steps are as follows:(1)
Input the MFCC eigenvector firstly(2)
The filters in convolution block 1 and convolution block 2 convolve the input MFCC feature vectors in a translation way, and after the calculation is completed, the data are pooled by using the maximum pooling layer in the convolution block [20](3)
After the data are processed by the convolution layer, they are input to the bidirectional LSTM network in the lower layer, that is, BLSTM, and the speech features are better extracted and fitted by the bidirectional LSTM network(4)
After feature extraction and fitting, the MeanOverTime layer is used to sum and average the corresponding positions ofN vectors with equal lengths output by the upper LSTM network, and finally, a one-dimensional vector is output(5)
Finally, the one-dimensional vector output by the MeanOverTime layer is fitted through the full connection layer, so as to output the corresponding speech score resultsFigure 3
Speech scoring model based on CNN + LSTM.As shown in Figure4, the text scoring model based on CNN + LSTM is mainly composed of five layers: word embedding layer, one-dimensional convolution layer, LSTM layer, MeanOverTime layer, and full connection layer. The main function of the MeanOverTime layer is to sum and average the corresponding positions of N equal length vectors (including all intermediate state outputs of the LSTM network) output by the LSTM layer and finally output a one-dimensional vector. Finally, the full connection layer combines the one-dimensional vector and outputs the corresponding speech scoring results.Figure 4
Text scoring model based on CNN + LSTM.The specific steps of text scoring design are as follows:(1)
Firstly, the word embedding matrix based on the GloVe model is designed, and the dimension size of the word embedding layer is set to 50. If words cannot be found in the matrix, the word embedding layer vector is set to 0.(2)
The main function of the one-dimensional convolution layer is to shorten the length of the network input sequence and better extract speech features.(3)
The convolution layer is followed by the LSTM layer, which uses the function of selecting “memory” and “forget” information of the LSTM network to better extract and fit text features.(4)
MeanOverTime layer in the text scoring model has the same function as the MeanOverTime layer in the speech scoring model.(5)
Finally, the one-dimensional vector output by the MeanOverTime layer is fitted by the full connection layer, so as to output the corresponding text scoring results.


## 4. Experimental Verification

### 4.1. Environmental Construction

To verify the feasibility of the speech scoring model and text scoring model based on the CNN + LSTM network, the experimental data are the recordings of 600 students’ open oral test in a university in Hebei province. The recording files are all in the MP3 form, and the audio attributes are 16 bits and 16 kHz sampling rates. The spoken pronunciation and content are scored by manual scoring. Before training and testing, the FFmpeg tool is used to convert the MP3 format of the recording to the PCM format. Finally, the dataset is divided into two parts: one is the training dataset and the other is the test dataset. Among them, the training set is 480 pieces of data, and the test set is 120 pieces of data.In the above, this study constructs a speech scoring model and a text scoring model based on the CNN + LSTM network. For these two models, this study uses the RMSProp optimizer and mean square error loss function to train. The training iteration times are set to epoch 50, the batch value is set to 10, and the learning rates are set to 0.001 and 0.01, respectively. On the contrary, because the activation function is used in the output layer of the two scoring models, only the scoring results in the range of 0–1 can be obtained. Therefore, this study needs to normalize the manual scoring results before training the two models. The normalization process is shown in the following:(17)x∗=x−xminxmax−xmin.


### 4.2. Data Preprocessing

In the scoring model based on CNN + LSTM constructed in this study, the working form of the scoring model is to transform spoken recording and speech recognition text into a numerical vector representation. In this study, too many datasets are selected, which will lead to jumbled experimental process and increase the difficulty of the experiment. Therefore, it is necessary to preprocess the data in order to achieve better experimental results.


#### 4.2.1. Data Cleaning

In the above dataset, due to the nonstandard spoken language of most students and the recognition defects of the speech recognition system itself, there are large errors in feature extraction of the tested speech recognition text, and the accuracy rate is relatively low, thus affecting the final recognition results. Therefore, this experiment cleans the experimental data, filters and removes some useless spoken words, which make the speech text structure clearer, and increases the effectiveness of the generated model. In this paper, adaptive filter (LMS) is used for data cleaning. Voice data formats include MP3 format, VOC format, and Au format. In the system, it is expressed as binary format coding. The system samples audio data in KB/s.


#### 4.2.2. Feature Extraction

Feature extraction is very important in the scoring model, and the reliability and accuracy of the scoring model are determined by feature extraction. In the neural network scoring model based on CNN + LSTM constructed in this study, two kinds of features, speech class and text class, are mainly extracted. Specific feature extraction is shown in Table1. Speech class feature extraction is directly based on the speech signal, and text class feature extraction is based on the output of the speech recognition engine.Table 1
Summary of the feature extraction of the speech class and text class.
Feature categoryName to be signedBrief description of characteristicsPhonetic classArticulation rateSpeed of speechNum silenceNumber of voice pausesPosterior scorePostpronunciation verification probability scoreSpeaking ratioRatio of effective speaking time to total recording timeText classContent lengthTotal number of words in the textUnique wordsNumber of nonrepeating words in the textSyntactic tree depthSum of all syntactic tree depths in the textSemantic similaritySemantic similarity between the text and topicGood grammar ratioCorrect rate of text grammarAccording to the summary of feature extraction categories in Table1, this study selects four phonetic features to evaluate students’ oral pronunciation quality, fluency, and content richness. Among them, the specific function of the rate of speech (ROS) is to describe oral fluency, and the expression is(18)ROS=Nwordst−ts.In equation (18), Nwords is the total number of words in students’ spoken English, T is the total duration of oral recording, and ts is the mute duration in recording.In this study, five text features are selected to evaluate the oral content of candidates. Text feature extraction can better reflect the richness of the oral content and effectively deal with special data. In text feature extraction, grammar is regarded as the standard to judge students’ oral English. Through part-of-speech tags, we can judge whether there are grammatical problems in the spoken content in vocabulary. The row label is shown in Table2.Table 2
Part-of-speech tags.
Part-of-speech tagsDescriptionNNNoun (singular)NNSNoun (plural)VBBy bar (prototype)VBDVerb (past tense)VUNMoving river (past participle)JJAdjectivesRBAdverbINSubordinate conjunctionsCCConjunctionsPRPPersonal pronounIn the text extraction of speech recognition, grammatical errors exist in sentences and are difficult to be found. In this study, the correct rate of text grammar is calculated by(19)correct_ratio=NgNs.In equation (19), NS represents the total number of ternary or quaternary tag combinations in the text, and Ng represents the total number of correct tag combinations.


#### 4.2.3. Data Conversion

In this study, a pretrained word embedding model is used to transform the speech recognition text into a vector representation, as shown in Figure5.Figure 5
Text vector flow.For spoken speech recording data, the extracted mel-frequency cepstrum coefficient (MFCC) is used as the input of the speech scoring model, as shown in Figure6.Figure 6
MFCC feature extraction flow.


### 4.3. Empirical Results

#### 4.3.1. Indicators for Evaluating System Performance

In the scoring system constructed in this study, the speech and text scoring model is based on the CNN + LSTM neural network. For the comprehensive evaluation and comparison of the scoring performance of the two models, this study sets up three evaluation indicators to quantitatively analyze the experimental results.(1)
Pearson’s correlation coefficient is the most widely used performance test index in the field of automatic scoring. The main function of this coefficient is to reflect the linear correlation between two sequences, as shown in the following:(20)ρX,Y=covX,YσXσY.In formula (20), covX,Y is denoted as the oblique variance of X and Y, and σX and σY are both standard deviations. X is the system score, Y is the teacher score, and the range of ρX,Y is [−1, 1]. If the result is positive, it indicates that X and Y are positively correlated, and if it is negative, it indicates that X and Y are negatively correlated. The closer the absolute value of this coefficient is to 1, the higher the correlation between X and Y.(2)
Average difference of the man-machine score: it mainly describes the deviation degree between the machine score and manual score. The indicator is represented by the letterd, as shown in the following:(21)d=ESMachine−SHuman.(3)
Accuracy: this experiment uses the method of the maximum error of man-machine scoring to judge whether the scoring result is accurate or not. If the difference between the manual score and machine score is less than the maximum error, it indicates that the score result is accurate.Finally, the ratio of the number of accurate scoring results in the test set to the total number of samples is calculated, which is used as the accuracy index of the scoring model. In practical applications, the error of man-machine scoring is less than 1 point, which shows that the machine prediction result is accurate.


#### 4.3.2. Analysis and Evaluation of the Scoring Model

In the CNN + LSTM neural network model, 9 features are extracted as the input of speech and text scoring models, and Pearson’s correlation coefficients between each feature and manual scoring are calculated, respectively. The calculated results are shown in Tables3 and 4.Table 3
Correlation between phonetic features and manual scores.
Phonetic featuresPearson’s correlation coefficientArticulation rate0.37Num silence0.44Posterior score0.31Speaking ratio0.42Table 4
Correlation between text class features and manual scoring.
Text class featurePearson’s correlation coefficientContent length0.57Unique words0.59Syntactic tree depth0.27Semantic similarity0.33Good grammar ratio0.24It can be seen from Tables3 and 4 that the Pearson correlation coefficients of num silence and speaking ratio are 0.44 and 0.42, respectively, which are highly correlated with manual scores. It shows that teachers pay more attention to students’ oral fluency and effective oral duration when grading oral English. The Pearson correlation coefficients of content length and unique words are 0.57 and 0.59, respectively, which have high correlation with manual scoring. It shows that teachers pay more attention to students’ mastery of vocabulary and the richness of the oral content.


#### 4.3.3. Test Results

As shown in Figures7 and 8, 120 test data were selected to test the scoring models of the BP neural network and CNN + LSTM neural network, and the performance of these two scoring models was comprehensively evaluated through the three evaluation indexes set above.Figure 7
Scoring results of the BP scoring model.Figure 8
Scoring results of the CNN + LSTM neural network scoring model.It can be seen from Figures7 and 8 that the CNN + LSTM neural network scoring model has a better fitting effect than the BP neural network scoring model, and the CNN + LSTM neural network scoring model has stronger adaptability. In Figures 7 and 8, the dotted line represents the standard actual test sample data. In the BP scoring model in Figure 7, it is quite different from the standard sample data, indicating that the evaluation effect is poor. In Figure 8, the difference between the CNN + LSTM model and the standard sample data is small, indicating that the evaluation effect is good.It can be seen from Table5 that the CNN + LSTM neural network model is better than the BP neural network model in the Pearson correlation coefficient and accuracy, and the difference between them in man-machine score correlation is obvious. CNN + LSTM is slightly better than the BP model in the average difference index.Table 5
Performance evaluation of the scoring model.
Pearson’s correlationAverage differenceAccuracy (%)Manual scoring model0.7640.485—CNN + LSTM model0.6940.63982.3BP model0.5330.60180.0


## 4.1. Environmental Construction

To verify the feasibility of the speech scoring model and text scoring model based on the CNN + LSTM network, the experimental data are the recordings of 600 students’ open oral test in a university in Hebei province. The recording files are all in the MP3 form, and the audio attributes are 16 bits and 16 kHz sampling rates. The spoken pronunciation and content are scored by manual scoring. Before training and testing, the FFmpeg tool is used to convert the MP3 format of the recording to the PCM format. Finally, the dataset is divided into two parts: one is the training dataset and the other is the test dataset. Among them, the training set is 480 pieces of data, and the test set is 120 pieces of data.In the above, this study constructs a speech scoring model and a text scoring model based on the CNN + LSTM network. For these two models, this study uses the RMSProp optimizer and mean square error loss function to train. The training iteration times are set to epoch 50, the batch value is set to 10, and the learning rates are set to 0.001 and 0.01, respectively. On the contrary, because the activation function is used in the output layer of the two scoring models, only the scoring results in the range of 0–1 can be obtained. Therefore, this study needs to normalize the manual scoring results before training the two models. The normalization process is shown in the following:(17)x∗=x−xminxmax−xmin.


## 4.2. Data Preprocessing

In the scoring model based on CNN + LSTM constructed in this study, the working form of the scoring model is to transform spoken recording and speech recognition text into a numerical vector representation. In this study, too many datasets are selected, which will lead to jumbled experimental process and increase the difficulty of the experiment. Therefore, it is necessary to preprocess the data in order to achieve better experimental results.


### 4.2.1. Data Cleaning

In the above dataset, due to the nonstandard spoken language of most students and the recognition defects of the speech recognition system itself, there are large errors in feature extraction of the tested speech recognition text, and the accuracy rate is relatively low, thus affecting the final recognition results. Therefore, this experiment cleans the experimental data, filters and removes some useless spoken words, which make the speech text structure clearer, and increases the effectiveness of the generated model. In this paper, adaptive filter (LMS) is used for data cleaning. Voice data formats include MP3 format, VOC format, and Au format. In the system, it is expressed as binary format coding. The system samples audio data in KB/s.


### 4.2.2. Feature Extraction

Feature extraction is very important in the scoring model, and the reliability and accuracy of the scoring model are determined by feature extraction. In the neural network scoring model based on CNN + LSTM constructed in this study, two kinds of features, speech class and text class, are mainly extracted. Specific feature extraction is shown in Table1. Speech class feature extraction is directly based on the speech signal, and text class feature extraction is based on the output of the speech recognition engine.Table 1
Summary of the feature extraction of the speech class and text class.
Feature categoryName to be signedBrief description of characteristicsPhonetic classArticulation rateSpeed of speechNum silenceNumber of voice pausesPosterior scorePostpronunciation verification probability scoreSpeaking ratioRatio of effective speaking time to total recording timeText classContent lengthTotal number of words in the textUnique wordsNumber of nonrepeating words in the textSyntactic tree depthSum of all syntactic tree depths in the textSemantic similaritySemantic similarity between the text and topicGood grammar ratioCorrect rate of text grammarAccording to the summary of feature extraction categories in Table1, this study selects four phonetic features to evaluate students’ oral pronunciation quality, fluency, and content richness. Among them, the specific function of the rate of speech (ROS) is to describe oral fluency, and the expression is(18)ROS=Nwordst−ts.In equation (18), Nwords is the total number of words in students’ spoken English, T is the total duration of oral recording, and ts is the mute duration in recording.In this study, five text features are selected to evaluate the oral content of candidates. Text feature extraction can better reflect the richness of the oral content and effectively deal with special data. In text feature extraction, grammar is regarded as the standard to judge students’ oral English. Through part-of-speech tags, we can judge whether there are grammatical problems in the spoken content in vocabulary. The row label is shown in Table2.Table 2
Part-of-speech tags.
Part-of-speech tagsDescriptionNNNoun (singular)NNSNoun (plural)VBBy bar (prototype)VBDVerb (past tense)VUNMoving river (past participle)JJAdjectivesRBAdverbINSubordinate conjunctionsCCConjunctionsPRPPersonal pronounIn the text extraction of speech recognition, grammatical errors exist in sentences and are difficult to be found. In this study, the correct rate of text grammar is calculated by(19)correct_ratio=NgNs.In equation (19), NS represents the total number of ternary or quaternary tag combinations in the text, and Ng represents the total number of correct tag combinations.


### 4.2.3. Data Conversion

In this study, a pretrained word embedding model is used to transform the speech recognition text into a vector representation, as shown in Figure5.Figure 5
Text vector flow.For spoken speech recording data, the extracted mel-frequency cepstrum coefficient (MFCC) is used as the input of the speech scoring model, as shown in Figure6.Figure 6
MFCC feature extraction flow.


## 4.2.1. Data Cleaning

In the above dataset, due to the nonstandard spoken language of most students and the recognition defects of the speech recognition system itself, there are large errors in feature extraction of the tested speech recognition text, and the accuracy rate is relatively low, thus affecting the final recognition results. Therefore, this experiment cleans the experimental data, filters and removes some useless spoken words, which make the speech text structure clearer, and increases the effectiveness of the generated model. In this paper, adaptive filter (LMS) is used for data cleaning. Voice data formats include MP3 format, VOC format, and Au format. In the system, it is expressed as binary format coding. The system samples audio data in KB/s.


## 4.2.2. Feature Extraction

Feature extraction is very important in the scoring model, and the reliability and accuracy of the scoring model are determined by feature extraction. In the neural network scoring model based on CNN + LSTM constructed in this study, two kinds of features, speech class and text class, are mainly extracted. Specific feature extraction is shown in Table1. Speech class feature extraction is directly based on the speech signal, and text class feature extraction is based on the output of the speech recognition engine.Table 1
Summary of the feature extraction of the speech class and text class.
Feature categoryName to be signedBrief description of characteristicsPhonetic classArticulation rateSpeed of speechNum silenceNumber of voice pausesPosterior scorePostpronunciation verification probability scoreSpeaking ratioRatio of effective speaking time to total recording timeText classContent lengthTotal number of words in the textUnique wordsNumber of nonrepeating words in the textSyntactic tree depthSum of all syntactic tree depths in the textSemantic similaritySemantic similarity between the text and topicGood grammar ratioCorrect rate of text grammarAccording to the summary of feature extraction categories in Table1, this study selects four phonetic features to evaluate students’ oral pronunciation quality, fluency, and content richness. Among them, the specific function of the rate of speech (ROS) is to describe oral fluency, and the expression is(18)ROS=Nwordst−ts.In equation (18), Nwords is the total number of words in students’ spoken English, T is the total duration of oral recording, and ts is the mute duration in recording.In this study, five text features are selected to evaluate the oral content of candidates. Text feature extraction can better reflect the richness of the oral content and effectively deal with special data. In text feature extraction, grammar is regarded as the standard to judge students’ oral English. Through part-of-speech tags, we can judge whether there are grammatical problems in the spoken content in vocabulary. The row label is shown in Table2.Table 2
Part-of-speech tags.
Part-of-speech tagsDescriptionNNNoun (singular)NNSNoun (plural)VBBy bar (prototype)VBDVerb (past tense)VUNMoving river (past participle)JJAdjectivesRBAdverbINSubordinate conjunctionsCCConjunctionsPRPPersonal pronounIn the text extraction of speech recognition, grammatical errors exist in sentences and are difficult to be found. In this study, the correct rate of text grammar is calculated by(19)correct_ratio=NgNs.In equation (19), NS represents the total number of ternary or quaternary tag combinations in the text, and Ng represents the total number of correct tag combinations.


## 4.2.3. Data Conversion

In this study, a pretrained word embedding model is used to transform the speech recognition text into a vector representation, as shown in Figure5.Figure 5
Text vector flow.For spoken speech recording data, the extracted mel-frequency cepstrum coefficient (MFCC) is used as the input of the speech scoring model, as shown in Figure6.Figure 6
MFCC feature extraction flow.


## 4.3. Empirical Results

### 4.3.1. Indicators for Evaluating System Performance

In the scoring system constructed in this study, the speech and text scoring model is based on the CNN + LSTM neural network. For the comprehensive evaluation and comparison of the scoring performance of the two models, this study sets up three evaluation indicators to quantitatively analyze the experimental results.(1)
Pearson’s correlation coefficient is the most widely used performance test index in the field of automatic scoring. The main function of this coefficient is to reflect the linear correlation between two sequences, as shown in the following:(20)ρX,Y=covX,YσXσY.In formula (20), covX,Y is denoted as the oblique variance of X and Y, and σX and σY are both standard deviations. X is the system score, Y is the teacher score, and the range of ρX,Y is [−1, 1]. If the result is positive, it indicates that X and Y are positively correlated, and if it is negative, it indicates that X and Y are negatively correlated. The closer the absolute value of this coefficient is to 1, the higher the correlation between X and Y.(2)
Average difference of the man-machine score: it mainly describes the deviation degree between the machine score and manual score. The indicator is represented by the letterd, as shown in the following:(21)d=ESMachine−SHuman.(3)
Accuracy: this experiment uses the method of the maximum error of man-machine scoring to judge whether the scoring result is accurate or not. If the difference between the manual score and machine score is less than the maximum error, it indicates that the score result is accurate.Finally, the ratio of the number of accurate scoring results in the test set to the total number of samples is calculated, which is used as the accuracy index of the scoring model. In practical applications, the error of man-machine scoring is less than 1 point, which shows that the machine prediction result is accurate.


### 4.3.2. Analysis and Evaluation of the Scoring Model

In the CNN + LSTM neural network model, 9 features are extracted as the input of speech and text scoring models, and Pearson’s correlation coefficients between each feature and manual scoring are calculated, respectively. The calculated results are shown in Tables3 and 4.Table 3
Correlation between phonetic features and manual scores.
Phonetic featuresPearson’s correlation coefficientArticulation rate0.37Num silence0.44Posterior score0.31Speaking ratio0.42Table 4
Correlation between text class features and manual scoring.
Text class featurePearson’s correlation coefficientContent length0.57Unique words0.59Syntactic tree depth0.27Semantic similarity0.33Good grammar ratio0.24It can be seen from Tables3 and 4 that the Pearson correlation coefficients of num silence and speaking ratio are 0.44 and 0.42, respectively, which are highly correlated with manual scores. It shows that teachers pay more attention to students’ oral fluency and effective oral duration when grading oral English. The Pearson correlation coefficients of content length and unique words are 0.57 and 0.59, respectively, which have high correlation with manual scoring. It shows that teachers pay more attention to students’ mastery of vocabulary and the richness of the oral content.


### 4.3.3. Test Results

As shown in Figures7 and 8, 120 test data were selected to test the scoring models of the BP neural network and CNN + LSTM neural network, and the performance of these two scoring models was comprehensively evaluated through the three evaluation indexes set above.Figure 7
Scoring results of the BP scoring model.Figure 8
Scoring results of the CNN + LSTM neural network scoring model.It can be seen from Figures7 and 8 that the CNN + LSTM neural network scoring model has a better fitting effect than the BP neural network scoring model, and the CNN + LSTM neural network scoring model has stronger adaptability. In Figures 7 and 8, the dotted line represents the standard actual test sample data. In the BP scoring model in Figure 7, it is quite different from the standard sample data, indicating that the evaluation effect is poor. In Figure 8, the difference between the CNN + LSTM model and the standard sample data is small, indicating that the evaluation effect is good.It can be seen from Table5 that the CNN + LSTM neural network model is better than the BP neural network model in the Pearson correlation coefficient and accuracy, and the difference between them in man-machine score correlation is obvious. CNN + LSTM is slightly better than the BP model in the average difference index.Table 5
Performance evaluation of the scoring model.
Pearson’s correlationAverage differenceAccuracy (%)Manual scoring model0.7640.485—CNN + LSTM model0.6940.63982.3BP model0.5330.60180.0


## 4.3.1. Indicators for Evaluating System Performance

In the scoring system constructed in this study, the speech and text scoring model is based on the CNN + LSTM neural network. For the comprehensive evaluation and comparison of the scoring performance of the two models, this study sets up three evaluation indicators to quantitatively analyze the experimental results.(1)
Pearson’s correlation coefficient is the most widely used performance test index in the field of automatic scoring. The main function of this coefficient is to reflect the linear correlation between two sequences, as shown in the following:(20)ρX,Y=covX,YσXσY.In formula (20), covX,Y is denoted as the oblique variance of X and Y, and σX and σY are both standard deviations. X is the system score, Y is the teacher score, and the range of ρX,Y is [−1, 1]. If the result is positive, it indicates that X and Y are positively correlated, and if it is negative, it indicates that X and Y are negatively correlated. The closer the absolute value of this coefficient is to 1, the higher the correlation between X and Y.(2)
Average difference of the man-machine score: it mainly describes the deviation degree between the machine score and manual score. The indicator is represented by the letterd, as shown in the following:(21)d=ESMachine−SHuman.(3)
Accuracy: this experiment uses the method of the maximum error of man-machine scoring to judge whether the scoring result is accurate or not. If the difference between the manual score and machine score is less than the maximum error, it indicates that the score result is accurate.Finally, the ratio of the number of accurate scoring results in the test set to the total number of samples is calculated, which is used as the accuracy index of the scoring model. In practical applications, the error of man-machine scoring is less than 1 point, which shows that the machine prediction result is accurate.


## 4.3.2. Analysis and Evaluation of the Scoring Model

In the CNN + LSTM neural network model, 9 features are extracted as the input of speech and text scoring models, and Pearson’s correlation coefficients between each feature and manual scoring are calculated, respectively. The calculated results are shown in Tables3 and 4.Table 3
Correlation between phonetic features and manual scores.
Phonetic featuresPearson’s correlation coefficientArticulation rate0.37Num silence0.44Posterior score0.31Speaking ratio0.42Table 4
Correlation between text class features and manual scoring.
Text class featurePearson’s correlation coefficientContent length0.57Unique words0.59Syntactic tree depth0.27Semantic similarity0.33Good grammar ratio0.24It can be seen from Tables3 and 4 that the Pearson correlation coefficients of num silence and speaking ratio are 0.44 and 0.42, respectively, which are highly correlated with manual scores. It shows that teachers pay more attention to students’ oral fluency and effective oral duration when grading oral English. The Pearson correlation coefficients of content length and unique words are 0.57 and 0.59, respectively, which have high correlation with manual scoring. It shows that teachers pay more attention to students’ mastery of vocabulary and the richness of the oral content.


## 4.3.3. Test Results

As shown in Figures7 and 8, 120 test data were selected to test the scoring models of the BP neural network and CNN + LSTM neural network, and the performance of these two scoring models was comprehensively evaluated through the three evaluation indexes set above.Figure 7
Scoring results of the BP scoring model.Figure 8
Scoring results of the CNN + LSTM neural network scoring model.It can be seen from Figures7 and 8 that the CNN + LSTM neural network scoring model has a better fitting effect than the BP neural network scoring model, and the CNN + LSTM neural network scoring model has stronger adaptability. In Figures 7 and 8, the dotted line represents the standard actual test sample data. In the BP scoring model in Figure 7, it is quite different from the standard sample data, indicating that the evaluation effect is poor. In Figure 8, the difference between the CNN + LSTM model and the standard sample data is small, indicating that the evaluation effect is good.It can be seen from Table5 that the CNN + LSTM neural network model is better than the BP neural network model in the Pearson correlation coefficient and accuracy, and the difference between them in man-machine score correlation is obvious. CNN + LSTM is slightly better than the BP model in the average difference index.Table 5
Performance evaluation of the scoring model.
Pearson’s correlationAverage differenceAccuracy (%)Manual scoring model0.7640.485—CNN + LSTM model0.6940.63982.3BP model0.5330.60180.0


## 5. Conclusion

To sum up, the open oral English scoring model based on CNN + LSTM designed in this study has better fitting effect and adaptability and can improve the intelligent level of open oral English learning. Compared with the traditional artificial scoring and BP neural network scoring model, the CNN + LSTM neural network scoring model has higher scoring accuracy and efficiency. Experimental results show that the CNN + LSTM neural network model has better scoring performance and has certain practicability when the training dataset is small.



---

*Source: 3928246-2021-09-22.xml*