# Automatic Scoring of Spoken English Based Principal Component Analysis and Convolutional Neural Network

Authors: Liu Chongya
Journal: Wireless Communications and Mobile Computing
 (2022)

---

## Abstract

Using computers to help people practice spoken language is a common method, but there are currently some problems. Firstly, because fluency feature is calculated depending on expert knowledge, the key information contained in the original data set may be lost. Secondly, optimize each model’s parameters separately to make the model’s performance in sub-optimal state. In order to solve these problems, a spoken English fluency scoring method based on convolutional neural network is proposed, in order to make the feature extraction consider the short-, medium-, and long-term characteristics of speech signal; three convolution layers are superimposed in this paper, which jointly learns feature extraction and scoring models from the original time-domain signal input. In the feature extraction process, we applied principal component analysis to make useful data extraction of audio features. The experimental results show that the scoring results of the proposed method are more accurate.


---

## Body

## 1. Introduction

Computer-aided instruction (CAI) is a variety of teaching activities carried out under the aid of computer. It discusses teaching content with students in a dialogue way and carries out teaching training. CAI can provide students with a good personalized learning environment and comprehensively use multimedia, artificial intelligence, educational data analysis, and other computer technologies. CAI can shorten the learning time and improve the quality of teaching, to achieve the optimization of teaching objectives. In recent years, computer-assisted language learning, as a method for non-native English students to improve their oral English ability, has attracted wide attention. To give valuable mentoring input and further develop scoring productivity, a computer-based automatic assessment model is needed to assess the quality of non-native English pronunciation, fluency, and specific errors they are prone to.English Language Learners (ELL) are students who are microeducated to learn English, usually from non-English-speaking families or backgrounds. With the acceleration of world integration, mastering English has become a necessary skill for people. Therefore, with the increase of the number of English learners, educators are confronted with the test of how to more readily meet the advancing necessities of these understudies, since a few English students experience issues in utilizing and getting English, which may hinder their academic performance, especially in the English teaching for children. This paper presents a neural network-based aid method for children’s English. Past work on programmed evaluation of youngsters’ oral perusing familiarity has included scoring systems at essay level or word level. A progression of open-finished talking assignments was utilized to evaluate style as well as understanding and utilization of language content. ASR of kids’ open discourse is more difficult than that of grown-up discourse, somewhat on account of the huge recurrence range and time changeability of youngsters’ discourse, which makes acoustic displaying more troublesome. In the ELL data in particular, stressed or mispronounced words, as well as students’ uncertainty about the task, often lead to hesitation, oral noise, and unclear pronunciation, which are additional challenges for the proposed assessment system.Most automatic scoring systems for spoken English fluency mainly include the part of the ASR, the part of the fluency feature extraction, and the part of the scoring model. ASR creates time-adjusted groupings of words for input speech. In the process of extracting fluency features, this paper selects features which are highly related to oral English fluency, and various characteristics of the scoring system are studied, among which long silence time, silence time, and phone duration are normal characteristics of English fluency [1]. Due to the different contributions of each feature to the final score, the feature selection technique used can only select few generally significant elements.Recently, computer science based teaching frameworks are already to be the hot exploration themes in the domains of software engineering and schooling [2]. In particular, in huge scope language assessments, they have started to step by step supplant instructors in checking and have turned into a significant change in the schooling area, that is called CALL frameworks [3]. Numerous computer science appraisal frameworks have been utilized for an enormous scope in real education and assessments, for example, English piece checking frameworks and computer science program stamping frameworks [4]. Such frameworks are more precise in evaluating, and, above all, they save HR and further develop effectiveness [5]. Nonetheless, some exam questions in many subjects like spoken are also scored by teachers currently [6]. There are numerous oral reviewing frameworks for perusing and once again referring to; however, there are not many evaluating frameworks for open-finished oral inquiries, for example, tests, redundancies, and individual reactions. Ordinate and speech specialist scoring frameworks are perceived as commonplace instances of robotized scoring; nonetheless, they do not address the issue for open-finished talking questions [7]. Lately, with the advancement of discourse acknowledgment innovation and the development of exposition checking frameworks, it is normal that in fact talking the programmed stamping of open-finished talking questions can be survived and arrives at a down to earth level [8]. There are some genuine situations wherein it is important to assess a speaker’s oral articulation capacity, for example, Mandarin tests, oral preparation, language showing assessments, and radio moderator tests [9]. As of now, these situations are as yet assessed by manual scoring, for example, averaging, casting a ballot, or one-vote casting a ballot, which are excessively emotional, regularly need reasonableness, and do not give objective input to the speaker [10]. Traditional oral assessment is inefficient, and for oral speech, there are many hidden problems exist in oral expression; the existence of these problems to a certain extent can lead to the learner’s learning efficiency [11]; for this, a method based on spectral density values in time domain and the distribution of the cognitive domain of oral evaluation model was put forward, and achieved good results [12]; in addition, some scholars use deep learning to evaluate the quality of spoken language [13], and it has been verified through experiments that the model based on deep learning is more effective than the traditional model [14]. Then, a combined evaluation system based on deep learning model and HMM model was established [15]. In addition, some scholars evaluated spoken language through tree-based models [16], and scholars found that when the size of the training data set was large enough, the output results of these models were almost the same [17]. At the same time, scholars proposed an automatic evaluation algorithm for oral English without subjective factors, and verified its effectiveness through experimental data [18]. Some other scholars constructed a new LSA model to analyze oral English quality evaluation [19]. The main points of contention for the model are the proper determination of explicit texts to be physically stamped and the general estimation of increased impacts [20]. The nature of text increase is estimated by increasing the text until the impact arrives at a limit [21]. As far as text choice, there are three choices: irregular choice, grouping, or choosing moderately comparative text for the labeled information. The benefit of Klein’s strategy is that the text that should be physically labeled can be chosen consequently with at least exertion. The shortcomings of the technique are apparent in the assessment of its effectiveness, which must be accomplished after the manual finishing of the total explanation of understudy reactions. One more issue is that the boundaries in the technique are autonomous; the semantic space aspect and the likeness limit boundary should be determined all the while.


## 2. Methodology

Oral English quality scoring system is described in Figure1; first of all, the input of the original speech signal is converted to a series of characteristics, and then to training evaluation model based on the expert experience and ASR system, final output score; in this article, we use PCA to process the raw input data, then using CNN to training characteristics, and finally it is concluded a more effective model.Figure 1
Oral English quality scoring system.


### 2.1. Principal Component Analysis

PCA is a multivariate statistical analysis method, which is widely used in data dimension reduction, signal processing, pattern recognition, and other fields [22]. On the premise of ensuring the accuracy of calculation, principal component analysis can obviously reduce the cost of calculation, eliminate the redundancy, and extract the main features of the data. Let sample set X=x1,x2,⋯xN. Based on the principle of maximum separability, the projection of sample point xi to the hyperplane in the new space is UTXI, where U is the projection matrix.The variance of sample pointxi behind the projection point is UTxixiTU. If the projection of all sample points is separated as far as possible, the variance of sample points after projection should be maximized. Therefore, the optimization objective can be expressed as follows:
(1)maxUtrUTXXTU,s.t.UTU=I.XXT is the divergence matrix; I is the identity matrix of order n. Using the Lagrange multiplier method for Equation (1):
(2)XXTU=λU,where λ is the divergence matrix. In order to obtain the maximum variance of the projected sample points, the divergence matrix needs to be decomposed by eigenvalues, and the obtained eigenvalues are sorted as follows: λ1≥λ2≥⋯≥λk, where the largest eigenvalue λ1 corresponds to the first major component, and so on, we can obtain the second major component, the third major component, etc. The number of principal components k can be obtained by the method of cumulative variance contribution rate. The contribution rate of the i principal component Qi and the cumulative contribution Qcv of the first m principal components are, respectively, defined as:
(3)Qi=λi/∑n=1kλni=1,2,⋯,kQcv=∑i=1mλi/∑n=1kλnm<n.


### 2.2. Convolutional Neural Network

In this part, the structure of CNN model, such as convolutional layer, pooling layer, full connection layer, and batch normalization, are briefly reviewed [23]. The schematic diagram of the whole CNN model is shown in Figure 2.Figure 2
The structure of the CNN model.


#### 2.2.1. Convolution Layer

Convolution layer is one of the important components of neural network, and its function is to extract the characteristics of the data input from the input layer through convolution calculation. In the process of each convolution movement, the characteristic valueH at this point will be calculated by summing the channel, and the specific calculation formula is shown in equation (4).
(4)hj,k=σ∑s=1Swjs∗xks+bj,k,where σ∙ is ReLU∙ nonlinearity; ∗ stands for linear convolution; bj,k is the deviation.


#### 2.2.2. Pooling Layer

The function of the pooling layer is to select and filter the feature information extracted from the convolutional layer, so as to prevent excessive data from affecting the running speed of the network and remove the interference of noise. Features shrink in size as they pass through the pooling layer.


#### 2.2.3. Full Connection Layer

The function of the full connection layer is to integrate the features obtained in the previous steps and transform the input feature vectorz by applying affine transformation, as shown below:
(5)hi=σWi×z+bi,where Wi and bi are the i-th weight vector and deviation, respectively. σ∙ is ReLU∙ nonlinearity and make use of softmax∙ to get the probability of class s of xt in the last layer.
(6)psxt=softmaxxt=expwsyL∑n=1NexpwsyL.


#### 2.2.4. Batch Normalization

Unlike layer normalization, which is horizontal normalization, batch normalization is vertical normalization. The goal of normalization is to make each neural unit in the same layer in the direction of batch have different mean and variance to prevent the generation of gradient explosion.


### 2.3. Proposed Method

#### 2.3.1. Automatic Scoring of Spoken English (ASSE)

The ASSE is used to give the proposed scorey to unknown quality x and is shown in Formula (7).
(7)fx;θ=y.


#### 2.3.2. Scoring Model Based on CNN

In this work, it is recommended to employ the following characteristics and method parametersθ, and x (N-dimensional integer vector represents the original time-domain speech segment x∈zN).In order to make the feature extraction consider the short-, medium-, and long-term characteristics of speech signal, up to three convolution layers can be superimposed. As shown in Figure3, the softmax layer maps the probability of the output to the desired fraction.Figure 3
The structure of the proposed method.


## 2.1. Principal Component Analysis

PCA is a multivariate statistical analysis method, which is widely used in data dimension reduction, signal processing, pattern recognition, and other fields [22]. On the premise of ensuring the accuracy of calculation, principal component analysis can obviously reduce the cost of calculation, eliminate the redundancy, and extract the main features of the data. Let sample set X=x1,x2,⋯xN. Based on the principle of maximum separability, the projection of sample point xi to the hyperplane in the new space is UTXI, where U is the projection matrix.The variance of sample pointxi behind the projection point is UTxixiTU. If the projection of all sample points is separated as far as possible, the variance of sample points after projection should be maximized. Therefore, the optimization objective can be expressed as follows:
(1)maxUtrUTXXTU,s.t.UTU=I.XXT is the divergence matrix; I is the identity matrix of order n. Using the Lagrange multiplier method for Equation (1):
(2)XXTU=λU,where λ is the divergence matrix. In order to obtain the maximum variance of the projected sample points, the divergence matrix needs to be decomposed by eigenvalues, and the obtained eigenvalues are sorted as follows: λ1≥λ2≥⋯≥λk, where the largest eigenvalue λ1 corresponds to the first major component, and so on, we can obtain the second major component, the third major component, etc. The number of principal components k can be obtained by the method of cumulative variance contribution rate. The contribution rate of the i principal component Qi and the cumulative contribution Qcv of the first m principal components are, respectively, defined as:
(3)Qi=λi/∑n=1kλni=1,2,⋯,kQcv=∑i=1mλi/∑n=1kλnm<n.


## 2.2. Convolutional Neural Network

In this part, the structure of CNN model, such as convolutional layer, pooling layer, full connection layer, and batch normalization, are briefly reviewed [23]. The schematic diagram of the whole CNN model is shown in Figure 2.Figure 2
The structure of the CNN model.


### 2.2.1. Convolution Layer

Convolution layer is one of the important components of neural network, and its function is to extract the characteristics of the data input from the input layer through convolution calculation. In the process of each convolution movement, the characteristic valueH at this point will be calculated by summing the channel, and the specific calculation formula is shown in equation (4).
(4)hj,k=σ∑s=1Swjs∗xks+bj,k,where σ∙ is ReLU∙ nonlinearity; ∗ stands for linear convolution; bj,k is the deviation.


### 2.2.2. Pooling Layer

The function of the pooling layer is to select and filter the feature information extracted from the convolutional layer, so as to prevent excessive data from affecting the running speed of the network and remove the interference of noise. Features shrink in size as they pass through the pooling layer.


### 2.2.3. Full Connection Layer

The function of the full connection layer is to integrate the features obtained in the previous steps and transform the input feature vectorz by applying affine transformation, as shown below:
(5)hi=σWi×z+bi,where Wi and bi are the i-th weight vector and deviation, respectively. σ∙ is ReLU∙ nonlinearity and make use of softmax∙ to get the probability of class s of xt in the last layer.
(6)psxt=softmaxxt=expwsyL∑n=1NexpwsyL.


### 2.2.4. Batch Normalization

Unlike layer normalization, which is horizontal normalization, batch normalization is vertical normalization. The goal of normalization is to make each neural unit in the same layer in the direction of batch have different mean and variance to prevent the generation of gradient explosion.


## 2.2.1. Convolution Layer

Convolution layer is one of the important components of neural network, and its function is to extract the characteristics of the data input from the input layer through convolution calculation. In the process of each convolution movement, the characteristic valueH at this point will be calculated by summing the channel, and the specific calculation formula is shown in equation (4).
(4)hj,k=σ∑s=1Swjs∗xks+bj,k,where σ∙ is ReLU∙ nonlinearity; ∗ stands for linear convolution; bj,k is the deviation.


## 2.2.2. Pooling Layer

The function of the pooling layer is to select and filter the feature information extracted from the convolutional layer, so as to prevent excessive data from affecting the running speed of the network and remove the interference of noise. Features shrink in size as they pass through the pooling layer.


## 2.2.3. Full Connection Layer

The function of the full connection layer is to integrate the features obtained in the previous steps and transform the input feature vectorz by applying affine transformation, as shown below:
(5)hi=σWi×z+bi,where Wi and bi are the i-th weight vector and deviation, respectively. σ∙ is ReLU∙ nonlinearity and make use of softmax∙ to get the probability of class s of xt in the last layer.
(6)psxt=softmaxxt=expwsyL∑n=1NexpwsyL.


## 2.2.4. Batch Normalization

Unlike layer normalization, which is horizontal normalization, batch normalization is vertical normalization. The goal of normalization is to make each neural unit in the same layer in the direction of batch have different mean and variance to prevent the generation of gradient explosion.


## 2.3. Proposed Method

### 2.3.1. Automatic Scoring of Spoken English (ASSE)

The ASSE is used to give the proposed scorey to unknown quality x and is shown in Formula (7).
(7)fx;θ=y.


### 2.3.2. Scoring Model Based on CNN

In this work, it is recommended to employ the following characteristics and method parametersθ, and x (N-dimensional integer vector represents the original time-domain speech segment x∈zN).In order to make the feature extraction consider the short-, medium-, and long-term characteristics of speech signal, up to three convolution layers can be superimposed. As shown in Figure3, the softmax layer maps the probability of the output to the desired fraction.Figure 3
The structure of the proposed method.


## 2.3.1. Automatic Scoring of Spoken English (ASSE)

The ASSE is used to give the proposed scorey to unknown quality x and is shown in Formula (7).
(7)fx;θ=y.


## 2.3.2. Scoring Model Based on CNN

In this work, it is recommended to employ the following characteristics and method parametersθ, and x (N-dimensional integer vector represents the original time-domain speech segment x∈zN).In order to make the feature extraction consider the short-, medium-, and long-term characteristics of speech signal, up to three convolution layers can be superimposed. As shown in Figure3, the softmax layer maps the probability of the output to the desired fraction.Figure 3
The structure of the proposed method.


## 3. Experiment and Evaluation

### 3.1. Experimental Environment and Data Set

In this paper, the experimental environment is a MacBook Pro laptop, the system is MAC 0S 10.14.1, and the computer hardware is I7 2.5 GHz Processor, 16GB ram. The experiment was implemented in the Pycharm development environment using Python 3.7 programming language. TensorFlow version 1.8 is used. The data set adopted was the oral English test data set of students in English classes of The School of Liberal Arts, Renmin University of China from 2015 to 2018, with a total of 32,364 pieces of data. The length of each piece of audio data varies from 30 to 60 seconds. Each student has at least two pieces of audio data, and the sample data content is shown in Table1.Table 1
Audio data content of one student.
Sentence1I believe that it takes a long time to know a person well because judging people in the first sight by their appearance is not objective. For example, I remember when I first met my friend Arnold in high school, I thought he was a guy that’s hard to get close to and difficult to communicate because he looked so quiet and introverted just like a nerd, you know, that type of person who cannot do anything except for studying but later when we were doing a group project together, we really did hit it off with each other and I found he’s so funny, he would make jokes that made me laugh out loud. So I would believe it takes long time to know a person well.2I agree with this statement that it is a bad behavior to interrupt people when they are talking because it’s impolite and it can make people mad. For example, I remember when I was in high school, the teacher assigned a group project for us to do. Then there’s this guy who always interrupted us while we were talking, we could not really convey our ideas and were really irritated at the moment, so we just told him, fine it’s all yours, we are done, consequently the project was unfinished. So interrupting people while others are talking had become the most notorious behavior on my list.Here, 75% of the data was selected as the training set and the remaining 25% as the test set to evaluate performance. From the training set, 500 items of data were selected and scored by IELTS speaking examiners. Each sentence is rated by four people on a scale of one to five. The scoring criteria are mainly based on fluency, pronunciation and intonation, grammatical wording, sentence meaning, and accuracy, with 5 being the highest score and 1 being the lowest score. Table2 shows Pearson’s correlations among raters.Table 2
Pearson correlation among raters.
rater1rater2rater3rater4rater11.000.840.760.80rater21.000.800.82rater31.000.77rater41.00


### 3.2. The Results of ASSE

Within the scope of this study, we choose different CNN models for comparison. Table3 lists the experimental results and configurations of CNN model. We hope to prove the superiority of our proposed model through CNN networks with different layers and different kernels. The convolution layer is represented by conv, and the fully connected layer is represented by fc. M_1 to M_17 represent the 17 models we have listed. Among them, the first eight models were configured as a single convolution layer in the experiment of this paper. The ninth, tenth, and eleventh models are considered with two convolution layers; The remaining five models contain three convolution layers.Table 3
The experiment result.
Name1234CorrM_1Conv: 1 × 320Fc——0.8836M_2Conv: 2 × 320Fc——0.8933M_3Conv: 2 × 320, orthogonalFc——0.8988M_4Conv: 3 × 320Fc——0.9058M_5Conv: 4 × 320Fc——0.9010M_6Conv: 4 × 320, orthogonalFc——0.9040M_7Conv: 16 × 320Fc——0.9064M_8Conv: 32 × 320Fc——0.9082M_9Conv: 4 × 320Conv: 4 × 50Fc—0.9010M_10Conv: 16 × 320Conv: 16 × 50Fc—0.9226M_11Conv: 32 × 320Conv: 32 × 50Fc—0.9304M_12Conv: 1 × 320Conv: 1 × 50Conv: 1 × 8Fc0.8841M_13Conv: 2 × 320Conv: 4 × 50Conv: 1 × 8Fc0.8098M_14Conv: 4 × 320Conv: 4 × 50Conv: 4 × 8Fc0.9010M_15Conv: 16 × 320Conv: 4 × 50Conv: 16 × 8Fc0.9373M_16Conv: 32 × 320Conv: 4 × 50Conv: 32 × 8Fc0.9496M_17Conv: 64 × 320Conv: 4 × 50Conv: 64 × 8Fc0.9670In this paper, we use Pearson’s correlation coefficient as an evaluation index to compare the advantages and disadvantages of the model. As can be seen in Table3, this correlation usually increases proportionately with the number of cores and layers. In particular, the correlation coefficient between M_16 and _17 reached about 0.95 at one point. M_17 had the highest correlation of 0.9670; it indicates that our proposed method get an outstanding result in the test data sets.


### 3.3. Fluency Characteristics and Visualization

To make the phonetic features more intuitive, the output of three convolutional layers in the network from a student’s spoken audio, scoring 1, 2, and 3, respectively, is visualized as shown in Figures4–6. Every sentence is as input to the training model and the output of the evolution layer is visualized. Figures 4–6 obtain the scoring 1, 2, and 3, respectively.Figure 4
Speech visualization with a score of 1.Figure 5
Speech visualization with a score of 2.Figure 6
Speech visualization with a score of 3.Figures7–9 show the output of the convolution layer in “M_12.” In the core of the first layer, “conv_1,” it almost to have learned to calculate energies, like the characteristics of the original input. At some aspect, energy-like features are highly correlated with speech duration and silence, and silence is considered to be an important feature of fluency in knowledge-based approaches.Figure 7
The output of the conv_1.
(a)(b)(c)Figure 8
The output of the conv_2.
(a)(b)(c)Figure 9
The output of the conv_3.
(a)(b)(c)


## 3.1. Experimental Environment and Data Set

In this paper, the experimental environment is a MacBook Pro laptop, the system is MAC 0S 10.14.1, and the computer hardware is I7 2.5 GHz Processor, 16GB ram. The experiment was implemented in the Pycharm development environment using Python 3.7 programming language. TensorFlow version 1.8 is used. The data set adopted was the oral English test data set of students in English classes of The School of Liberal Arts, Renmin University of China from 2015 to 2018, with a total of 32,364 pieces of data. The length of each piece of audio data varies from 30 to 60 seconds. Each student has at least two pieces of audio data, and the sample data content is shown in Table1.Table 1
Audio data content of one student.
Sentence1I believe that it takes a long time to know a person well because judging people in the first sight by their appearance is not objective. For example, I remember when I first met my friend Arnold in high school, I thought he was a guy that’s hard to get close to and difficult to communicate because he looked so quiet and introverted just like a nerd, you know, that type of person who cannot do anything except for studying but later when we were doing a group project together, we really did hit it off with each other and I found he’s so funny, he would make jokes that made me laugh out loud. So I would believe it takes long time to know a person well.2I agree with this statement that it is a bad behavior to interrupt people when they are talking because it’s impolite and it can make people mad. For example, I remember when I was in high school, the teacher assigned a group project for us to do. Then there’s this guy who always interrupted us while we were talking, we could not really convey our ideas and were really irritated at the moment, so we just told him, fine it’s all yours, we are done, consequently the project was unfinished. So interrupting people while others are talking had become the most notorious behavior on my list.Here, 75% of the data was selected as the training set and the remaining 25% as the test set to evaluate performance. From the training set, 500 items of data were selected and scored by IELTS speaking examiners. Each sentence is rated by four people on a scale of one to five. The scoring criteria are mainly based on fluency, pronunciation and intonation, grammatical wording, sentence meaning, and accuracy, with 5 being the highest score and 1 being the lowest score. Table2 shows Pearson’s correlations among raters.Table 2
Pearson correlation among raters.
rater1rater2rater3rater4rater11.000.840.760.80rater21.000.800.82rater31.000.77rater41.00


## 3.2. The Results of ASSE

Within the scope of this study, we choose different CNN models for comparison. Table3 lists the experimental results and configurations of CNN model. We hope to prove the superiority of our proposed model through CNN networks with different layers and different kernels. The convolution layer is represented by conv, and the fully connected layer is represented by fc. M_1 to M_17 represent the 17 models we have listed. Among them, the first eight models were configured as a single convolution layer in the experiment of this paper. The ninth, tenth, and eleventh models are considered with two convolution layers; The remaining five models contain three convolution layers.Table 3
The experiment result.
Name1234CorrM_1Conv: 1 × 320Fc——0.8836M_2Conv: 2 × 320Fc——0.8933M_3Conv: 2 × 320, orthogonalFc——0.8988M_4Conv: 3 × 320Fc——0.9058M_5Conv: 4 × 320Fc——0.9010M_6Conv: 4 × 320, orthogonalFc——0.9040M_7Conv: 16 × 320Fc——0.9064M_8Conv: 32 × 320Fc——0.9082M_9Conv: 4 × 320Conv: 4 × 50Fc—0.9010M_10Conv: 16 × 320Conv: 16 × 50Fc—0.9226M_11Conv: 32 × 320Conv: 32 × 50Fc—0.9304M_12Conv: 1 × 320Conv: 1 × 50Conv: 1 × 8Fc0.8841M_13Conv: 2 × 320Conv: 4 × 50Conv: 1 × 8Fc0.8098M_14Conv: 4 × 320Conv: 4 × 50Conv: 4 × 8Fc0.9010M_15Conv: 16 × 320Conv: 4 × 50Conv: 16 × 8Fc0.9373M_16Conv: 32 × 320Conv: 4 × 50Conv: 32 × 8Fc0.9496M_17Conv: 64 × 320Conv: 4 × 50Conv: 64 × 8Fc0.9670In this paper, we use Pearson’s correlation coefficient as an evaluation index to compare the advantages and disadvantages of the model. As can be seen in Table3, this correlation usually increases proportionately with the number of cores and layers. In particular, the correlation coefficient between M_16 and _17 reached about 0.95 at one point. M_17 had the highest correlation of 0.9670; it indicates that our proposed method get an outstanding result in the test data sets.


## 3.3. Fluency Characteristics and Visualization

To make the phonetic features more intuitive, the output of three convolutional layers in the network from a student’s spoken audio, scoring 1, 2, and 3, respectively, is visualized as shown in Figures4–6. Every sentence is as input to the training model and the output of the evolution layer is visualized. Figures 4–6 obtain the scoring 1, 2, and 3, respectively.Figure 4
Speech visualization with a score of 1.Figure 5
Speech visualization with a score of 2.Figure 6
Speech visualization with a score of 3.Figures7–9 show the output of the convolution layer in “M_12.” In the core of the first layer, “conv_1,” it almost to have learned to calculate energies, like the characteristics of the original input. At some aspect, energy-like features are highly correlated with speech duration and silence, and silence is considered to be an important feature of fluency in knowledge-based approaches.Figure 7
The output of the conv_1.
(a)(b)(c)Figure 8
The output of the conv_2.
(a)(b)(c)Figure 9
The output of the conv_3.
(a)(b)(c)


## 4. Conclusions

In order to enable non-native English students to learn oral English well, and be able to use unified judgment to score students’ oral English, based on the original time-domain waveform as input, this paper proposes a CNN oral English evaluation method combining feature selection by PCA and prediction model parameter optimization. The method comprehensively evaluates different aspects of students’ oral English and finally obtains the students’ oral English scores. In the experimental part, the performance of the proposed method is evaluated by using existing data sets, the optimal structure is found, and the method is verified.



---

*Source: 6576605-2022-05-02.xml*