# Automatic Scoring Model of Japanese Interpretation Based on Semantic Scoring

Authors: Qin Wang
Journal: Mathematical Problems in Engineering
 (2022)

---

## Abstract

In order to improve the current level of Japanese teaching and the difficulty of non-standard Japanese spoken language, the author proposes a method for the study of the automatic scoring model of Japanese ants for scoring. The author introduces a semantic scoring model that integrates the long short-term memory neural network and self-attention mechanism, which can be applied to keyword scoring and sentence semantic scoring. The scoring principle of the model is as follows: firstly, extract the word and sentence features and represent them in a vectorized form, then use a bidirectional long short-term memory neural network to optimize the feature vector, and then use the self-attention mechanism to obtain the semantic features of the word or sentence. Finally, the semantic score is calculated by a simple neural network. Experiments show that compared with the semantic scoring model based on a stretchable recursive autoencoder that performs better in semantic scoring, the average correlation between this model and the original score is 0.444; the lowest rate of agreement with the original score is 95%; and the highest rate of agreement with adjacent ones is 74%. The automatic scoring model for Japanese interpreting with semantic scoring is proved to be practical and has excellent results.


---

## Body

## 1. Introduction

In the traditional oral teaching classroom, a teacher faces dozens of students, and it is difficult for students to get one-on-one targeted guidance. Students get a lot of listening, reading, and writing-related training in traditional classrooms, but little training in speaking. Due to the lack of environment and atmosphere for students to practice oral English, although students can complete the tasks of listening, reading, and writing well, it is difficult for them to pass the oral examination. This social status quo has led researchers to think about how to improve learners’ oral English. Education management departments and teaching departments have also paid more and more attention to the development of relevant oral language teaching work, adding the oral language test to the large-scale foreign language test. They try to arouse students’ attention to oral language learning by means of examinations. However, at present, most of oral examinations use generalized scoring standards, and the scoring standards are not quantified, resulting in unclear scoring points. Such a scoring standard is not conducive to the scorer’s grasp of the scoring details [1]. The scoring method of the speaking test is mainly based on the manual overall scoring method. This traditional scoring method is easily affected by the subjective factors of the rater, such as the rater’s grading status, educational background, and teaching experience. In order to reduce the influence of the subjective factors of the raters on the scoring, scholars began to refine the generalized scoring criteria and develop a spoken automatic scoring system suitable for scoring large-scale speech samples [2].In 2017, the Central People’s Government will use intelligent technology to build a new education system and incorporate it into the artificial intelligence development plan. Among them, the new education system includes intelligent learning and interactive learning. The smart education innovation and development action is one of the eight implementation actions of the “Education Informatization 2.0 Action Plan” issued by the Ministry of Education in 2018. This development action includes the construction of an intelligent education platform, the intelligentization of the education process, the mining of education big data, and the construction of an education knowledge map [3]. The development plans and action plans successively issued by the government fully demonstrate that smart education is the general trend of education development in the Internet era.Automatic scoring is an important part of the intelligentization of the educational process, which can be used for both teaching assistance and teaching quality assessment.Using an automatic grading system in the teaching process, in terms of learning, it can not only provide students with a platform for after-school practice but also allow students to receive timely feedback and guidance. From the examination, it can not only reduce the workload of the graders but also avoid the inconsistency of the grades caused by the subjective factors of the graders and improve the objectivity and fairness of the grades.


## 2. Literature Review

Hayden’s survey pointed out that in order to cultivate applied and compound Japanese talents, many domestic colleges and universities have opened Japanese interpretation courses. For undergraduate students, because the learners’ Japanese listening and speaking abilities are low and they are still in a stage of ability development, interpretation has also become a difficult subject [4]. Sichkoriz. pointed out that in the teaching of interpretation at the postgraduate level, we should actively explore and focus on interpreting skills training, the special training teaching mode is supplemented by language ability improvement [5]. Han, T. discussed the current situation of college students participating in the national translation professional qualification (level) examination and analyzed some problems existing in the examination preparation process, as well as the experience and lessons in the examination process [6]. Farooque Research pointed out that candidates mainly have problems such as low listening ability, high data translation error rate, prominent information translation omission, and insufficient background knowledge reserve [7]. A. Patterson pointed out that interpretation is an information processing process with multitask parallel processing and high cognitive load. And in this process, working memory plays an important role. As an important cognitive ability of learners, working memory is closely related to second language listening and reading, which are processed in parallel with multiple tasks and is an important support for learners to give full play to their language ability [8]. Ansarin pointed out through empirical research that the capacity of working memory affects the content comprehension in the process of Japanese listening, and learners with high working memory capacity can improve the efficiency of information processing in the listening process [9]. In addition, in the process of second language listening learning, it is also an important cognitive ability for learners to quickly and accurately understand the information they hear. Ji pointed out that the learners can process the phonetic and semantic information of auditory cue words quickly and accurately, that is, the processing speed of auditory vocabulary is also closely related to the content comprehension in the listening process [10].On the basis of the current research, the author proposes a method to study the automatic scoring model of Japanese interpretation based on semantic scoring. The author focuses on the semantic scoring of spoken Chinese-to-English questions and introduces a semantic scoring model that integrates the long-term memory neural network and self-attention mechanism, which can be applied to keyword scoring and sentence semantic scoring. The scoring principle of the model is as follows: firstly, extract word and sentence features and represent them in vectorized form, then use a bidirectional long-term and short-term memory neural network to optimize the feature vector and then use the self-attention mechanism to obtain the semantic features of words or sentences. Finally, the semantic score is calculated by a simple neural network. Based on the research on semantic scoring, the author proposes a multi-parameter automatic scoring model for Japanese sentence-level interpretation based on semantic scoring. The automatic scoring model calculates the candidate’s Japanese sentence interpretation score by fusing the feature parameters at the two levels of speech and content. At the voice level, fluency is selected as the feature scoring parameter, and the automatic scoring model directly scores the fluency feature on the candidates’ recordings. At the content level, two feature scoring parameters, keyword and sentence semantics, are selected, and the content features are scored after the candidates’ recordings are converted into texts by manual conversion. Specifically, the zero-energy product method is used to extract speech features to calculate the fluency feature score. Use the semantic scoring model introduced by the author to calculate the keyword and sentence semantic feature scores. Finally, the random forest algorithm is used to fuse the above three feature scoring parameters to obtain the total score of Japanese interpretation quality. The performance of the automatic scoring model is tested by real test data, the scoring results of the automatic scoring model for Japanese interpretation proposed by the author have a high average consistency rate with the original score, which verifies the validity of the feature scoring parameters selected by the author and proves that the author’s research is effective, and the automatic scoring method of Japanese interpretation based on semantic scoring proposed by him is feasible [11].


## 3. Methods

### 3.1. Eigenvector Representation

#### 3.1.1. Discrete Representation

The discrete representation directly uses the combination of the number 0 and the number 1 to represent all words. One-hot encoding removes the repeated words in the corpus and arranges the words into a vocabulary list in a certain order. Assuming that the total number of words in the vocabulary is N, the length of the word vectors in the vocabulary is N. The position of each word corresponds to a value of 1, and all other positions are 0. Although the discrete representation program is simple to implement, this sparsely stored encoding representation method. When the number of words in the vocabulary increases, the problem of dimension explosion is prone to occur. Furthermore, with this vector representation, all words are discrete symbols, and we cannot understand the similarity between words through vectors. In real life, people use language and words, and there are semantic associations between words. Distributed representations were born to address these shortcomings of discrete representations [12].


#### 3.1.2. Distributed Representation

The idea of distributed representation stems from the distribution hypothesis. The hypothesis states that the context of the word determines the meaning of the word. If two words are in a similar context, their semantics are also similar. Therefore, the distributed representation of the vector is different from the discrete representation, which is a vector representation that uses a neural network to convert words into continuous values [13]. Researchers can represent the semantic similarity between different words by calculating the spatial distance between distributed vectors. Euclidean distance and cosine similarity are commonly used spatial distance calculation methods. As shown in Figure 1, the Word2Vec model is an important model for generating distributed vectors based on contextual information.Figure 1
Two model structures of Word2Vec.


#### 3.1.3. BERT Model

As shown in Figure2, feature extraction (Features Extract) and fine-tuning (Fine-Tunning) are currently the main applications of BERT. The BERT model is a model trained through massive text and can directly generate word vectors. Feature extraction is to use the trained model to convert new words or new texts into vector representations as input to the network model. Fine-tuning means that the BERT model can be adapted to the needs of the applied model, and appropriate adjustments are made to the internal parameters of the BERT model to get a better text vector representation.Figure 2
Structure diagram of the encoder part of the transformer.


### 3.2. Recurrent Neural Network

The specific structure of the RNN unit, shown in Figure3, represents the processing of data at a time. In addition to X, the hidden layer input at time t also includes the output Ht−1 of the hidden layer at time t−1. Therefore, the RNN model can use the previous information to process the t-th target word, which can better solve the problem of sequence prediction. The researchers use RNNs to process long text sequences, successively learn the texts divided into single word sequences and then save the learned text features into a fixed-size hidden layer. RNN can memorize historical information, so it has been widely used in time series modeling.Figure 3
RNN unit structure diagram.


#### 3.2.1. Long-Term Memory Model

The LSTM storage unit is mainly composed of four parts: memory cells, forgetting gates, input gates, and output gates. The forget gate is used to filter out old cell information and update the current memory cell according to the memory cell candidate information generated by the input gate. The sigmoid activation function in the forget gate processes the input information and outputs a value between [0, 1]. The output value represents the state in which the old memory cell information is retained, the output value of 0 means that all the old memory cell information is forgotten, and the output value of 1 means that all the old memory cell information is retained. The information state calculation formula of the forget gate is shown in formula (1) and the update formula is shown in formula (2).(1)ft=σWf⋅Ht−1,Xt+bf,(2)Ct=ft∗Ct−1+it∗C˜t.


#### 3.2.2. Bidirectional Long Short-Term Memory Model

Although the LSTM model can obtain historical information, it can only predict the output at the next moment based on the historical information, and the current output can only be combined with the previous sequence information. In practical tasks, the output at the current moment is not only related to historical information but may also be related to subsequent information at the current moment. For example, in order to predict the semantics of words in a sentence, not only need to judge according to the previous text but also need to consider the information behind the words. Only the joint context information can accurately judge the semantics of words, which is the meaning of the existence of the bidirectional long and short-term memory model [14]. The model saves the information before and after the text, which effectively solves the problem that LSTM can only save the previous information. The BiLSTM model is composed of a forward LSTM model and a backward LSTM model, and the unit outputs of the two LSTM models are spliced to generate a unit output with historical information and future information.


#### 3.2.3. Activation and Loss Functions in Neural Networks

Activation function is a nonlinear functional relationship between the upper and lower layers of nodes in a multilayer neural network. The existence of the activation function enables the neural network to have the ability to model nonlinear expressions. Activation function is the key to deep learning to obtain strong learning ability [15].


### 3.3. Application of the Semantic Scoring Model

In essence, the attention mechanism can be regarded as the mapping from query to key-value, and its calculation is mainly divided into three stages. The first stage is to obtain the weight by calculating the similarity between the query value and the key value, the calculation formula is shown in formula (3) [16]. Formula (3) is a commonly used function method, they are dot product (Dot), concatenation (Concatenation), and perceptron (Perceptron). The second stage uses the softmax function to normalize the weights obtained in the first stage processing, the calculation formula is shown in formula (4). In the third stage, the weight and the corresponding value are weighted and summed to obtain the final calculation result of attention, the calculation formula is shown in formula (5). In most natural language processing tasks key and value are the same, i.e., Key = Value. In the formula, Q represents the query, K represents the key, V represents the key value, and W and U are the parameter matrices during the training process.(3)fQki=QtkidotWαQ;kiconcatvαttanhWαQ+Uαkiperceptron,(4)αi=softmaxfQ,ki=expfQ,ki∑jexpfQ,ki,(5)AttentionQ,K,V=∑iαiVi.


#### 3.3.1. Keyword-Based Semantic Scoring

Japanese interpreters pay more attention to the key information points of answering questions. A good scoring model should consider not only the scores of keywords that are consistent with the reference answer but also the scores of synonyms related to the keywords [17].Word2vec can mine the semantics of keywords and their synonyms and express them in the form of vectors and can also be used to calculate the semantic similarity of words. In the application of word vector representation, the main difference between the Word2Vec model and the BERT model is that the Word2Vec model generates a static word vector, and each word has only a unique word vector representation. The BERT model can dynamic word vectors according to the context, and each word may have different representations in different contexts, which is conducive to the representation of polysemy. In order to compare the performance of the two models applied to keyword semantic scoring, when other layers of the semantic scoring model are unchanged, replace the BERT model in the word embedding layer with the Word2Vec model for control experiments. In the experiment, the word vector dimension of the Word2Vec model is set to 128. We will analyze the performance of the two models in keyword scoring from two evaluation indicators, the correlation between the model scoring results and the original scores, and the consistency between the model scoring results and the original scores [18].


#### 3.3.2. Sentence-Based Semantic Scoring

Sentence semantic scoring is not only a test of the overall understanding of the candidate’s sentence but also a reflection of the candidate’s ability to express the sentence. In the manual scoring, there is no specific item of sentence semantics, but the scorer will score the overall situation of the candidate’s translation. The rater can tell directly by listening to the recording, and the examinee uttered a complete sentence or a string of key words. However, computers cannot easily make such judgments. From the semantic level, sentences usually consist of key words and general words. Key words are words that can affect the meaning of a sentence, and are generally composed of words with specific meanings such as nouns, verbs, and adjectives. While general words do not have a decisive influence on understanding the meaning of the whole sentence, they are also an indispensable part of the sentence, such as prepositions, conjunctions, articles, and so on. In this section, we will use the sentence semantic scoring model based on the stretchable recurrent autoencoder neural network and the BiLSTM-AM-based semantic scoring model proposed by the author to conduct an experimental comparison and analyze the advantages and disadvantages of the two models in sentence semantic analysis [19].


## 3.1. Eigenvector Representation

### 3.1.1. Discrete Representation

The discrete representation directly uses the combination of the number 0 and the number 1 to represent all words. One-hot encoding removes the repeated words in the corpus and arranges the words into a vocabulary list in a certain order. Assuming that the total number of words in the vocabulary is N, the length of the word vectors in the vocabulary is N. The position of each word corresponds to a value of 1, and all other positions are 0. Although the discrete representation program is simple to implement, this sparsely stored encoding representation method. When the number of words in the vocabulary increases, the problem of dimension explosion is prone to occur. Furthermore, with this vector representation, all words are discrete symbols, and we cannot understand the similarity between words through vectors. In real life, people use language and words, and there are semantic associations between words. Distributed representations were born to address these shortcomings of discrete representations [12].


### 3.1.2. Distributed Representation

The idea of distributed representation stems from the distribution hypothesis. The hypothesis states that the context of the word determines the meaning of the word. If two words are in a similar context, their semantics are also similar. Therefore, the distributed representation of the vector is different from the discrete representation, which is a vector representation that uses a neural network to convert words into continuous values [13]. Researchers can represent the semantic similarity between different words by calculating the spatial distance between distributed vectors. Euclidean distance and cosine similarity are commonly used spatial distance calculation methods. As shown in Figure 1, the Word2Vec model is an important model for generating distributed vectors based on contextual information.Figure 1
Two model structures of Word2Vec.


### 3.1.3. BERT Model

As shown in Figure2, feature extraction (Features Extract) and fine-tuning (Fine-Tunning) are currently the main applications of BERT. The BERT model is a model trained through massive text and can directly generate word vectors. Feature extraction is to use the trained model to convert new words or new texts into vector representations as input to the network model. Fine-tuning means that the BERT model can be adapted to the needs of the applied model, and appropriate adjustments are made to the internal parameters of the BERT model to get a better text vector representation.Figure 2
Structure diagram of the encoder part of the transformer.


## 3.1.1. Discrete Representation

The discrete representation directly uses the combination of the number 0 and the number 1 to represent all words. One-hot encoding removes the repeated words in the corpus and arranges the words into a vocabulary list in a certain order. Assuming that the total number of words in the vocabulary is N, the length of the word vectors in the vocabulary is N. The position of each word corresponds to a value of 1, and all other positions are 0. Although the discrete representation program is simple to implement, this sparsely stored encoding representation method. When the number of words in the vocabulary increases, the problem of dimension explosion is prone to occur. Furthermore, with this vector representation, all words are discrete symbols, and we cannot understand the similarity between words through vectors. In real life, people use language and words, and there are semantic associations between words. Distributed representations were born to address these shortcomings of discrete representations [12].


## 3.1.2. Distributed Representation

The idea of distributed representation stems from the distribution hypothesis. The hypothesis states that the context of the word determines the meaning of the word. If two words are in a similar context, their semantics are also similar. Therefore, the distributed representation of the vector is different from the discrete representation, which is a vector representation that uses a neural network to convert words into continuous values [13]. Researchers can represent the semantic similarity between different words by calculating the spatial distance between distributed vectors. Euclidean distance and cosine similarity are commonly used spatial distance calculation methods. As shown in Figure 1, the Word2Vec model is an important model for generating distributed vectors based on contextual information.Figure 1
Two model structures of Word2Vec.


## 3.1.3. BERT Model

As shown in Figure2, feature extraction (Features Extract) and fine-tuning (Fine-Tunning) are currently the main applications of BERT. The BERT model is a model trained through massive text and can directly generate word vectors. Feature extraction is to use the trained model to convert new words or new texts into vector representations as input to the network model. Fine-tuning means that the BERT model can be adapted to the needs of the applied model, and appropriate adjustments are made to the internal parameters of the BERT model to get a better text vector representation.Figure 2
Structure diagram of the encoder part of the transformer.


## 3.2. Recurrent Neural Network

The specific structure of the RNN unit, shown in Figure3, represents the processing of data at a time. In addition to X, the hidden layer input at time t also includes the output Ht−1 of the hidden layer at time t−1. Therefore, the RNN model can use the previous information to process the t-th target word, which can better solve the problem of sequence prediction. The researchers use RNNs to process long text sequences, successively learn the texts divided into single word sequences and then save the learned text features into a fixed-size hidden layer. RNN can memorize historical information, so it has been widely used in time series modeling.Figure 3
RNN unit structure diagram.


### 3.2.1. Long-Term Memory Model

The LSTM storage unit is mainly composed of four parts: memory cells, forgetting gates, input gates, and output gates. The forget gate is used to filter out old cell information and update the current memory cell according to the memory cell candidate information generated by the input gate. The sigmoid activation function in the forget gate processes the input information and outputs a value between [0, 1]. The output value represents the state in which the old memory cell information is retained, the output value of 0 means that all the old memory cell information is forgotten, and the output value of 1 means that all the old memory cell information is retained. The information state calculation formula of the forget gate is shown in formula (1) and the update formula is shown in formula (2).(1)ft=σWf⋅Ht−1,Xt+bf,(2)Ct=ft∗Ct−1+it∗C˜t.


### 3.2.2. Bidirectional Long Short-Term Memory Model

Although the LSTM model can obtain historical information, it can only predict the output at the next moment based on the historical information, and the current output can only be combined with the previous sequence information. In practical tasks, the output at the current moment is not only related to historical information but may also be related to subsequent information at the current moment. For example, in order to predict the semantics of words in a sentence, not only need to judge according to the previous text but also need to consider the information behind the words. Only the joint context information can accurately judge the semantics of words, which is the meaning of the existence of the bidirectional long and short-term memory model [14]. The model saves the information before and after the text, which effectively solves the problem that LSTM can only save the previous information. The BiLSTM model is composed of a forward LSTM model and a backward LSTM model, and the unit outputs of the two LSTM models are spliced to generate a unit output with historical information and future information.


### 3.2.3. Activation and Loss Functions in Neural Networks

Activation function is a nonlinear functional relationship between the upper and lower layers of nodes in a multilayer neural network. The existence of the activation function enables the neural network to have the ability to model nonlinear expressions. Activation function is the key to deep learning to obtain strong learning ability [15].


## 3.2.1. Long-Term Memory Model

The LSTM storage unit is mainly composed of four parts: memory cells, forgetting gates, input gates, and output gates. The forget gate is used to filter out old cell information and update the current memory cell according to the memory cell candidate information generated by the input gate. The sigmoid activation function in the forget gate processes the input information and outputs a value between [0, 1]. The output value represents the state in which the old memory cell information is retained, the output value of 0 means that all the old memory cell information is forgotten, and the output value of 1 means that all the old memory cell information is retained. The information state calculation formula of the forget gate is shown in formula (1) and the update formula is shown in formula (2).(1)ft=σWf⋅Ht−1,Xt+bf,(2)Ct=ft∗Ct−1+it∗C˜t.


## 3.2.2. Bidirectional Long Short-Term Memory Model

Although the LSTM model can obtain historical information, it can only predict the output at the next moment based on the historical information, and the current output can only be combined with the previous sequence information. In practical tasks, the output at the current moment is not only related to historical information but may also be related to subsequent information at the current moment. For example, in order to predict the semantics of words in a sentence, not only need to judge according to the previous text but also need to consider the information behind the words. Only the joint context information can accurately judge the semantics of words, which is the meaning of the existence of the bidirectional long and short-term memory model [14]. The model saves the information before and after the text, which effectively solves the problem that LSTM can only save the previous information. The BiLSTM model is composed of a forward LSTM model and a backward LSTM model, and the unit outputs of the two LSTM models are spliced to generate a unit output with historical information and future information.


## 3.2.3. Activation and Loss Functions in Neural Networks

Activation function is a nonlinear functional relationship between the upper and lower layers of nodes in a multilayer neural network. The existence of the activation function enables the neural network to have the ability to model nonlinear expressions. Activation function is the key to deep learning to obtain strong learning ability [15].


## 3.3. Application of the Semantic Scoring Model

In essence, the attention mechanism can be regarded as the mapping from query to key-value, and its calculation is mainly divided into three stages. The first stage is to obtain the weight by calculating the similarity between the query value and the key value, the calculation formula is shown in formula (3) [16]. Formula (3) is a commonly used function method, they are dot product (Dot), concatenation (Concatenation), and perceptron (Perceptron). The second stage uses the softmax function to normalize the weights obtained in the first stage processing, the calculation formula is shown in formula (4). In the third stage, the weight and the corresponding value are weighted and summed to obtain the final calculation result of attention, the calculation formula is shown in formula (5). In most natural language processing tasks key and value are the same, i.e., Key = Value. In the formula, Q represents the query, K represents the key, V represents the key value, and W and U are the parameter matrices during the training process.(3)fQki=QtkidotWαQ;kiconcatvαttanhWαQ+Uαkiperceptron,(4)αi=softmaxfQ,ki=expfQ,ki∑jexpfQ,ki,(5)AttentionQ,K,V=∑iαiVi.


### 3.3.1. Keyword-Based Semantic Scoring

Japanese interpreters pay more attention to the key information points of answering questions. A good scoring model should consider not only the scores of keywords that are consistent with the reference answer but also the scores of synonyms related to the keywords [17].Word2vec can mine the semantics of keywords and their synonyms and express them in the form of vectors and can also be used to calculate the semantic similarity of words. In the application of word vector representation, the main difference between the Word2Vec model and the BERT model is that the Word2Vec model generates a static word vector, and each word has only a unique word vector representation. The BERT model can dynamic word vectors according to the context, and each word may have different representations in different contexts, which is conducive to the representation of polysemy. In order to compare the performance of the two models applied to keyword semantic scoring, when other layers of the semantic scoring model are unchanged, replace the BERT model in the word embedding layer with the Word2Vec model for control experiments. In the experiment, the word vector dimension of the Word2Vec model is set to 128. We will analyze the performance of the two models in keyword scoring from two evaluation indicators, the correlation between the model scoring results and the original scores, and the consistency between the model scoring results and the original scores [18].


### 3.3.2. Sentence-Based Semantic Scoring

Sentence semantic scoring is not only a test of the overall understanding of the candidate’s sentence but also a reflection of the candidate’s ability to express the sentence. In the manual scoring, there is no specific item of sentence semantics, but the scorer will score the overall situation of the candidate’s translation. The rater can tell directly by listening to the recording, and the examinee uttered a complete sentence or a string of key words. However, computers cannot easily make such judgments. From the semantic level, sentences usually consist of key words and general words. Key words are words that can affect the meaning of a sentence, and are generally composed of words with specific meanings such as nouns, verbs, and adjectives. While general words do not have a decisive influence on understanding the meaning of the whole sentence, they are also an indispensable part of the sentence, such as prepositions, conjunctions, articles, and so on. In this section, we will use the sentence semantic scoring model based on the stretchable recurrent autoencoder neural network and the BiLSTM-AM-based semantic scoring model proposed by the author to conduct an experimental comparison and analyze the advantages and disadvantages of the two models in sentence semantic analysis [19].


## 3.3.1. Keyword-Based Semantic Scoring

Japanese interpreters pay more attention to the key information points of answering questions. A good scoring model should consider not only the scores of keywords that are consistent with the reference answer but also the scores of synonyms related to the keywords [17].Word2vec can mine the semantics of keywords and their synonyms and express them in the form of vectors and can also be used to calculate the semantic similarity of words. In the application of word vector representation, the main difference between the Word2Vec model and the BERT model is that the Word2Vec model generates a static word vector, and each word has only a unique word vector representation. The BERT model can dynamic word vectors according to the context, and each word may have different representations in different contexts, which is conducive to the representation of polysemy. In order to compare the performance of the two models applied to keyword semantic scoring, when other layers of the semantic scoring model are unchanged, replace the BERT model in the word embedding layer with the Word2Vec model for control experiments. In the experiment, the word vector dimension of the Word2Vec model is set to 128. We will analyze the performance of the two models in keyword scoring from two evaluation indicators, the correlation between the model scoring results and the original scores, and the consistency between the model scoring results and the original scores [18].


## 3.3.2. Sentence-Based Semantic Scoring

Sentence semantic scoring is not only a test of the overall understanding of the candidate’s sentence but also a reflection of the candidate’s ability to express the sentence. In the manual scoring, there is no specific item of sentence semantics, but the scorer will score the overall situation of the candidate’s translation. The rater can tell directly by listening to the recording, and the examinee uttered a complete sentence or a string of key words. However, computers cannot easily make such judgments. From the semantic level, sentences usually consist of key words and general words. Key words are words that can affect the meaning of a sentence, and are generally composed of words with specific meanings such as nouns, verbs, and adjectives. While general words do not have a decisive influence on understanding the meaning of the whole sentence, they are also an indispensable part of the sentence, such as prepositions, conjunctions, articles, and so on. In this section, we will use the sentence semantic scoring model based on the stretchable recurrent autoencoder neural network and the BiLSTM-AM-based semantic scoring model proposed by the author to conduct an experimental comparison and analyze the advantages and disadvantages of the two models in sentence semantic analysis [19].


## 4. Results and Discussion

The results of the correlation analysis between the BERT model, the Word2Vec model, and the original score are shown in Table1 and Figure 4, it can be seen that the average correlation between the BERT model and the original score is 0.444, which is higher than the Word2Vec model of 0.349. In terms of the consistency rate with the original score, the lowest is 95% for the BERT model, and the lowest for the Word2Vec model is 86%. In terms of the adjacent consistency rate, the highest for the BERT model is 74%, and the highest for the Word2Vec model is 59%. It can be seen that the performance of the BERT model is due to the Word2Vec model in all aspects.Table 1
Comparison of BERT model, Word2vec model, and original score correlation.
TopicBERT-BLSTM-ATTWord2Vec-BiLSTM-ATTA_10.570.56A_20.350.25A_30.570.48A_40.460.32A_50.590.47B_10.610.47B_20.470.39B_30.320.21B_40.50.34B_50.450.35Figure 4
The consistency rate between the keyword semantic scoring model and the original score.In order to reduce the impact of artificial subjectivity on the scoring results, the recorded answer sheet data used in this experiment were independently scored by two raters, taking the average of the two teachers’ scores as the raw score. In order to ensure the authority of the original score and the validity of the experimental data, we eliminate the speech data with two scorers scoring more than 0.5 points and verify the validity of the machine scoring model by comparing the differences between the scores of the two scorers [20]. Figure 5 shows the distribution of manual scoring levels. Marker 1, Marker 2, and the original point all conform to the normal distribution law. Studies have shown that in general, the distribution of students’ grades obeys a normal distribution, which reflects the rationality of the two graders’ grades and the reliability of the original grades [21]. The scoring results of the automatic scoring model for Chinese-English interpretation based on semantic scoring are compared with the manual scoring, and the overall average similarity between the scoring results of this model and the original scores is 0.93% [22–25].Figure 5
Distribution of manual scoring levels.


## 5. Conclusion

The author proposes a method of automatic scoring model research for Japanese interpretation based on semantic scoring, which introduces a semantic scoring model based on BiLSTM-AM. The model extracts local and global features of sentence information by using the semantic representation ability and local feature extraction ability of the BERT model and uses the BiLSTM model to simplify the vector representation. A self-attention mechanism is introduced to obtain the relationship between words and appropriate weights are assigned to each word. Finally, a fully connected layer is used to obtain a semantic score. Through multiple sets of experiments, the effectiveness and advancement of the model in keyword scoring and sentence semantic scoring are effectively verified.



---

*Source: 3299549-2022-09-06.xml*