# Research on Neural Network Machine Translation Model Based on Entity Tagging Improvement

Authors: Xijun Xu
Journal: Mathematical Problems in Engineering
 (2022)

---

## Abstract

Machine translation, as an efficient tool, can achieve equivalent conversion between different languages while preserving the original semantics. At present, machine translation models based on deep neural networks have become a hot research topic in the fields of natural language processing and image processing. However, the randomness of neural networks leads to the existing neural network machine translation models unable to effectively reflect the linguistic dependencies and having unsatisfactory results when dealing with long sentence sequences. To solve these two problems, a new neural network machine translation model with entity tagging improvement is proposed. First, for the low-frequency word translation problem, UNK entity tags replacement is used to compensate for the weakness of the randomness of neural networks and the encoding/decoding strategy of entity tagging is improved. Then, on the basis of the LSTM translation model, an attention mechanism is introduced to dynamically adjust the degree of influence of the context at the source language end on the target language sequence to improve the feature learning ability of the translation model in processing long sentences. The analysis of the experimental results shows that the translation evaluation index BLEU of the proposed translation model is significantly improved compared with various translation models, which verifies its effectiveness.


---

## Body

## 1. Introduction

Machine translation (MT) is an important research direction combining natural language processing and artificial intelligence [1–4]. With the development of the Internet technology, international communication has become more and more frequent. Traditional manual translation does not achieve high efficiency while consuming a lot of human and financial resources, while computers can obtain translation results quickly and efficiently.Traditional machine translation was implemented mainly using statistical methods. In the statistical-based approach to machine translation, the translation task is abstracted as a probabilistic problem. It is necessary to establish a probability model [4–7] between any two sentences in the bilingual corpus, and the translation task is to find the corresponding target with the highest probability for the source-side utterance using a suitable method. Statistical machine translation solves the bottleneck problem of knowledge acquisition by learning from a bilingual corpus to obtain transformation rules and no longer requires human initiative to provide language rules. The field of statistical machine translation has produced many results, mainly including language models, hidden Markov models, and models based on the idea of maximum entropy [8–10]. However, to obtain perfect translation results, statistical-based machine translation methods still have many problems [11, 12]. Data preprocessing links such as word alignment, word segmentation, and rule extraction have a great influence, so experienced experts are needed for manual processing. Choi et al. [13] used semantic role tagging information to reorder the list of candidate translations and improved the accuracy of translation preprocessing by this method. Sevens et al. [14] used dynamic, static, and topic caching techniques to improve the selection of phrase translations based on the nature of articulation, hoping that the same phrases remain consistent throughout the document. Church [15] found that unknown words are an important factor in translation errors during cross-domain translation and proposed the use of dictionary mining techniques to solve the translation problem of unlogged words in new domains. Although statistical machine translation greatly reduces the labor cost and development cycle, the translation focuses on the conversion between source and target languages, ignoring the connection between contexts, resulting in a lack of semantic coherence in translation results.With the development of deep learning techniques, deep neural networks have achieved remarkable results in the fields of image processing and speech recognition. Deep learning techniques provide new solutions to the challenges related to machine translation. Sennrich et al. [16] proposed an “encoding-decoding” model of neural machine translation and explained the encoding rules. Liang and Du [17] used a convolutional neural network to construct the encoder and a recurrent neural network for the decoder to obtain historical information and process variable-length strings. The current mainstream neural network machine translation approach is an end-to-end codec translation system built using recurrent neural network (RNN) models. Although the implementation of machine translation based on neural network modelling achieves significantly better results compared with traditional methods, there are still some obvious problems that constrain the performance of neural network machine translation models after the analysis of the model structure and translation results. These two problems are as follows: (1) the problem of unregistered words and low-frequency words: as the neural network approach requires modelling of word vectors for all source and target words, unregistered words and low-frequency words cannot be trained in a normal model, and the importance of this problem is particularly apparent when the language is rich in word morphology or has a large vocabulary. (2) The long-distance dependency problem: neural network-based machine translation models tend to have better translation results for inputs that are closer to the decoder, while words that are farther away lead to inaccurate translation of long sentence sequences due to the loss of information transfer.Therefore, in order to solve the above 2 problems, a neural network machine translation model based on entity tagging improvement is proposed in this paper. The main work includes the following: (1) improvement for the low-frequency word problem. UNK entity tag replacement is used to compensate for the weakness of randomness of neural networks. Using the improved entity tagging coding/decoding strategy, all target words are mapped to a controlled size lexicon to achieve adaptation to the neural network model; (2) improvements for the model structure. The long short-term Memory (LSTM) decoder incorporates an attention mechanism, which solves the problem of gradient disappearance in the traditional RNN and solves the problem of long-distance dependence in the training of translation models through the control of three types of “gates.”The rest of the paper is organized as follows: in Section2, the low-frequency word processing based on improved entity tagging is studied in detail, while Section 3 provides a neural network translation model incorporating attentional mechanisms. Section 4 provides the results and discussion. Finally, the paper is concluded in Section 5.


## 2. Low-Frequency Word Processing Based on Entity Tagging Improvement

In the traditional neural machine translation model, all unregistered words or low-frequency words are marked as <UNK> symbols. For <UNK> symbols, the current mainstream approach searches for the corresponding aligned words by sequence backtracking at the source end and performs a fill-and-replace operation. This approach provides a feasible solution for regions that cannot be covered by the dictionary, but its effectiveness is hardly guaranteed.


### 2.1. Named Entity Location Tasks

For neural network machine translation models, the coding and decoding methods with the addition of entity tags reduce the uncertainty of variables in the neural network while imposing higher requirements on some performance at the machine translation level. For example, for the decoding stage, as entity tags need to be decoded at specific locations, their semantics and context may not be quite the same as other common words, leading to error cases such as mistranslation and omission. Therefore, we make specific enhancements to the named entity localization in the decoding stage.The main goal of the named entity localization task is to determine, during the decoding operation, whether it outputs a named entity and the class of the output named entity. This task has certain commonalities with the conventional named entity identification task, while there are differences [18–21]: (1) both are tasked with identifying named entities, but the latter operates on a determined input text and is a direct identification task. The localization task, on the other hand, determines the output type at decoding based on the encoding result of the sequence at the source and does not need to determine the specific result of decoding; (2) named entity recognition is annotated at the word level, and the goal is to annotate a single entity word or a group of entity words in multiple consecutive combinations. However, the localization task has already completed the recognition and combination of named entities during preprocessing. Therefore, only the semantics of that output is discriminated, and the need for contextual information is relatively small.Therefore, simple named entity localization can be implemented entirely based on the neural network machine translation model, and only the dimension of the last output layer is changed. Since the codec method proposed in this paper labels tags as 4 classes, the output dimension of the output layer is 4 + 1 = 5, corresponding to 4 classes of tags and other common words, respectively.


### 2.2. Encoding Strategy for Entity Tagging

For named entity words, more advanced extraction methods currently exist. Therefore, we can extend the encoding method of low-frequency words based on the features of named entity tagging. Based on the common named entity classification, we divide low-frequency words into four categories, as shown in Table1.Table 1
Tagging of low-frequency words.
CategoryTagsDescriptionPerson name<PER>Person names with low word frequencyGeographical name<LOC>Geographical names with low word frequencyOrganization name<ORG>Organization name with a low word frequency, possibly a phraseLow-frequency words<UNK>Other low-frequency wordsThe rules and constraints for the tagging of words are as follows:(1)
For the named entities obtained through recognition, only the individuals with lower word frequency are marked. For words like "China", "Smith," and others with high frequency, it is better to include them into the dictionary as normal words for translation.(2)
For numeric-related entity words, the current tool does not recognize numeric entities well because of the complex format of numbers and the existence of words, abbreviations, numbers, and other forms. In addition, numbers often require different translation methods depending on the scenario, even affecting the overall semantics, and are not suitable for excision and labeling. Therefore, for the operation of entities, we choose to discard numeric entities.According to the above rules and constraints, a dictionary of specified size is built based on the word frequency and then the entity words or phrases outside the dictionary are tagged according to the corresponding labels. Using the tagging method for low-frequency words does not mean abandoning the subword logic-based word separation method. For words other than tagged entities, there is still an option to preprocess them using double-byte encoding, thus extending the coverage of the lexicon as much as possible.After incorporating the subword construction strategy with double-byte encoding, the new corpus preprocessing method is as follows: (1) performing the normal word-sorting operation: for different languages, the cut-off method in word sense is identified; (2) the operation of named entity identification is performed on the training corpus. Each entity word is identified with its category; (3) statistical word frequency: for entity words with word frequencies below a specific read value, tagging operation is performed according to the corresponding category and the mapping of entity tags to entity words in each sentence is saved; (4) for untagged words, the operation of cutting subwords is performed according to the double-byte encoding. The final lexicon based on subword logic and the preprocessed corpus are obtained.For example, given a sentence “I visited Peking University in China in 2018.,” it is processed to get “I visit$ ed <LOC> in China in 20$ 18.”, and we can see that “Peking University” as an institutional entity word with low word frequency is replaced using the corresponding tag. The high-frequency word “China” is not. Among the other words, the past tense and the year numeral are cut into two words by the subword cutting method due to the dictionary size limitation, but the statistical-based features still keep them linguistically meaningful.


### 2.3. Decoding Strategy for Entity Tagging

For the problem of low-frequency words, we design an improved decoding rule for entity tags: (1) For the decoding step where the decoding output is an entity word tag, the source input sequence is sorted according to its attention score. The search is done according to the principle of “same entity tag > other words,” and the found words are translated by looking up the bilingual dictionary. (2) For other decoding steps with UNK tags, the search is carried out according to the principle of “UNK tags > other words,” and the others are similar.For example, when translating “Annie visited Peking University in China in 2018.,” the output sequence is mapped by dictionary to “<PER> visited <LOC> in China in 2018.” According to the above decoding rules, for the <PER> tag, it was found that the source-end sequence did not have a <PER> tag, so the output of this step was mapped to the source-end word “Annie” with the highest ATTENTION score. It is translated as “Annie” by regularization methods such as dictionary search. For the <LOC> tag, although the word with the highest attention score is “in,” the source sequence contains the same <LOC> tag at this time, so we locate the position and get the correct translation result of “Peking University” by word translation, as shown in Figure1.Figure 1
Example of coding and decoding process for entity tagging.


## 2.1. Named Entity Location Tasks

For neural network machine translation models, the coding and decoding methods with the addition of entity tags reduce the uncertainty of variables in the neural network while imposing higher requirements on some performance at the machine translation level. For example, for the decoding stage, as entity tags need to be decoded at specific locations, their semantics and context may not be quite the same as other common words, leading to error cases such as mistranslation and omission. Therefore, we make specific enhancements to the named entity localization in the decoding stage.The main goal of the named entity localization task is to determine, during the decoding operation, whether it outputs a named entity and the class of the output named entity. This task has certain commonalities with the conventional named entity identification task, while there are differences [18–21]: (1) both are tasked with identifying named entities, but the latter operates on a determined input text and is a direct identification task. The localization task, on the other hand, determines the output type at decoding based on the encoding result of the sequence at the source and does not need to determine the specific result of decoding; (2) named entity recognition is annotated at the word level, and the goal is to annotate a single entity word or a group of entity words in multiple consecutive combinations. However, the localization task has already completed the recognition and combination of named entities during preprocessing. Therefore, only the semantics of that output is discriminated, and the need for contextual information is relatively small.Therefore, simple named entity localization can be implemented entirely based on the neural network machine translation model, and only the dimension of the last output layer is changed. Since the codec method proposed in this paper labels tags as 4 classes, the output dimension of the output layer is 4 + 1 = 5, corresponding to 4 classes of tags and other common words, respectively.


## 2.2. Encoding Strategy for Entity Tagging

For named entity words, more advanced extraction methods currently exist. Therefore, we can extend the encoding method of low-frequency words based on the features of named entity tagging. Based on the common named entity classification, we divide low-frequency words into four categories, as shown in Table1.Table 1
Tagging of low-frequency words.
CategoryTagsDescriptionPerson name<PER>Person names with low word frequencyGeographical name<LOC>Geographical names with low word frequencyOrganization name<ORG>Organization name with a low word frequency, possibly a phraseLow-frequency words<UNK>Other low-frequency wordsThe rules and constraints for the tagging of words are as follows:(1)
For the named entities obtained through recognition, only the individuals with lower word frequency are marked. For words like "China", "Smith," and others with high frequency, it is better to include them into the dictionary as normal words for translation.(2)
For numeric-related entity words, the current tool does not recognize numeric entities well because of the complex format of numbers and the existence of words, abbreviations, numbers, and other forms. In addition, numbers often require different translation methods depending on the scenario, even affecting the overall semantics, and are not suitable for excision and labeling. Therefore, for the operation of entities, we choose to discard numeric entities.According to the above rules and constraints, a dictionary of specified size is built based on the word frequency and then the entity words or phrases outside the dictionary are tagged according to the corresponding labels. Using the tagging method for low-frequency words does not mean abandoning the subword logic-based word separation method. For words other than tagged entities, there is still an option to preprocess them using double-byte encoding, thus extending the coverage of the lexicon as much as possible.After incorporating the subword construction strategy with double-byte encoding, the new corpus preprocessing method is as follows: (1) performing the normal word-sorting operation: for different languages, the cut-off method in word sense is identified; (2) the operation of named entity identification is performed on the training corpus. Each entity word is identified with its category; (3) statistical word frequency: for entity words with word frequencies below a specific read value, tagging operation is performed according to the corresponding category and the mapping of entity tags to entity words in each sentence is saved; (4) for untagged words, the operation of cutting subwords is performed according to the double-byte encoding. The final lexicon based on subword logic and the preprocessed corpus are obtained.For example, given a sentence “I visited Peking University in China in 2018.,” it is processed to get “I visit$ ed <LOC> in China in 20$ 18.”, and we can see that “Peking University” as an institutional entity word with low word frequency is replaced using the corresponding tag. The high-frequency word “China” is not. Among the other words, the past tense and the year numeral are cut into two words by the subword cutting method due to the dictionary size limitation, but the statistical-based features still keep them linguistically meaningful.


## 2.3. Decoding Strategy for Entity Tagging

For the problem of low-frequency words, we design an improved decoding rule for entity tags: (1) For the decoding step where the decoding output is an entity word tag, the source input sequence is sorted according to its attention score. The search is done according to the principle of “same entity tag > other words,” and the found words are translated by looking up the bilingual dictionary. (2) For other decoding steps with UNK tags, the search is carried out according to the principle of “UNK tags > other words,” and the others are similar.For example, when translating “Annie visited Peking University in China in 2018.,” the output sequence is mapped by dictionary to “<PER> visited <LOC> in China in 2018.” According to the above decoding rules, for the <PER> tag, it was found that the source-end sequence did not have a <PER> tag, so the output of this step was mapped to the source-end word “Annie” with the highest ATTENTION score. It is translated as “Annie” by regularization methods such as dictionary search. For the <LOC> tag, although the word with the highest attention score is “in,” the source sequence contains the same <LOC> tag at this time, so we locate the position and get the correct translation result of “Peking University” by word translation, as shown in Figure1.Figure 1
Example of coding and decoding process for entity tagging.


## 3. A Neural Network Translation Model Incorporating Attentional Mechanisms

### 3.1. Neural Machine Translation Module

To date, various neural machine translation frameworks have been proposed. Among them, self-attentive-based frameworks achieve the most advanced translation performance. The self-attentive-based framework follows an encoder-decoder architecture, where the encoder converts a source sentenceX into a set of context vectors C. The decoder generates a target sentence Y from the context vectors C. Given a parallel dataset of sentence pairs D=X,Y, where X is the source sentence and Y is the target sentence, the loss function can be defined as follows:(1)LD;θ=∑X,Y∈DlogpY|X;θ,where p represents the probability function and θ represents the weight to be updated.


### 3.2. LSTM Module

Recurrent neural network (RNN) is a class of neural networks designed for the processing of serialised information. Traditional perceptrons and convolutional neural networks generally accept information as a whole and have difficulty modelling serialised information such as time and history, in which cases recurrent neural networks can be better adapted. Based on the basic ideas of recurrent neural networks, a number of quite effective improvements have been produced to address their common problems and limitations. An example is the Long Short-Term Memory Module [22, 23], also known as LSTM.Unlike the RNN network architecture, the LSTM has three additional "gates": a forgetting gate, an input gate, and an output gate. The forgetting gate is a probability vector based on the current input, so that the information from the previous step is passed on with a certain weight. The input gate is responsible for the selective transfer of long-distance dependent information, so that only part of the input information of the current step is involved in the state update. For the output values, the input information is used to generate output gates that filter the updated state information. The multiple gating settings strictly control the flow of information in the sequential network and improve the performance of the RNN in long sequences, as shown in Figure2.Figure 2
Architecture of long short-term memory.The output of the next moment in the LSTM is calculated together with the output of the last moment whenever there is a new word vector input to the network. The hidden layer loops and keeps the latest state. LSTM connects the hidden layer with a traditional feedforward network as the output layer. Each nodeyi in the output layer corresponds to the unnormalised log probability at the next moment, and the output y is then normalised by the softmax function. The equation for this is as follows:(2)y^t=softmaxWsht,where y^t is the probability distribution of the hidden layer calculated based on all vocabulary in each iteration. That is, the weights of all the predefined words in the document and the observed word vector xt will be determined when the model predicts the following words.The LSTM also needs to decide which information is forgotten by the neurons. For a datasetXt∈Rn×x with sample size n and feature vector dimension x at moment t, the state of the hidden layer at the previous moment is denoted as Ht−1∈Rn×h, at which point the forgetting gate is represented as follows:(3)ft=σXt⋅Wxf+Ht−1⋅Whf+bf.Here, Whf and Wxf are the learnable weight parameters and bf is the bias vector parameter.


### 3.3. Translation Models Using Attention Mechanisms

Attentional mechanisms can model the attentional model of the human brain. When a person sees a picture, the eye focuses on certain parts of the picture rather than the whole picture. The human brain pays attention to different parts of the picture differently, and this is at the heart of the attention mechanism.In this paper, we propose to incorporate an attention mechanism in the decoder part to solve the long-distance dependence problem of translation models dealing with long sequences. The proposed model is mainly based on an attention-based LSTM. Using the forgetting, input, and output gates of the LSTM reduces the number of features required for text encoding and allows for efficient encoding of long-range word dependencies. An LSTM incorporating an attention mechanism is able to access both previous and subsequent contextual information, treating the information obtained as two distinct text features. This feature is then fed into the attention layer, which selects the feature that is highly relevant to the entity marker, as shown in Figure3.Figure 3
Translation model using the attention mechanism.Using such processing, the attention mechanism can significantly improve the accuracy of the translation and reduce the number of learnable weights required to run and efficiently learn the contextual embedded meaning of variable-length sentences around the target word.The attention mechanism in this model uses the Bahdanau attention model, which calculates the contextual word vector for thei-th target word based on the hidden vector hj of the source word vector and the weights αij.(4)ci=∑j=1Tαijhj.The formula for calculating the weights is as follows:(5)αij=expeij∑k=1Texpeik.Here, eij denotes an alignment model that refers to the fraction of the input at position j that matches the output at position i.


### 3.4. Loss Functions

In this paper, the model is trained to learn a linear Softmax classifier model by minimizing the cross-entropy loss function [24].(6)LossW=−∑i∈Jt∑kyi,klnpyi,k|vi;W.For the proposed Chinese-English translation model,W denotes the network model weight, k denotes the number of entity tokens, vi denotes the entity token of the i-th text, n denotes the maximum number of texts, Jt denotes the set of training samples after text annotation, and yi,k denotes the entity token corresponding to the binary representation.


## 3.1. Neural Machine Translation Module

To date, various neural machine translation frameworks have been proposed. Among them, self-attentive-based frameworks achieve the most advanced translation performance. The self-attentive-based framework follows an encoder-decoder architecture, where the encoder converts a source sentenceX into a set of context vectors C. The decoder generates a target sentence Y from the context vectors C. Given a parallel dataset of sentence pairs D=X,Y, where X is the source sentence and Y is the target sentence, the loss function can be defined as follows:(1)LD;θ=∑X,Y∈DlogpY|X;θ,where p represents the probability function and θ represents the weight to be updated.


## 3.2. LSTM Module

Recurrent neural network (RNN) is a class of neural networks designed for the processing of serialised information. Traditional perceptrons and convolutional neural networks generally accept information as a whole and have difficulty modelling serialised information such as time and history, in which cases recurrent neural networks can be better adapted. Based on the basic ideas of recurrent neural networks, a number of quite effective improvements have been produced to address their common problems and limitations. An example is the Long Short-Term Memory Module [22, 23], also known as LSTM.Unlike the RNN network architecture, the LSTM has three additional "gates": a forgetting gate, an input gate, and an output gate. The forgetting gate is a probability vector based on the current input, so that the information from the previous step is passed on with a certain weight. The input gate is responsible for the selective transfer of long-distance dependent information, so that only part of the input information of the current step is involved in the state update. For the output values, the input information is used to generate output gates that filter the updated state information. The multiple gating settings strictly control the flow of information in the sequential network and improve the performance of the RNN in long sequences, as shown in Figure2.Figure 2
Architecture of long short-term memory.The output of the next moment in the LSTM is calculated together with the output of the last moment whenever there is a new word vector input to the network. The hidden layer loops and keeps the latest state. LSTM connects the hidden layer with a traditional feedforward network as the output layer. Each nodeyi in the output layer corresponds to the unnormalised log probability at the next moment, and the output y is then normalised by the softmax function. The equation for this is as follows:(2)y^t=softmaxWsht,where y^t is the probability distribution of the hidden layer calculated based on all vocabulary in each iteration. That is, the weights of all the predefined words in the document and the observed word vector xt will be determined when the model predicts the following words.The LSTM also needs to decide which information is forgotten by the neurons. For a datasetXt∈Rn×x with sample size n and feature vector dimension x at moment t, the state of the hidden layer at the previous moment is denoted as Ht−1∈Rn×h, at which point the forgetting gate is represented as follows:(3)ft=σXt⋅Wxf+Ht−1⋅Whf+bf.Here, Whf and Wxf are the learnable weight parameters and bf is the bias vector parameter.


## 3.3. Translation Models Using Attention Mechanisms

Attentional mechanisms can model the attentional model of the human brain. When a person sees a picture, the eye focuses on certain parts of the picture rather than the whole picture. The human brain pays attention to different parts of the picture differently, and this is at the heart of the attention mechanism.In this paper, we propose to incorporate an attention mechanism in the decoder part to solve the long-distance dependence problem of translation models dealing with long sequences. The proposed model is mainly based on an attention-based LSTM. Using the forgetting, input, and output gates of the LSTM reduces the number of features required for text encoding and allows for efficient encoding of long-range word dependencies. An LSTM incorporating an attention mechanism is able to access both previous and subsequent contextual information, treating the information obtained as two distinct text features. This feature is then fed into the attention layer, which selects the feature that is highly relevant to the entity marker, as shown in Figure3.Figure 3
Translation model using the attention mechanism.Using such processing, the attention mechanism can significantly improve the accuracy of the translation and reduce the number of learnable weights required to run and efficiently learn the contextual embedded meaning of variable-length sentences around the target word.The attention mechanism in this model uses the Bahdanau attention model, which calculates the contextual word vector for thei-th target word based on the hidden vector hj of the source word vector and the weights αij.(4)ci=∑j=1Tαijhj.The formula for calculating the weights is as follows:(5)αij=expeij∑k=1Texpeik.Here, eij denotes an alignment model that refers to the fraction of the input at position j that matches the output at position i.


## 3.4. Loss Functions

In this paper, the model is trained to learn a linear Softmax classifier model by minimizing the cross-entropy loss function [24].(6)LossW=−∑i∈Jt∑kyi,klnpyi,k|vi;W.For the proposed Chinese-English translation model,W denotes the network model weight, k denotes the number of entity tokens, vi denotes the entity token of the i-th text, n denotes the maximum number of texts, Jt denotes the set of training samples after text annotation, and yi,k denotes the entity token corresponding to the binary representation.


## 4. Experiment and Result Analysis

### 4.1. Experimental Environment and Dataset

In order to control the cost and ensure the effectiveness of the experiment, the CAsia2015 bilingual alignment corpus with good data quality was selected as the training dataset, containing a total of 105,000 aligned sentence pairs. The test set was selected from the newstest2017 bilingual alignment corpus, containing a total of 2002 sentence pairs. newstest2017 was equally divided into three subsets, newstest2017-1, newstest2017-2, and newstest2017-3. The neural translation system was built on the deep learning framework TensorFlow machine translation system. For the LSTM-based translation model, the encoder is a recurrent neural network with a forward-backward depth of 2 layers each and the decoder is a unidirectional recurrent neural network with a depth of 4 layers. Based on the convergence experience, the number of running steps is set to 8 epochs.The experiments were conducted using the deep learning framework TensorFlow to build a neural machine translation system, and the development language was Python 2.7. The hardware configuration was Intel Xeon E5-2678 CPU, 2.50 GHz main frequency, 64 GB RAM, NVIDIA GTX 1080ti GPU, and the operating system was CentOS 7. The relevant parameters are set as shown in Table2.Table 2
Experimental parameter setting.
Parameter typeValueBatch size64Maximum acceptable sequence length100Hidden variable size1000Word vector size500Dropout_probability0.3Attention_methodBahdanau


### 4.2. Evaluation Standards

Translation quality is the most general measure of a good or bad translation model. The most commonly used quality evaluation metric for machine translation tasks is called BLEU (bilingual evaluation understudy) [25], which is an algorithm designed based on the co-occurrence frequency of words and phrases, and the formula for calculating BLEU is shown as follows:(7)BLEU=BP⋅exp∑n=1Nωnlogpn,where BP denotes the length penalty factor, ωn denotes the weight of the n-word (usually taken as a constant value of 1/n), and pn is the matching accuracy of the n-word.(8)BP=1,lc>ls,exp1−lslc,lc≤ls,where lc is the length of the candidate sentence and ls is the length of the reference sentence.(9)Pn=∑i∑kminhkci,maxj∈mhksij∑kminhkci,where ci denotes the whole sentence to be translated and si denotes the standard answer. hkci denotes the number of occurrences of Wk in ci, hksij denotes the number of occurrences of Wk in sij, and maxj∈mhksij denotes the number of words that occur most frequently in multiple standard answers.


### 4.3. Convergence of the Named Entity Location

When using the encoding preprocessing operation of named entity tagging, we simultaneously trained the named entity localization task for the target language. The convergence of the named entity localization task was presented as a change in the prediction accuracy of the named entity labels during training. After multiple training sessions on a dataset, cross-drops of entity labels and targets were calculated on the validation set and the resulting statistics are shown in Figure4, with the number of training cycles as the horizontal coordinate and the cross-entropy values as the vertical coordinate.Figure 4
Cross-entropy change curve for named entity localization.It can be seen that the overall value of cross entropy shows a decreasing trend as training proceeds and the decreasing curve eventually converges unchanged, indicating that the proposed entity labeling improvement method effectively compensates for the weakness of the randomness of the neural network. The proposed method completes the normal training of low-frequency words, thus better adapting to the neural network machine translation model.


### 4.4. Performance Comparison of Translation Models

Different neural networks were trained separately in the translation models. The performance comparison results of the translation models with different network structures on the Chinese-English machine translation task are shown in Table3.Table 3
Performance comparison of translation models in BLEU.
Translation modelcasia2015newstest2017-1newstest2017-2newstest2017-3BPNN12.3911.8711.6311.79CNN14.4714.1513.8313.99RNN14.4914.1613.8514.01LSTM16.2115.8215.4915.58BPNN + Attention21.3416.7516.4116.62CNN + Attention23.2722.7422.4722.53RNN + Attention23.3122.7722.4922.55LSTM + Attention24.2823.8623.3723.41As can be seen from Table3, the performance of the LSTM-based translation models is better (larger BLEU values) compared to models with BPNN, CNN, and RNN structures without the use of the attention mechanism. The translation models incorporating the attention mechanism on the original structure each achieved improved performance. At this point, the LSTM model incorporating the attention mechanism achieves optimal performance. This is because the long-distance dependency problem is solved by enhancing the representation of contextual information at the source language end, thus improving the translation accuracy of long sentence sequences.Next, the LSTM + Attention model with entity tagging improvement and the LSTM + Attention model without entity tagging improvement were compared for the low-frequency word problem, and the results are shown in Figure5.Figure 5
Analysis of the effect of entity tagging improvement.It can be seen that although the addition of the attention mechanism leads to an improvement in translation quality, the increase in scores due to tag replacement after the entity tagging improvement leads to a further improvement in overall performance, with a mean BLEU value of 24.7%. This indicates that the use of entity labeling improvement can solve the problem of randomness in neural networks, enhance the training of low-frequency words, and thus better adapt the neural network model to improve the overall performance of machine translation.


## 4.1. Experimental Environment and Dataset

In order to control the cost and ensure the effectiveness of the experiment, the CAsia2015 bilingual alignment corpus with good data quality was selected as the training dataset, containing a total of 105,000 aligned sentence pairs. The test set was selected from the newstest2017 bilingual alignment corpus, containing a total of 2002 sentence pairs. newstest2017 was equally divided into three subsets, newstest2017-1, newstest2017-2, and newstest2017-3. The neural translation system was built on the deep learning framework TensorFlow machine translation system. For the LSTM-based translation model, the encoder is a recurrent neural network with a forward-backward depth of 2 layers each and the decoder is a unidirectional recurrent neural network with a depth of 4 layers. Based on the convergence experience, the number of running steps is set to 8 epochs.The experiments were conducted using the deep learning framework TensorFlow to build a neural machine translation system, and the development language was Python 2.7. The hardware configuration was Intel Xeon E5-2678 CPU, 2.50 GHz main frequency, 64 GB RAM, NVIDIA GTX 1080ti GPU, and the operating system was CentOS 7. The relevant parameters are set as shown in Table2.Table 2
Experimental parameter setting.
Parameter typeValueBatch size64Maximum acceptable sequence length100Hidden variable size1000Word vector size500Dropout_probability0.3Attention_methodBahdanau


## 4.2. Evaluation Standards

Translation quality is the most general measure of a good or bad translation model. The most commonly used quality evaluation metric for machine translation tasks is called BLEU (bilingual evaluation understudy) [25], which is an algorithm designed based on the co-occurrence frequency of words and phrases, and the formula for calculating BLEU is shown as follows:(7)BLEU=BP⋅exp∑n=1Nωnlogpn,where BP denotes the length penalty factor, ωn denotes the weight of the n-word (usually taken as a constant value of 1/n), and pn is the matching accuracy of the n-word.(8)BP=1,lc>ls,exp1−lslc,lc≤ls,where lc is the length of the candidate sentence and ls is the length of the reference sentence.(9)Pn=∑i∑kminhkci,maxj∈mhksij∑kminhkci,where ci denotes the whole sentence to be translated and si denotes the standard answer. hkci denotes the number of occurrences of Wk in ci, hksij denotes the number of occurrences of Wk in sij, and maxj∈mhksij denotes the number of words that occur most frequently in multiple standard answers.


## 4.3. Convergence of the Named Entity Location

When using the encoding preprocessing operation of named entity tagging, we simultaneously trained the named entity localization task for the target language. The convergence of the named entity localization task was presented as a change in the prediction accuracy of the named entity labels during training. After multiple training sessions on a dataset, cross-drops of entity labels and targets were calculated on the validation set and the resulting statistics are shown in Figure4, with the number of training cycles as the horizontal coordinate and the cross-entropy values as the vertical coordinate.Figure 4
Cross-entropy change curve for named entity localization.It can be seen that the overall value of cross entropy shows a decreasing trend as training proceeds and the decreasing curve eventually converges unchanged, indicating that the proposed entity labeling improvement method effectively compensates for the weakness of the randomness of the neural network. The proposed method completes the normal training of low-frequency words, thus better adapting to the neural network machine translation model.


## 4.4. Performance Comparison of Translation Models

Different neural networks were trained separately in the translation models. The performance comparison results of the translation models with different network structures on the Chinese-English machine translation task are shown in Table3.Table 3
Performance comparison of translation models in BLEU.
Translation modelcasia2015newstest2017-1newstest2017-2newstest2017-3BPNN12.3911.8711.6311.79CNN14.4714.1513.8313.99RNN14.4914.1613.8514.01LSTM16.2115.8215.4915.58BPNN + Attention21.3416.7516.4116.62CNN + Attention23.2722.7422.4722.53RNN + Attention23.3122.7722.4922.55LSTM + Attention24.2823.8623.3723.41As can be seen from Table3, the performance of the LSTM-based translation models is better (larger BLEU values) compared to models with BPNN, CNN, and RNN structures without the use of the attention mechanism. The translation models incorporating the attention mechanism on the original structure each achieved improved performance. At this point, the LSTM model incorporating the attention mechanism achieves optimal performance. This is because the long-distance dependency problem is solved by enhancing the representation of contextual information at the source language end, thus improving the translation accuracy of long sentence sequences.Next, the LSTM + Attention model with entity tagging improvement and the LSTM + Attention model without entity tagging improvement were compared for the low-frequency word problem, and the results are shown in Figure5.Figure 5
Analysis of the effect of entity tagging improvement.It can be seen that although the addition of the attention mechanism leads to an improvement in translation quality, the increase in scores due to tag replacement after the entity tagging improvement leads to a further improvement in overall performance, with a mean BLEU value of 24.7%. This indicates that the use of entity labeling improvement can solve the problem of randomness in neural networks, enhance the training of low-frequency words, and thus better adapt the neural network model to improve the overall performance of machine translation.


## 5. Conclusions

This paper presents a neural network machine translation model based on entity tagging improvement. An improved entity tag coding/decoding strategy is used to map all target words to a controlled size lexicon to achieve an adaptation to the neural network model. In addition, the LSTM decoder enhances the representation of contextual information at the source language end by incorporating an attention mechanism, thus improving the translation accuracy of long sentence sequences. The comparison results between multiple neural network machine translation models show that the proposed model has the highest performance with a mean BLEU value of 24.7%, effectively solving the low-frequency word problem and the long-distance dependency problem. Due to the limitation of hardware conditions, a very large amount of data cannot be used in the selection of training data and similarly the parameters such as the number of training sessions and data dimension cannot be set too large. Larger scale experimental tests will be conducted on more types of datasets later.



---

*Source: 8407437-2022-04-26.xml*