# Unsupervised English Intelligent Machine Translation in Wireless Network Environment

Authors: Bing Zhang
Journal: Security and Communication Networks
 (2022)

---

## Abstract

Researchers suggest unsupervised English machine translation to address the absence of parallel corpus in English translation. Unsupervised pretraining techniques, denoising autoencoders, back translation, and shared latent representation mechanisms are used to simulate the translation task using just monolingual corpora. This paper uses pseudo-parallel data to construct unsupervised neural machine translation (NMT) and dissimilar language pair analysis. This paper firstly analyzes the low performance of unsupervised translation on dissimilar language pairs from three aspects: bilingual word embedding quality, shared words, and word order. And artificial shared word replacement and preordering strategies are proposed to increase the shared words between dissimilar language pairs and reduce the difference in their syntactic structure, thereby improving the translation performance on dissimilar language pairs. The denoising autoencoder and shared latent representation mechanism in unsupervised English machine translation are only required in the early stage of training, and learning the shared latent representation limits the further improvement of performance in different directions. While training the denoising autoencoder by repeatedly altering the training data slows down the convergence of the model, this is especially true for divergent languages. This paper presents an unsupervised NMT model based on pseudo-parallel data to address this issue. It trains two standard supervised neural machine translation models using the pseudo-parallel corpus generated by the unsupervised neural machine translation system, which enhances translation performance and speeds convergence. Finally, the English intelligent translation model is deployed in the wireless network server, and users can access it through the wireless network.


---

## Body

## 1. Introduction

It is the language that connects us all. The intertranslation of global languages has become a necessary requirement for human-friendly exchanges around the world due to the rapid expansion of modern civilization and the progressive construction of global integration. Global integration relies heavily on cross-language communication, which is made easier via translation. Human translators have long been unable to keep up with today’s information explosion and high-speed operation, thanks to the development of the translation industry. The focus of English-language translation research has shifted to how to communicate information more efficiently, precisely, and conveniently across multiple languages. Since the beginning of time, hand translation has been the most common method of translation. As computer technology has advanced, so has machine translation technology, which has now entered the history books. Machine translation, as the name suggests, is a technology that uses the efficient computing power of computers to convert and express information between two languages. Machine translation technology has progressed from academic research to practical application, and it is now the driving force behind the world translation industry’s rapid growth. Neural machine translation systems are increasingly being used in everyday life. Many Internet companies such as Baidu, Google, and NetEase provide free online translation software. Machine translation is combined with image and speech technology to help us solve the language barriers we encounter in our lives. Although the accuracy of machine translation is not enough to replace human translation, machine translation can still play a very good auxiliary role in real life [1–5].There are three main approaches to machine translation that have been used for decades: rule-based, instance-based, and statistical. Translation technology was dominated by rule-based systems prior to 1988. Lexical rules, lexical transformation rules, and syntax generation rules are among the most common types of rules employed in rule-based machine translation. Thereby, related technologies such as text processing, syntactic analysis, and dictionary query of the source language are derived. And the conversion from the source language to the target language requires the help of a dictionary query. Instance-based machine translation techniques differ from rule-based approaches. This method relies more on a bilingual corpus instance database and obtains the target language by matching it with the corpus. The basic idea is to input the sentences to be translated into the EBMT system and then match them in the bilingual corpus instance database to find the source language sentences with the greatest similarity with the input sentences. The corresponding target language translation is generated according to the standard translation of the matching sentence in the instance library. The main idea of statistical-based machine translation methods is to collect large-scale bilingual parallel corpora. On this basis, statistics and analysis are performed to construct a useable statistical translation model. Previous statistically based machine translation techniques relied almost completely on word-level modeling with explicit segmentation due to data sparsity constraints. This is especially true for statistical machine translation n-gram networks [6–10], which represent a sentence as a series of characters rather than words.In recent years, computer hardware has been iteratively updated, and its performance has become better and better. Both computing power and computing consumption cost have been greatly improved. Artificial intelligence has developed since its initial sprouting, and the technology at the academic research level and practical application level has become more mature. Today, human life has long been closely related to artificial intelligence technology. Machine learning is popular research in artificial intelligence, and it is also a key development subject of cutting-edge computer technology. Machine translation has broad application prospects but also has a certain complexity. Therefore, both in academic research and industrial applications, machine translation is regarded as a key research task in natural language processing. With the advent of the era of artificial intelligence revolution, machine learning has undoubtedly injected new vitality into machine translation technology [11–15].This work is devoted to researching an efficient unsupervised intelligent English translation method and deploying the translation model in a wireless network. This can efficiently provide users with English intelligent translation services and improve translation quality.The paper’s organization paragraph is as follows. The related work is presented in Section2. Section 3 analyzes the methods of the proposed work. Section 4 discusses the experiments and results. Finally, in Section 5, the research work is concluded.


## 2. Related Work

The encoder-decoder model, first introduced in [16], is still widely used. The model encodes source language sentences into a fixed-length vector using a recurrent neural network. For each moment in time, a new target vocabulary is predicted by using a combination of the target vocabulary outputs from prior moments. The quality of the translations produced by this method is on par with that of more traditional statistically based methods. To account for the varying lengths of sentences, semantic information is represented as a fixed-length vector. Each phrase will have varied amounts of information in its corresponding vector, which directly influences the neural machine translation model’s translation quality for longer sentences. For the long-distance dependency problem, [17] suggested a bidirectional recurrent neural network model based on attention mechanisms. Using the bidirectional recurrent neural network, the model is able to learn not only the relationship between current and past words but also the relationship between the present word and the following words. The attention mechanism is a weight calculation mechanism between the encoder and the decoder. When decoding and predicting the current target vocabulary, not only the decoder output at the previous moment but also the weighted summation result obtained by calculating the weight according to the current hidden layer state as well as the hidden layer state for the encoder at all times is combined. Reference [18] improves the attention mechanism and proposes two concepts, global attention mechanism and local attention mechanism, to further alleviate the problem of long-distance dependence. Reference [19] proposes to use convolutional neural networks to implement machine translation models. The model uses the convolution method to continuously expand from local features to global features and the characteristics of parallel computing to optimize sequence-to-sequence machine translation tasks. Reference [20] proposed an NMT model based entirely on attention and proposed a self-attention mechanism to encode bilingual sentences and learn their semantic information through the self-attention mechanism.New machine translation methods were studied using a neural network encoder-decoder structure that was presented in [21]. Based on neural networks, the data sparsity issue in traditional statistical machine translation models was greatly improved. Their approach also provides the groundwork for future neural network-based machine translation applications. An end-to-end encoder-decoder model for machine translation was proposed in [22, 23]. When encoding a source-language text into a set of vector forms, an end-to-end model employs a convolution neural network as the encoder. As part of the end-to-end model structure, a recurrent neural network (RNN) serves as the decoder and is responsible for predicting the target language from the vector encoded by the encoder. The end-to-end model structure’s first use in machine translation heralds the birth of neural machine translation and illuminates the future course of that technology. The nonlinear structure of RNN-based neural machine translation differs from the linear structure of statistical machine translation models, which use state vectors connecting the end-to-end encoder and decoder to describe semantic equivalence. Furthermore, RNN can theoretically analyze sentences of any length, mine deep semantic information in phrases, and, to some extent, solve difficulties like long-distance rearrangement in statistical machine translation. However, investigations have revealed that when dealing with long-distance dependency problems, there is no matching memory unit in the recurrent neural network node. Due to the vanishing gradient in the training process, the perception of the new input information to the earlier input information decreases as time progresses. This will seriously affect the translation performance of the model when dealing with translation tasks, resulting in a decrease in translation quality [24]. Recurrent neural networks can be utilized as both encoders and decoders according to the principle of sequence-to-sequence [25]. Long Short-Term Memory (LSTM) for neural machine translation was also introduced by including a gate structure and memory cells in the model. The gradient explosion and disappearance during training may be managed in this way, which alleviates the problem of long-distance reliance to some extent. Reference [26] explained how to deal with a fixed encoder vector dimension by introducing an attention mechanism into a neural machine translation model. To compute the alignment probability of the target word, a decoder simply needs to calculate the weight of a portion of the source language words that are closely connected to the target language word. To improve translation quality, we used a content-based attention mechanism to produce a weighted context vector from the source language sentence. The neural network then uses this context vector to forecast the target word.


## 3. Method

This chapter looks at the impact of different languages on unsupervised NMT performance from three perspectives: bilingual word embedding quality, shared words, and word order. Compared with similar language pairs, there are very few shared words in dissimilar language pairs. And the syntactic structure between dissimilar language pairs is also very different. Based on this, this chapter proposes an artificial shared word replacement strategy and preranking strategy. It can increase the shared words between dissimilar language pairs and reduce the difference in their syntactic structure to improve the performance of unsupervised translation on dissimilar language pairs.The following chapter offers a simple general technique to increase translation performance across all language pairs. Autoencoders and shared latent representations are only required for pretraining neural machine translation, according to this study. And unsupervised NMT model architectures can quickly become inefficient throughout the training process. Learning shared latent representations limits further improvements in translation performance in different directions, especially for dissimilar language pairs. At the same time, the denoising autoencoder training by continuously modifying the training data also greatly slows down the convergence speed of the model. To avoid these problems, this chapter proposes an unsupervised English translation model (UETPPD) based on pseudo-parallel data. The unsupervised neural machine translation model is transformed into a standard supervised neural machine translation system after a period of training, where pseudo-parallel data generated by an unsupervised neural machine translation model, rather than real parallel corpora, is used to train the translation system. The English translation model that only relies on a monolingual corpus achieves faster training speed, and it also significantly outperforms the traditional unsupervised English translation model in translation quality.


### 3.1. Mechanisms of Unsupervised Neural Machine Translation

Unsupervised neural machine translation (NMT) models mainly include the following four mechanisms: unsupervised cross-language representation pretraining, denoising autoencoders, back translation, and shared latent representations. Unsupervised neural machine translation models have a dual structure that can generate translations in two different directions.Formally, given a monolingual corpus in two languagesL1 and L2, unsupervised neural machine translation is first initialized with bilingual word embeddings or masked language model pretraining and then trained by maximizing the objective function shown in the following formula:(1)Le=Ld+Lb,where Ld represents the objective function of the denoising autoencoder, and Lb represents the objective function of back translation. The entire training process of the unsupervised NMT model needs to consider the denoising autoencoder process of the respective language and the back translation process between the two languages. Unsupervised neural machine translation training alternates between sentences in L1 language and L2 language. Specifically, each unsupervised neural machine translation training iteration process includes L1 denoising autoencoder training, L2 denoising autoencoder training, L1−L2 back translation training, and L2−L1 back translation training.In contrast to NMT models, unsupervised models do not have supervised bilingual signals during training. Therefore, the initialization of the unsupervised translation model by the unsupervised pretraining model provides a naive bilingual signal before the unsupervised neural machine translation model training starts. This pretraining model is mainly divided into two types, bilingual word embedding pretraining and mask language model pretraining strategies.Unsupervised bilingual word embeddings can learn word pair translations from two nonparallel monolingual corpora. Typically, bilingual word embeddings are used to initialize the word embedding layers of unsupervised neural machine translation encoders and decoders. To create larger phrase representations, the encoder will need to learn how to integrate these language-independent word-level representations. The naive translation information provided by bilingual word embeddings enables the generation of pseudo-supervised bilingual signals through back translation. The encoder and decoder word embeddings are altered independently during unsupervised neural network translation training.The masked language model is used to encode two single-sentence sentences into a single common latent space. The encoder and decoder of the unsupervised neural machine translation model are initialized using this pretrained cross-lingual encoder. When a decoder is built, the encoder-decoder Attention layer is randomly set to begin with. Unsupervised neural machine translation models benefit from masked language model pretraining methods because they supply more cross-lingual information than bilingual word embedding pretraining methods.It is possible to train the unsupervised neural machine translation model directly to recreate its own input because of the dual structure of the model. Encoding a sentence in one language and then reconstructing the original text using that language’s decoder is one way to optimize an unsupervised neural machine translation model. Without adding some constraints, it will be difficult for a normal autoencoder to learn useful knowledge, and it will become a replication task, learning only one-to-one replications of the input words. To alleviate this problem, a denoising autoencoder strategy is introduced and adds noise to the input sentences to improve the model learning ability. Specifically, two different types of noise were selected to be added to the input sentences. Any word in the input sentence is deleted with a certain probability, the greater the probability, the more words are deleted in the input sentence. The second type of noise is a slight adjustment to the word order of the input sentence. A random permutation is used in the input sentence to change the word order of the original sentence so that the following conditions are met:(2)i−σi≤a.To generate a random permutation satisfying the above conditions for an input sentence, a random vector is generated:(3)q=i+U0,b.In this sense, an unsupervised neural machine translation system must be able to grasp the fundamental structure of the languages involved in order to recover the correct word order from a given source text. At the same time, actual disparities in word order between languages can be better addressed by preventing unsupervised neural machine translation systems from relying too much on the word order of input sentences. The denoising autoencoder encodes the above noisy input sentence and reconstructs it with the same language decoder. It optimizes by minimizing the objective function.The denoising autoencoder acts as a language model, trained in the same language, without considering the ultimate goal of translating between two languages. So back translation is introduced to train this unsupervised translation system using only monolingual corpus. Back translation can transform this unsupervised problem into a supervised learning task, albeit with noisy source sentences. Given an input sentence in one languageL1, in inference mode using L1 encoder for encoding and L2 decoder for decoding, greedy decoding is used to convert it into a translation in another language. In this way, pseudo-parallel sentence pairs are obtained and used to train an unsupervised neural machine translation system to predict the original sentence from this noisy synthetic translation. Standard back translation is to back-translate the entire corpus at once using a single model. When the unsupervised neural machine translation model is being trained, the model is utilized to back-translate each batch of words in real time because of the unsupervised neural machine translation system’s dual structure. Using back translation, better pseudo-parallel sentence pairings are generated as training advances and the model improves, which will aid in the model’s improvement in future rounds.The encoders and decoders of the two languages are shared or partially shared, and the two languages use the same vocabulary. Regardless of the input source language, the purpose of the shared encoder is to produce a language-independent latent representation of the input text. The decoder can use this latent representation to convert the target translation into the corresponding language. This ensures the benefit of the language model implemented by the denoising autoencoder, which can translate well from noisy source sentences to the corresponding target translations. Ultimately helping unsupervised neural machine translation models achieve smoother translations. Shared encoders are the key to making unsupervised translation models work, while shared decoders just introduce useful regularization.


### 3.2. Specific Language Analysis

Intuitively, we believe that languages belonging to the same or comparable language families should be similar. Measures like mutual understanding, lexical similarity, and syntactic similarity are commonly employed to assess the linguistic similarities of two languages from an objective standpoint. Language comprehension can also be asymmetrical, with asymmetric intelligibility referring to two languages that are only partially understandable to each other. But speakers of one language have more difficulty understanding the other than speakers of the other language. The problem of language pair similarity in machine translation is difficult to measure by mutual intelligibility, but it can be compared by lexical similarity. Lexical similarity is a linguistic measure of how similar two languages’ lexical structures are. No terms from either language are shared between languages with a lexical similarity of 1; conversely, no words from either language are shared. There are no exceptions to the rule of lexical similarity, which is symmetrical. The definition of lexical similarity can be defined in a variety of ways, with variable consequences. According to the Ethnologue, words that are similar in both form and meaning are counted in a regionally defined word list. Table1 shows the statistics of lexical similarity for some common languages.Table 1
Statistics of lexical similarity.
SimilarityEnGeFrEn1.000.600.27Ge0.601.000.29Fr0.270.291.00Through preexperiments, the performance of unsupervised neural machine translation in German-English and French-English directions is much better than that in Japanese-English and Chinese-English directions. In the German-English and French-English directions, the BLEU score of unsupervised neural machine translation can reach above 20. In the Japanese-English and Chinese-English directions, the BLEU scores of unsupervised neural machine translation are only 14.09 and 8.02, respectively. This chapter proposes methods to improve the performance of unsupervised neural machine translation on English translation tasks.Table2 shows how the precision of bilingual word embedding affects the performance of neural machine translation on the French-English and Japanese-English translation challenges (BWE). To produce bilingual word embedding with varying precisions, this part employs the bilingual word embedding of different VecMap training nodes to initialize the unsupervised neural machine translation model, resulting in varied precisions of bilingual word embedding.Table 2
UNMT performance using BWE.
Precision0520304560Ja-En1.3011.2012.1013.5013.9014.30Fr-En20.1021.0022.1023.3023.1023.70Unsupervised neural machine translation performance improves for both language pairs as the accuracy of bilingual word embeddings increases. This shows a link between unsupervised neural machine translation performance and the quality of bilingual word embeddings. Unsupervised neural machine translation’s performance in the French-to-English translation direction is still superior to the translation performance in the Japanese-to-English direction in the case of similar bilingual word embedding accuracy. On the French-English translation task, even if the unsupervised neural machine translation model is initialized with only monolingual word embeddings, the unsupervised neural machine translation model can still achieve good translation performance. In contrast, the BLEU score for Japanese-English translation under this initialization condition is below 1. As a result, unsupervised neural machine translation on Japanese-English and Chinese-English translation tasks performed poorly, even when using high-quality bilingual word embeddings.Similar language pairs, like German-English and French-English, have a lot of vocabulary in common. There are a few terms that can be used interchangeably in languages that are vastly different. This section includes an ablation preexperiment to assess the impact of shared phrases on unsupervised neural machine translation. A unique symbol is used to distinguish between words that have already been shared. This section uses both random and VecMap bilingual pretraining to train bilingual word embeddings for two different shared word settings. VecMap bilingual pretraining is the same for natural shared words and no shared phrases. Unsupervised neural machine translation training without addressing the impact of shared terms is the focus of this section. Random word embeddings were used to train an unsupervised neural machine translation model for all language pairs. It is required to pretrain multilingual word embeddings when there are no common terms to use. Using monolingual word embedding word pretraining to increase the performance of unsupervised neural machine translation models can be quite effective if there are enough shared words to do so. Unsupervised neural machine translation is significantly impacted by the performance of shared words. For unsupervised neural machine translation of dissimilar language pairs, shared words are the primary cause of poor performance. Comparing comparable and dissimilar language pairs, the quality of bilingual word embeddings tends to be better, but the reverse is also true: unsupervised neural machine translation performs well when the source and target languages have a high degree of similarity.The third premise is that English is the only language in which there is a major difference between it and other languages. In contrast to supervised neural machine translation, which uses only monolingual corpora, unsupervised neural machine translation lacks a bilingual signal. Furthermore, by replacing and rearranging words locally, the denoising algorithm only modifies the order of surrounding words. Learning syntactic information in language pairs that are vastly diverse is difficult for unsupervised neural machine translation. Unfortunately, unsupervised neural machine translation does not handle language pairs with wildly divergent syntactic structures particularly well.A preordering toolkit was used to modify word order in training and test data in order to verify the influence of word order. This method dramatically enhances the performance of neural machine translation models that do not require any prior training. It is more effective in the early phases of unsupervised neural machine translation training when compared to the preordering impact on supervised translation machine translations.


### 3.3. Unsupervised English Translation with Pseudo-Parallel Data

This section proposes an unsupervised neural machine translation model with pseudo-parallel data. The unsupervised NMT model is transformed into a standard supervised neural machine translation system after several epochs of training. This English translation system, named UETPPD, is trained on pseudo-parallel sentence pairs generated by an unsupervised NMT model. Compared with traditional unsupervised NMT models, the training process of unsupervised NMT based on pseudo-parallel data is simpler. The pipeline is illustrated in Figure1.Figure 1
UNMT and UETPPD training process.Two independent unsupervised neural machine translation models were trained using the pseudo-parallel corpus obtained by training an unsupervised neural machine translation model for a period of time. Each of the two unsupervised neural machine translation models is optimized by minimizing the objective function.This part employs the unsupervised neural machine translation model to build two pseudo-parallel corpora, which is the initial stage in training the unsupervised neural machine translation model based on pseudo-parallel data. Nonsupervised neural machine translation models are trained iteratively using pseudo-parallel corpora. Unsupervised neural machine translation models generate a pseudo-parallel corpus in the second stage, which is then utilized to separately train the model. It is possible to construct parallel pseudo-parallel corpora by training unsupervised neural machine translation models with pseudo-parallel data. The unsupervised neural machine translation model is trained independently using the pseudo-parallel corpus in the third stage. According to this approach, a pseudo-parallel corpus with superior translation quality can be obtained using an unsupervised neural machine translation model based on the use of parallel data. Unsupervised neural machine translation models based on pseudo-parallel data can be used to constantly develop these pseudo-parallel corpora. It is repeated in the fourth stage, but this time the pseudo-parallel corpus generated from trained unsupervised neural machine translation models with pseudo-parallel data is employed.


### 3.4. Combination with Wireless Network

With the change and development of wireless networks, more and more engineering tasks are closely associated with wireless networks. In this work, the English translation model is trained and deployed in a wireless network environment. External users can access the model through the wireless network and upload the content that needs to be translated. After the translation model is processed, the translated content is transmitted to the user. The model framework is illustrated in Figure2.Figure 2
Combination with a wireless network.


## 3.1. Mechanisms of Unsupervised Neural Machine Translation

Unsupervised neural machine translation (NMT) models mainly include the following four mechanisms: unsupervised cross-language representation pretraining, denoising autoencoders, back translation, and shared latent representations. Unsupervised neural machine translation models have a dual structure that can generate translations in two different directions.Formally, given a monolingual corpus in two languagesL1 and L2, unsupervised neural machine translation is first initialized with bilingual word embeddings or masked language model pretraining and then trained by maximizing the objective function shown in the following formula:(1)Le=Ld+Lb,where Ld represents the objective function of the denoising autoencoder, and Lb represents the objective function of back translation. The entire training process of the unsupervised NMT model needs to consider the denoising autoencoder process of the respective language and the back translation process between the two languages. Unsupervised neural machine translation training alternates between sentences in L1 language and L2 language. Specifically, each unsupervised neural machine translation training iteration process includes L1 denoising autoencoder training, L2 denoising autoencoder training, L1−L2 back translation training, and L2−L1 back translation training.In contrast to NMT models, unsupervised models do not have supervised bilingual signals during training. Therefore, the initialization of the unsupervised translation model by the unsupervised pretraining model provides a naive bilingual signal before the unsupervised neural machine translation model training starts. This pretraining model is mainly divided into two types, bilingual word embedding pretraining and mask language model pretraining strategies.Unsupervised bilingual word embeddings can learn word pair translations from two nonparallel monolingual corpora. Typically, bilingual word embeddings are used to initialize the word embedding layers of unsupervised neural machine translation encoders and decoders. To create larger phrase representations, the encoder will need to learn how to integrate these language-independent word-level representations. The naive translation information provided by bilingual word embeddings enables the generation of pseudo-supervised bilingual signals through back translation. The encoder and decoder word embeddings are altered independently during unsupervised neural network translation training.The masked language model is used to encode two single-sentence sentences into a single common latent space. The encoder and decoder of the unsupervised neural machine translation model are initialized using this pretrained cross-lingual encoder. When a decoder is built, the encoder-decoder Attention layer is randomly set to begin with. Unsupervised neural machine translation models benefit from masked language model pretraining methods because they supply more cross-lingual information than bilingual word embedding pretraining methods.It is possible to train the unsupervised neural machine translation model directly to recreate its own input because of the dual structure of the model. Encoding a sentence in one language and then reconstructing the original text using that language’s decoder is one way to optimize an unsupervised neural machine translation model. Without adding some constraints, it will be difficult for a normal autoencoder to learn useful knowledge, and it will become a replication task, learning only one-to-one replications of the input words. To alleviate this problem, a denoising autoencoder strategy is introduced and adds noise to the input sentences to improve the model learning ability. Specifically, two different types of noise were selected to be added to the input sentences. Any word in the input sentence is deleted with a certain probability, the greater the probability, the more words are deleted in the input sentence. The second type of noise is a slight adjustment to the word order of the input sentence. A random permutation is used in the input sentence to change the word order of the original sentence so that the following conditions are met:(2)i−σi≤a.To generate a random permutation satisfying the above conditions for an input sentence, a random vector is generated:(3)q=i+U0,b.In this sense, an unsupervised neural machine translation system must be able to grasp the fundamental structure of the languages involved in order to recover the correct word order from a given source text. At the same time, actual disparities in word order between languages can be better addressed by preventing unsupervised neural machine translation systems from relying too much on the word order of input sentences. The denoising autoencoder encodes the above noisy input sentence and reconstructs it with the same language decoder. It optimizes by minimizing the objective function.The denoising autoencoder acts as a language model, trained in the same language, without considering the ultimate goal of translating between two languages. So back translation is introduced to train this unsupervised translation system using only monolingual corpus. Back translation can transform this unsupervised problem into a supervised learning task, albeit with noisy source sentences. Given an input sentence in one languageL1, in inference mode using L1 encoder for encoding and L2 decoder for decoding, greedy decoding is used to convert it into a translation in another language. In this way, pseudo-parallel sentence pairs are obtained and used to train an unsupervised neural machine translation system to predict the original sentence from this noisy synthetic translation. Standard back translation is to back-translate the entire corpus at once using a single model. When the unsupervised neural machine translation model is being trained, the model is utilized to back-translate each batch of words in real time because of the unsupervised neural machine translation system’s dual structure. Using back translation, better pseudo-parallel sentence pairings are generated as training advances and the model improves, which will aid in the model’s improvement in future rounds.The encoders and decoders of the two languages are shared or partially shared, and the two languages use the same vocabulary. Regardless of the input source language, the purpose of the shared encoder is to produce a language-independent latent representation of the input text. The decoder can use this latent representation to convert the target translation into the corresponding language. This ensures the benefit of the language model implemented by the denoising autoencoder, which can translate well from noisy source sentences to the corresponding target translations. Ultimately helping unsupervised neural machine translation models achieve smoother translations. Shared encoders are the key to making unsupervised translation models work, while shared decoders just introduce useful regularization.


## 3.2. Specific Language Analysis

Intuitively, we believe that languages belonging to the same or comparable language families should be similar. Measures like mutual understanding, lexical similarity, and syntactic similarity are commonly employed to assess the linguistic similarities of two languages from an objective standpoint. Language comprehension can also be asymmetrical, with asymmetric intelligibility referring to two languages that are only partially understandable to each other. But speakers of one language have more difficulty understanding the other than speakers of the other language. The problem of language pair similarity in machine translation is difficult to measure by mutual intelligibility, but it can be compared by lexical similarity. Lexical similarity is a linguistic measure of how similar two languages’ lexical structures are. No terms from either language are shared between languages with a lexical similarity of 1; conversely, no words from either language are shared. There are no exceptions to the rule of lexical similarity, which is symmetrical. The definition of lexical similarity can be defined in a variety of ways, with variable consequences. According to the Ethnologue, words that are similar in both form and meaning are counted in a regionally defined word list. Table1 shows the statistics of lexical similarity for some common languages.Table 1
Statistics of lexical similarity.
SimilarityEnGeFrEn1.000.600.27Ge0.601.000.29Fr0.270.291.00Through preexperiments, the performance of unsupervised neural machine translation in German-English and French-English directions is much better than that in Japanese-English and Chinese-English directions. In the German-English and French-English directions, the BLEU score of unsupervised neural machine translation can reach above 20. In the Japanese-English and Chinese-English directions, the BLEU scores of unsupervised neural machine translation are only 14.09 and 8.02, respectively. This chapter proposes methods to improve the performance of unsupervised neural machine translation on English translation tasks.Table2 shows how the precision of bilingual word embedding affects the performance of neural machine translation on the French-English and Japanese-English translation challenges (BWE). To produce bilingual word embedding with varying precisions, this part employs the bilingual word embedding of different VecMap training nodes to initialize the unsupervised neural machine translation model, resulting in varied precisions of bilingual word embedding.Table 2
UNMT performance using BWE.
Precision0520304560Ja-En1.3011.2012.1013.5013.9014.30Fr-En20.1021.0022.1023.3023.1023.70Unsupervised neural machine translation performance improves for both language pairs as the accuracy of bilingual word embeddings increases. This shows a link between unsupervised neural machine translation performance and the quality of bilingual word embeddings. Unsupervised neural machine translation’s performance in the French-to-English translation direction is still superior to the translation performance in the Japanese-to-English direction in the case of similar bilingual word embedding accuracy. On the French-English translation task, even if the unsupervised neural machine translation model is initialized with only monolingual word embeddings, the unsupervised neural machine translation model can still achieve good translation performance. In contrast, the BLEU score for Japanese-English translation under this initialization condition is below 1. As a result, unsupervised neural machine translation on Japanese-English and Chinese-English translation tasks performed poorly, even when using high-quality bilingual word embeddings.Similar language pairs, like German-English and French-English, have a lot of vocabulary in common. There are a few terms that can be used interchangeably in languages that are vastly different. This section includes an ablation preexperiment to assess the impact of shared phrases on unsupervised neural machine translation. A unique symbol is used to distinguish between words that have already been shared. This section uses both random and VecMap bilingual pretraining to train bilingual word embeddings for two different shared word settings. VecMap bilingual pretraining is the same for natural shared words and no shared phrases. Unsupervised neural machine translation training without addressing the impact of shared terms is the focus of this section. Random word embeddings were used to train an unsupervised neural machine translation model for all language pairs. It is required to pretrain multilingual word embeddings when there are no common terms to use. Using monolingual word embedding word pretraining to increase the performance of unsupervised neural machine translation models can be quite effective if there are enough shared words to do so. Unsupervised neural machine translation is significantly impacted by the performance of shared words. For unsupervised neural machine translation of dissimilar language pairs, shared words are the primary cause of poor performance. Comparing comparable and dissimilar language pairs, the quality of bilingual word embeddings tends to be better, but the reverse is also true: unsupervised neural machine translation performs well when the source and target languages have a high degree of similarity.The third premise is that English is the only language in which there is a major difference between it and other languages. In contrast to supervised neural machine translation, which uses only monolingual corpora, unsupervised neural machine translation lacks a bilingual signal. Furthermore, by replacing and rearranging words locally, the denoising algorithm only modifies the order of surrounding words. Learning syntactic information in language pairs that are vastly diverse is difficult for unsupervised neural machine translation. Unfortunately, unsupervised neural machine translation does not handle language pairs with wildly divergent syntactic structures particularly well.A preordering toolkit was used to modify word order in training and test data in order to verify the influence of word order. This method dramatically enhances the performance of neural machine translation models that do not require any prior training. It is more effective in the early phases of unsupervised neural machine translation training when compared to the preordering impact on supervised translation machine translations.


## 3.3. Unsupervised English Translation with Pseudo-Parallel Data

This section proposes an unsupervised neural machine translation model with pseudo-parallel data. The unsupervised NMT model is transformed into a standard supervised neural machine translation system after several epochs of training. This English translation system, named UETPPD, is trained on pseudo-parallel sentence pairs generated by an unsupervised NMT model. Compared with traditional unsupervised NMT models, the training process of unsupervised NMT based on pseudo-parallel data is simpler. The pipeline is illustrated in Figure1.Figure 1
UNMT and UETPPD training process.Two independent unsupervised neural machine translation models were trained using the pseudo-parallel corpus obtained by training an unsupervised neural machine translation model for a period of time. Each of the two unsupervised neural machine translation models is optimized by minimizing the objective function.This part employs the unsupervised neural machine translation model to build two pseudo-parallel corpora, which is the initial stage in training the unsupervised neural machine translation model based on pseudo-parallel data. Nonsupervised neural machine translation models are trained iteratively using pseudo-parallel corpora. Unsupervised neural machine translation models generate a pseudo-parallel corpus in the second stage, which is then utilized to separately train the model. It is possible to construct parallel pseudo-parallel corpora by training unsupervised neural machine translation models with pseudo-parallel data. The unsupervised neural machine translation model is trained independently using the pseudo-parallel corpus in the third stage. According to this approach, a pseudo-parallel corpus with superior translation quality can be obtained using an unsupervised neural machine translation model based on the use of parallel data. Unsupervised neural machine translation models based on pseudo-parallel data can be used to constantly develop these pseudo-parallel corpora. It is repeated in the fourth stage, but this time the pseudo-parallel corpus generated from trained unsupervised neural machine translation models with pseudo-parallel data is employed.


## 3.4. Combination with Wireless Network

With the change and development of wireless networks, more and more engineering tasks are closely associated with wireless networks. In this work, the English translation model is trained and deployed in a wireless network environment. External users can access the model through the wireless network and upload the content that needs to be translated. After the translation model is processed, the translated content is transmitted to the user. The model framework is illustrated in Figure2.Figure 2
Combination with a wireless network.


## 4. Experimental Results and Analysis

### 4.1. Comparison with Other Methods

To verify the validity and correctness of UETPPD, it is compared with other methods. The experimental results are illustrated in Table3.Table 3
Comparison with other methods.
MethodDe-EnEn-DeDr-EnEn-FrJa-EnEn-JaCh-EnEn-ChRef [27]——15.6015.10————Ref [28]13.309.6014.3015.00————Ref [29]14.6010.9015.6017.00————Ref [30]21.0017.2024.2025.10————UETPPD24.2018.9027.5028.9019.1027.0011.3010.10The classic unsupervised neural machine translation models are outperformed by a pseudo-parallel data-based model. Traditional unsupervised neural machine translation models have a lower average BLEU score in similar language pairs. Traditional unsupervised neural machine translation models have a lower average BLEU score when translating between languages that are very different from one other. Thus, in existing unsupervised neural machine translation models, learning of shared latent representations influences translation performance in two opposite directions, as suggested by this finding. Unsupervised neural machine translation based on pseudo-parallel data without taking into account shared latent representation learning can increase translation efficiency, particularly for language pairs that are different.


### 4.2. Efficiency Analysis

To build pseudo-parallel corpora, this step selects three early training nodes of an unsupervised neural machine translation model. 400,000 batches were iterated during the training procedure for the entire unsupervised neural machine translation baseline system and the pseudo-parallel data-based unsupervised neural machine translation model. Compared with the unsupervised neural machine translation model that has been iteratively trained with the same 400,000 batches, the unsupervised neural machine translation model based on pseudo-parallel data uses the pseudo-parallel corpus generated by the unsupervised neural machine translation model of different training nodes for iterative training of the remaining batches; the experimental results are shown in Figures3 and 4. UUUU stands for unsupervised neural machine translation model. UUUP refers to training 100,000 batches of pseudo-parallel data-based unsupervised neural machine translation models using pseudo-parallel corpora generated by unsupervised neural machine translation models after 300,000 batches. UUPP refers to training a 200,000 batch unsupervised neural machine translation model based on pseudo-parallel data with pseudo-parallel corpus generated by the unsupervised neural machine translation model after 200,000 batches. UPPP refers to using a pseudo-parallel corpus generated by an unsupervised neural machine translation model through 100,000 batches to train a 300,000-batch unsupervised NMT model based on pseudo-parallel data.Figure 3
Results for En-Frh and Fr-En translation.Figure 4
Results for En-Ge and Ge-En translation.The performance of unsupervised neural machine translation models based on pseudo-parallel data gradually gets better as the number of iterations increases. And the convergence speed of the unsupervised neural machine translation model based on pseudo-parallel data is faster than that of the unsupervised neural machine translation model. In unsupervised neural machine translation models based on pseudo-parallel data, UUUP outperforms UUPP and UPPP. This implies that the quality of the pseudo-parallel corpora utilized for training unsupervised neural machine translation models based on pseudo-parallel data is important. The performance of the unsupervised neural machine translation model with 100000 batches iteration and the unsupervised neural machine translation model based on pseudo-parallel data with a batch of 100000 batches was significantly better than the performance of the original unsupervised neural machine translation model with an iteration of 400000 batches.


### 4.3. Back Translation Analysis

Table4 shows the performance of the unsupervised NMT model based on pseudo-parallel data at different epochs. This section selects the training nodes of 500,000 batches of unsupervised neural machine translation model to generate a pseudo-parallel corpus. In this section, one iteration is set to 100000 batches, and then, the pseudo-parallel corpus is regenerated.Table 4
Back translation analysis.
RoundDe-EnEn-DeDr-EnEn-FrJa-EnEn-JaCh-EnEn-Ch021.7017.6025.0025.9014.6022.1008.5007.30123.6018.6026.8028.7018.4024.4010.7009.30224.2018.9027.5028.9019.1027.0011.3010.10324.3019.0027.5029.1019.1027.0011.3010.50The unsupervised NMT model’s BLEU score is used in the first round. As the number of rounds rises, unsupervised NMT models based on pseudo-parallel data perform better. Unsupervised NMT model training on pseudo-parallel data gets better and better as the quality of the pseudo-parallel corpus increases. The unsupervised NMT model based on pseudo-parallel data, on the other hand, improves less and less with time. In the third round, unsupervised NMT models based on pseudo-parallel data performed similarly to the second round of models. Because of the substantial computational cost of building a pseudo-parallel corpus, this component only conducts two rounds of training for other unsupervised NMT models that employ parallel data.


### 4.4. Combined Experimental Analysis

This subsection combines the dissimilar language pair-specific method introduced in the previous subsection with an unsupervised NMT method based on pseudo-parallel data to further study the experimental results on dissimilar language pairs. Specifically, ASW_PO is used to produce pseudo-parallel corpora, which will be utilized to train new unsupervised NMT models based on pseudo-parallel data. The experimental results are shown in Figure5. It is obvious that the joint method proposed in this work can achieve the highest BLEU score.Figure 5
BLEU scores of different methods.


## 4.1. Comparison with Other Methods

To verify the validity and correctness of UETPPD, it is compared with other methods. The experimental results are illustrated in Table3.Table 3
Comparison with other methods.
MethodDe-EnEn-DeDr-EnEn-FrJa-EnEn-JaCh-EnEn-ChRef [27]——15.6015.10————Ref [28]13.309.6014.3015.00————Ref [29]14.6010.9015.6017.00————Ref [30]21.0017.2024.2025.10————UETPPD24.2018.9027.5028.9019.1027.0011.3010.10The classic unsupervised neural machine translation models are outperformed by a pseudo-parallel data-based model. Traditional unsupervised neural machine translation models have a lower average BLEU score in similar language pairs. Traditional unsupervised neural machine translation models have a lower average BLEU score when translating between languages that are very different from one other. Thus, in existing unsupervised neural machine translation models, learning of shared latent representations influences translation performance in two opposite directions, as suggested by this finding. Unsupervised neural machine translation based on pseudo-parallel data without taking into account shared latent representation learning can increase translation efficiency, particularly for language pairs that are different.


## 4.2. Efficiency Analysis

To build pseudo-parallel corpora, this step selects three early training nodes of an unsupervised neural machine translation model. 400,000 batches were iterated during the training procedure for the entire unsupervised neural machine translation baseline system and the pseudo-parallel data-based unsupervised neural machine translation model. Compared with the unsupervised neural machine translation model that has been iteratively trained with the same 400,000 batches, the unsupervised neural machine translation model based on pseudo-parallel data uses the pseudo-parallel corpus generated by the unsupervised neural machine translation model of different training nodes for iterative training of the remaining batches; the experimental results are shown in Figures3 and 4. UUUU stands for unsupervised neural machine translation model. UUUP refers to training 100,000 batches of pseudo-parallel data-based unsupervised neural machine translation models using pseudo-parallel corpora generated by unsupervised neural machine translation models after 300,000 batches. UUPP refers to training a 200,000 batch unsupervised neural machine translation model based on pseudo-parallel data with pseudo-parallel corpus generated by the unsupervised neural machine translation model after 200,000 batches. UPPP refers to using a pseudo-parallel corpus generated by an unsupervised neural machine translation model through 100,000 batches to train a 300,000-batch unsupervised NMT model based on pseudo-parallel data.Figure 3
Results for En-Frh and Fr-En translation.Figure 4
Results for En-Ge and Ge-En translation.The performance of unsupervised neural machine translation models based on pseudo-parallel data gradually gets better as the number of iterations increases. And the convergence speed of the unsupervised neural machine translation model based on pseudo-parallel data is faster than that of the unsupervised neural machine translation model. In unsupervised neural machine translation models based on pseudo-parallel data, UUUP outperforms UUPP and UPPP. This implies that the quality of the pseudo-parallel corpora utilized for training unsupervised neural machine translation models based on pseudo-parallel data is important. The performance of the unsupervised neural machine translation model with 100000 batches iteration and the unsupervised neural machine translation model based on pseudo-parallel data with a batch of 100000 batches was significantly better than the performance of the original unsupervised neural machine translation model with an iteration of 400000 batches.


## 4.3. Back Translation Analysis

Table4 shows the performance of the unsupervised NMT model based on pseudo-parallel data at different epochs. This section selects the training nodes of 500,000 batches of unsupervised neural machine translation model to generate a pseudo-parallel corpus. In this section, one iteration is set to 100000 batches, and then, the pseudo-parallel corpus is regenerated.Table 4
Back translation analysis.
RoundDe-EnEn-DeDr-EnEn-FrJa-EnEn-JaCh-EnEn-Ch021.7017.6025.0025.9014.6022.1008.5007.30123.6018.6026.8028.7018.4024.4010.7009.30224.2018.9027.5028.9019.1027.0011.3010.10324.3019.0027.5029.1019.1027.0011.3010.50The unsupervised NMT model’s BLEU score is used in the first round. As the number of rounds rises, unsupervised NMT models based on pseudo-parallel data perform better. Unsupervised NMT model training on pseudo-parallel data gets better and better as the quality of the pseudo-parallel corpus increases. The unsupervised NMT model based on pseudo-parallel data, on the other hand, improves less and less with time. In the third round, unsupervised NMT models based on pseudo-parallel data performed similarly to the second round of models. Because of the substantial computational cost of building a pseudo-parallel corpus, this component only conducts two rounds of training for other unsupervised NMT models that employ parallel data.


## 4.4. Combined Experimental Analysis

This subsection combines the dissimilar language pair-specific method introduced in the previous subsection with an unsupervised NMT method based on pseudo-parallel data to further study the experimental results on dissimilar language pairs. Specifically, ASW_PO is used to produce pseudo-parallel corpora, which will be utilized to train new unsupervised NMT models based on pseudo-parallel data. The experimental results are shown in Figure5. It is obvious that the joint method proposed in this work can achieve the highest BLEU score.Figure 5
BLEU scores of different methods.


## 5. Conclusion

Language is the most crucial link in the communication, and interlanguage translation has become an essential condition for human-friendly contacts all over the world. English translation, as the oldest form of communication in human history, now plays a critical role in the global integration process. Machine translation technology has gradually reached the stage of history, thanks to the maturation of computer technology and the rapid development of the Internet. Machine translation has moved from the academic discussion stage to the industrial application stage, which is of great significance to the study of English machine translation. Traditional unsupervised neural machine translation models train two different translation directions for a language pair simultaneously during the training process. Interlanguage translation has become an essential condition for human-friendly communications all over the world since language is the most important link in the communication. As the oldest method of communication in human history, English translation presently plays a key part in the global integration process. Thanks to the advancement of computer technology and the rapid growth of the Internet, machine translation technology has progressively progressed to the stage of history. The training data of this model is a pseudo-parallel corpus generated by an unsupervised neural machine translation system, rather than a real parallel corpus, and it only relies on monolingual data throughout the training process. Experiments on a variety of translation problems show that the strategy is effective and that it not only improves translation quality but also beats typical unsupervised neural machine translation models in terms of training speed. Finally, the English intelligent translation model is deployed in the wireless network environment and provides functional access to the outside world.



---

*Source: 8208242-2022-05-21.xml*