Improving Autoregressive Grammatical Error Correction with Non-autoregressive Models
Year: 2023

Abstract:
Grammatical Error Correction (GEC) aims to correct grammatical errors in sentences. We find that autoregressive models tend to assign low probabilities to tokens that need corrections. Here we introduce additional signals to the training of GEC models so that these systems can learn to better predict at ambiguous positions. To do this, we use a non-autoregressive model as an auxiliary model, and develop a new regularization term of training by considering the difference in predictions between the autoregressive and nonautoregressive models. We experiment with this method on both English and Chinese GEC tasks. Experimental results show that our GEC system outperforms the baselines on all the data sets significantly.

heading: 1 Introduction
text: Grammatical Error Correction (GEC) has attracted much attention in recent years, which aims to correct grammatical errors in a given text automatically. It is widely applied to natural language processing scenarios such as Automatic Speech Recognition (ASR) (Kubis et al., 2020; Wang et al., 2020), writing assistant and language learning platforms, etc. The GEC task is characterized by a significant overlap between input and output sentences with only a few errors requiring modification.
Since the transformer-based autoregressive (Vaswani et al., 2017) (AR) model with sequence-to-sequence (seq2seq) architecture has been successful in many generation tasks, a few works (Chollampatt and Ng, 2018) have applied it to the GEC task by taking the incorrect text as the source language and the text without errors as the target language, which has become a mainstream paradigm. However, in the GEC task, the overlap of source and target sentences makes the AR model simply copy most of the tokens
∗Corresponding author.
AR Decoder
<sos> y1 y2 y3
y1’ y2’ y3’ y4’
NAR Decoder
y1 y2 [mask] y4
y1’’ y2’’ y3’’ y4’’
T1: AR( I | X, <sos> ) T2: AR( have | X, <sos> I ) T3: AR( an| X, <sos> I have ) T4: AR( apple| X, <sos> I have an ) T1: NAR ( I have apple | X, _ _ _ _ ) T2: NAR( an | X, I have _ apple )
X: I have a apple Y: I have an apple
y4
<eos>
AR or NAR
over from the input to the output. We further find that the AR has high confidence for the tokens that are unchanged between the source and target sentence, while it usually has low confidence for correcting operations such as insertion, deletion, and substitution. Figure 1 is an example to illustrate this phenomenon. Intuitively, we believe that the reasonable cause of this phenomenon is the class imbalance issue (Li and Shi, 2021). With the influence of this problem, the AR model cannot confidently predict these incorrect tokens according to only the local context. Therefore, a natural idea is to improve the model performance by exploiting the global information, which can be captured by the non-autoregressive (NAR) (Gu et al., 2018; Lee et al., 2018) model. Although prior works have explored combining the two approaches through joint training, a combination for the GEC task is still missing. Besides, due to the inconsistency between AR and NAR output, a simple combination of them will lead to poor performance.
In this paper, we propose a simple yet novel approach to focus on incorrect tokens and integrate global information with the non-autoregressive model. Specifically, by masking the tokens in the golden target sentence corresponding to the low confidence positions of the AR output, we construct the input for NAR method. We combine the AR and NAR generation mechanisms to effectively
12014
utilize global information by constraining the consistency of their output distribution.
We conduct experiments on standard English GEC datasets and evaluate the system against strong baselines. Experimental results show that our approach can consistently achieve better results without relying on any resources other than the training data. Furthermore, we compare with a combination method of AR and NAR to verify whether the proposed model is more favorable for the GEC task. Here we use the Chinese GEC dataset as a benchmark to validate the generalization ability of the model. Meanwhile, we also conduct comparative ablation studies to illustrate the effectiveness of our proposed method.

heading: 2 Related Work
text: Seq2seq for GEC In recent years, a number of Transformer-based AR methods have been developed for GEC tasks. Junczys-Dowmunt et al. (2018) adapt several methods from low-resource machine translation to GEC by regarding GEC as low-resource machine translation. Zhao et al. (2019) aim to copy the words that overlap between the source and target sentence. They propose a copy-augmented architecture for GEC task which is pre-trained with unlabeled data. A series of work focus on data augmentation (Grundkiewicz et al., 2019; Ge et al., 2018; Lichtarge et al., 2019), Xie et al. (2018) propose to synthesize “realistic” parallel corpus with grammatical errors by back-translation. Zhao and Wang (2020) add a dynamic masking method to the original source sentence during training, which enhances the model performance without requiring additional data. With the help of large pre-trained language models (Kaneko et al., 2020), the performance of Transformer based AR models can be improved effectively. Meanwhile, the NAR approach emerges as a competitive alternative, which can correct the errors by modeling the whole sentence information. Li and Shi (2021) apply a Conditional Random Fields (CRF) layer to conduct non-autoregressive sequence prediction by modeling the dependencies among neighbor tokens.
Combination of AR and NAR The combination of AR and NAR modeling mechanisms has been discussed in other tasks. Wei et al. (2019) use a pre-trained AR model to supervise the decoding state of NAR, which can alleviate the problem of
large search space. Li et al. (2019) propose that learning the hidden representation and attention distribution of AR by hints from the hidden representation can effectively improve the performance of NAR. Several approaches (Guo et al., 2020; Liu et al., 2020) are proposed to gradually guide the model transition from AR to NAR by designing the decoder input and semi-autoregressive tasks as courses. Some other works (Sun and Yang, 2020; Hao et al., 2021; Wang et al., 2022) attempt to utilize a unified framework to train AR and NAR jointly so that the NAR can be enhanced. Besides, Zhou et al. (2020) have also explored using the output of NAR to improve the AR performance. Unlike them, we focus on the GEC task and introduce the NAR model to utilize the global information to help the model understand the context around incorrect tokens.

heading: 3 Methodology
text: In this section, we elaborate on our proposed framework for GEC. As shown in figure 2, we introduce the CMLM-based NAR model to integrate more context information into our single model.

heading: 3.1 Overview
text: Given the training dataset (X,Y ), the definition of the GEC task is to correct the original erroneous source X and generate a sentence Y without grammatical error, where X = (x1, x2, ..., xK) and Y = (y1, y2, ..., yN ). Specifically, the transformer encoder takes the source sentence X as input. Different from previous seq2seq works, our decoder consists of two components: AR decoder and NAR decoder. We keep the AR decoder as a traditional seq2seq decoder without any change. For the NAR decoder, we mask the tokens in the input that corresponds to the positions of the low confidence tokens from the output distribution of the AR decoder. Then, we regenerate tokens in masked positions more accurately by bidirectional semantic modeling of the NAR decoder. Finally, we try to decrease the output distribution distance which is in masked positions between two manners to further improve the model performance during the training stage.

heading: 3.2 Mask low Confidence
text: Here, it is important that the output probability represents whether the model is confident in the prediction. As for the GEC task, there are only a
Algorithm 1 Mask strategy Input: The AR decoder output yAR, the golden
target ytgt, the mask ratio δ Output: The NAR input xNAR
1: while not converged do 2: Select the max probability for each token; 3: max_logit← get_max_logit(yAR); 4: Lt ← get_target_length(ytgt); 5: Lm = Lt × δ; 6: Get the Lm index of the low confidence positions in maxim logit; 7: Index← select_index(max_logit, Lm); 8: Replace with <mask> in the ytgt corresponding position; 9: xNAR ← mask_target(Index, ytgt);
10: end while
few tokens that need to be modified (About 10%). Therefore, the model tends to focus on high confidence correct tokens that need to be kept, but not so much on low confidence tokens that need to be modified. As mentioned above, we choose the low confidence positions in the AR output distribution and substitute them with special symbols <mask> as the input of the NAR decoder. In this way, the NAR decoder is forced to learn the knowledge of low confidence tokens from the bidirectional context in hidden layers, which helps to boost the performance.
To construct the input effectively, we design a special mask strategy. Details are described in Al-
gorithm 1. Specifically, we select the maximum probability of each token from the AR decoder output distribution. Then, we reorder each token in the output sentence from low confidence to high confidence to get a specific number of positions for the low confidence tokens. We introduce a special token <mask> to mask the token at the corresponding position in the golden target, which serves as a placeholder to represent the position where the target token needs to be regenerated. The golden target after the masking operation is used as input to the NAR decoder to introduce bidirectional contextual information.

heading: 3.3 Restrict Output Consistency
text: The objective of our model is to overcome the limitations of AR models by introducing the NAR generation mechanism, and then correct sentences with grammatical errors. A common way is to implicitly pass the information learned by the NAR branch to the AR branch using the parameter-sharing method. Specifically, we share the parameters of the Transformer layer in both manners. However, there is a huge difference between the AR manner and the NAR manner in the training process, as shown in Equation 1 and Equation 2, where the AR generation process is more concerned with local dependencies, while the NAR generation process is more concerned with global dependencies.
P (Y |X) = N∏
i=1
PAR(yi|X,Y<i), (1)
P (Y |X) = N∏
i=1
PNAR(yi|X). (2)
The inconsistency between the two generation methods can lead to direct parameter sharing between the two branches without enabling the AR manner to obtain the exact information provided by the NAR manner. This sharing method only implicitly considers the correlation of model parameters and ignores the inconsistency between the two generation methods, which seriously hinders performance.
In contrast, in our work, to make the AR manner learn the information from NAR in a way that is more adapted to AR generation, we take an explicit approach to constrain the two manners. This approach can avoid the inconsistency caused by the different ways of AR and NAR generation and break the performance bottleneck. In practice, we accomplish explicit information modeling by using bidirectional Kullback-Leibler (KL) divergence to force the AR and NAR output distributions at the mask positions to be consistent with each other. Fortunately, Liang et al. (2022) also use KL divergence to combine the advantages of AR and NAR, which gives us much inspiration.

heading: 3.4 Training and Inference
text: Multi-Task Framework We learn the GEC model under the multi-task learning framework, including an AR primary task and a NAR auxiliary task. It should be noted that AR and NAR manners are regarded as two different tasks. For the AR task, we employ the negative log-likelihood (NLL) as the loss function which is akin to the traditional seq2seq. Therefore, the optimization objective is:
LAR = − N∑
i=1
logPAR(yi|X,Y<i), (3)
where N is the target length, and Y<i represents the tokens before the i-th time step. PAR(yi|X,Y<i) represents the output probability of the AR decoder, which will be used in the later process.
For the NAR task, we obtain the positions of the specified number of low confidence tokens based on the mask ratio δ, and replace the tokens with the special symbols <mask> at the corresponding positions of the golden target. The loss function LNAR for NAR task is to minimize the sum of
negative log-likelihood in masked positions:
LNAR = − M∑
i=1
logPNAR(yi|X,Ymask), (4)
where M is the number of the masked tokens, and Ymask is the set of the tokens in masked. In this way, the NAR decoder regenerates the masked tokens with more context information to help the AR task. Then the loss function of the multi-task framework is:
Lm = λtLNAR + (1− λt)LAR, (5)
where λt is the important factor to balance the weight of AR and NAR tasks during training. We will present the design details in the following paragraphs.
Curriculum Learning Compared with the AR task, the NAR task is more complex, and unreasonable weight setting will make training difficult. For example, the excessive weight of the NAR task will disturb the parameter learning of the AR primary task at the beginning. Inspired by curriculum learning (Bengio et al., 2009), which is to imitate the human learning process, we propose the dynamic weight strategy. More concretely, we start with λt = 0 and gradually increase the NAR task weight λt to introduce learning signals. The dynamic weight scheme is:
λt = t
T , (6)
where t and T are the current and total steps of training. We increase the weight linearly in all the experiments.
It is not enough to use only the hard parameter sharing method mentioned above, we regularize the two output distributions PAR and PNAR for unconfident words with the token-level bidirectional Kullback-Leibler divergence to further transfer the knowledge of NAR:
LKL = ∑
Ymask KL(PAR||PNAR)+ ∑
Ymask
KL(PNAR||PAR). (7)
The final training objective for our GEC model is a combination of the three terms reviewed above as:
L = λtLNAR + (1− λt)LAR + αLKL. (8)
Inference During the inference stage, we use the AR decoder to generate the correct sentences, and the inference efficiency is the same as the traditional seq2seq model since the NAR decoder is only used in training.

heading: 4 Experimental Setup

heading: 4.1 Datasets
text: To validate the effectiveness of our proposed GEC model, we conduct a set of experiments on both the restricted track of the BEA-2019 GEC shared task (Bryant et al., 2019) and NLPCC 2018 Task 2 (Zhao et al., 2018).
BEA-2019 GEC shared task This is a public dataset for the English GEC task, we follow the setting of (Chollampatt and Ng, 2018) and take the FCE training set (Yannakoudakis et al., 2011), Lang-8 Corpus of Learner English (Mizumoto et al., 2011), NUCLE (Dahlmeier et al., 2013) and W&I+LOCNESS (Granger, 2014; Bryant et al., 2019) as the training set. The development set is a subset of NUCLE, and our model is evaluated on the CoNLL-2014 (Ng et al., 2014), which is a wellknown English GEC benchmark test set. Specifically, we use pre-processed script1 in (Chollampatt and Ng, 2018) to obtain the parallel corpus.
NLPCC 2018 Task 2 It is the first and latest benchmark dataset for Chinese GEC. We combine
1https://github.com/nusnlp/mlconvgec2018/tree/ master/data
the incorrect sentence with each corrected sentence to build the parallel sentence pairs as described in (Zhao and Wang, 2020) and get 1.2 million sentence pairs in all. Next, we randomly sample 5,000 training instances as the development set. The official test set extracted from PKU Chinese Learner Corpus contains 2,000 samples. We use the combination of two group annotations that mark the golden edits of grammatical errors in these sentences to evaluate our model. Following the setting of NLPCC 2018 Task (Zhao et al., 2018), the tokenization of training data is implemented with the PKUNLP tool2.

heading: 4.2 Settings
text: While in the training process, we use the base model configuration of the Transformer for the Chinese GEC task, with 6 layers, the number of self-attention heads is set to 8, the embedding dimension is 512 and the size of FFN layer is 2048, the dropout and weight decay is 0.3 and 0.01 respectively. In the English GEC task, we use the big Transformer setting, which contains 6 layers and 16 self-attention heads, the size of word vectors on the source side and the target side are 1024, the FFN layer size is 4096, the dropout is applied with a probability of 0.1 and the weight decay value is set to be 0.0001. We adopt Adam (Kingma and Ba, 2015) optimizer with initial learning rate 0.0005 and 0.0007 for Chinese and English GEC tasks respectively, and a beta value of (0.9, 0.98). We use
2https://github.com/zhaoyyoo/NLPCC2018_GEC
learning rate schedule as in (Vaswani et al., 2017), 10,000 warmup steps for the Chinese GEC task and 4,000 for the English GEC task. Lable smoothing is added with an epsilon value of 0.1. We use 32K Byte Pair Encoding (BPE) (Sennrich et al., 2016) for tokenization on Chinese and English GEC tasks. We save the checkpoint for each epoch and select the best checkpoint based on the loss on the development set. The beam size is 5 during the inference stage. All experiments are based on fairseq (Ott et al., 2019).

heading: 4.3 Baselines
text: We compare the performance of the proposed model with several representative baseline methods on both English and Chinese GEC tasks. Specifically, for the English GEC task, Transformer Big is the typical AR model. LaserTagger proposes to predict tags with a smaller vocabulary (Malmi et al., 2019). Adversarial-GEC presents an adversarial learning approach to generate realistic texts in a generator-discriminator framework (Raheja and Alikaniotis, 2020). ESD+ESC is a pipeline model (Chen et al., 2020). SAD employs a new decoding method with a shallow decoder to conduct the prediction (Sun et al., 2021). S2A proposes to integrate action probabilities into token prediction probabilities to obtain the final results (Li et al., 2022). Levenshtein Transformer (Gu et al., 2019) and CMLM (Ghazvininejad et al., 2019) are NAR models, which achieve excellent performance with an iterative generation paradigm. In addition, we also compare with JANUS (Liang et al., 2022), which joints AR and NAR training for sequence generation.
For the Chinese GEC task, we compare our model to all previous systems in the NLPCC 2018
dataset. YouDao corrects the sentences independently by utilizing five different mixture models (Fu et al., 2018). AliGM combines three approaches, including NMT-based, SMT-based, and rule-based together (Zhou et al., 2018). BLCU is based on a multi-layer convolutional seq2seq model (Ren et al., 2018).

heading: 4.4 Evaluation Metrics
text: Following the typical previous works (Chen et al., 2020; Li et al., 2022), we use the official MaxMatch (M2) (Dahlmeier and Ng, 2012) scorer for evaluation of our grammatical error correction system. M2 scorer computes the sequence of phrase-level edits between a source sentence and a system hypothesis that achieves the maximal overlap with the gold standard annotation. Given the set of system edits and the set of gold edits for all sentences, the value of precision, recall, and F0.5 are computed by m2scorer 3.

heading: 5 Results

heading: 5.1 Main Results
text: The results of our proposed approach and recent models on English GEC task are shown in Table 1. We can see that our approach significantly outperforms the baselines mentioned above. Our model achieves an improvement above Transformer Big by nearly 3.1 in F0.5 score, and performs better than the strong baseline S2A, by a large margin of 1.6 F0.5. Moreover, the proposed model surpasses the recent JANUS model by F0.5 score of 2.3, which shows excellent performance on multiple tasks by combining AR and NAR. This result implies that our designed joint training method is more suit-
3https://github.com/nusnlp/m2scorer
able for the GEC task. It is noteworthy that our model with Transformer base settings still consistently exceeds the baselines with Transformer big settings. These results all support that our proposed approach can effectively improve the AR GEC by using a NAR model.
To validate the effectiveness of our approach, we conduct experiments on the Chinese GEC task and present the results in Table 2. These results demonstrate that the Chinese GEC task is more challenging than the English GEC. Despite this, the proposed model yields a higher F0.5 than the listed methods. Moreover, we can observe that all the top three models are ensemble models, including YouDao, AliGM, and BLCU, but our single model still surpasses them. This result means that our model is generalizable.

heading: 5.2 Fix more Grammar Errors
text: We carefully investigate the number of different types of errors corrected in the two datasets, and find that most of the corrected grammar errors are the same between the proposed method and the AR model. To show the advantages of our model intuitively, we propose a Correction Coincidence Rate, which is the number of overlaps of correction errors to the total number of respective correction errors. The results are summarized in Table 4. For computational convenience, the errors are broadly categorized into insertion, deletion, and substitution. From Table 4, the overlap rate of our proposed
method on all types of error modifications is much lower. For instance, the percentage of deletion decreases by 25%. This indicates that our model is able to correct more grammar errors while maintaining the ability of the AR model.

heading: 5.3 Ablation Analysis
text: Effect of Mask Ratio In this section, we present exhaustive investigations on the impact of mask ratio. Here we vary the mask ratios in {0.1, 0.15, 0.2, 0.3, 0.35} and conduct experiments in BEA-2019 and NLPCC 2018. The corresponding results are provided in Table 3. It can be observed that all mask ratios outperform the Transformer baseline. A reasonable reason is that the masking operation makes the model focus more on incorrect tokens, and the model is forced to capture more context information, which facilitates error correction. On the other hand, a small mask ratio (e.g., 0.1) cannot perform as well as a large one (e.g., 0.15), which means that there is a fraction of incorrect tokens that are not focused on. However, too much masking ratio is also not good. It will result in many correct words being masked, which may prevent the correction of incorrect tokens. Note that the choice of mask ratio is distinct for different datasets, and the best balance choices for BEA-2019 and NLPCC-2018 are 0.2 and 0.15 respectively.
Effect of KL Loss Weight α We explore the effect of KL-divergence loss weight α in Equation 8. The result is illustrated in Table 6. By comparing the performance with KL loss and without KL loss, we can see that the performance of the former is consistently better than the performance of the latter, which suggests that KL loss can be further combined with information from AR and NAR to correct errors. In addition, the performance is lower
wave :::: line are errors, while tokens with underline and in green are the corrections made
by the gold target or our model.
than the baseline when the value of α is 0, i.e., the model is fused using only the simple method of parameter sharing. It indicates that simple fusion will lead to poor performance. We also find that the performance is not optimal when α is set too small or too large. We believe that the model does not learn enough information when α is set too small, while setting it too large leads to the introduction of too much noise.

heading: 5.4 Case Study
text: In order to qualitatively show the effectiveness of global context information, we conduct case studies with Transformer and our proposed model. We pick the cases from the CoNLL-2014 English GEC
test set. The results are listed in Table 5. Generally, it is easy to see that both approaches can copy most of the correct tokens from the source to the target. Nevertheless, when correcting grammatical errors, our approach can predict more accurately by considering more context information. For example, as shown in the third sample in Table 5, the AR model generates the phase "share on" which tends to be consistent with the source language, while our model can delete the token "on" by utilizing more context information. This again confirms that our method can make use of the global information to correct errors.

heading: 6 Conclusion
text: In this work, we propose a joint AR and NAR learning objective for the GEC, using a multi-task learning framework. To better predict tokens at low-confidence positions, we introduce additional signals to the training of GEC models by using the NAR model as an auxiliary model. Meanwhile, we develop a new regularization term of training to constrain the inconsistency between the two manners. Through our experiments in the English and Chinese GEC task, the proposed approach can significantly improve the GEC model performance without additional inference costs.
In the future, we are also interested in introducing syntax and lexical knowledge to focus on incorrect tokens to further improve performance.

heading: 7 Limitations
text: In this work, we achieve a noticeable improvement in the GEC task by introducing additional context information with a NAR model. However, in order to focus on incorrect tokens, the input of the NAR is required to be constructed based on the AR output distribution. In this way, the AR and NAR model perform sequentially, which leads to much time consumption in the training stage. In the future, we will apply a layer dropout strategy to speed up model training. On the other hand, due to the limitation of computation resources, all experiments are conducted on two Nvidia TITAN V GPUs with 12GB VRAM. Therefore, we could not compare with the state-of-the-art models which are pre-trained with 100M synthetic parallel examples (Li et al., 2022). We left it as our future work.

heading: 8 Acknowledgments
text: This work was supported in part by the National Science Foundation of China (No.62276056), the National Key R&D Program of China, the China HTRD Center Project (No.2020AAA0107904), the Natural Science Foundation of Liaoning Province of China (2022-KF-16-01), the Yunnan Provincial Major Science and Technology Special Plan Projects (No.202103AA080015), the Fundamental Research Funds for the Central Universities (Nos.N2216016, N2216001, and N2216002), and the Program of Introducing Talents of Discipline to Universities, Plan 111 (No.B16009).

heading: A For every submission:
text: 3 A1. Did you describe the limitations of your work?
7
3 A2. Did you discuss any potential risks of your work? 7
3 A3. Do the abstract and introduction summarize the paper’s main claims? 1
7 A4. Have you used AI writing assistants when working on this paper? Left blank.
B 7 Did you use or create scientific artifacts? Left blank.
B1. Did you cite the creators of artifacts you used? No response.
B2. Did you discuss the license or terms for use and / or distribution of any artifacts? No response.
B3. Did you discuss if your use of existing artifact(s) was consistent with their intended use, provided that it was specified? For the artifacts you create, do you specify intended use and whether that is compatible with the original access conditions (in particular, derivatives of data accessed for research purposes should not be used outside of research contexts)? No response.
B4. Did you discuss the steps taken to check whether the data that was collected / used contains any information that names or uniquely identifies individual people or offensive content, and the steps taken to protect / anonymize it? No response.
B5. Did you provide documentation of the artifacts, e.g., coverage of domains, languages, and linguistic phenomena, demographic groups represented, etc.? No response.
B6. Did you report relevant statistics like the number of examples, details of train / test / dev splits, etc. for the data that you used / created? Even for commonly-used benchmark datasets, include the number of examples in train / validation / test splits, as these provide necessary context for a reader to understand experimental results. For example, small differences in accuracy on large test sets may be significant, while on small test sets they may not be. No response.
C 3 Did you run computational experiments? 5

heading: 3 C1. Did you report the number of parameters in the models used, the total computational budget
text: (e.g., GPU hours), and computing infrastructure used? 7
The Responsible NLP Checklist used at ACL 2023 is adopted from NAACL 2022, with the addition of a question on AI writing assistance.

heading: 3 C2. Did you discuss the experimental setup, including hyperparameter search and best-found
text: hyperparameter values? 5.3

heading: 3 C3. Did you report descriptive statistics about your results (e.g., error bars around results, summary
text: statistics from sets of experiments), and is it transparent whether you are reporting the max, mean, etc. or just a single run? 4

heading: 3 C4. If you used existing packages (e.g., for preprocessing, for normalization, or for evaluation), did
text: you report the implementation, model, and parameter settings used (e.g., NLTK, Spacy, ROUGE, etc.)? 4.4
D 7 Did you use human annotators (e.g., crowdworkers) or research with human participants? Left blank.
D1. Did you report the full text of instructions given to participants, including e.g., screenshots, disclaimers of any risks to participants or annotators, etc.? No response.
D2. Did you report information about how you recruited (e.g., crowdsourcing platform, students) and paid participants, and discuss if such payment is adequate given the participants’ demographic (e.g., country of residence)? No response.
D3. Did you discuss whether and how consent was obtained from people whose data you’re using/curating? For example, if you collected data via crowdsourcing, did your instructions to crowdworkers explain how the data would be used? No response.
D4. Was the data collection protocol approved (or determined exempt) by an ethics review board? No response.
D5. Did you report the basic demographic and geographic characteristics of the annotator population that is the source of the data? No response.

heading: Future Work
text: In the future, we will apply a layer dropout strategy to speed up model training. On the other hand, due to the limitation of computation resources, all experiments are conducted on two Nvidia TITAN V GPUs with 12GB VRAM. Therefore, we could not compare with the state-of-the-art models which are pre-trained with 100M synthetic parallel examples (Li et al. , 2022). We left it as our future work