SlowGuess/NAACL_2022_OCR huggingface dataset:
Pengzhi Gao, Zhongjun He, Hua Wu, and Haifeng Wang
Baidu Inc. No. 10, Shanghai 10th Street, Beijing, 100085, China
{gaopengzhi,hezhongjun,wu_hua,wanghaifeng}@baidu.com
We introduce Bi-SimCut: a simple but effective training strategy to boost neural machine translation (NMT) performance. It consists of two procedures: bidirectional pretraining and unidirectional finetuning. Both procedures utilize SimCut, a simple regularization method that forces the consistency between the output distributions of the original and the cutoff sentence pairs. Without leveraging extra dataset via back-translation or integrating large-scale pretrained model, Bi-SimCut achieves strong translation performance across five translation benchmarks (data sizes range from 160K to 20.2M): BLEU scores of 31.16 for en  $\rightarrow$  de and 38.37 for de  $\rightarrow$  en on the IWSLT14 dataset, 30.78 for en  $\rightarrow$  de and 35.15 for de  $\rightarrow$  en on the WMT14 dataset, and 27.17 for zh  $\rightarrow$  en on the WMT17 dataset. SimCut is not a new method, but a version of Cutoff (Shen et al., 2020) simplified and adapted for NMT, and it could be considered as a perturbation-based method. Given the universality and simplicity of SimCut and Bi-SimCut, we believe they can serve as strong baselines for future NMT research.
The state of the art in machine translation has been dramatically improved over the past decade thanks to the neural machine translation (NMT) (Wu et al., 2016), and Transformer-based models (Vaswani et al., 2017) often deliver state-of-the-art (SOTA) translation performance with large-scale corpora (Ott et al., 2018). Along with the development in the NMT field, consistency training (Bachman et al., 2014) has been widely adopted and shown great promise to improve NMT performance. It simply regularizes the NMT model predictions to be invariant to either small perturbations applied to the inputs (Sato et al., 2019; Shen et al., 2020) and hidden states (Chen et al., 2021) or the model randomness and variance existed in the training procedure (Liang et al., 2021).
Specifically, Shen et al. (2020) introduce a set of cutoff data augmentation methods and utilize Jensen-Shannon (JS) divergence loss to force the consistency between the output distributions of the original and the cutoff augmented samples in the training procedure. Despite its impressive performance, finding the proper values for the four additional hyper-parameters introduced in cutoff augmentation seems to be tedious and time-consuming if there are limited resources available, which hinders its practical value in the NMT field.
In this paper, our main goal is to provide a simple, easy-to-reproduce, but tough-to-beat strategy for training NMT models. Inspired by cutoff augmentation (Shen et al., 2020) and virtual adversarial regularization (Sato et al., 2019) for NMT, we firstly introduce a simple yet effective regularization method named SimCut. Technically, SimCut is not a new method and can be viewed as a simplified version of Token Cutoff proposed in Shen et al. (2020). We show that bidirectional backpropagation in Kullback-Leibler (KL) regularization plays a key role in improving NMT performance. We also regard SimCut as a perturbation-based method and discuss its robustness to the noisy inputs. At last, motivated by bidirectional training (Ding et al., 2021) in NMT, we present Bi-SimCut, a two-stage training strategy consisting of bidirectional pretraining and unidirectional finetuning equipped with SimCut regularization.
- We propose a simple but effective regularization method, SimCut, for improving the generalization of NMT models. SimCut could be regarded as a perturbation-based method and serves as a strong baseline for the approaches of robustness. We also show the compatibility of SimCut with the pretrained language models such as mBART (Liu et al., 2020).
Human Language Technologies, pages 3938 - 3948
- We propose Bi-SimCut, a training strategy for NMT that consists of bidirectional pretraining and unidirectional finetuning with SimCut regularization.
- Our experimental results show that NMT training with Bi-SimCut achieves significant improvements over the Transformer model on five translation benchmarks (data sizes range from 160K to 20.2M), and outperforms the current SOTA method BiBERT (Xu et al., 2021) on several benchmarks.
The NMT model refers to a neural network with an encoder-decoder architecture, which receives a sentence as input and returns a corresponding translated sentence as output. Assume  $\mathbf{x} = x_{1},\ldots ,x_{I}$  and  $\mathbf{y} = y_{1},\ldots ,y_{J}$  that correspond to the source and target sentences with lengths  $I$  and  $J$  respectively. Note that  $y_{J}$  denotes the special end-of-sentence symbol  $\langle eos\rangle$ . The encoder first maps a source sentence  $\mathbf{x}$  into a sequence of word embeddings  $e(\mathbf{x}) = e(x_1),\dots,e(x_I)$  where  $e(\mathbf{x})\in \mathbb{R}^{d\times I}$ , and  $d$  is the embedding dimension. The word embeddings are then encoded to the corresponding hidden representations  $\mathbf{h}$ . Similarly, the decoder maps a shifted copy of the target sentence  $\mathbf{y}$ , i.e.,  $\langle bos\rangle,y_1,\dots,y_{J - 1}$  into a sequence of word embeddings  $e(\mathbf{y}) = e(\langle bos\rangle),e(y_1),\dots,e(y_{J - 1})$ , where  $\langle bos\rangle$  denotes a special beginning-of-sentence symbol, and  $e(\mathbf{y})\in \mathbb{R}^{d\times J}$ . The decoder then acts as a conditional language model that operates on the word embeddings  $e(\mathbf{y})$  and the hidden representations  $\mathbf{h}$  learned by the encoder.
Given a parallel corpus  $S = \{\mathbf{x}^i,\mathbf{y}^i\}_{i = 1}^{|S|}$ , the standard training objective is to minimize the empirical risk:
$$
\mathcal {L} _ {c e} (\theta) = \underset {(\mathbf {x}, \mathbf {y}) \in \mathcal {S}} {\mathbb {E}} [ \ell (f (\mathbf {x}, \mathbf {y}; \theta), \ddot {\mathbf {y}}) ], \tag {1}
$$
where  $\ell$  denotes the cross-entropy loss,  $\theta$  is a set of model parameters,  $f(\mathbf{x},\mathbf{y};\theta)$  is a sequence of probability predictions, i.e.,
$$
f _ {j} (\mathbf {x}, \mathbf {y}; \theta) = P (y | \mathbf {x}, \mathbf {y} _ {<   j}; \theta), \tag {2}
$$
and  $\ddot{\mathbf{y}}$  is a sequence of one-hot label vectors for  $\mathbf{y}$ .
2.2 Cutoff Augmentation
Shen et al. (2020) introduce a set of cutoff methods which augments the training by creating the partial views of the original sentence pairs and propose Token Cutoff for the machine translation task. Given a sentence pair  $(\mathbf{x},\mathbf{y})$ ,  $N$  cutoff samples  $\{\mathbf{x}_{\mathrm{cut}}^i,\mathbf{y}_{\mathrm{cut}}^i\}_{i = 1}^N$  are constructed by randomly setting the word embeddings of  $x_{1},\ldots ,x_{I}$  and  $y_{1},\ldots ,y_{J}$  to be zero with a cutoff probability  $p_{\mathrm{cut}}$ . For each sentence pair, the training objective of Token Cutoff is then defined as:
$$
\mathcal {L} _ {\text {t o k c u t}} (\theta) = \mathcal {L} _ {c e} (\theta) + \alpha \mathcal {L} _ {\text {c u t}} (\theta) + \beta \mathcal {L} _ {k l} (\theta), \tag {3}
$$
$$
\mathcal {L} _ {c e} (\theta) = \ell (f (\mathbf {x}, \mathbf {y}; \theta), \ddot {\mathbf {y}}), \tag {4}
$$
$$
\mathcal {L} _ {c u t} (\theta) = \frac {1}{N} \sum_ {i = 1} ^ {N} \ell \left(f \left(\mathbf {x} _ {\mathrm {c u t}} ^ {i}, \mathbf {y} _ {\mathrm {c u t}} ^ {i}; \theta\right), \ddot {\mathbf {y}}\right), \tag {5}
$$
$$
\begin{array}{l} \mathcal {L} _ {k l} (\theta) = \frac {1}{N + 1} \{\sum_ {i = 1} ^ {N} \mathrm {K L} (f (\mathbf {x} _ {\mathrm {c u t}} ^ {i}, \mathbf {y} _ {\mathrm {c u t}} ^ {i}; \theta) \| p _ {\mathrm {a v g}}) \\ + \operatorname {K L} \left(f (\mathbf {x}, \mathbf {y}; \theta) \| p _ {\text {a v g}}\right) \}, \tag {6} \\ \end{array}
$$
$$
\begin{array}{l} p _ {\mathrm {a v g}} = \frac {1}{N + 1} \{\sum_ {i = 1} ^ {N} f (\mathbf {x} _ {\mathrm {c u t}} ^ {i}, \mathbf {y} _ {\mathrm {c u t}} ^ {i}; \theta) \\ + f (\mathbf {x}, \mathbf {y}; \theta) \}, \tag {7} \\ \end{array}
$$
in which  $\mathrm{KL}(\cdot \| \cdot)$  denotes the Kullback-Leibler (KL) divergence of two distributions, and  $\alpha$  and  $\beta$  are the scalar hyper-parameters that balance  $\mathcal{L}_{ce}(\theta)$ ,  $\mathcal{L}_{cut}(\theta)$  and  $\mathcal{L}_{kl}(\theta)$ .
3 Datasets and Baseline Settings
In this section, we describe the datasets used in experiments as well as the model configurations. For fair comparisons, we keep our experimental settings consistent with previous works.
Datasets We initially consider a low-resource (IWSLT14 en  $\leftrightarrow$  de) scenario and then show further experiments in standard (WMT14 en  $\leftrightarrow$  de) and high (WMT17 zh  $\rightarrow$  en) resource scenarios in Sections 5 and 6. The detailed information of the datasets are summarized in Table 1. We here conduct experiments on the IWSLT14 English-German dataset<sup>1</sup>, which has 160K parallel bilingual
$^{1}$ https://github.com/pytorch/fairseq/blob/main/examples/translation/prepare-iwslt14.sh
IWSLT	WMT
en↔de	en↔de	zh→en
train	160239	4468840	20184941
valid	7283	6003	2002
test	6750	3003	2001
Table 1: Number of sentence pairs used in our machine translation experiments.
sentence pairs. Following the common practice, we lowercase all words in the dataset. We build a shared dictionary with 10K byte-pair-encoding (BPE) (Sennrich et al., 2016) types.
Settings We implement our approach on top of the Transformer (Vaswani et al., 2017). We apply a Transformer with 6 encoder and decoder layers, 4 attention heads, embedding size 512, and FFN layer dimension 1024. We apply cross-entropy loss with label smoothing rate 0.1 and set max tokens per batch to be 4096. We use Adam optimizer with Beta (0.9, 0.98), 4000 warmup updates, and inverse square root learning rate scheduler with initial learning rates  $5e^{-4}$ . We use dropout rate 0.3 and beam search decoding with beam size 5 and length penalty 1.0. We apply the same training configurations in both pretraining and finetuning stages which will be discussed in the following sections. We use multi-bleu.pl² for BLEU (Papineni et al., 2002) evaluation. We train all models until convergence on a single NVIDIA Tesla V100 GPU. All reported BLEU scores are from a single model. For all the experiments below, we select the saved model state with the best validation performance.
4 Bi-SimCut
In this section, we formally propose Bidirectional Pretrain and Unidirectional Finetune with Simple Cutoff Regularization (Bi-SimCut), a simple but effective training strategy that can greatly enhance the generalization of the NMT model. Bi-SimCut consists of a simple cutoff regularization and a two-phase pretraining and finetuning strategy. We introduce the details of each part below.
4.1 SimCut: A Simple Cutoff Regularization for NMT
Despite the impressive performance reported in Shen et al. (2020), finding the proper hyperparameters  $(p_{\mathrm{cut}},\alpha ,\beta ,N)$  in Token Cutoff seems
to be tedious and time-consuming if there are limited resources available, which hinders its practical value in the NMT community. To reduce the burden in hyper-parameter searching, we propose SimCut, a simple regularization method that forces the consistency between the output distributions of the original sentence pairs and the cutoff samples.
Our problem formulation is motivated by Virtual Adversarial Training (VAT), where Sato et al. (2019) introduces a KL-based adversarial regularization that forces the output distribution of the samples with adversarial perturbations  $\delta_{\mathbf{x}} \in \mathbb{R}^{d \times I}$  and  $\delta_{\mathbf{y}} \in \mathbb{R}^{d \times J}$  to be consistent with that of the original samples:
$$
\operatorname {K L} (f (e (\mathbf {x}), e (\mathbf {y}); \theta) \| f (e (\mathbf {x}) + \delta_ {\mathbf {x}}, e (\mathbf {y}) + \delta_ {\mathbf {y}}; \theta)).
$$
Instead of generating perturbed samples by gradient-based adversarial methods, for each sentence pair  $(\mathbf{x},\mathbf{y})$ , we only generate one cutoff sample  $(\mathbf{x}_{\mathrm{cut}},\mathbf{y}_{\mathrm{cut}})$  by following the same cutoff strategy used in Token Cutoff. For each sentence pair, the training objective of SimCut is defined as:
$$
\mathcal {L} _ {\text {s i m c u t}} (\theta) = \mathcal {L} _ {c e} (\theta) + \alpha \mathcal {L} _ {\text {s i m k l}} (\theta), \tag {8}
$$
$$
\mathcal {L} _ {s i m k l} (\theta) = \mathrm {K L} (f (\mathbf {x}, \mathbf {y}; \theta) \| f (\mathbf {x} _ {\mathrm {c u t}}, \mathbf {y} _ {\mathrm {c u t}}; \theta)).
$$
There are only two hyper-parameters  $\alpha$  and  $p_{\mathrm{cut}}$  in SimCut, which greatly simplifies the hyperparameter searching step in Token Cutoff. Note that VAT only allows the gradient to be backpropagated through the right-hand side of the KL divergence term, while the gradient is designed to be backpropagated through both sides of the KL regularization in SimCut. We can see that the constraints introduced by  $\mathcal{L}_{\mathrm{cut}}(\theta)$  and  $\mathcal{L}_{kl}(\theta)$  in (3) still implicitly hold in (8):
-  $\mathcal{L}_{cut}(\theta)$  in Token Cutoff is designed to guarantee that the output of the cutoff sample should close to the ground-truth to some extent. In SimCut,  $\mathcal{L}_{ce}(\theta)$  requires the outputs of the original sample close to the ground-truth, and  $\mathcal{L}_{simkl}(\theta)$  requires the output distributions of the cutoff sample close to that of the original sample. The constraint introduced by  $\mathcal{L}_{cut}(\theta)$  then implicitly holds.
-  $\mathcal{L}_{kl}(\theta)$  in Token Cutoff is designed to guarantee that the output distributions of the original sample and  $N$  different cutoff samples
$^{2}$ https://github.com/moses-smt/mosesdecoder/master/scripts/generic/multi-bleu.perl
Method	en→de	de→en
Transformer	28.70	34.99
VAT	29.45	35.52
R-Drop	30.73	37.30
Token Cutoff	30.89	37.61
SimCut	30.98	37.81
Table 2: SimCut achieves the superior or comparable performance on IWSLT14 en  $\leftrightarrow$  de translation tasks over the strong baselines such as VAT, R-Drop, and Token Cutoff.
should be consistent with each other. In SimCut,  $\mathcal{L}_{simkl}(\theta)$  guarantees the consistency between the output distributions of the original and cutoff samples. Even though SimCut only generates one cutoff sample at each time, different cutoff samples of the same sentence pair will be considered in different training epochs. Such constraint raised by  $\mathcal{L}_{kl}(\theta)$  still implicitly holds.
4.2 Analysis on SimCut
4.2.1 How Does the Simplification Affect Performance?
We here investigate whether our simplification on Token Cutoff hurts its performance on machine translation tasks. We compare SimCut with Token Cutoff, VAT, and R-Drop (Liang et al., 2021), a strong regularization baseline that forces the output distributions of different sub-models generated by dropout to be consistent with each other. Table 2 shows that SimCut achieves superior or comparable performance over VAT, R-Drop, and Token Cutoff, which clearly shows the effectiveness of our method. To further compare SimCut with other strong baselines in terms of training cost, we summarize the validation BLEU score along the training time on IWSLT14 de→en translation task in Table 3. From the table, we can see that the BLEU score of SimCut continuously increases in the first 1500 minutes. The results on VAT are consistent with the previous studies on adversarial overfitting, i.e., virtual adversarial training easily suffering from overfitting (Rice et al., 2020). Though SimCut needs more training time to converge, the final NMT model is much better than the baseline. For the detailed training cost for each epoch, Token Cutoff costs about 148 seconds per epoch, while SimCut costs about 128 seconds per epoch. Note that the training cost of Token Cutoff is greatly influenced by the hyper-parameter  $N$ . We set  $N$  to
be 1 in our experiments. With the increasing of  $N$ , the training time of Token Cutoff will be much longer. Due to the tedious and time-consuming hyper-parameter searching in Token Cutoff, we will not include its results in the following sections and show the results of SimCut directly.
4.2.2 How Does the Bidirectional Backpropagation Affect Performance?
Even though the problem formulation of SimCut is similar to that of VAT, one key difference is that the gradients are allowed to be backpropagated bidirectionally in the KL regularization in SimCut. We here investigate the impact of the bidirectional backpropagation in the regularization term on the NMT performance. Table 4 shows the translation results of VAT and SimCut with or without bidirectional backpropagation. We can see that both VAT and SimCut benefit from the bidirectional gradient backpropagation in the KL regularization.
4.2.3 Performance on Perturbed Inputs
Given the similar problem formulations of VAT and SimCut, it is natural to regard cutoff operation as a special perturbation and consider SimCut as a perturbation-based method. We here investigate the robustness of NMT models on the perturbed inputs. As discussed in Takase and Kiyono (2021), simple techniques such as word replacement and word drop can achieve comparable performance to sophisticated perturbations. We hence include them as baselines to show the effectiveness of our method as follows:
- UniRep: Word replacement approach constructs a new sequence whose tokens are randomly replaced with sampled tokens. For each token in the source sentence  $\mathbf{x}$ , we sample  $\hat{x}_i$  uniformly from the source vocabulary, and use it for the new sequence  $\mathbf{x}'$  with probability  $1 - p'$ :
$$
x _ {i} ^ {\prime} = \left\{ \begin{array}{l l} x _ {i}, & \text {w i t h p r o b a b i l i t y} p ^ {\prime}, \\ \hat {x} _ {i}, & \text {w i t h p r o b a b i l i t y} 1 - p ^ {\prime}. \end{array} \right. \tag {9}
$$
We construct  $\mathbf{y}'$  from the target sentence  $\mathbf{y}$  in the same manner. Following the curriculum learning strategy used in Bengio et al. (2015), we adjust  $p'$  with the inverse sigmoid decay:
$$
p _ {t} ^ {\prime} = \max  \left(q, \frac {k}{k + \exp \left(\frac {t}{k}\right)}\right), \tag {10}
$$
where  $q$  and  $k$  are hyper-parameters.  $p_t^\prime$  decreases to  $q$  from 1, depending on the training epoch num
Minutes	10	30	60	90	150	300	600	900	1200	1500
Transformer	11.51	31.20	34.19	34.88	35.17	34.86	34.43	34.28	34.23	33.95
VAT	1.87	20.08	31.69	33.95	35.41	35.78	35.81	35.63	35.17	34.99
R-Drop	2.11	26.32	32.81	34.25	35.88	36.91	37.18	37.43	37.52	37.43
Token Cutoff	2.16	28.88	32.82	34.61	35.90	36.84	37.70	37.81	37.93	37.83
SimCut	1.99	25.12	32.21	33.66	34.93	36.37	37.31	37.62	37.89	38.10
Table 3: On the IWSLT14 de→en validation set, the BLEU score increases over time in model training using SimCut. In contrast, the BLEU scores of the other strong baselines all stop increasing before 1500 minutes. The results suggest that the use of SimCut can effectively alleviate the model training from overfitting.
Method	en→de	de→en
VAT	29.45	35.52
+ Bi-backpropagation	29.69	36.26
SimCut	30.98	37.81
- Bi-backpropagation	30.29	36.91
Table 4: Bidirectional backpropagation achieves better performance on IWSLT14 en  $\leftrightarrow$  de translation tasks compared with unidirectional backpropagation in the KL regularization.
Method	probability
0.00	0.01	0.05	0.10
Transformer	34.99	34.01	30.38	25.70
UniRep	35.67	34.91	31.54	27.24
WordDrop	35.65	34.73	31.22	26.46
VAT	35.52	34.65	30.48	25.44
R-Drop	37.30	36.24	32.27	27.19
SimCut	37.81	36.94	33.16	27.93
Table 5: The model trained by SimCut achieves high robustness on the perturbed test set and high performance on the clean test set. Entries represent BLEU scores on IWSLT14 de→en test set when we inject perturbations to source sentences with different probability.
ber  $t$ . We use  $p_t'$  as  $p'$  in epoch  $t$ . We set  $q$  and  $k$  to be 0.9 and 25 respectively in the experiments.
- **WordDrop:** Word drop randomly applies the zero vector instead of the word embedding  $e(x_{i})$  or  $e(y_{i})$  for the input token  $x_{i}$  or  $y_{i}$  (Gal and Ghahramani, 2016). For each token in both source and target sentences, we keep the original embedding with the probability  $\beta$  and set it to be the zero vector otherwise. We set  $\beta$  to be 0.9 in the experiments.
We construct noisy inputs by randomly replacing words in the source sentences based on a predefined probability. If the probability is 0.0, we use the original source sentence. If the probabil
ity is 1.0, we use completely different sentences as source sentences. We set the probability to be 0.00, 0.01, 0.05, and 0.10 in our experiments. We randomly replace each word in the source sentence with a word uniformly sampled from the vocabulary. We apply this procedure to IWSLT14 de→en test set. Table 5 shows the BLEU scores of each method on the perturbed test set. Note that the BLEU scores are calculated against the original reference sentences. We can see that all methods improve the robustness of the NMT model, and SimCut achieves the best performance among all the methods on both the clean and perturbed test sets. The performance results indicate that SimCut could be considered as a strong baseline for the perturbation-based method for the NMT model.
As shown in Table 6, the baseline model completely ignores the translation of "in spielien (in games)" due to the replacement of "denken (think)" with "festgelegten (determined)" in the source sentence. In contrast, our model successfully captures the translation of "inspielien" under the noisy input. This result shows that our model is more robust to small perturbations in an authentic context.
4.2.4 Effects of  $\alpha$  and  $p_{\mathrm{cut}}$
We here investigate the impact of the scalar hyperparameters  $\alpha$  and  $p_{\mathrm{cut}}$  in SimCut.  $\alpha$  is a penalty parameter that controls the regularization strength in our optimization problem.  $p_{\mathrm{cut}}$  controls the percentage of the cutoff perturbations in SimCut. We here vary  $\alpha$  and  $p_{\mathrm{cut}}$  in  $\{1,2,3,4,5\}$  and  $\{0.00,0.05,0.10,0.15,0.20\}$  respectively and conduct the experiments on the IWSLT14 de  $\rightarrow$  en dataset. Note that SimCut is simplified to R-Drop approximately when  $p_{\mathrm{cut}} = 0.00$ . The test BLEU scores are reported in Figure 1. By checking model performance under different combinations of  $\alpha$  and  $p_{\mathrm{cut}}$ , we have the following observations: 1) A too small  $\alpha$  (e.g., 1) cannot achieve good performance as larger  $\alpha$  (e.g., 3), indicating a certain de
Input	wir denken (festgelegten), dass wir in der realität nicht so gut sind wie in spielen.
Reference	we feel that we are not as good in reality as we are in games.
Vaswani et al. (2017) on Input	we think we’re not as good in reality as we are in games.
on Noisy Input	we realized that we weren’t as good as we were in real life.
SimCut on Input	we think in reality, we’re not as good as we do in games.
on Noisy Input	we realized that we’re not as good in reality as we are in games.
gree of regularization strength during NMT model training is conducive to generalization. Meanwhile, an overwhelming regularization  $(\alpha = 5)$  is not plausible for learning NMT models. 2) When  $\alpha = 3$ , the best performance is achieved when  $p_{\mathrm{cut}} = 0.05$ , and  $p_{\mathrm{cut}} = 0.00$  performs suboptimal among all selected probabilities. Such an observation demonstrates that the cutoff perturbation in SimCut can effectively promote the generalization compared with R-Drop.
Figure 1: BLEU scores with different  $\alpha$  and  $p_{\mathrm{cut}}$  on IWSLT14 de  $\rightarrow$  en dataset.
4.2.5 Is SimCut Compatible with the Pretrained Language Model?
The multilingual sequence-to-sequence pretrained language models (Song et al., 2019; Liu et al., 2020; Xue et al., 2021) have shown impressive performance on machine translation tasks, where the pretrained models generally learn the knowledge from the large-scale monolingual data. It is interesting to investigate whether SimCut can gain performance improvement based on the pretrained language model. We adopt mBART (Liu