SlowGuess/NAACL_2021_OCR huggingface dataset:
sho takase@nlp.c.titech.ac.jp
Shun Kiyono
RIKEN / Tohoku University
shun.kiyono@riken.jp
We often use perturbations to regularize neural models. For neural encoder-decoders, previous studies applied the scheduled sampling (Bengio et al., 2015) and adversarial perturbations (Sato et al., 2019) as perturbations but these methods require considerable computational time. Thus, this study addresses the question of whether these approaches are efficient enough for training time. We compare several perturbations in sequence-to-sequence problems with respect to computational time. Experimental results show that the simple techniques such as word dropout (Gal and Ghahramani, 2016) and random replacement of input tokens achieve comparable (or better) scores to the recently proposed perturbations, even though these simple methods are faster. Our code is publicly available at https://github.com/takase/rethink_perturbations.
Recent advances in neural encoder-decoders have driven tremendous success for sequence-to-sequence problems including machine translation (Sutskever et al., 2014), summarization (Rush et al., 2015), and grammatical error correction (GEC) (Ji et al., 2017). Since neural models can be too powerful, previous studies have proposed various regularization methods to avoid over-fitting.
To regularize neural models, we often apply a perturbation (Goodfellow et al., 2015; Miyato et al., 2017), which is a small difference from a correct input. During the training process, we force the model to output the correct labels for both perturbed inputs and unmodified inputs. In sequence-to-sequence problems, existing studies regard the following as perturbed inputs: (1) sequences containing tokens replaced from correct ones (Bengio et al., 2015; Cheng et al., 2019), (2) embeddings injected small differences (Sato et al., 2019). For example, Bengio et al. (2015) proposed the scheduled sampling that samples a token from the output
probability distribution of a decoder and uses it as a perturbed input for the decoder. Sato et al. (2019) applied an adversarial perturbation, which significantly increases the loss value of a model, to the embedding spaces of neural encoder-decoders.
Those studies reported that their methods are effective to construct robust encoder-decoders. However, their methods are much slower than the training without using such perturbations because they require at least one forward computation to obtain the perturbation. In fact, we need to run the decoder the same times as the required number of perturbations in the scheduled sampling (Bengio et al., 2015). For adversarial perturbations (Sato et al., 2019), we have to compute the backpropagation in addition to forward computation because we use gradients to obtain perturbations.
Those properties seriously affect the training budget. For example, it costs approximately 1,800 USD for each run when we train Transformer (big) with adversarial perturbations (Sato et al., 2019) on the widely used WMT English-German training set in AWS EC $^{2}$ . Most studies conduct multiple runs for the hyper-parameter search and/or model ensemble to achieve better performance (Barrault et al., 2019), which incurs a tremendous amount of training budget for using such perturbations. Strubell et al. (2019) and Schwartz et al. (2019) indicated that recent neural approaches increase computational costs substantially, and they encouraged exploring a cost-efficient method. For instance, Li et al. (2020) explored a training strategy to obtain the best model in a given training time. However, previous studies have paid little attention to the costs of computing perturbations.
Thus, we rethink a time efficient perturbation method. In other words, we address the question whether perturbations proposed by recent studies as effective methods are time efficient. We compare
<sup>1</sup>We assume that we use on-demand instances having 8 V100 GPUs.
Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 5767-5780 June 6-11, 2021. ©2021 Association for Computational Linguistics
several perturbation methods for neural encoder-decoders in terms of computational time. We introduce light computation methods such as word dropout (Gal and Ghahramani, 2016) and using randomly sampled tokens as perturbed inputs. These methods are sometimes regarded as baseline methods (Bengio et al., 2015), but experiments on translation datasets indicate that these simple methods surprisingly achieve comparable scores to those of previous effective perturbations (Bengio et al., 2015; Sato et al., 2019) in a shorter training time. Moreover, we indicate that these simple methods are also effective for other sequence-to-sequence problems: GEC and summarization.
2 Definition of Encoder-Decoder
In this paper, we address sequence-to-sequence problems such as machine translation with neural encoder-decoders, and herein we provide a definition of encoder-decoders.
In sequence-to-sequence problems, neural encoder-decoders generate a sequence corresponding to an input sequence. Let  $x_{1:I}$  and  $y_{1:J}$  be input and output token sequences whose lengths are  $I$  and  $J$ , respectively:  $x_{1:I} = x_1,\dots,x_I$  and  $y_{1:J} = y_1,\dots,y_J$ . Neural encoder-decoders compute the following conditional probability:
$$
p (\boldsymbol {Y} | \boldsymbol {X}) = \prod_ {j = 1} ^ {J + 1} p \left(y _ {j} \mid \boldsymbol {y} _ {0: j - 1}, \boldsymbol {X}\right), \tag {1}
$$
where  $y_0$  and  $y_{J + 1}$  are special tokens representing beginning-of-sentence (BOS) and end-of-sentence (EOS) respectively,  $\pmb {X} = \pmb{x}_{1:I}$ , and  $\pmb {Y} = \pmb{y}_{1:J + 1}$ .
In the training phase, we optimize the parameters  $\pmb{\theta}$  to minimize the negative log-likelihood in the training data. Let  $\mathcal{D}$  be the training data consisting of a set of pairs of  $\pmb{X}_n$  and  $\pmb{Y}_n$ :  $\mathcal{D} = \{(X_n, Y_n)\}_{n=1}^{|\mathcal{D}|}$ . We minimize the following loss function:
$$
\mathcal {L} (\boldsymbol {\theta}) = - \frac {1}{| \mathcal {D} |} \sum_ {(\boldsymbol {X}, \boldsymbol {Y}) \in \mathcal {D}} \log p (\boldsymbol {Y} | \boldsymbol {X}; \boldsymbol {\theta}). \tag {2}
$$
3 Definition of Perturbations
This section briefly describes perturbations used in this study. This study focuses on three types of perturbations: word replacement, word dropout, and adversarial perturbations. Figure 1 shows perturbations used in this study. As shown in this figure, we can use all types of perturbations in the
Figure 1: Overview of perturbations used in this study. We can combine perturbations as shown in this figure because each type of perturbation is orthogonal.
same time because perturbations are orthogonal to each other. In fact, we combine word replacement with word dropout in our experiments.
3.1 Word Replacement: REP
For any approach that uses a sampled token instead of a correct token, such as the scheduled sampling (Bengio et al., 2015), we refer to this as a word replacement approach. In this approach, we construct a new sequence whose tokens are randomly replaced with sampled tokens. For the construction from the sequence  $\mathbf{X}$ , we sample  $\hat{x}_i$  from a distribution  $Q_{x_i}$  and use it for the new sequence  $\mathbf{X}'$  with the probability  $\alpha$ :
$$
\hat {x} _ {i} \sim Q _ {x _ {i}}, \tag {3}
$$
$$
x _ {i} ^ {\prime} = \left\{ \begin{array}{l l} x _ {i} & \text {w i t h p r o b a b i l i t y} \alpha \\ \hat {x} _ {i} & \text {w i t h p r o b a b i l i t y} 1 - \alpha . \end{array} \right. \tag {4}
$$
We construct  $\mathbf{Y}'$  from the sequence  $\mathbf{Y}$  in the same manner.
Bengio et al. (2015) used a curriculum learning strategy to adjust  $\alpha$ , and thus proposed several functions to decrease  $\alpha$  based on the training step. Their strategy uses correct tokens frequently at the beginning of training, whereas it favors sampled tokens frequently at the end of training. We also adjust  $\alpha$  with their use of the inverse sigmoid decay:
$$
\alpha_ {t} = \max  \left(q, \frac {k}{k + \exp \left(\frac {t}{k}\right)}\right) \tag {5}
$$
where  $q$  and  $k$  are hyper-parameters. In short,  $\alpha_{t}$  decreases to  $q$  from 1, depending on the training step  $t$ . We use  $\alpha_{t}$  as  $\alpha$  at  $t$ .
For  $Q_{x_i}$ , we prepare three types of distributions: conditional probability, uniform, and similarity.
Conditional Probability: REP(SS) Bengio et al. (2015) proposed the scheduled sampling which uses predicted tokens during training to address the gap between training and inference. Formally, the scheduled sampling uses the following conditional probability as  $Q_{y_i}$ :
$$
p \left(\hat {y} _ {i} \mid \boldsymbol {y} _ {0: i - 1} ^ {\prime}, X\right). \tag {6}
$$
Since the scheduled sampling is the method to compute the perturbation for the decoder side only, it uses the correct sequence as the input of the encoder side. In other words, the scheduled sampling does not provide any function for  $Q_{x_i}$ .
The original scheduled sampling repeats the decoding for each of the tokens on the decoder side, and thus, requires computational time in proportion to the length of the decoder-side input sequence. To address this issue, Duckworth et al. (2019) proposed a more time efficient method: parallel scheduled sampling which computes output probability distributions corresponding to each position simultaneously. In this study, we use parallel scheduled sampling instead of the original method.
Uniform: REP(UNI) The scheduled sampling is slow even if we use parallel scheduled sampling because it requires decoding at least once to compute Equation (6). Thus, we introduce two faster methods to explore effective perturbations from the perspective of computational time. In uniform, we use the uniform distributions on each vocabulary as  $Q_{x_i}$  and  $Q_{y_i}$ , respectively. For example, we randomly pick up a token from the source-side vocabulary and use the token as  $\hat{x}_i$  in Equation (4) to construct the source-side perturbed input. This method is used as the baseline in the previous study (Bengio et al., 2015).
Similarity: REP(SIM) We also explore more sophisticated way than the uniform distribution. We assume that the conditional probability of Equation (6) assigns high probabilities to tokens that are similar to the correct input token. Based on this assumption, we construct a distribution that enables us to sample similar tokens frequently. Let  $\mathcal{V}_x$  be the source-side vocabulary,  $E_x \in \mathbb{R}^{|\mathcal{V}_x| \times d_x}$  be the  $d_x$  dimensional embedding matrix, and  $e(x_i)$  be the function returning the embedding of  $x_i$ . We use the following probability distribution as  $Q_{x_i}$ :
$$
\operatorname {s o f t m a x} \left(\boldsymbol {E} _ {x} \boldsymbol {e} \left(x _ {i}\right)\right), \tag {7}
$$
where  $\mathrm{softmax}(.)$  is the softmax function. Thus, Equation (7) assigns high probabilities to tokens
whose embeddings are similar to  $e(x_{i})$ . In other words, Equation (7) is the similarity against  $x_{i}$  without considering any context. We compute the probability distribution for the target side by using  $e(y_{i})$  in the same manner.
3.2 Word Dropout: WDROP
We apply the word dropout technique to compute the perturbed input. Word dropout randomly uses the zero vector instead of the embedding  $e(x_{i})$  for the input token  $x_{i}$  (Gal and Ghahramani, 2016):
$$
b _ {x _ {i}} \sim \operatorname {B e r n o u l l i} (\beta), \tag {8}
$$
$$
\operatorname {W D r o p} \left(x _ {i}, b _ {x _ {i}}\right) = b _ {x _ {i}} e \left(x _ {i}\right), \tag {9}
$$
where Bernoulli  $(\beta)$  returns 1 with the probability  $\beta$  and 0 otherwise. Thus,  $\mathrm{WDrop}(x_i, b_{x_i})$  returns  $e(x_i)$  with the probability  $\beta$  and the zero vector otherwise. We apply Equation (9) to each token in the input sequence. Then, we use the results as the perturbed input.
3.3 Adversarial Perturbation: ADV
Miyato et al. (2017) proposed a method to compute adversarial perturbations in the embedding space. Their method adds adversarial perturbations to input embeddings instead of replacing correct input tokens with others. Sato et al. (2019) applied this approach to neural encoder-decoders and reported its effectiveness. Thus, this study follows the methods used in Sato et al. (2019).
The method seeks the adversarial perturbation, which seriously damages the loss value, based on the gradient of the loss function  $\mathcal{L}(\pmb{\theta})$ . Then, we add the adversarial perturbation to the input token embedding. Let  $r_{x_i} \in \mathbb{R}^{d_x}$  be the adversarial perturbation vector for the input token  $x_i$ . We obtain the perturbed input embedding  $e'(x_i)$  with the following equations:
$$
\boldsymbol {e} ^ {\prime} \left(x _ {i}\right) = \boldsymbol {e} \left(x _ {i}\right) + \boldsymbol {r} _ {x _ {i}}, \tag {10}
$$
$$
\boldsymbol {r} _ {x _ {i}} = \epsilon \frac {\boldsymbol {c} _ {x _ {i}}}{\left| \left| \boldsymbol {c} _ {x _ {i}} \right| \right|}, \tag {11}
$$
$$
\boldsymbol {c} _ {x _ {i}} = \nabla_ {\boldsymbol {e} (x _ {i})} \mathcal {L} (\boldsymbol {\theta}), \tag {12}
$$
where  $\epsilon$  is a hyper-parameter to control the norm of the adversarial perturbation. We apply the above equations to all tokens in the input sequence.
In the training using word replacement and/or word dropout perturbations, we search the parameters
predicting the correct output sequence from the perturbed input. For example, in the word replacement approach, we minimize the following negative log-likelihood:
$$
\begin{array}{l} \mathcal {L} ^ {\prime} (\pmb {\theta}) = - \frac {1}{| \mathcal {D} |} \sum_ {\mathcal {D}} \log p (\pmb {Y} | \pmb {X} ^ {\prime}, \pmb {Y} ^ {\prime}; \pmb {\theta}), \\ = - \frac {1}{| \mathcal {D} |} \sum_ {\mathcal {D}} \sum_ {j = 1} ^ {J + 1} \log p \left(y _ {j} \mid \boldsymbol {y} _ {0: j - 1} ^ {\prime}, \boldsymbol {X} ^ {\prime}; \boldsymbol {\theta}\right). \tag {13} \\ \end{array}
$$
Virtual Adversarial Training When we use adversarial perturbations, we train parameters of the neural encoder-decoder to minimize both Equation (2) and a loss function  $\mathcal{A}(\theta)$  composed of perturbed inputs:
$$
\mathcal {J} (\boldsymbol {\theta}) = \mathcal {L} (\boldsymbol {\theta}) + \lambda \mathcal {A} (\boldsymbol {\theta}), \tag {14}
$$
where  $\lambda$  is a hyper-parameter to control the balance of two loss functions. This calculation seems to be reasonably time efficient because adversarial perturbations require computing Equation (2).
Sato et al. (2019) used the virtual adversarial training originally proposed in Miyato et al. (2016) as a loss function for perturbed inputs. In the virtual adversarial training, we regard the output probability distributions given the correct input sequence as positive examples:
$$
\mathcal {A} (\boldsymbol {\theta}) = \frac {1}{| \mathcal {D} |} \sum_ {\mathcal {D}} \operatorname {K L} (p (\cdot | \boldsymbol {X}; \boldsymbol {\theta}) | | p (\cdot | \boldsymbol {X}, \boldsymbol {r}; \boldsymbol {\theta})) \tag {15}
$$
where  $r$  represents a concatenated vector of adversarial perturbations for each input token, and  $\mathrm{KL}(\cdot ||\cdot)$  denotes the Kullback-Leibler divergence.
4 Experiments on Machine Translation
To obtain findings on sequence-to-sequence problems, we conduct experiments on various situations: different numbers of training data and multiple tasks. We mainly focus on translation datasets because machine translation is a typical sequence-to-sequence problem. We regard the widely used WMT English-German dataset as a standard setting. In addition, we vary the number of training data in machine translation: high resource in Section 4.2 and low resource in Section 4.3. Table 1 summarizes the number of training data in each configuration. Moreover, we conduct experiments
Setting	Genuine	Synthetic
Standard	4.5M	-
High Resource	4.5M	20.0M
Low Resource	160K	-
Table 1: Sizes of training datasets on our machine translation experiments.
on other sequence-to-sequence problems: grammatical error correction (GEC) in Section 5 and summarization in Appendix A to confirm whether the findings from machine translation are applicable to other tasks.
4.1 Standard Setting
Datasets We used the WMT 2016 English-German training set, which contains 4.5M sentence pairs, in the same as Ott et al. (2018), and followed their pre-processing. We used newstest2013 as a validation set, and newstest2010-2012, and 2014-2016 as test sets. We measured case-sensitive detokenized BLEU with SacreBLEU (Post, 2018) $^{2}$ .
Methods We used Transformer (Vaswani et al., 2017) as a base neural encoder-decoder model because it is known as a strong neural encoder-decoder model. We used two parameter sizes: base and big settings in Vaswani et al. (2017).
We applied perturbations described in Section 3 for comparison. For parallel scheduled sampling (Duckworth et al., 2019), we can compute output probability distributions multiple times but we used the first decoding result only because it is the fastest approach. We set  $q = 0.9$ ,  $k = 1000$  and  $\beta = 0.9$ . For ADV, we used the same hyperparameters as in Sato et al. (2019). Our implementation is based on  $\text{fairseq}^3$  (Ott et al., 2019). We trained each model for a total of 50,000 steps.
Preliminary: To which sides do we apply perturbations? As described, perturbations based on  $\mathrm{REP}(\mathrm{SS})$  can be applied to the decoder side only. Sato et al. (2019) reported their method was the most effective when they applied their ADV to both encoder and decoder sides. However, we do not have evidence for suitable sides in applying other perturbations. Thus, we applied  $\mathrm{REP}(\mathrm{UNI})$ ,
As reported in Ott et al. (2018), the BLEU score from SacreBLEU is often lower than the score from multi-bleu.perl but SacreBLEU is suitable for scoring WMT datasets (Post, 2018).
Method	Position	2010	2011	2012	2013	2014	2015	2016	Average
Transformer (base)
w/o perturbation	-	24.27	22.06	22.43	26.11	27.13	29.70	34.40	26.59
REP(UNI)	enc	24.26	21.95	22.33	25.76	26.70	29.08	34.61	26.38
dec	24.27	21.99	22.29	26.31	27.28	29.74	34.42	26.61
both	24.30	22.20	22.43	26.06	26.82	29.42	34.13	26.48
REP(SIM)	enc	24.12	22.02	22.14	26.21	27.01	29.33	34.56	26.48
dec	24.32	21.96	22.55	26.36	27.23	29.86	34.33	26.66
both	23.94	21.85	22.29	25.84	26.61	29.50	34.20	26.32
WDROP	enc	24.31	22.12	22.45	26.20	27.09	29.95	34.58	26.67
dec	23.96	22.08	22.22	26.36	27.08	29.91	33.98	26.51
both	24.33	22.14	22.35	26.10	26.82	29.51	34.51	26.54
Transformer (big)
w/o perturbation	-	24.22	22.11	22.69	26.60	28.46	30.50	33.58	26.88
REP(UNI)	enc	24.79	22.49	23.10	27.07	28.39	30.52	34.51	27.27
dec	24.33	22.34	22.63	26.93	28.22	30.36	33.41	26.89
both	24.75	22.68	23.32	27.01	28.89	31.38	34.94	27.57
REP(SIM)	enc	24.68	22.91	23.13	27.03	28.25	30.81	34.40	27.32
dec	24.51	22.22	22.83	26.46	28.64	30.68	33.58	26.99
both	24.77	22.50	23.10	26.91	28.98	31.03	34.29	27.37
WDROP	enc	24.60	22.32	23.27	27.07	28.40	31.00	34.61	27.32
dec	24.53	22.33	22.75	27.00	28.56	30.58	33.20	26.99
both	24.92	22.71	23.40	27.11	28.73	30.99	34.80	27.52
REP(UNI)+WDROP	both	24.82	22.82	23.38	27.30	28.56	30.65	35.02	27.51
REP(SIM)+WDROP	both	24.83	22.95	23.40	27.23	28.65	30.88	35.05	27.57
Table 2: BLEU scores on newstest2010-2016 and averaged scores. Bolds are better scores than w/o perturbations.
$\mathrm{REP}(\mathrm{SIM})$  , and WDROP to the encoder side, decoder side, and both as preliminary experiments.
Table 2 shows BLEU scores on newstest2010-2016 and averaged scores when we varied the position of the perturbations. In this table, we indicate better scores than the original Transformer (Vaswani et al., 2017) (w/o perturbation) in bold. This table shows that it is better to apply word replacement (REP(UNI) and REP(SIM)) to the decoder side in Transformer (base). For WDROP, applying the encoder side is slightly better than other positions in Transformer (base). In contrast, applying perturbations to both sides achieved the best averaged BLEU scores for all methods in Transformer (big). These results imply that it is better to apply to word replacement and/or word dropout to both encoder and decoder sides if we prepare enough parameters for neural encoder-decoders. Based on these results, we select methods to compare against scheduled sampling (REP(SS)) and adversarial perturbations (ADV).
Table 2 also shows the results when we combined each word replacement with word dropout  $(\mathrm{REP}(\mathrm{UNI}) + \mathrm{WDROP}$  and  $\mathrm{REP}(\mathrm{SIM}) + \mathrm{WDROP})$ $\mathrm{REP}(\mathrm{SIM}) + \mathrm{WDROP}$  slightly outperformed the separated settings.
Results We compare each perturbation in view of computational time. Table 3 shows BLEU scores
of each method and computational speeds $^4$  based on Transformer (base) without any perturbations, i.e., larger is faster. In this table, we indicate the best score of each column for Transformer (base) and (big) settings in bold. This table indicates that Transformer without perturbations achieved a comparable score to previous studies (Vaswani et al., 2017; Ott et al., 2018) on newtest2014 in base and big settings. Thus, we consider that our trained Transformer models (w/o perturbation) can be regarded as strong baselines. This table shows that ADV achieved the best averaged score in Transformer (base), but this method required twice as much training time as the original Transformer (base). In contrast, REP(SIM) and WDROP achieved comparable scores to ADV although they slightly affected the computational time. REP(UNI) also achieved a slightly better averaged score than the original Transformer (base).
In the Transformer (big) setting, all perturbations surpassed the performance of w/o perturbation in the averaged score. REP(SS) and ADV improved the performance, but other methods outperformed these two methods with a small training time. Moreover, REP(UNI) and REP(SIM)+WDROP achieved the best averaged score.
Figure 2 illustrates the negative log-likelihood
4We regard processed tokens per second as the computational speed of each method.
Method	Position	2010	2011	2012	2013	2014	2015	2016	Average	Speed
Transformer (base)
w/o perturbation	-	24.27	22.06	22.43	26.11	27.13	29.70	34.40	26.59	×1.00
REP(UNI)	dec	24.27	21.99	22.29	26.31	27.28	29.74	34.42	26.61	×0.99
REP(SIM)	dec	24.32	21.96	22.55	26.36	27.23	29.86	34.33	26.66	×0.95
WDROP	enc	24.31	22.12	22.45	26.20	27.09	29.95	34.58	26.67	×1.00
REP(SS)	dec	24.18	22.03	22.38	26.04	27.15	29.77	34.24	26.54	×0.88
ADV	both	24.34	22.19	22.58	26.19	27.10	29.78	34.89	26.72	×0.44
Transformer (big)
w/o perturbation	-	24.22	22.11	22.69	26.60	28.46	30.50	33.58	26.88	×0.60
REP(UNI)	both	24.75	22.68	23.32	27.01	28.89	31.38	34.94	27.57	×0.60
REP(SIM)	both	24.77	22.50	23.10	26.91	28.98	31.03	34.29	27.37	×0.55
WDROP	both	24.92	22.71	23.40	27.11	28.73	30.99	34.80	27.52	×0.60
REP(UNI)+WDROP	both	24.82	22.82	23.38	27.30	28.56	30.65	35.02	27.51	×0.60
REP(SIM)+WDROP	both	24.83	22.95	23.40	27.23	28.65	30.88	35.05	27.57	×0.55
REP(SS)	dec	24.44	21.97	22.74	26.77	28.44	30.83	33.71	26.99	×0.52
ADV	both	24.71	22.60	23.23	26.98	28.97	30.49	34.40	27.34	×0.20
Table 3: BLEU scores on newestest2010-2016,averaged scores,and computational speeds based on Transformer (base) w/o perturbation. Scores in bold denote the best result for each set for Transformer (base) and (big) settings.
(a) Valid NLL
(b) BLEU score
(c) Time to achieve BLEU score of Transformer w/o perturbation
Figure 2: Negative log-likelihood (NLL) values, BLEU scores of each method, and time to achieve BLEU score of Transformer w/o perturbation on validation set (newstest2013).