SlowGuess/ACL_2020_OCR huggingface dataset:
Luoxin Chen, Weitong Ruan, Xinyue Liu, Jianhua Lu
Amazon Alexa AI, Cambridge, MA, US
{luoxchen, weiton, luxnyu, jianhual}@amazon.com
Virtual adversarial training (VAT) is a powerful technique to improve model robustness in both supervised and semi-supervised settings. It is effective and can be easily adopted on lots of image classification and text classification tasks. However, its benefits to sequence labeling tasks such as named entity recognition (NER) have not been shown as significant, mostly, because the previous approach can not combine VAT with the conditional random field (CRF). CRF can significantly boost accuracy for sequence models by putting constraints on label transitions, which makes it an essential component in most state-of-the-art sequence labeling model architectures. In this paper, we propose SeqVAT, a method which naturally applies VAT to sequence labeling models with CRF. Empirical studies show that SeqVAT not only significantly improves the sequence labeling performance over baselines under supervised settings, but also outperforms state-of-the-art approaches under semi-supervised settings.
While having achieved great success on various computer vision and natural language processing tasks, deep neural networks, even state-of-the-art models, are usually vulnerable to tiny input perturbations (Szegedy et al., 2014; Goodfellow et al., 2015). To improve the model robustness against perturbations, Goodfellow et al. (2015) proposed to train neural networks on both original training examples and adversarial examples (examples generated by adding small but worst-case perturbations to the original examples). This approach, named adversarial training (AT), has been reported to be highly effective on image classification (Goodfellow et al., 2015), text classification (Miyato et al., 2017), as well as sequence labeling (Yasunaga et al., 2018).
However, AT is limited to a supervised scenario,
which uses the labels to compute adversarial losses. To make use of unlabeled data, virtual adversarial training (VAT) was proposed to extend AT to semi-supervised settings (Miyato et al., 2019). Unlike AT which treats adversarial examples as new training instances that have the same labels as original examples, VAT minimizes the KL divergence between estimated label distribution of original examples and that of adversarial examples. In this manner, both labeled and unlabeled data can be used in training to improve accuracy and robustness. As a semi-supervised learning algorithm, VAT was reported to be effective on both image (Goodfellow et al., 2015; Miyato et al., 2019) and text classifications (Miyato et al., 2017). Moreover, a recent study (Oliver et al., 2018) conducted comprehensive comparisons on various popular semi-supervised learning algorithms. VAT turned out to be the most effective one.
Despite its success in classification tasks, VAT has not shown similar effectiveness in sequence labeling tasks. In the conventional classification task, the model learns a mapping between a sentence (sequence of tokens) and a label. Nevertheless, in sequence labeling task, the target function becomes a mapping from a sequence of tokens to a sequence of labels. To apply VAT on sequence labeling, Clark et al. (2018) proposed to use a softmax layer on the top of token representations to obtain label probability distributions for each token. In this fashion, VAT could take KL divergence between tokens at the same position of the original sequence and the adversarial sequence as the adversarial losses. This approach shows marginal improvements over baseline models on several benchmarks, but fails to achieve comparable performance as other state-of-the-art models (Clark et al., 2018; Akbik et al., 2018; Peters et al., 2018; Devlin et al., 2019).
Although the approach above applies VAT on the entire sequence, it locally normalizes the label probability per token and assumes all transitions
8801
Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 8801-8811 July 5 - 10, 2020. ©2020 Association for Computational Linguistics
between labels have equal possibilities. But in sequence labeling tasks, label transition probabilities are not always the same. For example, a song name is more likely to appear after a singer name, compared to a travel company.
To incorporate label transitions into sequence models, Lafferty et al. (2001) proposed conditional random field (CRF). CRF models the probability distribution of the whole label sequence given the input sequence, instead of yielding a label probability distribution for each token. It takes account of both token features and transition features. Most state-of-the-art sequence labeling models apply a CRF on top of token representations as a decoder. Such neural-CRF models usually outperform models without CRF (Ma and Hovy, 2016; Akbik et al., 2018; Peters et al., 2018; Yasunaga et al., 2018).
To apply the conventional VAT on a model with CRF, one can calculate the KL divergence on the label distribution of each token between the original examples and adversarial examples. However, it is sub-optimal because the transition probabilities are not taken into account.
To better address these issues, we proposed SeqVAT, a variant of VAT that can be used along with CRF. Our evaluation demonstrates that SeqVAT brings significant improvements in supervised settings, rather than marginal improvements reported from previous VAT-based approaches Clark et al.. In the semi-supervised settings, SeqVAT also outperforms many widely used methods such as self-training (ST) (Yarowsky, 1995) and entropy minimization (EM) (Grandvalet and Bengio, 2004), as well as the state-of-the-art semi-supervised sequence labeling algorithm, cross-view training (CVT) (Clark et al., 2018).
2.1 Sequence Labeling
Sequence labeling is a series of common natural language processing tasks that predicts a label for each token within a sequence, rather than a label for the whole sequence. Such tasks include named entity recognition, chunking and part-of-speech (POS) tagging etc. Most state-of-the-art sequence labeling models are based on a neural-CRF architecture (Ma and Hovy, 2016; Akbik et al., 2018; Peters et al., 2018; Yasunaga et al., 2018). More precisely, the general design is to use bidirectional recurrent neural network (RNN) layers for encoding and a CRF layer for decoding. In
addition, usually one or more convolutional neural network (CNN) or RNN layers are applied before the neural-CRF architecture to encode character-level information as part of the input. In this paper, we adapt the neural-CRF architecture by a CNN-LSTM-CRF model, which consists of one CNN layer to generate character embeddings, two layers of bidirectional long short-term memory (LSTM) as the encoder and a CRF layer as the decoder.
2.2 Semi-Supervised Learning
Semi-supervised learning is an important approach to improve model performance without enough labeled data. It utilizes unlabeled data to get more information which might be beneficial for supervised tasks. For semi-supervised learning, two robust and widely used approaches are self-training (ST) (Yarowsky, 1995) and entropy minimization (EM) (Grandvalet and Bengio, 2004). In natural language processing, ST has been successfully applied to word sense disambiguation (Yarowsky, 1995) and parsing (McClosky et al., 2006), and EM also has successful application in text classification (Sachan et al., 2019).
Recently, a powerful semi-supervised approach, cross-view training (CVT), has achieved state-of-the-art on several semi-supervised language tasks, including dependency parsing, machine translation and chunking (Clark et al., 2018). CVT forces the model to make consistent predictions when using the full input or partial input. Hence, it does not require label information and can be used for semi-supervised learning. In order to validate the effectiveness of our approach on semi-supervised sequence labeling, we make fair comparisons to those three semi-supervised learning methods in the experiments.
2.3 Virtual Adversarial Training
Adversarial training (Goodfellow et al., 2015) is a regularization method that enhances model robustness against input perturbations. It generates adversarial examples by injecting worst-case perturbations bounded by a small norm into the original examples, and adds them into training. As a consequence, model predictions would be consistent regardless of the perturbations. Prior to AT, several papers investigated various ways of perturbations (Xie et al., 2017). Adversarial training was demonstrated to be more effective since it introduces the perturbations which leading to the largest increase on model loss, respective to a constrained
8802
Figure 1: Sequence Labeling Model Architecture.
size (Goodfellow et al., 2015). Goodfellow et al. (2015) proved the effect of adversarial training in enhancing model robustness especially towards unseen samples for image classification. In addition to computer vision tasks, adversarial training also demonstrated its effectiveness on language tasks, such as text classification, POS tagging, named entity recognition and chunking (Miyato et al., 2017; Yasunaga et al., 2018).
To extend AT to semi-supervised settings, Miyato et al. (2019) proposed virtual adversarial training (VAT). "Virtual" means label information is not required in this new adversarial training approach and consequently it could be applied to both labeled or unlabeled training instances. VAT achieved state-of-the-art performance for image classification tasks (Miyato et al., 2019), and proved to be more efficient than traditional semi-supervised approaches, such as entropy minimization (Grandvalet and Bengio, 2004) and self-training (Yarowsky, 1995), from a recent study (Oliver et al., 2018).
However, despite the successful applications on text classification (Miyato et al., 2017), VAT has not shown great benefits to semi-supervised sequence labeling tasks, due to its incompatibility with CRF. In this paper, SeqVAT is proposed to make VAT compatible with CRF, and achieves significant improvements in sequence labeling.
Our baseline model architecture is illustrated in Fig.1. It adopts the basic architecture for several state-of-the-art sequence labeling models (Ma and Hovy, 2016; Peters et al., 2017; Akbik et al., 2018; Peters et al., 2018), called CNN-LSTM-CRF (CLC) in this paper. We apply a CNN layer to extract character information and concatenate its output with word embeddings as input features. Then, we feed the input features into LSTM layers, and decode with a CRF layer.
3.1.1 Word Embeddings
300-dimension randomly initialized word embeddings serve as word-level input. However, the model could learn embeddings with large norm, which makes the effects of adversarial perturbations with small norm insignificant (Miyato et al., 2017). To avoid such effect, we normalize the word embeddings at the beginning of each epoch. Denote  $v = \{v_{i} | i = 1, 2, \dots, n\}$  as the embeddings set, where  $n$  is vocabulary size, a specific embedding  $v_{i}$  is normalized by:
$$
\hat {v _ {i}} \quad = \quad \frac {v _ {i} - E (v)}{\sqrt {D (v)}} \tag {1}
$$
$$
w h e r e \quad E (v) = \frac {1}{n} \sum_ {i = 1} ^ {n} v _ {i}
$$
$$
a n d \quad D (v) = \frac {1}{n} \sum_ {i = 1} ^ {n} (v _ {i} - E (v)) ^ {2}
$$
After normalization, word embeddings have zero mean and unit variance.
3.1.2 Character CNN Layer
Character-level information has proved to help improve the sequence labeling accuracy by capturing morphological features (Ma and Hovy, 2016). In this paper, 32-dimension embeddings are randomly initialized for each character. To ensure that adversarial perturbations have significant effects, character embeddings are also normalized at the beginning of each epoch in the same way as word embeddings. Suppose  $u = \{u_i | i = 1, 2, \dots, m\}$  where  $m$  is the number of unique characters show up in the dataset, a specific embedding  $u_i$  is randomly initialized and normalized by:
8803
for  $w$  and  $c$  are:
$$
\begin{array}{l} \hat {u _ {i}} \quad = \quad \frac {u _ {i} - E (u)}{\sqrt {D (u)}} \tag {2} \\ w h e r e \quad E (u) = \frac {1}{m} \sum_ {i = 1} ^ {m} u _ {i} \\ a n d \quad D (u) = \frac {1}{m} \sum_ {i = 1} ^ {m} (u _ {i} - E (u)) ^ {2} \\ \end{array}
$$
A CNN layer with 16 unigram, 16 bigram and 32 trigram filters is applied on top of all 32-dimension embeddings for one word. Hence, each word has 64-dimension character embeddings which are the output of CNN layer.
3.1.3 LSTM Layer
After concatenating character embeddings and word embeddings as input, all those features pass through two bidirectional LSTM layers with 256 neurons per direction to encode information for the whole sequence.
3.1.4 CRF Layer
To incorporate the probabilities of label transitions, the outputs of LSTM layers are fed into a linear-chain CRF decoder (Lafferty et al., 2001). Negative log-likelihood is computed as the training loss and Viterbi algorithm (Viterbi, 1967) is used for decoding.
Adversarial training (Goodfellow et al., 2015) is an effective method to improve model robustness over input perturbations. AT first generates adversarial examples, which are close to the original examples but model is not likely to correctly predict their labels (i.e. leading to most significant loss increase). Then, the model is trained with both original examples and adversarial examples. The loss on adversarial examples are treated as adversarial loss. In this paper, adversarial perturbations are added to word and character embeddings respectively. To prevent vanishing effects of adversarial perturbations explained in section 3.1.1 and 3.1.2, embeddings are normalized at the beginning of each epoch. Denote  $w$  and  $c$  as normalized word and character embeddings of the whole input sequence,  $\theta$  is parameter of model,  $y$  is a vector of labels for all tokens in the sequence, and  $Loss$  is the loss (i.e. negative log-likelihood) for the whole sequence. Given the bounded norms  $\delta_w$  and  $\delta_c$  respectively, the worst-case perturbations  $d_w$  and  $d_c$
$$
d _ {w} = \underset {\epsilon , | | \epsilon | | _ {2} \leq \delta_ {w}} {\operatorname {a r g m a x}} L o s s (y; w + \epsilon , c, \hat {\theta}) \tag {3}
$$
$$
d _ {c} = \underset {\tau , | | \tau | | _ {2} \leq \delta_ {c}} {\operatorname {a r g m a x}} L o s s (y; w, c + \tau , \hat {\theta}) \tag {4}
$$
Note that all variables,  $y$ ,  $w$ ,  $c$ ,  $d_w$  and  $d_c$  here are vectors for the whole sequence, since the last layer, CRF, is modeling the whole label sequence. In addition,  $\hat{\theta}$  is current estimation of  $\theta$ . The purpose for using constant value  $\hat{\theta}$  instead of  $\theta$  is to emphasize that the gradient should not propagate during generation of adversarial examples.
Hence, the worst-case perturbations  $d_w$  and  $d_c$  against current model can be calculated through (3) and (4) at each training step, and model can be trained on examples plus those perturbations to improve robustness against them. Yet, computing exact value of those perturbations with maximization is intractable for complex DNN models. As proposed by Goodfellow et al. (2015), first order approximation is applied to approximate the value of  $d_w$  and  $d_c$ . With this approximation,  $d_w$  and  $d_c$  can be calculated by:
$$
d _ {w} = \frac {g _ {w}}{\left\| g _ {w} \right\| _ {2}} \delta_ {w} \tag {5}
$$
$$
d _ {c} = \frac {g _ {c}}{\left\| g _ {c} \right\| _ {2}} \delta_ {c} \tag {6}
$$
$$
w h e r e g _ {w} = \nabla_ {w} L o s s (y; w, c, \hat {\theta}),
$$
$$
{ a n d } { g _ { c } } = { \nabla _ { c } L o s s ( y ; w , c , \hat { \theta } ) }
$$
Then, the adversarial loss  $L_{adv}$  is formed by:
$$
L _ {a d v} = \operatorname {L o s s} (y; w + d _ {w}, c + d _ {c}, \hat {\theta}) \tag {7}
$$
3.3 Virtual Adversarial Training
Nevertheless, adversarial training cannot be applied to unlabeled data since label information is required to generate adversarial examples and compute adversarial loss. Virtual adversarial training is proposed (Miyato et al., 2019) to adapt adversarial training to semi-supervised settings. In VAT, instead of using the regular loss on perturbed examples as adversarial loss, the discrepancy (KL divergence) between predictions of original examples and those of adversarial examples acts as the adversarial loss. With this modification, label information is not needed in the computation of adversarial loss.
Indeed, the adversarial loss for VAT is written as:
8804
$$
L _ {a d v} = K L \left(P _ {o r i} \mid \mid P _ {a d v}\right) \tag {8}
$$
$$
w h e r e P _ {o r i} = P (\hat {y}; w, c, \hat {\theta}),
$$
$$
{ a n d } { P _ { a d v } } = { P ( \hat { y } ; w + d _ { w } , c + d _ { c } , \hat { \theta } ) }
$$
Here,  $\hat{y}$  is to emphasize that the computation of KL divergence takes current estimation of distribution over  $y$ , so that label information is not required.  $P_{ori}$  and  $P_{adv}$  are the estimated probability distributions of labels on original examples and adversarial examples respectively. As explained in section 1, VAT is not compatible with CRF. Hence,  $P_{ori}$  and  $P_{adv}$  here stand for sets of label distributions for tokens, computed by applying a softmax on top of LSTM output representations. As a consequence, the function  $P$  to estimate probability distributions of labels here is:
$$
P (\hat {y}; w, c, \hat {\theta}) = C L S (w, c, \hat {\theta}) \tag {9}
$$
where  $CLS$  means applying softmax on top of CNN-LSTM encoder.
However, to compute worst-case perturbations  $d_w$  and  $d_c$ , label information  $y$  is still needed, as in equation (3), (4), (5) and (6). To get rid of the label information, the worst-case perturbations are now computed based on KL divergence between  $P_{ori}$  and  $P_{adv}$ , given the bounded norms  $\delta_w$  and  $\delta_c$ .
So word perturbation  $d_w$  is now defined by:
$$
\underset {\epsilon , \left| \left| \epsilon \right| \right| _ {2} \leq \delta_ {w}} {\operatorname {a r g m a x}} K L (P (\hat {y}; w, c, \hat {\theta}) | | P (\hat {y}; w + \epsilon , c, \hat {\theta})) \tag {10}
$$
While character perturbation  $d_{c}$  is:
$$
\underset {\tau , \| \tau \| _ {2} \leq \delta_ {c}} {\operatorname {a r g m a x}} K L (P (\hat {y}; w, c, \hat {\theta}) \| P (\hat {y}; w, c + \tau , \hat {\theta})) \tag {11}
$$
Those two computations are still intractable for gradient descent. By applying second-order approximation and a single iteration of power method, as in (Miyato et al., 2019), the word perturbation and character perturbation can be estimated with:
$$
d _ {w} = \frac {g _ {w}}{\left\| g _ {w} \right\| _ {2}} \delta_ {w} \tag {12}
$$
$$
d _ {c} = \frac {g _ {c}}{\left\| g _ {c} \right\| _ {2}} \delta_ {c} \tag {13}
$$
$$
g _ {w} = \nabla_ {\epsilon} K L (P (\hat {y}; w, c, \hat {\theta}) | | P (\hat {y}; w + \epsilon , c, \hat {\theta})),
$$
$$
g _ {c} = \nabla_ {\tau} K L (P (\hat {y}; w, c, \hat {\theta}) | | P (\hat {y}; w, c + \tau , \hat {\theta}))
$$
3.4 SeqVAT
Because of its incompatibility with CRF, adapting VAT to sequence labeling is not yet successful (Clark et al., 2018). To fully release the power of VAT to sequence labeling models with CRF, we propose a CRF-friendly VAT, named SeqVAT.
CRF models the conditional probability of the whole label sequence given the whole input sequence. Consequently, instead of using the label distribution over individual token, we could use the probability distribution for the whole label sequence, to compute KL divergence. The probability distribution can be denoted by:
$$
P (\hat {y}; w, c, \hat {\theta}) = C L C (w, c, \hat {\theta}) \tag {14}
$$
where  $\hat{y}$  is the whole label sequence, and CLC indicates the full CLC model.
Nevertheless, given a sequence with  $t$  tokens and  $l$  possible labels for each token, the total number of possible label sequences is  $l^t$ . Considering the substantial number of possible label sequences, it is not possible to compute the full probability distribution over all possible label sequences. To make the computation of such distribution possible, we estimate the full distribution by only considering the probabilities of  $k$  most possible label sequences, with one additional dimension to represent all the rest label sequences. Thus, the estimation of the probability distribution is  $(k + 1)$  dimensions and feasible to compute.
To get the most possible label sequences, we apply a k-best Viterbi decoding (Huang and Chiang, 2005) on the original sequence in each training step. Denote  $S = (s_1, s_2, \dots, s_k)$  as the k-best label sequences of current input embeddings  $w$  and  $c$ , and  $p_{crf}$  as the function to get probability of a label sequence. Given the current parameters  $\hat{\theta}$ , the probability distribution estimation  $P'$  can be written as:
$$
P ^ {\prime} (S; w, c, \hat {\theta}) = \left(p _ {1} ^ {\prime}, p _ {2} ^ {\prime},.., p _ {k} ^ {\prime}, 1 - \sum_ {i = 1} ^ {k} p _ {i} ^ {\prime}\right) (1 5)
$$
$$
w h e r e p _ {i} ^ {\prime} = p _ {c r f} (s _ {i}; w, c, \hat {\theta}), i \in [ 1, k ]
$$
Then,  $P_{ori}$  and  $P_{adv}$  can be denoted as:
$$
P _ {o r i} = P ^ {\prime} (S; w, c, \hat {\theta}) \tag {16}
$$
$$
P _ {a d v} = P ^ {\prime} \left(S; w + d _ {w}, c + d _ {c}, \hat {\theta}\right) \tag {17}
$$
8805
Here,  $d_w$  and  $d_c$  can be computed using the same approximation as VAT by:
$$
d _ {w} = \frac {g _ {w}}{\left\| g _ {w} \right\| _ {2}} \delta_ {w} \tag {18}
$$
$$
d _ {c} = \frac {g _ {c}}{\left\| g _ {c} \right\| _ {2}} \delta_ {c} \tag {19}
$$
$$
g _ {w} = \nabla_ {\epsilon} K L (P ^ {\prime} (S; w, c, \hat {\theta}) | | P ^ {\prime} (S; w + \epsilon , c, \hat {\theta})),
$$
$$
{g _ {c}} = {\nabla_ {\tau} K L (P ^ {\prime} (S; w, c, \hat {\theta}) | | P ^ {\prime} (S; w, c + \tau , \hat {\theta}))}
$$
The adversarial loss for SeqVAT can be computed by:
$$
L _ {a d v} = K L \left(P _ {o r i} \| P _ {a d v}\right) \tag {20}
$$
3.5 Training with Adversarial Loss
Regardless of the adversarial training method we use (AT, VAT or SeqVAT), sequence labeling loss is computed for all labeled data at each training step:
$$
L _ {l a b e l} = \operatorname {L o s s} (y; w, c, \eta , \hat {\theta}) \tag {21}
$$
In addition, in every training step, adversarial examples are generated and adversarial loss  $L_{adv}$  is calculated based on the corresponding adversarial training algorithm. To combine the sequence labeling loss and adversarial loss, the total loss is a summation of those two loss:
$$
L _ {\text {t o t a l}} = L _ {\text {l a b e l}} + \lambda L _ {\text {a d v}} \tag {22}
$$
Here, weight  $\lambda$  is introduced to balance the model accuracy (sequence labeling loss) and robustness (adversarial loss). This objective function is optimized with respect to  $\theta$ .
Note, unlabeled data might be leveraged in VAT and SeqVAT, and they do not have sequence labeling loss due to lack of annotation. Hence, the sequence labeling loss  $L_{label}$  would be set to 0 for unlabeled data.
Our proposed method is evaluated on three datasets: CoNLL 2000 (Sang and Buchholz, 2000) for chunking, CoNLL 2003 (Sang and Meulder, 2003) for named entity recognition (NER) and an internal natural language understanding (NLU) dataset for slot filling.
For chunking and NER, One Billion Word Language Model Benchmark (Chelba et al., 2014) is
Domain	Labels	Train	Test	Unlabeled
Cook	66	306155	55368	416348
Joke	20	230835	10311	586509
Booking	32	121067	5691	218864
News	12	116841	9607	339790
Assist	15	164364	5922	199383
Sporting	14	26763	3119	16034
Table 1: Number of sentences and labels in our internal NLU dataset.
used as unlabeled data pool for semi-supervised learning. Considering the relatively small size of those two datasets, we randomly sampled  $1\%$  of the benchmark as the unlabeled dataset. We still have 20 times more data than training sets of CoNLL 2000 and 2003. For slot filling, our NLU dataset contains labeled and unlabeled sentences for 6 domains (detailed information is shown in Table.1). We directly use the unlabeled data for semi-supervised experiments.