SlowGuess/ACL_2020_OCR huggingface dataset:
All parameters are randomly initialized. All hyperparameters are chosen by grid search on the development set. Variational dropout (Blum et al., 2015) with rate 0.2 is applied to the input and output of each LSTM layer. The perturbation sizes for word and character embeddings,  $\delta_w$  and  $\delta_c$ , are 0.4 and 0.2 respectively. The weight for adversarial loss (i.e.  $\lambda$ ) is set to 0.6.  $k$  is set to 3 for CoNLL datasets and 9 for our NLU dataset.
Sequence labeling model is optimized by Adam optimizer (Kingma and Ba, 2015) with batch size 64, learning rate 0.0006 and decay rate 0.992. Early stopping is applied based on model performance on the development set.
All sequence labeling tasks are evaluated with "slot-F1" metric, which is used in CoNLL 2000 and CoNLL 2003 shared tasks (Sang and Buchholz, 2000; Sang and Meulder, 2003).
5.1 Supervised Sequence Labeling
We evaluate our proposed SeqVAT technique in supervised settings and compare the results with other techniques designed to improve model robustness, including AT (Miyato et al., 2017), VAT (Miyato et al., 2019) and CVT (Clark et al., 2018).
To demonstrate the effectiveness of CRF, we compare results from models with or without CRF using each training technique mentioned above.
8806
Method	Cook	Joke	Booking	News	Assist	Sporting	CoNLL 2000	CoNLL 2003
Baseline w/o. CRF	88.25	87.39	92.32	89.55	83.55	82.73	94.68	90.59
AT w/o. CRF	88.51	88.05	93.03	89.36	84.14	84.25	95.03	90.95
VAT w/o. CRF	88.40	88.04	92.84	89.75	83.92	84.21	94.89	90.87
CVT w/o. CRF	88.47	87.99	92.97	89.41	83.50	84.07	94.76	91.02
Baseline	88.53	87.97	93.04	90.32	84.99	86.67	95.18	91.20
AT	88.93	88.32	93.21	90.46	85.26	87.66	95.30	91.63
VAT	88.62	88.19	93.11	90.38	85.05	87.20	95.21	91.55
CVT	88.86	88.24	93.18	90.36	85.12	87.63	95.26	91.47
SeqVAT	88.90	88.46	93.23	90.81	85.28	87.79	95.45	91.76
ST†	88.73	88.69	93.42	91.29	85.13	86.73	95.27	91.66
EM†	88.68	88.70	93.45	91.21	85.09	86.79	95.91	91.69
VAT†	88.92	88.30	93.66	91.34	84.97	87.82	96.12	91.70
CVT†	88.81	88.75	93.57	91.31	85.58	87.80	96.19	92.08
SeqVAT†	89.05	88.87	93.74	91.57	85.86	88.43	96.34	92.27
Table 2: Slot F1 on all domains and datasets. "w/o. CRF" indicates CRF is excluded in the model architecture. † indicates semi-supervised sequence labeling.
In Table.2, the first set of results corresponds to models without CRF, while the second utilizes CRF. Note, based on the characteristics of each training technique, the added adversarial loss varies. Since AT is compatible with CRF, and thus its adversarial loss is computed on top of CRF. But as explained in Sec.1, the adversarial loss of conventional VAT cannot be calculated on top of CRF. Consequently, VAT in the second set of Table.2 only applies CRF for label loss. It uses adversarial loss without CRF.
As shown in Table.2, regardless of the training techniques, models with CRF consistently perform better than those without it. This demonstrates that CRF is a crucial component in sequence labeling. Hence, we conduct the rest of our evaluation only on models with CRF.
Moreover, except that AT performs slightly better than SeqVAT in Cook domain, SeqVAT can outperform all approaches in all the other domains/datasets. All improvements of SeqVAT over other approaches are statistically significant (with p-value  $< 0.05$  in t-test). Compared with VAT used by Clark et al. (2018), SeqVAT consistently shows more significant improvements, which indicates that SeqVAT is a better way of adopting virtual adversarial loss to sequence labeling.
5.2 Semi-Supervised Sequence Labeling
VAT has been proved to be very effective in semi-supervised learning (Oliver et al., 2018). Our proposed SeqVAT preserves the ability of utilizing unlabeled data. In this work, we also compare SeqVAT with two widely used semi-supervised learning algorithms: self-training (ST) (Yarowsky, 1995), entropy minimization (EM) (Grandvalet
and Bengio, 2004), and one state-of-the-art semi-supervised sequence labeling approach, cross-view training (CVT) (Clark et al., 2018). Detailed results are tabulated in the third set of Table.2. From this comparison, SeqVAT consistently outperforms conventional VAT, ST, EM, and CVT. The improvements over other approaches are also statistically significant with p-value  $< 0.05$ . These results suggest that SeqVAT is also highly effective at utilizing unlabeled data.
5.3 K-best Selection in SeqVAT
To choose the optimal  $k$  in k-best decoding, we conduct experiments with different  $k$ s on supervised sequence labeling. The F1 score from each  $k$  is plotted in Fig.2. From these plots, we observe that each dataset has its own optimal  $k$  for SeqVAT, and there is no unique  $k$  that gives the best results across datasets.
To get a better generalization over all datasets and tasks, we avoid selecting the optimal  $k$  for each dataset/domain. However, different sources of language have different characteristics, including vocabulary, sentence length, syntax etc. Using the same  $k$  for different types of text might limit the effects of SeqVAT. To make a balance between generalization and effectiveness, we use different  $k$  for different types of text, but the same  $k$  for all datasets/domains with the same source. We use  $k = 3$  for CoNLL 2000 and 2003 (news), and  $k = 9$  for our internal NLU dataset (spoken language).
5.4 Impact of Unlabeled Data
To further understand the effect of unlabeled data in semi-supervised learning, we analyze the corre
8807
(a) Cook
(b) Joke
(c) Booking
(d) News
(g) CoNLL 2000
Figure 2: K-best results with different values of  $k$  for all domains and datasets.
(e) Assist
(f) Sportinging
(h) CoNLL 2003
lation between the amount of augmented unlabeled data and model performance on both CoNLL 2000 and 2003 datasets. For this analysis, we specifically focus ourselves on CVT and SeqVAT, which show the best accuracy across all datasets in Table.2. As shown in Fig.3, the amount of unlabeled data is a crucial factor for the performance of those two approaches. More specifically, the performance of those two approaches increases with more unlabeled data. For the CoNLL 2000 dataset, CVT has better performance when the unlabeled data is limited while SeqVAT gradually outperforms with more unlabeled data. As for the CoNLL 2003 dataset, SeqVAT shows consistently superior performance. This experiment shows that both approaches can provide significant benefits with a large amount of unlabeled data. In addition, SeqVAT has better utilization of unlabeled data, especially when having substantial unlabeled data.
5.5 Comparison on Semi-Supervised Approaches
ST utilizes the unlabeled data by augmenting training data with the teacher model predictions, while EM makes the model more confident on the predictions for unlabeled data. Hence, both approaches are trying to force the model to trust predictions from the teacher model. If the teacher initially makes wrong predictions, the error would propagate to the student model.
Unlike them, CVT and VAT/SeqVAT construct similar sentences which might have the same labels, and force the model to make consistent predictions on them. If the model makes incorrect prediction for the original sentence, CVT and VAT/SeqVAT can form a "discussion" to reach an agreement among the prediction of the original sentence and that of the similar sentences. If the model can make correct predictions for some similar utterances, it would have a chance to fix the error. Consequently, CVT and VAT/SeqVAT are generally expected to be more effective than ST and EM on the use of unlabeled data.
8808
(b) CoNLL 2003
Figure 3: Model performance with different amount of augmented unlabeled sentences.
The major difference between CVT and VAT is the mechanism of selecting similar sentences. CVT takes segments of the original sentence while VAT/SeqVAT generates new sentences by replacing tokens in the original sentence with their neighbors in the embedding space. Each approach has its own benefits and problems: 1) CVT can handle different tokens in the similar context, but would produce noise when the key words for meaning are not in the segments; 2) VAT generates truly similar sentences, but it might not be able to cover synonyms which have large distances in the embedding space. Hence, the effectiveness of them highly depends on the data. As in Table.2, CVT and VAT might outperform each other on different domains/datasets.
The improvements of SeqVAT over CVT and VAT can be explained by its compatibility with CRF, because CRF is a critical component for some sequence labeling tasks (including the three in this paper). The compatibility with CRF would largely affect the effectiveness of semi-supervised approaches. In other tasks where label transitions are important, we might not see significant gains from SeqVAT over VAT or CVT.
5.6 Insights from K-best Estimation
To make VAT compatible with CRF, we propose an idea to estimate the label sequence distribution using k-best estimation. This idea provides a view to optimize the label sequence level distribution
directly rather than work on the label distribution per token. This idea could be beneficial for tasks needing distribution transfer on sequence models, such as knowledge distillation, multi-source transfer learning.
In this paper, we propose a CRF compatible VAT training algorithm and demonstrate that sequence labeling tasks can greatly benefit from it. Our proposed method, SeqVAT, has strong effects to improve model robustness and accuracy on supervised sequence labeling tasks. In addition, SeqVAT is also highly effective in semi-supervised settings and outperforms traditional semi-supervised algorithms (ST and EM) as well as a state-of-the-art approach (CVT). Overall, our approach is highly effective for chunking, NER and slot filling, and can be easily extended to solve other sequence labeling problems in both supervised and semi-supervised settings.
We want to thank all the anonymous reviewers for the helpful feedback and suggestions on this work.
Alan Akbik, Duncan Blythe, and Roland Vollgraf. 2018. Contextual string embeddings for sequence labeling. In Proceedings of the 27th International Conference on Computational Linguistics, COLING 2018, Santa Fe, New Mexico, USA, August 20-26, 2018, pages 1638-1649.
Avrim Blum, Nika Haghtalab, and Ariel D. Procaccia. 2015. Variational dropout and the local reparameterization trick. In Advances in Neural Information Processing Systems 28: Annual Conference on Neural Information Processing Systems 2015, December 7-12, 2015, Montreal, Quebec, Canada, pages 2575-2583.
Ciprian Chelba, Tomas Mikolov, Mike Schuster, Qi Ge, Thorsten Brants, Phillip Koehn, and Tony Robinson. 2014. One billion word benchmark for measuring progress in statistical language modeling. In INTERSPEECH 2014, 15th Annual Conference of the International Speech Communication Association, Singapore, September 14-18, 2014, pages 2635-2639.
Kevin Clark, Minh-Thang Luong, Christopher D. Manning, and Quoc V. Le. 2018. Semi-supervised sequence modeling with cross-view training. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, Brussels, Bel
8809
gium, October 31 - November 4, 2018, pages 1914-1925.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171-4186, Minneapolis, Minnesota. Association for Computational Linguistics.
Ian J. Goodfellow, Jonathon Shlens, and Christian Szegedy. 2015. Explaining and harnessing adversarial examples. In 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings.
Yves Grandvalet and Yoshua Bengio. 2004. Semi-supervised learning by entropy minimization. In Advances in Neural Information Processing Systems 17 [Neural Information Processing Systems, NIPS 2004, December 13-18, 2004, Vancouver, British Columbia, Canada], pages 529-536.
Liang Huang and David Chiang. 2005. Better k-best parsing. In Proceedings of the Ninth International Workshop on Parsing Technology, Parsing '05, pages 53-64, Stroudsburg, PA, USA. Association for Computational Linguistics.
Diederik P. Kingma and Jimmy Ba. 2015. Adam: A method for stochastic optimization. In 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings.
John D. Lafferty, Andrew McCallum, and Fernando C. N. Pereira. 2001. Conditional random fields: Probabilistic models for segmenting and labeling sequence data. In Proceedings of the Eighteenth International Conference on Machine Learning (ICML 2001), Williams College, Williamstown, MA, USA, June 28 - July 1, 2001, pages 282-289.
Xuezhe Ma and Eduard H. Hovy. 2016. End-to-end sequence labeling via bi-directional LSTM-cnns-crf. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics, ACL 2016, August 7-12, 2016, Berlin, Germany, Volume 1: Long Papers.
David McClosky, Eugene Charniak, and Mark Johnson. 2006. Effective self-training for parsing. In Human Language Technology Conference of the North American Chapter of the Association of Computational Linguistics, Proceedings, June 4-9, 2006, New York, New York, USA.
Takeru Miyato, Andrew M. Dai, and Ian J. Goodfellow. 2017. Adversarial training methods for semi-supervised text classification. In 5th International Conference on Learning Representations,
ICLR 2017, Toulon, France, April 24-26, 2017, Conference Track Proceedings.
Takeru Miyato, Shin-ichi Maeda, Masanori Koyama, and Shin Ishii. 2019. Virtual adversarial training: A regularization method for supervised and semi-supervised learning. IEEE Trans. Pattern Anal. Mach. Intell., 41(8):1979-1993.
Avital Oliver, Augustus Odena, Colin Raffel, Ekin Dogus Cubuk, and Ian J. Goodfellow. 2018. Realistic evaluation of deep semi-supervised learning algorithms. In Advances in Neural Information Processing Systems 31: Annual Conference on Neural Information Processing Systems 2018, NeurIPS 2018, 3-8 December 2018, Montréal, Canada., pages 3239-3250.
Matthew E. Peters, Waleed Ammar, Chandra Bhagavatula, and Russell Power. 2017. Semi-supervised sequence tagging with bidirectional language models. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics, ACL 2017, Vancouver, Canada, July 30 - August 4, Volume 1: Long Papers, pages 1756-1765.
Matthew E. Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. 2018. Deep contextualized word representations. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT 2018, New Orleans, Louisiana, USA, June 1-6, 2018, Volume 1 (Long Papers), pages 2227-2237.
Devendra Singh Sachan, Manzil Zaheer, and Ruslan Salakhutdinov. 2019. Revisiting LSTM networks for semi-supervised text classification via mixed objective function. In The Thirty-Third AAAI Conference on Artificial Intelligence, AAAI 2019, The Thirty-First Innovative Applications of Artificial Intelligence Conference, IAAI 2019, The Ninth AAAI Symposium on Educational Advances in Artificial Intelligence, EAAI 2019, Honolulu, Hawaii, USA, January 27 - February 1, 2019, pages 6940-6948.
Erik F. Tjong Kim Sang and Sabine Buchholz. 2000. Introduction to the conll-2000 shared task chunking. In Fourth Conference on Computational Natural Language Learning, CoNLL 2000, and the Second Learning Language in Logic Workshop, LLL 2000, Held in cooperation with ICGI-2000, Lisbon, Portugal, September 13-14, 2000, pages 127-132.
Erik F. Tjong Kim Sang and Fien De Meulder. 2003. Introduction to the conll-2003 shared task: Language-independent named entity recognition. In Proceedings of the Seventh Conference on Natural Language Learning, CoNLL 2003, Held in cooperation with HLT-NAACL 2003, Edmonton, Canada, May 31 - June 1, 2003, pages 142-147.
Christian Szegedy, Wojciech Zaremba, Ilya Sutskever, Joan Bruna, Dumitru Erhan, Ian J. Goodfellow, and
8810
Rob Fergus. 2014. Intriguing properties of neural networks. In 2nd International Conference on Learning Representations, ICLR 2014, Banff, AB, Canada, April 14-16, 2014, Conference Track Proceedings.
Andrew J. Viterbi. 1967. Error bounds for convolutional codes and an asymptotically optimum decoding algorithm. IEEE Trans. Information Theory, 13(2):260-269.
Ziang Xie, Sida I. Wang, Jiwei Li, Daniel Lévy, Aiming Nie, Dan Jurafsky, and Andrew Y. Ng. 2017. Data noising as smoothing in neural network language models. In 5th International Conference on Learning Representations, ICLR 2017, Toulon, France, April 24-26, 2017, Conference Track Proceedings.
David Yarowsky. 1995. Unsupervised word sense disambiguation rivaling supervised methods. In 33rd Annual Meeting of the Association for Computational Linguistics, 26-30 June 1995, MIT, Cambridge, Massachusetts, USA, Proceedings, pages 189-196. Morgan Kaufmann Publishers / ACL.
Michihiro Yasunaga, Jungo Kasai, and Dragomir R. Radev. 2018. Robust multilingual part-of-speech tagging via adversarial training. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT 2018, New Orleans, Louisiana, USA, June 1-6, 2018, Volume 1 (Long Papers), pages 976-986.
8811