SlowGuess/NAACL_2022_OCR huggingface dataset:
on meta structure: Multi-task tagging and parsing in low-resource scenarios. In The Thirty-Fourth AAAI Conference on Artificial Intelligence, AAAI 2020, The Thirty-Second Innovative Applications of Artificial Intelligence Conference, IAAI 2020, The Tenth AAAI Symposium on Educational Advances in Artificial Intelligence, EAAI 2020, New York, NY, USA, February 7-12, 2020, pages 8344-8351. AAAI Press.
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, M. Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized bert pretraining approach. ArXiv, abs/1907.11692.
Ilya Loshchilov and Frank Hutter. 2019. Decoupled weight decay regularization. In 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019. OpenReview.net.
Xiaofei Ma, Peng Xu, Zhiguo Wang, Ramesh Nallapati, and Bing Xiang. 2019. Domain adaptation with BERT-based domain classification and data selection. In Proceedings of the 2nd Workshop on Deep Learning Approaches for Low-Resource NLP (DeepLo 2019), pages 76–83, Hong Kong, China. Association for Computational Linguistics.
Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. 2018. Towards deep learning models resistant to adversarial attacks. In 6th International Conference on Learning Representations, ICLR 2018, Vancouver, BC, Canada, April 30 - May 3, 2018, Conference Track Proceedings. OpenReview.net.
Julian J. McAuley, Christopher Targett, Qinfeng Shi, and Anton van den Hengel. 2015. Image-based recommendations on styles and substitutes. In Proceedings of the 38th International ACM SIGIR Conference on Research and Development in Information Retrieval, Santiago, Chile, August 9-13, 2015, pages 43-52. ACM.
David McClosky, Eugene Charniak, and Mark Johnson. 2006. Reranking and self-training for parser adaptation. In Proceedings of the 21st International Conference on Computational Linguistics and 44th Annual Meeting of the Association for Computational Linguistics, pages 337-344, Sydney, Australia. Association for Computational Linguistics.
Timothy Miller. 2019. Simplified neural unsupervised domain adaptation. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 414-419, Minneapolis, Minnesota. Association for Computational Linguistics.
Takeru Miyato, Shin-ichi Maeda, Masanori Koyama, and Shin Ishii. 2018. Virtual adversarial training:
a regularization method for supervised and semi-supervised learning. IEEE transactions on Pattern Analysis and Machine Intelligence, 41(8):1979-1993.
Myle Ott, Sergey Edunov, Alexei Baevski, Angela Fan, Sam Gross, Nathan Ng, David Grangier, and Michael Auli. 2019. *fairoseq: A fast, extensible toolkit for sequence modeling*. In *Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics* (Demonstrations), pages 48-53, Minneapolis, Minnesota. Association for Computational Linguistics.
Sinno Jialin Pan, Xiaochuan Ni, Jian-Tao Sun, Qiang Yang, and Zheng Chen. 2010. Cross-domain sentiment classification via spectral feature alignment. In Proceedings of the 19th International Conference on World Wide Web, WWW 2010, Raleigh, North Carolina, USA, April 26-30, 2010, pages 751-760. ACM.
Changhwa Park, Jonghyun Lee, Jaeyoon Yoo, Minhoe Hur, and Sungroh Yoon. 2020. Joint contrastive learning for unsupervised domain adaptation. ArXiv preprint, abs/2006.10297.
Lis Pereira, Xiaodong Liu, Hao Cheng, Hoifung Poon, Jianfeng Gao, and Ichiro Kobayashi. 2021. Targeted adversarial training for natural language understanding. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 5385-5393, Online. Association for Computational Linguistics.
Yanru Qu, Dinghan Shen, Yelong Shen, Sandra Sajeev, Weizhu Chen, and Jiawei Han. 2021. Coda: Contrast-enhanced and diversity-promoting data augmentation for natural language understanding. In 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021. OpenReview.net.
Alan Ramponi and Barbara Plank. 2020. Neural unsupervised domain adaptation in NLP—A survey. In Proceedings of the 28th International Conference on Computational Linguistics, pages 6838-6855, Barcelona, Spain (Online). International Committee on Computational Linguistics.
Kuniaki Saito, Yoshitaka Ushiku, and Tatsuya Harada. 2017. Asymmetric tri-training for unsupervised domain adaptation. In Proceedings of the 34th International Conference on Machine Learning, ICML 2017, Sydney, NSW, Australia, 6-11 August 2017, volume 70 of Proceedings of Machine Learning Research, pages 2988-2997. PMLR.
Nikunj Saunshi, Orestis Plevrakis, Sanjeev Arora, Mikhail Khodak, and Hrishikesh Khandeparkar. 2019. A theoretical analysis of contrastive unsupervised representation learning. In Proceedings of the 36th International Conference on Machine Learning, ICML 2019, 9-15 June 2019, Long Beach, California,
USA, volume 97 of Proceedings of Machine Learning Research, pages 5628-5637. PMLR.
Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016. Improving neural machine translation models with monolingual data. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 86-96, Berlin, Germany. Association for Computational Linguistics.
Darsh Shah, Tao Lei, Alessandro Moschitti, Salvatore Romeo, and Preslav Nakov. 2018. Adversarial domain adaptation for duplicate question detection. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 1056-1063, Brussels, Belgium. Association for Computational Linguistics.
Jian Shen, Yanru Qu, Weinan Zhang, and Yong Yu. 2018. Wasserstein distance guided representation learning for domain adaptation. In Proceedings of the Thirty-Second AAAI Conference on Artificial Intelligence, (AAAI-18), the 30th innovative Applications of Artificial Intelligence (IAAI-18), and the 8th AAAI Symposium on Educational Advances in Artificial Intelligence (EAAI-18), New Orleans, Louisiana, USA, February 2-7, 2018, pages 4058-4065. AAAI Press.
Rui Wang, Zuxuan Wu, Zejia Weng, Jingjing Chen, Guo-Jun Qi, and Yu-Gang Jiang. 2021. Cross-domain contrastive learning for unsupervised domain adaptation. ArXiv preprint, abs/2106.05528.
Zirui Wang, Zihang Dai, Barnabás Póczos, and Jaime G. Carbonell. 2019. Characterizing and avoiding negative transfer. In IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2019, Long Beach, CA, USA, June 16-20, 2019, pages 11293-11302. Computer Vision Foundation / IEEE.
Xing Wu, Tao Zhang, Liangjun Zang, Jizhong Han, and Songlin Hu. 2019a. Mask and infill: Applying masked language model for sentiment transfer. In Proceedings of the Twenty-Eighth International Joint Conference on Artificial Intelligence, IJCAI 2019, Macao, China, August 10-16, 2019, pages 5271-5277. ijcai.org.
Yifan Wu, Ezra Winston, Divyansh Kaushik, and Zachary C. Lipton. 2019b. Domain adaptation with asymmetrically-relaxed distribution alignment. In Proceedings of the 36th International Conference on Machine Learning, ICML 2019, 9-15 June 2019, Long Beach, California, USA, volume 97 of Proceedings of Machine Learning Research, pages 6872-6881. PMLR.
Zhuofeng Wu, Sinong Wang, Jiatao Gu, Madian Khabsa, Fei Sun, and Hao Ma. 2020. Clear: Contrastive learning for sentence representation. ArXiv preprint, abs/2012.15466.
Yuanmeng Yan, Rumei Li, Sirui Wang, Fuzheng Zhang, Wei Wu, and Weiran Xu. 2021. ConSERT: A contrastive framework for self-supervised sentence representation transfer. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 5065-5075, Online. Association for Computational Linguistics.
Zhilin Yang, Zihang Dai, Yiming Yang, Jaime G. Carbonell, Ruslan Salakhutdinov, and Quoc V. Le. 2019. Xlnet: Generalized autoregressive pretraining for language understanding. In Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019, NeurIPS 2019, December 8-14, 2019, Vancouver, BC, Canada, pages 5754-5764.
David Yarowsky. 1995. Unsupervised word sense disambiguation rivaling supervised methods. In 33rd Annual Meeting of the Association for Computational Linguistics, pages 189-196, Cambridge, Massachusetts, USA. Association for Computational Linguistics.
Hai Ye, Qingyu Tan, Ruidan He, Juntao Li, Hwee Tou Ng, and Lidong Bing. 2020. Feature adaptation of pre-trained language models across languages and domains with robust self-training. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 7386-7399, Online. Association for Computational Linguistics.
Zhi-Hua Zhou and Ming Li. 2005. Tri-training: Exploiting unlabeled data using three classifiers. IEEE Transactions on knowledge and Data Engineering, 17(11):1529-1541.
Chen Zhu, Yu Cheng, Zhe Gan, Siqi Sun, Tom Goldstein, and Jingjing Liu. 2020. Freelb: Enhanced adversarial training for natural language understanding. In 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenReview.net.
Fuzhen Zhuang, Xiaohu Cheng, Ping Luo, Sinno Jialin Pan, and Qing He. 2015. Supervised representation learning: Transfer learning with deep autoencoders. In Proceedings of the Twenty-Fourth International Joint Conference on Artificial Intelligence, IJCAI 2015, Buenos Aires, Argentina, July 25-31, 2015, pages 4119-4125. AAAI Press.
Yftah Ziser and Roi Reichart. 2017. Neural structural correspondence learning for domain adaptation. In Proceedings of the 21st Conference on Computational Natural Language Learning (CoNLL 2017), pages 400-410, Vancouver, Canada. Association for Computational Linguistics.
We obtain the Amazon review datasets from He et al. (2018). This dataset does not remove neutral labels and will not be problematic in UDA situation where the label information of the target domain is not available. In addition, reserving neutral labels also bring challenges for pre-trained language model, making it more favorable for self-supervised representation learning. Summary of this dataset is available in Table 4.
Domain	#Neg	#Neu	#Pos	Total
Book	Set 1	2000	2000	2000	6000
Set 2	513	663	4824	6000
Electronics	Set 1	2000	2000	2000	6000
Set 2	694	489	4817	6000
Beauty	Set 1	2000	2000	2000	6000
Set 2	616	675	4709	6000
Music	Set 1	2000	2000	2000	6000
Set 2	785	774	4441	6000
Each domain contains two sets, set 1 contains 6000 instances with balanced class labels, and set 2 contains instances that are randomly sampled from the larger dataset (McAuley et al., 2015), preserving authentic label distribution, examples in these two datasets do not overlap. Following (He et al., 2018), we use set 1 from the source domain as the training set for all our experiments. Since label distribution in the target domain is unpredictable and out of control in real life, so it's more reasonable to use set 2 from the target domain as the unlabeled set, lastly the model will be evaluated in set 1 from target domain. For data split, we randomly sample 1000 instances from the source labeled dataset as validation set. When running UDA experiments, the model will train on 5000 source labeled examples and 6000 target unlabeled examples, then validate on 1000 source labeled examples.
For Amazon Benchmark(Blitzer et al., 2007), it also contains four domains: Book (BK), DVD (D), Electronics (E) and Kitchen housewares (K). There are 2000 balanced labeled data for each domain, we randomly select 20000 unlabeled reviews for BK, D and E. For K, only 17856 unlabeled reviews are available, statistics of Amazon Benchmark can be find in Table.5. For data split, 1600 balanced samples are randomly sampled from the source labeled dataset, and 400 for validation.
Table 4: Amazon review dataset
Domains	Labeled	Unlabeled
Book	2000	973194
DVD	2000	122438
Electronics	2000	21009
Kitchen & housewares	2000	17856
Table 5: Amazon Benchmark statistics
B DANN
Domain Adversarial Neural Network Ganin et al. (2016) proposes Domain Adversarial Neural Network (DANN), which learns domain invariant and discriminative features simultaneously. This approach is motivated by the idea that an adaptation algorithm could learn good representations for cross-domain transfer if it cannot differentiate the domain of the input observations. The optimization objective is:
$$
\min  _ {\theta_ {f}, \theta_ {y}} \sum_ {(x, y) \sim \mathcal {D} ^ {S}} [ \mathcal {L} (f (x; \theta_ {f}, \theta_ {y}), y) + \lambda R _ {\theta_ {f}} ], \tag {11}
$$
$$
R _ {\theta_ {f}} = \max  _ {\theta_ {d}} \sum_ {(x, d) \sim \mathcal {D} ^ {S}, \mathcal {D} ^ {T}} [ - \mathcal {L} (f (x; \theta_ {f}, \theta_ {d}), d) ], \tag {12}
$$
where  $\theta_{d}$  is the parameter corresponding to the computation of the domain classification,  $d$  is the domain label,  $R_{\theta_f}$  is a regularizer weighted by  $\lambda$ . Objective (11) learns task discrimination by minimizing task classification loss and tries to make features similar across domains by maximizing the domain classification loss.
Although the domain classifier with parameters  $\theta_{d}$  could perfectly classify different domains, the balance between two terms in (11) is hard to maintain. Hence, the training process becomes unstable, requiring an elaborate adaptation rate  $\lambda$  tuning (Shen et al., 2018; Shah et al., 2018; Du et al., 2020). Furthermore, the encoder could learn trivial solutions (Karouzos et al., 2021) which produce features with flipped domain predictions.