SlowGuess/NAACL_2022_OCR huggingface dataset:
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized BERT pretraining approach. CoRR, abs/1907.11692.
Yixin Liu and Pengfei Liu. 2021. SimCLS: A simple framework for contrastive learning of abstractive summarization. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 2: Short Papers), pages 1065-1072, Online. Association for Computational Linguistics.
Yu Meng, Chenyan Xiong, Payal Bajaj, Saurabh Tiwary, Paul Bennett, Jiawei Han, and Xia Song. 2021a. Coco-lm: Correcting and contrasting text sequences for language model pretraining. arXiv preprint arXiv:2102.08473.
Yuxian Meng, Wei Wu, Fei Wang, Xiaoya Li, Ping Nie, Fan Yin, Muyu Li, Qinghong Han, Xiaofei Sun, and Jiwei Li. 2019. Glyce: Glyph-vectors for chinese character representations. In Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019, NeurIPS 2019, December 8-14, 2019, Vancouver, BC, Canada, pages 2742-2753.
Zaiqiao Meng, Fangyu Liu, Ehsan Shareghi, Yixuan Su, Charlotte Collins, and Nigel Collier. 2021b. Rewire-then-probe: A contrastive recipe for probing biomedical knowledge of pre-trained language models. CoRR, abs/2110.08173.
Aaron van den Oord, Yazhe Li, and Oriol Vinyals. 2018. Representation learning with contrastive predictive coding. arXiv preprint arXiv:1807.03748.
Lin Pan, Chung-Wei Hang, Haode Qi, Abhishek Shah, Saloni Potdar, and Mo Yu. 2021. Multilingual bert post-pretraining alignment. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 210-219.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. 2021. Learning transferable visual models from natural language supervision. arXiv preprint arXiv:2103.00020.
Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019. Language models are unsupervised multitask learners.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2020. Exploring the limits of transfer learning with a unified text-to-text transformer. J. Mach. Learn. Res., 21:140:1-140:67.
Pranav Rajpurkar, Robin Jia, and Percy Liang. 2018. Know what you don't know: Unanswerable questions for squad. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics, ACL 2018, Melbourne, Australia, July 15-20, 2018, Volume 2: Short Papers, pages 784-789. Association for Computational Linguistics.
Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. 2016. Squad: 100,  $000+$  questions for machine comprehension of text. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, EMNLP 2016, Austin, Texas, USA, November 1-4, 2016, pages 2383-2392. The Association for Computational Linguistics.
Nils Reimers and Iryna Gurevych. 2019. SentenceBERT: Sentence embeddings using Siamese BERTnetworks. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 3982-3992, Hong Kong, China. Association for Computational Linguistics.
Pierre Sermanet, Corey Lynch, Yevgen Chebotar, Jasmine Hsu, Eric Jang, Stefan Schaal, and Sergey Levine. 2018. Time-contrastive networks: Self-supervised learning from video. In 2018 IEEE International Conference on Robotics and Automation, ICRA 2018, Brisbane, Australia, May 21-25, 2018, pages 1134-1141. IEEE.
Yixuan Su, Deng Cai, Yan Wang, David Vandyke, Simon Baker, Piji Li, and Nigel Collier. 2021a. Non-autoregressive text generation with pre-trained language models. In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume, EACL 2021, Online, April 19 - 23, 2021, pages 234-243. Association for Computational Linguistics.
Yixuan Su, Deng Cai, Qingyu Zhou, Zibo Lin, Simon Baker, Yunbo Cao, Shuming Shi, Nigel Collier, and Yan Wang. 2021b. Dialogue response selection with hierarchical curriculum learning. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing, ACL/IJCNLP 2021, (Volume 1: Long Papers), Virtual Event, August 1-6, 2021, pages 1740-1751. Association for Computational Linguistics.
Yixuan Su, Tian Lan, Yan Wang, Dani Yogatama, Lingpeng Kong, and Nigel Collier. 2022. A contrastive framework for neural text generation. CoRR, abs/2202.06417.
Yixuan Su, Zaiqiao Meng, Simon Baker, and Nigel Collier. 2021c. Few-shot table-to-text generation with prototype memory. In *Findings of the Association for Computational Linguistics: EMNLP* 2021, Virtual Event / Punta Cana, Dominican Republic, 16-20 November, 2021, pages 910-917. Association for Computational Linguistics.
Yixuan Su, Lei Shu, Elman Mansimov, Arshit Gupta, Deng Cai, Yi-An Lai, and Yi Zhang. 2021d. Multi-task pre-training for plug-and-play task-oriented dialogue system. CoRR, abs/2109.14739.
Yixuan Su, David Vandyke, Simon Baker, Yan Wang, and Nigel Collier. 2021e. Keep the primary, rewrite the secondary: A two-stage approach for paraphrase generation. In Findings of the Association for Computational Linguistics: ACL/IJCNLP 2021, Online Event, August 1-6, 2021, volume ACL/IJCNLP 2021 of Findings of ACL, pages 560-569. Association for Computational Linguistics.
Yixuan Su, David Vandyke, Sihui Wang, Yimai Fang, and Nigel Collier. 2021f. Plan-then-generate: Controlled data-to-text generation via planning. In Findings of the Association for Computational Linguistics: EMNLP 2021, Virtual Event / Punta Cana, Dominican Republic, 16-20 November; 2021, pages 895-909. Association for Computational Linguistics.
Yixuan Su, Yan Wang, Deng Cai, Simon Baker, Anna Korhonen, and Nigel Collier. 2021g. PROTOTYPE-TO-STYLE: dialogue generation with style-aware editing on retrieval memory. IEEE ACM Trans. Audio Speech Lang. Process., 29:2152-2161.
Ivan Vulic, Edoardo Maria Ponti, Anna Korhonen, and Goran Glavaš. 2021. LexFit: Lexical fine-tuning of pretrained language models. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 5269-5283, Online. Association for Computational Linguistics.
Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R. Bowman. 2019. GLUE: A multi-task benchmark and analysis platform for natural language understanding. In 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019. OpenReview.net.
Shufan Wang, Laure Thompson, and Mohit Iyyer. 2021. Phrase-bert: Improved phrase embeddings from bert with an application to corpus exploration. *ArXiv*, abs/2109.06304.
Xiaolong Wang and Abhinav Gupta. 2015. Unsupervised learning of visual representations using videos. In 2015 IEEE International Conference on Computer Vision, ICCV 2015, Santiago, Chile, December 7-13, 2015, pages 2794-2802. IEEE Computer Society.
Ralph Weischedel, Sameer Pradhan, Lance Ramshaw, Martha Palmer, Nianwen Xue, Mitchell Marcus, Ann Taylor, Craig Greenberg, Eduard Hovy, and Robert Belvin. 2011. Ontonotes release 4.0. ldc2011t03. In Philadelphia, Penn.: Linguistic Data Consortium.
Zhuofeng Wu, Sinong Wang, Jiatao Gu, Madian Khabsa, Fei Sun, and Hao Ma. 2020. Clear: Contrastive learning for sentence representation. arXiv preprint arXiv:2012.15466.
Hu Xu, Gargi Ghosh, Po-Yao Huang, Dmytro Okhonko, Armen Aghajanyan, Florian Metze, Luke Zettle-moyer, and Christoph Feichtenhofer. 2021. Videoclip: Contrastive pre-training for zero-shot video-text understanding. arXiv preprint arXiv:2109.14084.
Hu Xu, Bing Liu, Lei Shu, and Philip Yu. 2019. BERT post-training for review reading comprehension and aspect-based sentiment analysis. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 2324-2335, Minneapolis, Minnesota. Association for Computational Linguistics.
Yuanmeng Yan, Rumei Li, Sirui Wang, Fuzheng Zhang, Wei Wu, and Weiran Xu. 2021. ConSERT: A contrastive framework for self-supervised sentence representation transfer. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 5065-5075, Online. Association for Computational Linguistics.
Jianwei Yang, Yonatan Bisk, and Jianfeng Gao. 2021. Taco: Token-aware cascade contrastive learning for video-text alignment. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 11562-11572.
Zhilin Yang, Zihang Dai, Yiming Yang, Jaime G. Carbonell, Ruslan Salakhutdinov, and Quoc V. Le. 2019. Xlnet: Generalized autoregressive pretraining for language understanding. In Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019, NeurIPS 2019, December 8-14, 2019, Vancouver, BC, Canada, pages 5754-5764.
Yue Zhang and Jie Yang. 2018. Chinese NER using lattice LSTM. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics, ACL 2018, Melbourne, Australia, July 15-20, 2018, Volume 1: Long Papers, pages 1554-1564. Association for Computational Linguistics.
Ming Zhong, Yang Liu, Yichong Xu, Chenguang Zhu, and Michael Zeng. 2021. Dialoglm: Pre-trained model for long dialogue understanding and summarization. CoRR, abs/2109.02492.
A Statistics of Evaluated Benchmarks
A.1 English Benchmarks
Dataset	Train	Test	Evaluation Metric
CoLA	8.5k	1k	Matthews correlation
SST-2	67k	1.8k	accuracy
MRPC	3.7k	1.7K	accuracy
STS-B	7k	1.4k	Spearman correlation
QQP	364k	391k	accuracy
MNLI	393k	20k	matched/mismatched accuracy
QNLI	105k	5.4k	accuracy
RTE	2.5k	3k	accuracy
Table 3: GLUE Statistics
Dataset	Train	Dev	Evaluation Metric
1.1	87.6k	10.6k	Exact-Match/F1
2.0	130.3k	11.9k	Exact-Match/F1
A.2 Chinese Benchmarks
Table 4: SQuAD Statistics
Dataset	Train	Dev	Test	Evaluation Metric
Ontonotes	15.7k	4.3k	4.3k	F1
MSRA	37.0k	9.3k	4.4k	F1
Resume	3.8k	0.5k	0.5k	F1
Weibo	1.4k	0.3k	0.3k	F1
Pre-trained Language Models. Since the introduction of BERT (Devlin et al., 2019), the research community has witnessed remarkable progress in the field of language model pre-training on a large amount of free text. Such advancements have led to significant progresses in a wide range of natural language understanding (NLU) tasks (Liu et al., 2019; Yang et al., 2019; Clark et al., 2020; Lan et al., 2021) and text generation tasks (Radford et al., 2019; Lewis et al., 2020; Raffel et al., 2020; Su et al., 2021a,e,g,d,f,c; Zhong et al., 2021)
Contrastive Learning. Generally, contrastive learning methods distinguish observed data points from fictitious negative samples. They have been widely applied to various computer vision areas, including image (Chopra et al., 2005; Oord et al., 2018) and video (Wang and Gupta, 2015; Sermanet et al., 2018). Recently, Chen et al. (2020) proposed a simple framework for contrastive learning of visual representations (SimCLR) based on multiclass N-pair loss. Radford et al. (2021); Jia et al.
Table 5: NER Dataset Statistics
Dataset	Train	Test	Evaluation Metric
PKU	19.1k	1.9k	F1
CityU	53.0k	1.5k	F1
AS	708.9k	14.4k	F1
Table 6: CWS Dataset Statistics
(2021) applied the contrastive learning approach for language-image pretraining. Xu et al. (2021); Yang et al. (2021) proposed a contrastive pre-training approach for video-text alignment.
In the field of NLP, numerous approaches have been proposed to learn better sentence-level (Reimers and Gurevych, 2019; Wu et al., 2020; Meng et al., 2021a; Liu et al., 2021b; Gao et al., 2021; Su et al., 2021b) and lexical-level (Liu et al., 2021a; Vulic et al., 2021; Liu et al., 2021c; Wang et al., 2021) representations using contrastive learning. Different from our work, none of these studies specifically investigates how to utilize contrastive learning for improving general-purpose token-level representations. Beyond representation learning, contrastive learning has also been applied to other NLP applications such as NER (Das et al., 2021) and summarisation (Liu and Liu, 2021), knowledge probing for pre-trained language models (Meng et al., 2021b), and open-ended text generation (Su et al., 2022).
Continual Pre-training. Many researchers (Xu et al., 2019; Gururangan et al., 2020; Pan et al., 2021) have investigated how to continually pretrain the model to alleviate the task- and domain-discrepancy between the pre-trained models and the specific target task. In contrast, our proposed approach studies how to apply continual pretraining to directly improve the quality of model representations which is transferable and beneficial to a wide range of benchmark tasks.
C More Self-similarity Visualizations
In Figure 4, 5, and 6, we provide three more comparisons between the self-similarity matrix produced by TaCL and BERT (the example sentences are randomly sampled from Wikipedia). From the figures, we can draw the same conclusion as in section §3.2.2, that the token representations of BERT follow an anisotropic distribution and are less discriminative. On the other hand, the token representations of TaCL better follow an isotropic distribution, therefore different tokens become more
6All results are generated by models with base size.
(b) Self-similarity Visualization of TaCL
Figure 4: Example 2: self-similarity matrix visualization of (a) BERT and (b) TaCL. (best viewed in color)
distinguishable with respect to each other.
(b) Self-similarity Visualization of TaCL
Figure 5: Example 3: self-similarity matrix visualization of (a) BERT and (b) TaCL. (best viewed in color)
(b) Self-similarity Visualization of TaCL
Figure 6: Example 4: self-similarity matrix visualization of (a) BERT and (b) TaCL. (best viewed in color)