SlowGuess/NAACL_2022_OCR huggingface dataset:
/Punta Cana, Dominican Republic, 7-11 November, 2021, pages 4381-4394. Association for Computational Linguistics.
Jian Tang, Meng Qu, and Qiaozhu Mei. 2015. PTE: Predictive text embedding through large-scale heterogeneous text networks. In Proceedings of the ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, 2015-August:1165-1174.
Aäron van den Oord, Yazhe Li, and Oriol Vinyals. 2018. Representation learning with contrastive predictive coding. CoRR, abs/1807.03748.
Laurens Van der Maaten and Geoffrey Hinton. 2008. Visualizing data using t-sne. Journal of machine learning research, 9(11).
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Advances in Neural Information Processing Systems, pages 5998-6008.
Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R. Bowman. 2019. GLUE: A multi-task benchmark and analysis platform for natural language understanding. In 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019. OpenReview.net.
Dong Wang, Ning Ding, Piji Li, and Haitao Zheng. 2021. CLINE: contrastive learning with semantic negative examples for natural language understanding. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing, ACL/IJCNLP 2021, (Volume 1: Long Papers), Virtual Event, August 1-6, 2021, pages 2332-2342. Association for Computational Linguistics.
Guoyin Wang, Chunyuan Li, Wenlin Wang, Yizhe Zhang, Dinghan Shen, Xinyuan Zhang, Ricardo Henao, and Lawrence Carin. 2018. Joint embedding of words and labels for text classification. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 2321-2331.
Lingxiao Wang, Jing Huang, Kevin Huang, Ziniu Hu, Guangtao Wang, and Ququan Gu. 2020. Improving neural language generation with spectrum control. In 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenReview.net.
Tongzhou Wang and Phillip Isola. 2020. Understanding contrastive representation learning through alignment and uniformity on the hypersphere. In Proceedings of the 37th International Conference on Machine Learning, ICML 2020, 13-18 July 2020, Virtual Event, volume 119 of Proceedings of Machine Learning Research, pages 9929-9939. PMLR.
Frank Wilcoxon. 1945. Individual comparisons by ranking methods. Biometrics bulletin, 1(6):80-83.
Lin Xiao, Xin Huang, Boli Chen, and Liping Jing. 2019. Label-specific document representation for multi-label text classification. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 466-475.
Liang Xu, Hai Hu, Xuanwei Zhang, Lu Li, Chenjie Cao, Yudong Li, Yechen Xu, Kai Sun, Dian Yu, Cong Yu, Yin Tian, Qianqian Dong, Weitang Liu, Bo Shi, Yiming Cui, Junyi Li, Jun Zeng, Rongzhao Wang, Weijian Xie, Yanting Li, Yina Patterson, Zuoyu Tian, Yiwen Zhang, He Zhou, Shaoweihua Liu, Zhe Zhao, Qipeng Zhao, Cong Yue, Xinrui Zhang, Zhengliang Yang, Kyle Richardson, and Zhenzhong Lan. 2020. CLUE: A Chinese language understanding evaluation benchmark. In Proceedings of the 28th International Conference on Computational Linguistics, Barcelona, Spain (Online). International Committee on Computational Linguistics.
Liang Xu, Xiaojing Lu, Chenyang Yuan, Xuanwei Zhang, Hu Yuan, Huilin Xu, Guoao Wei, Xiang Pan, and Hai Hu. 2021. Few clue: A chinese few-shot learning evaluation benchmark. CoRR, abs/2107.07498.
Yuanmeng Yan, Rumei Li, Sirui Wang, Fuzheng Zhang, Wei Wu, and Weiran Xu. 2021. Consent: A contrastive framework for self-supervised sentence representation transfer. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing, ACL/IJCNLP 2021, (Volume 1: Long Papers), Virtual Event, August 1-6, 2021, pages 5065-5075. Association for Computational Linguistics.
Honglun Zhang, Liqiang Xiao, Wenqing Chen, Yongkun Wang, and Yaohui Jin. 2018. Multi-task label embedding for text classification. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 4545-4553.
A.1 Accumulative Ablation Study
In this section, we supplement more ablation results by adding each proposed CL loss cumulatively. We conduct experiments on MRPC, RTE, and CoLA datasets and keep the setting consistent with Section 3.4. Table 6 demonstrates that the contribution of each component in more details.
A.2 Experimental Results of More Datasets
We supplement more experimental results on the remaining datasets of GLUE and CLUE benchmarks.
Methods	MRPC	RTE	CoLA
LICL	86.2±1.5	67.8±1.1	61.2±0.9
LCI	86.9±1.1	68.2±0.7	61.2±1.5
LCL	87.0±1.7	68.3±1.0	61.1±1.3
ICL + LCL	87.3±1.1	70.2±1.3	62.2±1.6
ICL + LER	87.1±0.6	70.5±0.8	61.2±1.1
LaCon-vanilla	87.5±0.8	71.4±0.7	62.4±1.1
Table 6: Ablation study via adding losses cumulatively.
We follow the same experimental setup with Section 3.1. Please note that SST-B is a regression task that is beyond the capacity of the proposed LaCon. The official CLUE benchmark has replaced CMNLI with OCNLI dataset, and the CSL dataset is a keyword recognition task, which is not suitable for our proposed model. Thus, we omit the experiments on above three datasets and report the performance on the remaining language understanding tasks including SST-2, MNLI, AFQMC, OCNLI and IFLYTEK.
Table 7 shows that, LaCon-vanilla consistently outperforms BERT fine-tuned with CE, and LaConfusion still beats the baselines among all datasets, which further demonstrates the superiority of our proposed method.
In this section, we conduct the theoretical analysis to prove the rationality and necessity of our proposed ICL and LCL losses. We also explain why these two losses are complementary. Finally, we analyze the computational efficiency of ICL and LCL compared to InfoNCE (van den Oord et al., 2018) and SCL (Khosla et al., 2020).
The recent researches (Li et al., 2020; Gao et al., 2019) reveal that the anisotropy problem of pretrained language models, which shows that the learnt embeddings occupy a narrow cone in the dense vector space, harming the uniformity of the models and limiting the representation capacity. The singular values of the contextual embeddings decay drastically with most of them nearly zeros (Wang and Isola, 2020). CL is proposed to eliminate the long-tail distribution problem of singular values, aiming to enhance the representation capacity (Gao et al., 2021; Yan et al., 2021; Gunel et al., 2021). From the spectrum perspective (Wang et al., 2020; Wang and Isola, 2020) that analyzes the distribution and uniformity of the learned embedding space, CL flattens singular values of the embeddings thus improves the capacity of language
Methods	SST-2	MNLI	AFQMC	OCNLI	IFLYTEK
Baseline*	91.4±0.3	73.1±0.3	74.5±0.3	68.3±0.7	61.5±0.5
CE	91.2±0.1	72.5±0.3	70.9±0.5	66.8±0.4	60.6±0.2
LaCon-vanilla	91.4±0.3	73.3±0.4	72.5±0.3	67.4±0.3	60.9±0.2
LaCon-fusion	92.5±0.2	73.9±0.3	74.8±0.5	69.1±0.6	63.5±0.7
Table 7: Performance on the remaining datasets of GLUE and CLUE. Baseline* means the best performance of our baselines. The evaluation metrics are the same as the official GLUE (Wang et al., 2019) and CLUE (Xu et al., 2020) benchmarks (all with significance value  $p < 0.05$ ).
$$
\begin{array}{l} \mathcal {L} _ {I C L} = - \frac {1}{\tau} E _ {(x, y) \sim A (y)} \left(H _ {x} L _ {y}\right) \tag {10} \\ + E _ {x \sim A (y)} [ l o g E _ {y ^ {-} \notin A (y)} (e ^ {H _ {x} L _ {y ^ {-}} / \tau}) ] \\ \end{array}
$$
$$
\begin{array}{l} E _ {x \sim A (y)} [ \log E _ {y ^ {-} \notin A (y)} \left(e ^ {H _ {x} L _ {y ^ {-}} / \tau}\right) ] \\ = \frac {1}{N} \sum_ {i = 1} ^ {i = N} \log \left(\frac {1}{C - 1} \sum_ {j \neq i} ^ {1 \leq j \leq C} e ^ {H _ {x _ {i}} L _ {y _ {j}} / \tau}\right) \tag {11} \\ \geq \frac {1}{N (C - 1) \tau} \sum_ {i = 1} ^ {i = N} \sum_ {j = 1, j \neq i} ^ {j = C} H _ {x} L _ {y ^ {-}} \\ \end{array}
$$
Therefore, we can form an asymptotic equivalent objective of the  $\mathcal{L}_{ICL}$  (Equation 3) as Equation 10.  $(x,y)\sim A(y)$  denotes instances (i.e.  $x$ ) with corresponding label (i.e.  $y$ ) and  $y^{-}$ denotes the label that is different from  $y$ . The first item keeps instances and corresponding labels similar and the second item pushes the mismatched instances and labels apart. We can further derive Equation 11 using Jensen's inequality because  $e(.)$  is convex. Therefore, minimizing the  $\mathcal{L}_{ICL}$  equals to minimization of summation of all elements in  $H L^{T}\in R^{N\times C}$ . Because both  $H$  and  $L$  are normalized,  $tr(H^TL)$  is a constant due to all diagonal elements are ones.  $sum(HLT)$  is an upper bound of the largest singular value (Merikoski, 1984) and minimization of the  $sum(HLT)$  will flatten the singular values distribution of  $HLT$ . As the  $HLT$  is a non-squared matrix, we need to optimize both the left and right singular values using  $HLT$  and  $LHT$  in order to effectively eliminate the anisotropy and promote the uniformity of pre-trained language models in classification tasks to enhance the model capacity. Thus, we also need to optimize the label-centered contrastive loss  $\mathcal{L}_{LCL}$  at the same time. From above analysis, we can see that  $\mathcal{L}_{ICL}$  and  $\mathcal{L}_{LCL}$  are complementary to each other. Similarly, we can derive that minimizing  $\mathcal{L}_{LCL}$  results in the minimization of  $sum(LHT)\in R^{C\times N}$ .
Although both ICL and LCL calculate the  $N \times C$  similarity scores for a mini-batch, they are different. The ICL is the average of the instance-level per sample loss while the LCL is the per label loss. The ICL intends to align each instance to corresponding label correctly. The LCL makes the instances of different labels far away from each other and instances of the same label more compact. They consider different aspects of instance and label representation through operating the  $N \times C$  similarity scores differently according to Equation 3 and 5.
Compared to InfoNCE, ICL improves the computational efficiency from  $O(N^2)$  to  $O(NC)$  because we only need to contrast the instance representation with corresponding label representations for per example loss, which is extremely useful for language understanding tasks (Wang et al., 2019; Xu et al., 2020) that commonly consist of 2 or 3 labels. Similarly, LCL is also more computationally efficient as it only contrasts one label representation to several instances rather than computes all pairs of instances belonging to a given label in the mini-batch. Thus it improves the complexity from  $O(N^2)$  to  $O(CN)$  compared with SCL too.