Venue: NAACL 2022

# Label Anchored Contrastive Learning for Language Understanding

Zhenyu Zhang, Yuming Zhao, Meng Chen, Xiaodong He

JD AI, Beijing, China {zhangzhenyu47,zahoyuming3,chenmeng20,xiaodong.he}@jd.com

###### Abstract

Contrastive learning (CL) has achieved astonishing progress in computer vision, speech, and natural language processing fields recently with self-supervised learning. However, CL approach to the supervised setting is not fully explored, especially for the natural language understanding classification task. Intuitively, the class label itself has the intrinsic ability to perform hard positive/negative mining, which is crucial for CL. Motivated by this, we propose a novel label anchored contrastive learning approach (denoted as LaCon) for language understanding. Specifically, three contrastive objectives are devised, including a multi-head instance-centered contrastive loss (ICL), a label-centered contrastive loss (LLC), and a label embedding regularizer (LER). Our approach does not require any specialized network architecture or any extra data augmentation, thus it can be easily plugged into existing powerful pre-trained language models. Compared to the state-of-the-art baselines, LaCon obtains up to 4.1% improvement on the popular datasets of GLUE and CLUE benchmarks. Besides, LaCon also demonstrates significant advantages under the few-shot and data imbalance settings, which obtains up to 9.4% improvement on the FewGLUE and FewCLUE benchmarking tasks.

+
Footnote †: July 10-15, 2022 ©2022 Association for Computational Linguistics

## 1 Introduction

In recent years, contrastive learning (CL) has been widely applied to self-supervised representation learning and led to major advances across computer vision (CV) [8, 6], speech [22, 13], and natural language processing (NLP) [17, 18, 19]. The basic idea behind these works is to pull together an anchor and a "positive" sample in the embedding space, and to push apart the anchor from many "negative" samples. Since no labels are available, a positive pair often consists of data augmentations of the sample (a.k.a "views"), and negative pairs are formed by the anchor and randomly chosen samples from the mini-batch. In visual representations, an effective solution to generate data augmentations is to take two random transformations of the same image (e.g., cropping, flipping, distortion and rotation) [6, 13, 14]. For natural language, similar approaches are adopted such as word deletion, reordering, substitution, and back-translation etc. [17, 18] However, data augmentation in NLP is inherently difficult because of its discrete nature. Therefore, some previous works [18, 19] also use dropout technique [20] to obtain sentence augmentations.

Unlike self-supervised setting, some researchers propose supervised contrastive learning (SCL) [13, 14, 15] which can construct positive pairs by leveraging label information. Examples from the same class are pulled closer than the examples from different classes, leveraging the semantics of labels to construct negatives and positives rather than shallow lexical information via data augmentation. Despite the aforementioned advantages brought by SCL, we argue that CL under supervised learning is not fully explored because the label information can be better utilized. On the one hand, labels are usually not merely categorical indices in the label vocabulary, but also contain specific semantic meanings, especially in the language understanding tasks. Thus labels can be used as positive/negative samples or anchors when calculating contrastive loss. On the other hand, label embedding enjoys a built-in ability to leverage alternative sources of information related to labels, such as class hierarchies or textual descriptions. Once we obtain representative label embeddings, they can be utilized to enhance the image/text representations, and finally facilitate the classification task. Previous la

bel embedding based classification models (Wang et al., 2018; Xiao et al., 2019) have demonstrated the effectiveness of leveraging label information.

Motivated by above analysis, we propose a novel label anchored supervised contrastive learning approach (denoted as LaCon), which combines the advantages of both contrastive learning and label embedding techniques. Specifically, we have the following three novel designs: 1) Instance-centered contrastive loss (ICL), which uses the InfoNCE (van den Oord et al., 2018) to encourage each text representation and its corresponding label representation to be closer while pushing far away mismatched instance-label pairs. We further apply a multi-head mechanism to catch different aspects of text semantics. 2) Label-centered contrastive loss (LCL), which takes label as anchor, and encourages the label representation to be more similar to the corresponding instances belonging to the same class in a mini-batch than the instances with different labels. 3) Label embedding regularizer (LER), which keeps the inter-label similarity as low as possible thus the feature space of each class is more dispersed to prevent representation degeneration. By combining above three losses, LaCon can learn good semantic representations within the same space for both input instances and labels. It's also well aligned with the two key properties related to CL: alignment and uniformity (Wang and Isola, 2020), where alignment favors encoders that assign similar features to similar samples. Uniformity prefers a feature distribution that preserves maximal information, i.e., the uniform distribution on the unit hypersphere.

To validate the effectiveness of LaCon, we perform extensive experiments on eight language understanding tasks. We take the popular pre-trained language model BERT-base (Devlin et al., 2019) as text encoder without loss of generality. For simplicity, we predict the classification label by matching the instance representation with label embeddings directly. Since our approach does not require any specialized network architecture or any extra data augmentation, LaCon can be easily plugged into other pre-trained language models. Additionally, we also explore the capability of LaCon under more difficult task settings, including few-shot learning and data imbalance situations.

To summarize, our contributions are as follows:

* We propose a novel label anchored contrastive learning approach for language understanding, which is equipped with a multi-head instance-centered contrastive loss, a label-centered contrastive loss, and a label embedding regularizer. All three contrastive objectives help the model learn the joint semantic representations for both input instances and labels.
* We conduct extensive experiments on eight public language understanding tasks from GLUE (Wang et al., 2019) and CLUE (Xu et al., 2020) benchmarks, and experimental results show the competitiveness of LaCon. Additionally, we also experiment on more difficult settings including few-shot learning and data imbalance situations. LaCon experimentally obtains up to 9.4% improvement over BERT-base on FewGLUE (Schick and Schutze, 2021) and FewCLUE (Xu et al., 2021) benchmark tasks.
* We analyze the contribution of each ingredient of LaCon, and also visualize the learned instance and label representations, showing the necessity of each loss component and the advantage of LaCon on representation learning over BERT fine-tuned with cross entropy.

## 2 Model

In this section, we introduce the details of LaCon. We focus on the language understanding classification tasks. For a multi-class classification problem with \(C\) classes, we work with a batch of training examples \(\{x_{i},y_{i}\}\), where \(1\leq i\leq N\) and \(1\leq y_{i}\leq C\). Our target is to learn discriminative representations for both instances and class labels. As Figure 1 shows, we propose three supervised CL based objectives, including the instance-centered contrastive loss, the label-centered contrastive loss, and the label embedding regularizer loss.

### The Input Encoder

The input of LaCon contains two parts consisting of the text and all the labels for the task. Since SOTA language understanding classification models follow the "pre-training then fine-tuning" two-stage paradigm, here we take the prevalent pre-trained language model (PtLM) as input encoder. In this paper, we select BERT-base (Devlin et al., 2019) as the backbone for PLM (denoted as \(f\)) without loss of generality. Given a text \(\mathbf{x}=\{w_{1},w_{2},...,w_{M}\}\) containing \(M\) tokens, the output of PtLM (i.e. BERT) is \(\mathbf{E}=f_{PLM}([CLS],w_{1},...,w_{M})\) where 

the \([CLS]\) token is the inserted sentence representation token and \({\bf E}\in R^{(M+1)\times d}\). \(d\) is the dimension of the model. We use the first token output \(E_{[CLS]}\in R^{d}\) to represent the whole input text. We then apply a projection head (denoted as \(g\)) that is a 3-layer MLP with ReLU activation function for each hidden layer to the \(E_{[CLS]}\), where the dimension of the \(g\) is also \(d\). For a mini-batch \(X\) with \(N\) training samples, the text representations can be obtained as Equation 1.

\[H=g\circ f_{PtLM}(X),~{}\text{where} \tag{1}\] \[X=x_{1},x_{2},...,x_{N}~{}and~{}H\in R^{N\times d}\]

Along with each mini-batch, LaCon also maps the \(C\) classes into \(C\) label embeddings. For simplicity, we look up in a learnable weight matrix \(W_{emb}\) and map the \(k^{th}\) label into the \(k^{th}\) row of \(W_{emb}\), where the \(W_{emb}\) is randomly initialized. The dimension \(d\) is the same as PtLM. We then normalize the vectors for both \(L\) and \(H\) using the \(l_{2}\) norm.

\[L=lookup([1,2,...,C],W_{emb}) \tag{2}\] \[\text{where}~{}L\in R^{C\times d}~{}and~{}W_{emb}\in R^{C\times d}\]

### Instance-centered Contrastive Loss

Given a mini-batch of input text and corresponding label \((x_{i},y_{i})\), as shown in Figure 1 (a), the instance-centered contrastive loss (ICL) takes each instance \(x_{i}\) as anchor, and mines positive and negative samples from class labels. ICL aims to encourage each text representation and corresponding label representation to be closer while pushing far away mismatched instance-label pairs. As shown in Equation 3, we modify the InfoNCE van den Oord et al. (2018) to calculate the loss. Here we leverage the cosine similarity function as the distance metric \(sim\). \(\tau\) is the temperature hyper-parameter, which can be tuned to improve the performance. Lower temperature increases the influence of examples that are harder to separate, effectively creating harder negatives. Similar to the self-supervised CL, ICL also takes only one positive and many negatives. Differently, the positive is not generated from data augmentation, and the negatives are not randomly sampled from the same mini-batch. By treating the class labels as data samples, ICL can mine better positive and negatives with the supervision signal. By minimizing the ICL, the instance representation is aligned to its label representation in the same semantic space, which encourages the model to learn a more representative embedding for each class label.

\[\mathcal{L}_{ICL}=-\frac{1}{N}\sum_{x_{i},y_{i}}log\frac{exp(sin( H_{x_{i}},L_{p_{i}})/\tau)}{\sum_{1\leq p\leq C}exp(sim(h_{x_{i}},L_{p})/\tau)} \tag{3}\]

Inspired by the image-augmented views proposed in CV He et al. (2019); Chen et al. (2020b, c); Grill et al. (2020), we also leverage the multi-head mechanism proposed in Transformer Vaswani et al. (2017) to compute the ICL for each head representation with smaller representation dimension. Each head can be regarded as a clipped local view of the instance or label representation. Suppose we have \(m\) heads for both instance representation and label representation, then for the \(k_{th}\) head, the corresponding representations for training sample (\(x_{i}\), \(y_{i}\)) are \(h_{x_{i}}^{k}\) and \(l_{y_{i}}^{k}\), and the dimension of each vector becomes \(d^{\prime}=d/m\). Then, we apply the contrastive loss for each head by following Equation 4. Compared with InfoNCE, \(\mathcal{L}_{ICL}\) and \(\mathcal{L}_{ICL}^{{}^{\prime}}\) do not suffer from small batch size issue He et al. (2019); Chen et al. (2020c) because we only need to contrast the instance representation with corresponding label representation for per example loss.

\[\mathcal{L}_{ICL}^{{}^{\prime}}=-\frac{1}{N}\sum_{k=1}^{m}\sum_{ x_{i},y_{i}}log\frac{exp(sim(h_{x_{i}}^{k},l_{y_{i}}^{k})/\tau)}{\sum_{1\leq p \leq C}exp(sim(h_{x_{i}}^{k},l_{y}^{k})/\tau)} \tag{4}\]

Figure 1: Overview of LaCon. The full line is the similarity between a instance and corresponding label, and the dash line is the similarity between the mismatched instance and label. The lines with the same color denote the per-instance or per-label loss.

 



### Label-centered Contrastive Loss

As shown in Figure 1 (b), we can take the class label in a mini-batch as anchor, and mine positive/negative samples from corresponding instances. Suppose there are \(|P|\) classes in the batch, where \(P=\{p|1\leq p\leq C\wedge|A(p)|>0\}\). We define that \(A(p)\) denotes the set of indices of all positive instances whose label is \(p\), i.e. \(A(p)=\{x_{i}|y_{i}=p\}\). And \(B(p)\) represents the set of negative instances whose label is not \(p\), i.e. \(B(p)=\{x_{j}|y_{j}\neq p\}\). Then we can calculate the label-centered contrastive loss (LCL) as Equation 5, which promotes the instances of a specific label to be more similar than the others for each label. Similar to the previous SCL Khosla et al. (2020); Gunel et al. (2021), LCL also contains many positives per anchor and many negatives. Different from SCL which sums up all the softmax scores among all pairs of instances of the same class in a batch, LCL is based on comparing a specific label representation with corresponding instances (i.e. \(A(p)\)). LCL is more stable as the label representation serves as the anchor which can be stably updated.

\[\mathcal{L}_{LCL}=-\frac{1}{|P|}\sum_{p\in P}\sum_{s\in A(p)}log\frac{exp(sim( L_{p},H_{a})/\tau)}{\sum_{d\in B(p)}exp(sim(L_{p},H_{l})/\tau)} \tag{5}\]

ICL and LCL are complementary to each other and more computationally efficient than previous SCL. We conduct the detailed theoretical analysis in Appendix A.3 due to space limitation.

### Label Embedding Regularizer

Recent researches Wang and Isola (2020) demonstrate that it is common and useful to add a regularization term during training to eliminate the anisotropy problem. Inspired by this, We devise a label embedding regularizer as shown in Equation 6 to promote the uniformity of our model and prevent model degeneration. As illustrated in Figure 1 (c), the label embedding regularizer (LER) encourages the label representations to be dispersed in the unit hypersphere uniformly. The LER loss is the exponential mean of the cosine similarity for all pairs of label representations. As \(-1\leq sim(L_{i},L_{j})\leq 1\), it is quite sensitive to the loss change as the gradient is larger than 1 for \(exp(x)\) w.r.t \(x\geq 0\). Thus, we add 1.0 to the cosine similarity so that the value of LER varies from 0 to \(e^{2}\) - 1.

\[\mathcal{L}_{LER}=avg(\sum_{i\neq j}(exp(1.0+sim(L_{i},L_{j}))-1.0) \tag{6}\]

Finally, the overall loss function of LaCon is summarized as follows:

\[\mathcal{L}=\mathcal{L}^{{}^{\prime}}_{ICL}+\mathcal{L}_{LCL}+\lambda* \mathcal{L}_{LER} \tag{7}\]

where \(\lambda\) is a hyper-parameter to balance the influence of our regularization term.

### Matching based Class Prediction

Since LaCon is capable of learning instance and label representations jointly, we can predict the class by matching the instance representation to all label representations directly during inference, just as shown in Equation 8. We denote this simple and direct approach as **LaCon-vanilla**. \(H_{x}\) is the instance representation and \(L_{j}\) is the label representation of Class \(j\). \(sim\) denotes the cosine similarity and \(1\leq j\leq C\) denotes the corresponding label. The advantage of LaCon-vanilla is that it does not require any complicated network architecture and can be easily plugged into the mainstream PtLMs. As a result, our inference-time model contains exactly the same number of parameters as the model using the same encoder but trained with cross entropy loss.

\[pred=\arg_{max}(j)\{score_{j}|sim(H_{x},L_{j})\} \tag{8}\]

### LaCon with Label Fusion

Previous researches Akata et al. (2016); Wang et al. (2018); Xiao et al. (2019); Pappas and Henderson (2019); Miyazaki et al. (2020) have proved that incorporating the label semantics into the sentence representation can improve the model performance because the label information can highlight the alignment of input tokens and label information via carefully designed fusion mechanism. Inspired by LEAM Wang et al. (2018), here we design a fusion block to enhance the instance representations by utilizing the learnt discriminative label embeddings. We firstly calculate the cosine similarity interaction matrix \(G\) between words and labels, and then apply a convolution then max-pooling layer (\(conv_{max}\)) to measure the attention score (\(\beta_{i}\)) for each word attending the instance representation. The fusion process is illustrated as Equation 9. Then the fused vector \(z\) is fed into the projection head \(g\) to get the enhanced instance representation.

\[m=conv_{max}(G),\text{ where }G_{ij}=\frac{<L_{i},E_{j}>}{||L_{i}|| \cdot||E_{j}||} \tag{9}\] \[z=\sum_{i}\beta_{i}E_{i},\text{ where }\beta=softmax(m)\]

To distinguish with the vanilla model above, we name this approach as **LaCon-fusion**. Please note that the fusion block is just applied between the text encoder and projection head, so the class prediction keeps the same as the LaCon-vanilla. Since the fusion block is not the main focus of this paper, we leave exploring more advanced fusion networks to future work.

## 3 Experiments

### Experimental Setup

**Datasets.** We experiment on 8 public datasets listed in Table 1. which are from GLUE Wang et al. (2019), CLUE Xu et al. (2020), DBpedia, and Yelp Dataset Challenge 2015. They cover five representative tasks including sentiment analysis (senti), genre classification (genre), paragraph identification (PI), natural language inference (NLI) and linguistic acceptability (LA). To improve the comparability and experiment confidence of the models, we follow the experimental setup in Chen et al. (2020) and use part of the training sets via sampling and the full original test sets for evaluation. We randomly sample without replacement at most 5K (binary-class) / 1K (multi-class) training instances per class from the whole datasets except for MRPC, RTE, and CoLA. We use the wilcoxon rank test Wilcoxon (1945) to check the statistic significance. The results of 10 runs are reported for each dataset in the format as "avg\(\pm std.dev\)".

**Training & Evaluation.** During training, we run experiments for MRPC, RTE and CoLA with 10 random seeds on the whole training datasets and run the sampling strategy with 10 repeats for the remaining datasets. The average evaluation metrics are reported to avoid the noise and unstable randomness of a single run. We use the AdamW optimizer with initial learning rate as {1e-5, 2e-5, 3e-5} with linear learning scheduler, 6% of warm-up steps of total optimization steps, and batch size as {8,16,32,64,96}, where the hyper-parameters are tuned for different datasets. For evaluation, we leverage accuracy (ACC), Macro-F1 score and Matthew's corr (M' corr) metrics to evaluate the performance. We run 10 epochs for all the datasets1 and then evaluate the models on dev set. Our implementation is based on Huggingface Transformers2.

Footnote 1: We split 5% of training set as validation for early stop.

Footnote 2: [https://github.com/huggingface/transformers](https://github.com/huggingface/transformers)

### Baselines

Since LaCon is based on CL and label embedding technique, we compare with several SOTA models in language understanding including BERT-base fine-tuned with cross-entropy (CE) loss, label embedding based models, and self-supervised CL and supervised CL models.

* **CE**: we directly follow the instructions of original paper Devlin et al. (2019) to finetune BERT for both English and Chinese langauge understanding tasks.
* **LEAM**Wang et al. (2018) apply cosine similarity to get matching scores between words and labels and use CNN on the matching matrix to get the label-aware attention weighted text representation for classification.
* **LSAN**Xiao et al. (2019) propose a label specific attention network that leverages label-attention and self-attention mechanism with an adaptive attention fusion strategy for multi-label classification. We use _softmax_ instead of _sigmoid_ for the model output due to our multi-class classification setting.
* **CE+CL**Yan et al. (2021) propose to learn sentence representations by joint fine-tuning PtLM with InfoNCE and cross-entropy based on feature augmentation. Here we leverage the framework of ConSERT Yan et al. (2021) and the feature augmentation in SimCSE Gao et al. (2021) to finetune and evaluate the PtLM on classification datasets.
* **CE+SCL**Gunel et al. (2021) propose to boost sentence representation learning by fine-tuning PtLM with both supervised contrastive learning loss Khosla et al. (2020) and cross entropy loss. We follow the instructions in Gunel et al. (2021) to set hyper-parameters.

\begin{table}
\begin{tabular}{c c c c c c} \hline
**Datasets** & **type** & **class** & **metric** & **train** & **dev** \\ \hline DBPedia & genre & 14 & ACC & 14K & 70K \\ Traus & genre & 15 & F1 & 14.2K & 10K \\ OQP & PI & 2 & ACC & 10K & 40.4K \\ MRPC & PI & 2 & ACC & 4.07K & 1.7K \\ QNLI & NLI & 2 & ACC & 10K & 5.4K \\ RTE & NLI & 2 & ACC & 2.5K & 278 \\ CoLA & LA & 2 & M’corr & 8.5K & 1K \\ YelpRev & senti & 2 & ACC & 10K & 10K \\ \hline \end{tabular}
\end{table}
Table 1: The statistics of datasets that are from GLUE Wang et al. (2019) and CLUE Xu et al. (2020) 



**LEAM** and **LSAN** are label embedding based methods while **CE+CL** and **CE+SCL** are contrastive learning based methods. To compare all models fairly, we use BERT-base encoder for all the baselines and our proposed model.

### Main Results

We report the experimental results of eight language understanding tasks in Table 2. It's observed that, LaCon-vanilla outperforms all the baselines in 7 datasets except MRPC, and LaCon-fusion achieves the best performance across all datasets. Specifically, 1) LaCon-vanilla outperforms BERT fine-tuned with CE by 4.1%, 2.3%, 1.9%, and 1.5% on RTE, Tnews, MRPC, and CoLA respectively, which indicates our proposed novel CL approach can facilitate the representation learning; 2) Compared with previous supervised contrastive learning method (CE+SCL), LaCon-fusion can still obtain very exciting improvements of 3.1%, 2.1%, 1.7%, 1.2% points on RTE, Tnews, YelpRev, QQP, which demonstrates the label fusion block can enhance the instance representations effectively; 3) Compared to previous label embedding methods (LEAM and LSAN) which are also equipped with the label fusion block, LaCon-fusion outperforms them with a large margin, which proves that LaCon can learn more discriminative joint representations for both labels and instances.

### Ablation Study

In this section, we conduct three groups of ablation studies to investigate the contribution of each component in LaCon. We only conduct experiments on MRPC, RTE, and CoLA datasets due to space limitation. The experimental results are shown in Table 3. First, we replace the multi-head ICL with single head version (LaCon w/ \({\mathcal{L}_{ICL}}\)). Table 3 shows the performance drops on all three datasets. We conjecture that the multi-head version can learn different parts of the local features of the representation, which can catch the text semantics in more fine-grained granularity. Second, we remove each of our proposed CL loss separately, and the results in the second part of Table 3 demonstrate that ICL plays a more important role while LCL and NER are complementary to further improve the performance. We also try to add each CL loss in an accumulative way, please refer to Appendix A.1 for more details. Finally, we try to remove the projection head \(g\) from LaCon and the performance degrades significantly, which indicates \(g\) is critical in CL. Previous researches Chen et al. (2020c) also find the projector head can eliminate the non-task relevant features of the encoder in CL and benefit the downstream tasks. Meanwhile, Table 3 shows that it is basically useless by adding \(g\) to BERT directly (BERT w/ \(g\)), indicating that the projector head needs to be used with CL.

## 4 Discussion

In this section, we conduct further experiments under more challenging few-shot and data imbalance settings. We also discuss the hyper-parameter tuning and the impact of class number on LaCon.

### LaCon for Few-shot Learning

Few-shot learning is critical for applications of language understanding models because the high-quality human annotated datasets are usually costly and limited. Previous researches Liang et al. (2021); Gunel et al. (2021); Aghajanyan et al. (2021) find that fine-tuning PtLM with cross entropy loss

\begin{table}
\begin{tabular}{c|l l l l l l l l} \hline
**Methods** & **YelpRev** & **DBPedia** & **Tnews** & **QNLI** & **RTE** & **QQP** & **MRPC** & **CoLA** \\ \hline CE & 82.0\(\pm\)0.5 & 98.7\(\pm\)0.3 & 54.5\(\pm\)0.3 & 87.1\(\pm\)0.2 & 67.3\(\pm\)1.9 & 82.2\(\pm\)0.5 & 85.6\(\pm\)1.6 & 60.9\(\pm\)0.8 \\ LEAM & 82.1\(\pm\)0.6 & 98.7\(\pm\)0.5 & 54.1\(\pm\)0.3 & 87.2\(\pm\)0.7 & 67.3\(\pm\)1.3 & 81.9\(\pm\)0.5 & 85.6\(\pm\)1.3 & 60.9\(\pm\)1.0 \\ LSAN & 82.2\(\pm\)0.6 & 98.7\(\pm\)0.7 & 54.9\(\pm\)0.8 & 87.1\(\pm\)0.3 & 69.7\(\pm\)1.0 & 81.2\(\pm\)0.5 & 86.1\(\pm\)0.7 & 61.6\(\pm\)0.9 \\ CE+CL & 82.2\(\pm\)0.6 & 98.5\(\pm\)0.5 & 53.9\(\pm\)0.5 & 87.3\(\pm\)0.3 & 67.8\(\pm\)1.5 & 82.4\(\pm\)0.3 & 83.1\(\pm\)0.7 & 61.1\(\pm\)0.7 \\ CE+SCL & 81.4\(\pm\)0.8 & 98.5\(\pm\)0.6 & 54.6\(\pm\)0.2 & 87.7\(\pm\)0.1 & 69.1\(\pm\)2.2 & 82.5\(\pm\)0.6 & 88.1\(\pm\)0.9 & 62.3\(\pm\)0.6 \\ \hline LaCon-vanilla & 82.3\(\pm\)0.5 & 98.9\(\pm\)0.5 & **56.8\(\pm\)0.6** & 88.1\(\pm\)0.2 & 71.4\(\pm\)0.7 & 82.8\(\pm\)0.5 & 87.5\(\pm\)0.9 & 62.4\(\pm\)1.0 \\ LaCon-fusion & **83.1\(\pm\)0.8** & **99.5\(\pm\)0.2** & 56.7\(\pm\)0.3 & **88.4\(\pm\)0.3** & **72.2\(\pm\)0.9** & **83.7\(\pm\)0.5** & **88.6\(\pm\)0.7** & **62.8\(\pm\)0.5** \\ \hline \end{tabular}
\end{table}
Table 2: The experimental results for the Language Understanding Tasks. Best scores for each dataset are highlight in **bold** (all with significance value \(p<0.05\)).

\begin{table}
\begin{tabular}{c c c c} \hline
**Methods** & **MRPC** & **RTE** & **CoLA** \\ \hline LaCon-vanilla & **87.5\(\pm\)0.8** & **71.4\(\pm\)0.7** & **62.4\(\pm\)1.1** \\ LaCon w/ \({\mathcal{L}_{ICL}}\) & 87.0\(\pm\)1.2 & 69.2\(\pm\)1.4 & 61.5\(\pm\)0.9 \\ \hline \(-{\mathcal{L}_{ICL}}\) & 86.6\(\pm\)1.3 & 68.1\(\pm\)0.9 & 61.3\(\pm\)0.7 \\ \(-{\mathcal{L}_{ICL}}\) & 87.1\(\pm\)0.6 & 70.5\(\pm\)0.8 & 61.2\(\pm\)1.1 \\ \(-{\mathcal{L}_{LRER}}\) & 87.3\(\pm\)1.1 & 70.2\(\pm\)1.3 & 62.2\(\pm\)1.6 \\ \hline \(-g\) & 86.8\(\pm\)0.6 & 69.6\(\pm\)0.6 & 62.2\(\pm\)0.9 \\ BERT w/ \(g\) & 84.9\(\pm\)1.7 & 66.5\(\pm\)2.1 & 61.0\(\pm\)1.2 \\ \hline \end{tabular}
\end{table}
Table 3: Ablation study. Best scores for each dataset are highlight in **bold** (all with significance test \(p<0.05\)).



in NLP tends to be unstable across different runs especially when supervised data is limited. This limitation can result in model degeneration and model shift. Besides, some researches (Muller et al., 2019) also demonstrate that the cross-entropy optimization goal is not reachable due to the bounding of the gradient, which can also easily result in overfitting. Since LaCon is equipped by CL, it's interesting to validate if LaCon can overcome the shortcomings of CE under few-shot learning settings.

We conduct further experiments with vanilla LaCon on 5 public English datasets from FewGLUE (Schick and Schutze, 2021) and 3 public Chinese datasets (Tnews, EPRSTMT and BUSTM) from FewCLUE (Xu et al., 2021). We build all the few-shot learning datasets by sampling **20 samples** for each class to form training set. We also held out the same amount of samples for validation set but keep the whole test set unchanged. We train the model for 20 epochs and select the best model based on validation set. Table 4 shows that LaCon significantly outperforms the BERT-base fine-tuned with CE loss with a huge margin. Specifically, we observe 9.4%, 7%, and 6.2% absolute improvement on BUSTM, QQP, and EPRSTMT.

Additionally, we also conduct more strict experiments by changing the number of samples per class from {10, 20, 50, 100}. Figure 2 demonstrates that the smaller the sample size per class is, the larger gain the model obtains. All above results indicate that the similarity-based CL losses in LaCon are able to hone in on the important dimensions of the multidimensional hidden representations hence lead to better and more stable few-shot learning results when fine-tuning PtLM.

### LaCon for Data Imbalance Setting

The real-world datasets are usually imbalanced for different classes (Cao et al., 2019; Bao et al., 2020), where several dominant classes contain most of the samples while the rest minority classes only hold a handful of samples. In this section, we conduct experiments to validate the capacity of LaCon under data imbalance setting. We follow the previous research (Cao et al., 2019) to construct imbalanced classification training datasets with different imbalance degree (\(\rho=|class_{max}|/|class_{min}|\), where \(|class_{max}|\) / \(|class_{min}|\) denotes number of samples in maximum / minimum class). For space limitation, we conduct experiments with vanilla LaCon on QNLI and CoLA. The minority class contains 32 samples and the majority class contains \(32\times\rho\) in our experiments. As shown in Figure 3, we vary the imbalance degree (\(\rho\)) from {1, 3, 5, 10, 20} and observe that LaCon outperforms BERT with CE consistently, demonstrating that LaCon also has advantage on the data imbalance setting.

We argue that LaCon may alleviate data imbalance issue on two aspects: 1) For the infrequent classes, treating labels as anchor or positive/negative may mitigate the data insufficient issue to some extent. 2) Label representations are shared across the whole dataset during training, which may transfer the knowledge from frequent classes to infrequent classes. To validate above conjecture, we present the performance on the test sets for majority and minorti

Figure 3: LaCon with Different Imbalance Degree (\(\rho\)).

\begin{table}
\begin{tabular}{c c c c c} \hline
**Model** & **YelpRev** & **Tnews** & **EPRSTMT** & **BUSTM** \\ \hline CE & 59.0 & 52.5 & 84.4 & 65.6 \\ LaCon & **65.0** & **55.8** & **90.6** & **75.0** \\ \hline
**Model** & **QNLI** & **RTE** & **MRPC** & **QQP** \\ \hline CE & 73.0 & 54.0 & 65.8 & 64.0 \\ LaCon & **76.0** & **60.0** & **69.0** & **71.0** \\ \hline \end{tabular}
\end{table}
Table 4: Performance under few-shot learning settings.

Figure 2: Few-shot learning with different number of training samples.

 

Table 5 shows that LaCon outperforms the baseline on both majority and minority classes and the gain on minority class is much larger.

### Visualization

To demonstrate the effectiveness of LaCon on representation learning, we visualize the learned instance representations of LaCon and CE on the MRPC and CoLA dataset. In Figure 4, we use t-SNE Van der Maaten and Hinton (2008) to visualize both the high dimensional representations of the instances and labels on a 2D map. Different classes are depicted by different colors. As shown in Figure 4 (a), the instances of class A and class B are sparsely located and overlapped in a large area, making it hard to find a hyper-plane to separate them. However, in Figure 4 (b), the instances gather into two compact clusters and the instances stay close to the corresponding class. For CoLA, Figure 4 (c) and (d) show the similar trends. It indicates that LaCon can learn more discriminative instance representations than CE. Besides, in Figure 4 (b), the instances are near the corresponding label anchor, proving that LaCon can also learn a representative label embedding for each class.

### Hyper-parameter Tuning

In this section, we take the RTE dataset as an example for illustrating the hyper-parameter tuning process. The similar hyper-parameter tuning strategy is applied for other datasets. The tuning scripts will be released in our source code. Figure 5 shows the influence of different hyper-parameters.

For each experiment, we conduct a grid-based hyper-parameter sweep for \(\tau\) between 0.05 and 0.5 with step 0.05, \(\lambda\) between 0.1 and 1.0 with step 0.1, and select the best hyper-parameter for the given dataset. The \(\tau\) is the most influential hyper-parameter that needs to be tuned carefully with minimum step 0.05. Larger \(\tau\) results in lower accuracy in LaCon and the recommended value is around 0.1 and 0.2. Figure 5 (b) illustrates that the optimal number of heads in Equation 4 is 6 and both the most and fewest heads result in low accuracy while heads with middle sizes get relatively better accuracy scores. Small number of head shows little diversity in feature clipping while larger one results in very short vectors with poor representation capacity. The label embedding regularizer weight \(\lambda\) in Figure 5(c) can be set in a wide range, where either without \(\mathcal{L}_{LER}\) or large \(\lambda\) will result in poor performance.

### The Impact of Class Number

As the number of classes influence the difficulty of classification task directly, in this section, we discuss the impact of class number on our proposed model LaCon. We pick the DBPedia dataset for experiment. The original DBPedia dataset includes 14 labels. We gradually increase the label number from 2 to 14 and randomly select 1000 samples for each label in our experiment as training set. Meanwhile, we keep the whole samples for the chosen labels in the evaluation set unchanged. Figure 6 demonstrates that with the increase of the labels, the performance of all models degrades as the task becomes more difficult. However, LaConfusion outperforms CE+SCL consistently on different number of labels, which shows the advantage of leveraging labels as anchors or positive/negative samples during contrastive learning.

## 5 Related Work

### Contrastive Learning

Contrastive Learning has become a rising domain and achieved significant success in various CV, speech and NLP tasks He et al. (2019); Chen et al. (2020); Fang and Xie (2020); Han et al. (2021); Saeed et al. (2021); Gunel et al. (2021); Gao et al. (2021); Yan et al. (2021). There are two kinds of CL approaches, which are self-supervised CL and supervised CL. The self-supervised CL contrasts a

Figure 4: Visualization of label and instance representations for MRPC (a&b) and CoLA (c&d) using T-SNE.

 

single positive for each anchor (i.e., an augmented version of the same image) against a set of negatives consisting of the entire remainder of the batch. However, due to the intrinsic discrete nature of natural language, data augmentations are less effective than that in CV. Recently, researchers Khosla et al. (2020); Gunel et al. (2021) propose supervised CL, which contrasts the set of all samples from the same class as positives against the negatives from the remainder of the batch. Suresh and Ong (2021) propose label-aware SCL method via assigning weights to instances of different labels, which treats the negative samples differently.

LaCon belongs to the scope of supervised CL. Different from Khosla et al. (2020); Gunel et al. (2021), LaCon can take the labels as anchors or mine negative/positive from labels, which does not need to construct positive pairs from the data augmentation. Meanwhile, Gunel et al. (2021) combine CL and CE losses at the same time, but LaCon is purely equipped with three CL objectives, including the instance-centered contrastive loss, the label-centered contrastive loss and the label embedding regularizer.

### Label Representation Learning

Label representation learning aims to learn the embeddings of labels in classification tasks and has been proven to be effective in various CV (Frome et al., 2013; Akata et al., 2016) and NLP tasks Tang et al. (2015); Pappas and Henderson (2019); Nam et al. (2016); Zhang et al. (2018); Wang et al. (2018); Xiao et al. (2019); Miyazaki et al. (2020). In this work, we compare with two representative label embedding based models, which are LEAM Wang et al. (2018) and LSAX Xiao et al. (2019). Both learn label embeddings and sentence representations in a joint space based on attention mechanism and fuse them to improve the classification. Differently, LaCon learns the label and instance representations jointly via purely supervised contrastive learning. Besides, our experiments also verify that after obtaining the discriminative label and instance representations, even simple fusion block can facilitate the language understanding tasks.

## 6 Conclusions

In this paper, we proposed a novel supervised contrastive learning approach for language understanding. To utilize the class labels sufficiently, we devise three novel contrastive objectives, including a multi-head instance-centered contrastive loss, a label-centered contrastive loss, and a label embedding regularizer. Extensive experiments were conducted on eight public datasets from GLUE and CLUE benchmarks, showing the competitiveness of LaCon against various strong baselines. Besides, we also demonstrate the strong capacity of LaCon on more challenging few-shot and data imbalance settings, which leads up to \(9.4\%\) improvement on the FewGLUE and FewCLUE benchmarks. LaCon does not require any complicated network architecture or any extra data augmentation, and can be easily plugged into mainstream pre-trained language models. In the future, we will explore more advanced representation fusion approaches to enhance the capability of LaCon and plan to extend LaCon to the computer vision and speech fields.

Figure 5: Illustration of hyper-parameters tuning (RTE is taken for example and other datasets are similar).

Figure 6: The impact of class number on LaCon. Experiments were conducted on DBPedia.

 



## References

* A. Aghajanyan, A. Shrivastava, N. Gupta, N. Goyal, L. Zettlemoyer, and S. Gupta (2021)Better fine-tuning by reducing representational collapse. In 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021, External Links: Link Cited by: SS4.
* Z. Akata, F. Perronnin, Z. Harchaoui, and C. Schmid (2016)Label-embedding for image classification. IEEE Transactions on Pattern Analysis and Machine Intelligence38 (7), pp. 1425-1438. Cited by: SS1.
* Y. Bao, M. Wu, S. Chang, and R. Barzilay (2020)Few-shot text classification with distributional signatures.. In 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020, External Links: Link Cited by: SS1.
* K. Cao, C. Wei, A. Gaidon, N. Arechiga, and T. Ma (2019)Learning imbalanced datasets with label-distribution-aware margin loss. In Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019, NeurIPS 2019, December 8-14, 2019, Vancouver, BC, Canada, pp. 1565-1576. Cited by: SS1.
* J. Chen, Z. Yang, and D. Yang (2020)Mixtext: linguistically-informed interpolation of hidden space for semi-supervised text classification. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, ACL 2020, Online, July 5-10, 2020, pp. 2147-2157. External Links: Link Cited by: SS1.
* T. Chen, S. Kornblith, M. Norouzi, and G. Hinton (2020)A simple framework for contrastive learning of visual representations. arXiv preprint arXiv:2002.05709. Cited by: SS1.
* T. Chen, S. Kornblith, K. Swersky, M. Norouzi, and G. Hinton (2020)Big self-supervised models are strong semi-supervised learners. arXiv preprint arXiv:2006.10029. Cited by: SS1.
* Y. Chen, J. Zhao, W. Wang, M. Fang, H. Kang, L. Wang, T. Wei, J. Ma, S. Wang, and J. Xiao (2021)SEQ-cpc : sequential contrastive predictive coding for automatic speech recognition. In IEEE International Conference on Acoustics, Speech and Signal Processing, ICASSP 2021, Toronto, ON, Canada, June 6-11, 2021, External Links: Link Cited by: SS1.
* J. Devlin, M. Chang, K. Lee, and K. Toutanova (2019)BERT: pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics, pp. 4171-4186. Cited by: SS1.
* H. Fang and P. Xie (2020)CERT: contrastive self-supervised learning for language understanding. CoRRabs/2005.12766. Cited by: SS1.
* A. Frome, G. S. Corrado, J. Shlens, S. Bengio, J. Dean, M. Runzaro, and T. Mikolov (2013)Devise: a deep visual-semantic embedding model. In Advances in Neural Information Processing Systems 26: 27th Annual Conference on Neural Information Processing Systems 2013. Proceedings of a meeting held December 5-8, 2013, Lake Tahoe, Nevada, United States, pp. 2121-2129. Cited by: SS1.
* J. Gao, D. He, X. Tan, T. Qin, L. Wang, and T. Liu (2019)Representation degeneration problem in training natural language generation models. In International Conference on Learning Representations, Cited by: SS1.
* T. Gao, X. Yao, and D. Chen (2021)SimCSE: simple contrastive learning of sentence embeddings. CoRRabs/2104.08821. Cited by: SS1.
* a new approach to self-supervised learning. In Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual, External Links: Link Cited by: SS1.
* B. Gunel, J. Du, A. Conneau, and V. Stoyanov (2021)Supervised contrastive learning for pre-trained language model fine-tuning. In 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021, External Links: Link Cited by: SS1.
* C. Han, M. Wang, H. Ji, and L. Li (2021)Learning shared semantic space for speech-to-text translation. In Findings of the Association for Computational Linguistics: ACL/IJCNLP 2021, Online Event, August 1-6, 2021, External Links: Link Cited by: SS1.
* K. He, H. Fan, Y. Wu, S. Xie, and R. Girshick (2019)Momentum contrast for unsupervised visual representation learning. arXiv preprint arXiv:1911.05722. Cited by: SS1.
* P. Khosla, P. Teterwak, C. Wang, A. Sarna, Y. Tian, P. Isola, A. Maschinot, C. Liu, and D. Krishnan (2020)Supervised contrastive learning. In Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual, External Links: Link Cited by: SS1.
* B. Li, H. Zhou, J. He, M. Wang, Y. Yang, and L. Li (2020)On the sentence embeddings from pre-trained language models. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP),

pages 9119-9130, Online. Association for Computational Linguistics.
* Liang et al. (2021) Xiaobo Liang, Lijun Wu, Juntao Li, Yue Wang, Qi Meng, Tao Qin, Wei Chen, Min Zhang, and Tie-Yan Liu. 2021. R-drop: Regularized dropout for neural networks. _CoRR_, abs/2106.14448.
* Merikoski (1984) Jorma Kaarlo Merikoski. 1984. On the trace and the sum of elements of a matrix. _Linear Algebra and its Applications_, 60:177-185.
* Miyazaki et al. (2020) Taro Miyazaki, Kiminobu Makino, Yuka Takei, Hiroki Okamoto, and Jun Goto. 2020. Label embedding using hierarchical structure of labels for twitter classification. _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and 9th International Joint Conference on Natural Language Processing, Proceedings of the Conference_, pages 6317-6322.
* Muller et al. (2019) Rafael Muller, Simon Kornblith, and Geoffrey E. Hinton. 2019. When does label smoothing help? In _Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019, NeurIPS 2019, December 8-14, 2019, Vancouver, BC, Canada_, pages 4696-4705.
* Nam et al. (2016) Jinseok Nam, Eneldo Loza Mencia, and Johannes Furnkranz. 2016. All-in text: Learning document, label, and word representations jointly. In _Proceedings of the thirtieth AAAI conference on artificial intelligence_, pages 1948-1954.
* Pappas and Henderson (2019) Nikolaos Pappas and James Henderson. 2019. GILE: A Generalized Input-Label Embedding for Text Classification. _Transactions of the Association for Computational Linguistics_, 7:139-155.
* Saeed et al. (2021) Aaqib Saeed, David Grangier, and Neil Zeghidour. 2021. Contrastive learning of general-purpose audio representations. In _IEEE International Conference on Acoustics, Speech and Signal Processing, ICASSP 2021, Toronto, ON, Canada, June 6-11, 2021_, pages 3875-3879. IEEE.
* Schick and Schutze (2021) Timo Schick and Hinrich Schutze. 2021. It's not just size that matters: Small language models are also few-shot learners. In _Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT 2021, Online, June 6-11, 2021_, pages 2339-2352. Association for Computational Linguistics.
* Srivastava et al. (2014) Nitish Srivastava, Geoffrey E. Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. 2014. Dropout: a simple way to prevent neural networks from overfitting. _J. Mach. Learn. Res._, 15(1):1929-1958.
* Suresh and Cong (2021) Varsha Suresh and Desmond C. Ong. 2021. Not all negatives are equal: Label-aware contrastive loss for fine-grained text classification. In _Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, EMNLP 2021, Virtual Event / Punta Cana, Dominican Republic, 7-11 November, 2021_, pages 4381-4394. Association for Computational Linguistics.
* Tang et al. (2015) Jian Tang, Meng Qu, and Qiaozhu Mei. 2015. PTE: Predictive text embedding through large-scale heterogeneous text networks. _In Proceedings of the ACM SIGKDD International Conference on Knowledge Discovery and Data Mining_, 2015-August:1165-1174.
* van den Oord et al. (2018) Aaron van den Oord, Yazhe Li, and Oriol Vinyals. 2018. Representation learning with contrastive predictive coding. _CoRR_, abs/1807.03748.
* Van der Maaten and Hinton (2008) Laurens Van der Maaten and Geoffrey Hinton. 2008. Visualizing data using t-sne. _Journal of machine learning research_, 9(11).
* Vaswani et al. (2017) Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In _Advances in Neural Information Processing Systems_, pages 5998-6008.
* Wang et al. (2019) Alex Wang, Amampreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R. Bowman. 2019. GLUE: A multi-task benchmark and analysis platform for natural language understanding. In _7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019_. OpenReview.net.
* Wang et al. (2021) Dong Wang, Ning Ding, Fiji Li, and Haitao Zheng. 2021. CLINE: contrastive learning with semantic negative examples for natural language understanding. In _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing, ACL/IJCNLP 2021, (Volume 1: Long Papers), Virtual Event, August 1-6, 2021_, pages 2332-2342. Association for Computational Linguistics.
* Wang et al. (2018) Guoyin Wang, Chunyuan Li, Wenlin Wang, Yizhe Zhang, Dinghan Shen, Xinyuan Zhang, Ricardo Henao, and Lawrence Carin. 2018. Joint embedding of words and labels for text classification. In _Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_, pages 2321-2331.
* Wang et al. (2020) Lingxiao Wang, Jing Huang, Kevin Huang, Ziniu Hu, Guangtao Wang, and Quanquan Gu. 2020. Improving neural language generation with spectrum control. In _8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020_. OpenReview.net.
* Wang and Isola (2020) Tongzhou Wang and Phillip Isola. 2020. Understanding contrastive representation learning through alignment and uniformity on the hypersphere. In _Proceedings of the 37th International Conference on Machine Learning, ICML 2020, 13-18 July 2020, Virtual Event_, volume 119 of _Proceedings of Machine Learning Research_, pages 9929-9939. PMLR 

* Wilcoxon (1945) Frank Wilcoxon. 1945. Individual comparisons by ranking methods. _Biometrics bulletin_, 1(6):80-83.
* Xiao et al. (2019) Lin Xiao, Xin Huang, Boli Chen, and Liping Jing. 2019. Label-specific document representation for multi-label text classification. In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_, pages 466-475.
* Xu et al. (2020) Liang Xu, Hai Hu, Xuanwei Zhang, Lu Li, Chenjie Cao, Yudong Li, Yechen Xu, Kai Sun, Dian Yu, Cong Yu, Yin Tian, Qianqian Dong, Weitang Liu, Bo Shi, Yiming Cui, Junyi Li, Jun Zeng, Rongzhao Wang, Weijian Xie, Yanting Li, Yina Patterson, Zuoyun Tian, Yiwen Zhang, He Zhou, Shaoweihua Liu, Zhe Zhao, Qipeng Zhao, Cong Yue, Xinrui Zhang, Zhengliang Yang, Kyle Richardson, and Zhengzhong Lan. 2020. CLUE: A Chinese language understanding evaluation benchmark. In _Proceedings of the 28th International Conference on Computational Linguistics_, Barcelona, Spain (Online). International Committee on Computational Linguistics.
* Xu et al. (2021) Liang Xu, Xiaojing Lu, Chenyang Yuan, Xuanwei Zhang, Hu Yuan, Hulin Xu, Guoao Wei, Xiang Pan, and Hai Hu. 2021. Fewclue: A chinese few-shot learning evaluation benchmark. _CoRR_, abs/2107.07498.
* Yan et al. (2021) Yuanmeng Yan, Romei Li, Sirui Wang, Fuzheng Zhang, Wei Wu, and Weiran Xu. 2021. Consert: A contrastive framework for self-supervised sentence representation transfer. In _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing, ACL/IJCNLP 2021, (Volume 1: Long Papers), Virtual Event, August 1-6, 2021_, pages 5065-5075. Association for Computational Linguistics.
* Zhang et al. (2018) Honglun Zhang, Liqiang Xiao, Wenqing Chen, Yongkun Wang, and Yaohui Jin. 2018. Multi-task label embedding for text classification. In _Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing_, pages 4545-4553.

## Appendix A Appendix

### Accumulative Ablation Study

In this section, we supplement more ablation results by adding each proposed CL loss cumulatively. We conduct experiments on MRPC, RTE, and CoLA datasets and keep the setting consistent with Section 3.4. Table 6 demonstrates that the contribution of each component in more details.

### Experimental Results of More Datasets

We supplement more experimental results on the remaining datasets of GLUE and CLUE benchmarks. We follow the same experimental setup with Section 3.1. Please note that SST-B is a regression task that is beyond the capacity of the proposed LaCon. The official CLUE benchmark has replaced CMNLI with OCNLI dataset, and the CSL dataset is a keyword recognition task, which is not suitable for our proposed model. Thus, we omit the experiments on above three datasets and report the performance on the remaining language understanding tasks including SST-2, MNLI, AFQMC, OCNLI and IFLYTEK.

Table 7 shows that, LaCon-vanilla consistently outperforms BERT fine-tuned with CE, and LaConfusion still beats the baselines among all datasets, which further demonstrates the superiority of our proposed method.

### Theoretical Analysis

In this section, we conduct the theoretical analysis to prove the rationality and necessity of our proposed ICL and LCL losses. We also explain why these two losses are complementary. Finally, we analyze the computational efficiency of ICL and LCL compared to InfoNCE (van den Oord et al., 2018) and SCL (Khosla et al., 2020).

The recent researches (Li et al., 2020; Gao et al., 2019) reveal that the anisotropy problem of pre-trained language models, which shows that the learnt embeddings occupy a narrow cone in the dense vector space, harming the uniformity of the models and limiting the representation capacity. The singular values of the contextual embeddings decay drastically with most of them nearly zeros (Wang and Isola, 2020). CL is proposed to eliminate the long-tail distribution problem of singular values, aiming to enhance the representation capacity (Gao et al., 2021; Yan et al., 2021; Gunel et al., 2021). From the spectrum perspective (Wang et al., 2020; Wang and Isola, 2020) that analyzes the distribution and uniformity of the learned embedding space, CL flattens singular values of the embeddings thus improves the capacity of language

\begin{table}
\begin{tabular}{c c c c} \hline
**Methods** & **MRPC** & **RTE** & **CoLA** \\ \hline \(\mathcal{L}_{ICL}\) & 86.2\(\pm\)1.5 & 67.8\(\pm\)1.1 & 61.2\(\pm\)0.9 \\ \(\mathcal{L}^{\prime}_{ICKL}\) & 86.9\(\pm\)1.1 & 68.2\(\pm\)0.7 & 61.2\(\pm\)1.5 \\ \(\mathcal{L}_{LCIL}\) & 87.0\(\pm\)1.7 & 68.3\(\pm\)1.0 & 61.1\(\pm\)1.3 \\ \(\mathcal{L}^{\prime}_{ICI+LCLR}\) & 87.3\(\pm\)1.1 & 70.2\(\pm\)1.3 & 62.2\(\pm\)1.6 \\ \(\mathcal{L}_{ICIL}+\mathcal{L}_{LRER}\) & 87.1\(\pm\)0.6 & 70.5\(\pm\)0.8 & 61.2\(\pm\)1.1 \\ \hline
**LaCon-vanilla** & **87.5\(\pm\)0.8** & **71.4\(\pm\)0.7** & **62.4\(\pm\)1.1** \\ \hline \end{tabular}
\end{table}
Table 6: Ablation study via adding losses cumulatively.

 

models.

\[\begin{split}&\mathcal{L}_{ICL}=-\frac{1}{\tau}E_{(x,y)\sim A(y)}(H _{x}L_{y})\\ +& E_{x\sim A(y)}[logE_{y^{-}\notin A(y)}(e^{H_{x}L_{ y^{-}/\tau}})]\end{split} \tag{10}\]

\[\begin{split}& E_{x\sim A(y)}[logE_{y^{-}\notin A(y)}(e^{H_{x} L_{y^{-}/\tau}})]\\ =&\frac{1}{N}\sum_{i=1}^{i=N}log(\frac{1}{C-1}\sum_{ j\neq i}^{1\leq j\leq C}e^{H_{x}L_{y_{j}}/\tau})\\ &\geq\frac{1}{N(C-1)\tau}\sum_{i=1}^{i=N}\sum_{j=1,j\neq i}^{j=C} H_{x}L_{y^{-}}\end{split} \tag{11}\]

Therefore, we can form an asymptotic equivalent objective of the \(\mathcal{L}_{ICL}\) (Equation 3) as Equation 10. \((x,y)\sim A(y)\) denotes instances (i.e. \(x\)) with corresponding label (i.e. \(y\)) and \(y^{-}\) denotes the label that is different from \(y\). The first item keeps instances and corresponding labels similar and the second item pushes the mismatched instances and labels apart. We can further derive Equation 11 using Jensen's inequality because \(e(.)\) is convex. Therefore, minimizing the \(\mathcal{L}_{ICL}\) equals to minimization of summation of all elements in \(HL^{T}\in R^{N\times C}\). Because both \(H\) and \(L\) are normalized, \(tr(H^{T}L)\) is a constant due to all diagonal elements are ones. \(sum(HL^{T})\) is an upper bound of the largest singular value Merikoski (1984) and minimization of the \(sum(HL^{T})\) will flatten the singular values distribution of \(HL^{T}\). As the \(HL^{T}\) is a non-squared matrix, we need to optimize both the left and right singular values using \(HL^{T}\) and \(LH^{T}\) in order to effectively eliminate the anisotropy and promote the uniformity of pre-trained language models in classification tasks to enhance the model capacity. Thus, we also need to optimize the label-centered contrastive loss \(\mathcal{L}_{LCL}\) at the same time. From above analysis, we can see that \(\mathcal{L}_{ICL}\) and \(\mathcal{L}_{LCL}\) are complementary to each other. Similarly, we can derive that minimizing \(\mathcal{L}_{LCL}\) results in the minimization of \(sum(LH^{T})\in R^{C\times N}\).

Although both ICL and LCL calculate the \(N\times C\) similarity scores for a mini-batch, they are different. The ICL is the average of the instance-level per sample loss while the LCL is the per label loss. The ICL intends to align each instance to corresponding label correctly. The LCL makes the instances of different labels far away from each other and instances of the same label more compact. They consider different aspects of instance and label representation through operating the \(N\times C\) similarity scores differently according to Equation 3 and 5.

Compared to InfoNCE, ICL improves the computational efficiency from \(O(N^{2})\) to \(O(NC)\) because we only need to contrast the instance representation with corresponding label representations for per example loss, which is extremely useful for language understanding tasks Wang et al. (2019); Xu et al. (2020) that commonly consist of 2 or 3 labels. Similarly, LCL is also more computationally efficient as it only contrasts one label representation to several instances rather than computes all pairs of instances belonging to a given label in the mini-batch. Thus it improves the complexity from \(O(N^{2})\) to \(O(CN)\) compared with SCL too.

## Appendix B

\begin{table}
\begin{tabular}{c c c|c c c} \hline
**Methods** & **SST-2** & **MNLI** & **AFQMC** & **OCNLI** & **IFLYTEK** \\ \hline Baseline* & 91.4\(\pm\)0.3 & 73.1\(\pm\)0.3 & 74.5\(\pm\)0.3 & 68.3\(\pm\)0.7 & 61.5\(\pm\)0.5 \\ CE & 91.2\(\pm\)0.1 & 72.5\(\pm\)0.3 & 70.9\(\pm\)0.5 & 66.8\(\pm\)0.4 & 60.6\(\pm\)0.2 \\ LaCon-vanilla & 91.4\(\pm\)0.3 & 73.3\(\pm\)0.4 & 72.5\(\pm\)0.3 & 67.4\(\pm\)0.3 & 60.9\(\pm\)0.2 \\
**LaCon-fusion** & **92.5\(\pm\)0.2** & **73.9\(\pm\)0.3** & **74.8\(\pm\)0.5** & **69.1\(\pm\)0.6** & **63.5\(\pm\)0.7** \\ \hline \end{tabular}
\end{table}
Table 7: Performance on the remaining datasets of GLUE and CLUE. Baseline* means the best performance of our baselines. The evaluation metrics are the same as the official GLUE Wang et al. (2019) and CLUE Xu et al. (2020) benchmarks (all with significance value \(p<0.05\)).