SlowGuess/NAACL_2022_OCR huggingface dataset:
Zhenyu Zhang, Yuming Zhao, Meng Chen, Xiaodong He  
JD AI, Beijing, China
{zhangzhenyu47,zhaoyuming3,chenmeng20,xiaodong.he}@jd.com
Contrastive learning (CL) has achieved astonishing progress in computer vision, speech, and natural language processing fields recently with self-supervised learning. However, CL approach to the supervised setting is not fully explored, especially for the natural language understanding classification task. Intuitively, the class label itself has the intrinsic ability to perform hard positive/negative mining, which is crucial for CL. Motivated by this, we propose a novel label anchored contrastive learning approach (denoted as LaCon) for language understanding. Specifically, three contrastive objectives are devised, including a multi-head instance-centered contrastive loss (ICL), a label-centered contrastive loss (LCL), and a label embedding regularizer (LER). Our approach does not require any specialized network architecture or any extra data augmentation, thus it can be easily plugged into existing powerful pre-trained language models. Compared to the state-of-the-art baselines, LaCon obtains up to  $4.1\%$  improvement on the popular datasets of GLUE and CLUE benchmarks. Besides, LaCon also demonstrates significant advantages under the few-shot and data imbalance settings, which obtains up to  $9.4\%$  improvement on the FewGLUE and FewCLUE benchmarking tasks.
In recent years, contrastive learning (CL) has been widely applied to self-supervised representation learning and led to major advances across computer vision (CV) (He et al., 2019; Chen et al., 2020b), speech (Saeed et al., 2021; Chen et al., 2021), and natural language processing (NLP) (Fang and Xie, 2020; Gao et al., 2021; Yan et al., 2021). The basic idea behind these works is to pull together an anchor and a “positive” sample in the embedding space, and to push apart the anchor from many “negative” samples. Since no labels are available, a positive pair often consists of data augmentations
of the sample (a.k.a "views"), and negative pairs are formed by the anchor and randomly chosen samples from the mini-batch. In visual representations, an effective solution to generate data augmentations is to take two random transformations of the same image (e.g., cropping, flipping, distortion and rotation) (Chen et al., 2020b; Grill et al., 2020; Chen et al., 2020c). For natural language, similar approaches are adopted such as word deletion, reordering, substitution, and back-translation etc. (Fang and Xie, 2020; Wang et al., 2021) However, data augmentation in NLP is inherently difficult because of its discrete nature. Therefore, some previous works (Gao et al., 2021; Yan et al., 2021) also use dropout technique (Srivastava et al., 2014) to obtain sentence augmentations.
Unlike self-supervised setting, some researchers propose supervised contrastive learning (SCL) (Khosla et al., 2020; Gunel et al., 2021; Suresh and Ong, 2021) which can construct positive pairs by leveraging label information. Examples from the same class are pulled closer than the examples from different classes, leveraging the semantics of labels to construct negatives and positives rather than shallow lexical information via data augmentation. Despite the aforementioned advantages brought by SCL, we argue that CL under supervised learning is not fully explored because the label information can be better utilized. On the one hand, labels are usually not merely categorical indices in the label vocabulary, but also contain specific semantic meanings, especially in the language understanding tasks. Thus labels can be used as positive/negative samples or anchors when calculating contrastive loss. On the other hand, label embedding enjoys a built-in ability to leverage alternative sources of information related to labels, such as class hierarchies or textual descriptions. Once we obtain representative label embeddings, they can be utilized to enhance the image/text representations, and finally facilitate the classification task. Previous la
Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 1437 - 1449 July 10-15, 2022 ©2022 Association for Computational Linguistics
bel embedding based classification models (Wang et al., 2018; Xiao et al., 2019) have demonstrated the effectiveness of leveraging label information.
Motivated by above analysis, we propose a novel label anchored supervised contrastive learning approach (denoted as LaCon), which combines the advantages of both contrastive learning and label embedding techniques. Specifically, we have the following three novel designs: 1) Instance-centered contrastive loss (ICL), which uses the InfoNCE (van den Oord et al., 2018) to encourage each text representation and its corresponding label representation to be closer while pushing far away mismatched instance-label pairs. We further apply a multi-head mechanism to catch different aspects of text semantics. 2) Label-centered contrastive loss (LCL), which takes label as anchor, and encourages the label representation to be more similar to the corresponding instances belonging to the same class in a mini-batch than the instances with different labels. 3) Label embedding regularizer (LER), which keeps the inter-label similarity as low as possible thus the feature space of each class is more dispersed to prevent representation degeneration. By combining above three losses, LaCon can learn good semantic representations within the same space for both input instances and labels. It's also well aligned with the two key properties related to CL: alignment and uniformity (Wang and Isola, 2020), where alignment favors encoders that assign similar features to similar samples. Uniformity prefers a feature distribution that preserves maximal information, i.e., the uniform distribution on the unit hypersphere.
To validate the effectiveness of LaCon, we perform extensive experiments on eight language understanding tasks. We take the popular pre-trained language model BERT-base (Devlin et al., 2019) as text encoder without loss of generality. For simplicity, we predict the classification label by matching the instance representation with label embeddings directly. Since our approach does not require any specialized network architecture or any extra data augmentation, LaCon can be easily plugged into other pre-trained language models. Additionally, we also explore the capability of LaCon under more difficult task settings, including few-shot learning and data imbalance situations.
- We propose a novel label anchored contrastive learning approach for language understanding,
which is equipped with a multi-head instance-centered contrastive loss, a label-centered contrastive loss, and a label embedding regularizer. All three contrastive objectives help the model learn the joint semantic representations for both input instances and labels.
- We conduct extensive experiments on eight public language understanding tasks from GLUE (Wang et al., 2019) and CLUE (Xu et al., 2020) benchmarks, and experimental results show the competitiveness of LaCon. Additionally, we also experiment on more difficult settings including few-shot learning and data imbalance situations. LaCon experimentally obtains up to  $9.4\%$  improvement over BERT-base on FewGLUE (Schick and Schütze, 2021) and FewCLUE (Xu et al., 2021) benchmark tasks.
- We analyze the contribution of each ingredient of LaCon, and also visualize the learned instance and label representations, showing the necessity of each loss component and the advantage of LaCon on representation learning over BERT fine-tuned with cross entropy.
In this section, we introduce the details of LaCon. We focus on the language understanding classification tasks. For a multi-class classification problem with  $C$  classes, we work with a batch of training examples  $\{x_{i},y_{i}\}$ , where  $1\leq i\leq N$  and  $1\leq y_{i}\leq C$ . Our target is to learn discriminative representations for both instances and class labels. As Figure 1 shows, we propose three supervised CL based objectives, including the instance-centered contrastive loss, the label-centered contrastive loss, and the label embedding regularizer loss.
2.1 The Input Encoder
The input of LaCon contains two parts consisting of the text and all the labels for the task. Since SOTA language understanding classification models follow the "pre-training then fine-tuning" two-stage paradigm, here we take the prevalent pre-trained language model (PtLM) as input encoder. In this paper, we select BERT-base (Devlin et al., 2019) as the backbone for PtLM (denoted as  $f$ ) without loss of generality. Given a text  $\mathbf{x} = \{w_1, w_2, \dots, w_M\}$  containing  $M$  tokens, the output of PtLM (i.e. BERT) is  $\mathbf{E} = f_{PtLM}([CLS], w_1, \dots, w_M)$  where
(a) Instance-centered Contrastive Loss
(b) Label-centered Contrastive Loss
Figure 1: Overview of LaCon. The full line is the similarity between an instance and corresponding label, and the dash line is the similarity between the mismatched instance and label. The lines with the same color denote the per-instance or per-label loss.
(c) Label Embedding Regularizer Loss
the  $[CLS]$  token is the inserted sentence representation token and  $\mathbf{E} \in R^{(M + 1)\times d}$ .  $d$  is the dimension of the model. We use the first token output  $E_{[CLS]} \in R^d$  to represent the whole input text. We then apply a projection head (denoted as  $g$ ) that is a 3-layer MLP with ReLU activation function for each hidden layer to the  $E_{[CLS]}$ , where the dimension of the  $g$  is also  $d$ . For a mini-batch  $X$  with  $N$  training samples, the text representations can be obtained as Equation 1.
$$
\begin{array}{c} H = g \circ f _ {P t L M} (X), \text {w h e r e} \\ X = x _ {1}, x _ {2}, \dots , x _ {N} \text {a n d} H \in R ^ {N \times d} \end{array} \tag {1}
$$
Along with each mini-batch, LaCon also maps the  $C$  classes into  $C$  label embeddings. For simplicity, we look up in a learnable weight matrix  $W_{emb}$  and map the  $k^{th}$  label into the  $k^{th}$  row of  $W_{emb}$ , where the  $W_{emb}$  is randomly initialized. The dimension  $d$  is the same as PtLM. We then normalize the vectors for both  $L$  and  $H$  using the  $l_2$  norm.
$$
\begin{array}{c} L = \operatorname {l o o k u p} ([ 1, 2, \dots , C ], W _ {\text {e m b}}) \\ \text {w h e r e} L \in R ^ {C \times d} \text {a n d} W _ {\text {e m b}} \in R ^ {C \times d} \end{array} \tag {2}
$$
2.2 Instance-centered Contrastive Loss
Given a mini-batch of input text and corresponding label  $(x_{i},y_{i})$ , as shown in Figure 1 (a), the instance-centered contrastive loss (ICL) takes each instance  $x_{i}$  as anchor, and mines positive and negative samples from class labels. ICL aims to encourage each text representation and corresponding label representation to be closer while pushing far away mismatched instance-label pairs. As shown in Equation 3, we modify the InfoNCE (van den Oord et al., 2018) to calculate the loss. Here we leverage the cosine similarity function as the distance metric sim.  $\tau$  is the temperature hyper-parameter, which can be tuned to improve the performance.
Lower temperature increases the influence of examples that are harder to separate, effectively creating harder negatives. Similar to the self-supervised CL, ICL also takes only one positive and many negatives. Differently, the positive is not generated from data augmentation, and the negatives are not randomly sampled from the same mini-batch. By treating the class labels as data samples, ICL can mine better positive and negatives with the supervision signal. By minimizing the ICL, the instance representation is aligned to its label representation in the same semantic space, which encourages the model to learn a more representative embedding for each class label.
$$
\mathcal {L} _ {I C L} = - \frac {1}{N} \sum_ {x _ {i}, y _ {i}} \log \frac {\exp \left(\operatorname {s i m} \left(H _ {x _ {i}} , L _ {y _ {i}}\right) / \tau\right)}{\sum_ {1 \leq p \leq C} \exp \left(\operatorname {s i m} \left(H _ {x _ {i}} , L _ {p}\right) / \tau\right)} \tag {3}
$$
Inspired by the image-augmented views proposed in CV (He et al., 2019; Chen et al., 2020b,c; Grill et al., 2020), we also leverage the multi-head mechanism proposed in Transformer (Vaswani et al., 2017) to compute the ICL for each head representation with smaller representation dimension. Each head can be regarded as a clipped local view of the instance or label representation. Suppose we have  $m$  heads for both instance representation and label representation, then for the  $k_{th}$  head, the corresponding representations for training sample  $(x_i, y_i)$  are  $h_{x_i}^k$  and  $l_{y_i}^k$ , and the dimension of each vector becomes  $d' = d / m$ . Then, we apply the contrastive loss for each head by following Equation 4. Compared with InfoNCE,  $\mathcal{L}_{ICL}$  and  $\mathcal{L}_{ICL}'$  do not suffer from small batch size issue (He et al., 2019; Chen et al., 2020c) because we only need to contrast the instance representation with corresponding label representation for per example loss.
$$
\mathcal {L} _ {I C L} ^ {\prime} = - \frac {1}{N} \sum_ {k = 1} ^ {m} \sum_ {x _ {i}, y _ {i}} \log \frac {\exp \left(\sin \left(h _ {x _ {i}} ^ {k} , l _ {y _ {i}} ^ {k}\right) / \tau\right)}{\sum_ {1 \leq p \leq C} \exp \left(\sin \left(h _ {x _ {i}} ^ {k} , l _ {p} ^ {k}\right) / \tau\right)} \tag {4}
$$
2.3 Label-centered Contrastive Loss
As shown in Figure 1 (b), we can take the class label in a mini-batch as anchor, and mine positive/negative samples from corresponding instances. Suppose there are  $|P|$  classes in the batch, where  $P = \{p \mid 1 \leq p \leq C \land |A(p)| > 0\}$ . We define that  $A(p)$  denotes the set of indices of all positive instances whose label is  $p$ , i.e.  $A(p) = \{x_i | y_i = p\}$ . And  $B(p)$  represents the set of negative instances whose label is not  $p$ , i.e.  $B(p) = \{x_j | y_j \neq p\}$ . Then we can calculate the label-centered contrastive loss (LCL) as Equation 5, which promotes the instances of a specific label to be more similar than the others for each label. Similar to the previous SCL (Khosla et al., 2020; Gunel et al., 2021), LCL also contains many positives per anchor and many negatives. Different from SCL which sums up all the softmax scores among all pairs of instances of the same class in a batch, LCL is based on comparing a specific label representation with corresponding instances (i.e.  $A(p)$ ). LCL is more stable as the label representation serves as the anchor which can be stably updated.
$$
\mathcal {L} _ {L C L} = - \frac {1}{| P |} \sum_ {p \in P} \sum_ {a \in A (p)} \log \frac {\exp \left(\operatorname {s i m} \left(L _ {p} , H _ {a}\right) / \tau\right)}{\sum_ {b \in B (p)} \exp \left(\operatorname {s i m} \left(L _ {p} , H _ {b}\right) / \tau\right)} (5)
$$
ICL and LCL are complementary to each other and more computationally efficient than previous SCL. We conduct the detailed theoretical analysis in Appendix A.3 due to space limitation.
2.4 Label Embedding Regularizer
Recent researches (Wang and Isola, 2020) demonstrate that it is common and useful to add a regularization term during training to eliminate the anisotropy problem. Inspired by this, We devise a label embedding regularizer as shown in Equation 6 to promote the uniformity of our model and prevent model degeneration. As illustrated in Figure 1 (c), the label embedding regularizer (LER) encourages the label representations to be dispersed in the unit hypersphere uniformly. The LER loss is the exponential mean of the cosine similarity for all pairs of label representations. As  $-1 \leq \text{sim}(L_i, L_j) \leq 1$ , it is quite sensitive to the loss change as the gradient is larger than 1 for  $\exp(x)$  w.r.t  $x \geq 0$ . Thus, we add 1.0 to the cosine similarity so that the value of LER varies from 0 to  $e^2 - 1$ .
$$
\mathcal {L} _ {L E R} = a v g \left(\sum_ {i \neq j} \left(e x p \left(1. 0 + s i m \left(L _ {i}, L _ {j}\right)\right) - 1. 0\right) \right. \tag {6}
$$
Finally, the overall loss function of LaCon is summarized as follows:
$$
\mathcal {L} = \mathcal {L} _ {I C L} ^ {\prime} + \mathcal {L} _ {L C L} + \lambda * \mathcal {L} _ {L E R} \tag {7}
$$
where  $\lambda$  is a hyper-parameter to balance the influence of our regularization term.
2.5 Matching based Class Prediction
Since LaCon is capable of learning instance and label representations jointly, we can predict the class by matching the instance representation to all label representations directly during inference, just as shown in Equation 8. We denote this simple and direct approach as LaCon-vanilla.  $H_{x}$  is the instance representation and  $L_{j}$  is the label representation of Class  $j$ . sim denotes the cosine similarity and  $1 \leq j \leq C$  denotes the corresponding label. The advantage of LaCon-vanilla is that it does not require any complicated network architecture and can be easily plugged into the mainstream PtLMs. As a result, our inference-time model contains exactly the same number of parameters as the model using the same encoder but trained with cross entropy loss.
$$
p r e d = \arg_ {m a x} (j) \left\{\operatorname {s c o r e} _ {j} \mid \operatorname {s i m} \left(H _ {x}, L _ {j}\right) \right\} \tag {8}
$$
2.6 LaCon with Label Fusion
Previous researches (Akata et al., 2016; Wang et al., 2018; Xiao et al., 2019; Pappas and Henderson, 2019; Miyazaki et al., 2020) have proved that incorporating the label semantics into the sentence representation can improve the model performance because the label information can highlight the alignment of input tokens and label information via carefully designed fusion mechanism. Inspired by LEAM (Wang et al., 2018), here we design a fusion block to enhance the instance representations by utilizing the learnt discriminative label embeddings. We firstly calculate the cosine similarity interaction matrix  $G$  between words and labels, and then apply a convolution then max-pooling layer  $(conv_{max})$  to measure the attention score  $(\beta_{i})$  for each word attending the instance representation. The fusion process is illustrated as Equation 9. Then the fused vector  $z$  is fed into the projection head  $g$  to get the enhanced instance representation.
$$
\begin{array}{l} m = \operatorname {c o n v} _ {\max } (G), \text {w h e r e} G _ {i j} = \frac {<   L _ {i} , E _ {j} >}{\left| \left| L _ {i} \right| \right| \cdot \left| \left| E _ {j} \right| \right|} \tag {9} \\ z = \sum_ {i} \beta_ {i} E _ {i}, \text {w h e r e} \beta = \operatorname {s o f t m a x} (m) \\ \end{array}
$$
To distinguish with the vanilla model above, we name this approach as LaCon-fusion. Please note that the fusion block is just applied between the text encoder and projection head, so the class prediction keeps the same as the LaCon-vanilla. Since the fusion block is not the main focus of this paper, we leave exploring more advanced fusion networks to future work.
Datasets	type	class	metric	train	dev
DBPedia	genre	14	ACC	14K	70K
Tnews	genre	15	F1	14.2K	10K
QQP	PI	2	ACC	10K	40.4K
MRPC	PI	2	ACC	4.07K	1.7K
QNLI	NLI	2	ACC	10K	5.4K
RTE	NLI	2	ACC	2.5K	278
CoLA	LA	2	M' corr	8.5K	1K
YelpRev	senti	2	ACC	10K	10K
Table 1: The statistics of datasets that are from GLUE (Wang et al., 2019) and CLUE (Xu et al., 2020)
Datasets. We experiment on 8 public datasets listed in Table 1. which are from GLUE (Wang et al., 2019), CLUE (Xu et al., 2020), DBpedia, and Yelp Dataset Challenge 2015. They cover five representative tasks including sentiment analysis (senti), genre classification (genre), paragraph identification (PI), natural language inference (NLI) and linguistic acceptability (LA). To improve the comparability and experiment confidence of the models, we follow the experimental setup in (Chen et al., 2020a) and use part of the training sets via sampling and the full original test sets for evaluation. We randomly sample without replacement at most 5K (binary-class) / 1K (multi-class) training instances per class from the whole datasets except for MRPC, RTE, and CoLA. We use the wilcoxon rank test (Wilcoxon, 1945) to check the statistic significance. The results of 10 runs are reported for each dataset in the format as "avg  $\pm$  std.dev".
Training & Evaluation. During training, we run experiments for MRPC, RTE and CoLA with 10 random seeds on the whole training datasets and run the sampling strategy with 10 repeats for the remaining datasets. The average evaluation metrics are reported to avoid the noise and unstable randomness of a single run. We use the AdamW optimizer with initial learning rate as {1e-5, 2e-5, 3e-5} with linear learning scheduler,  $6\%$  of warm-up steps of total optimization steps, and batch size
as  $\{8,16,32,64,96\}$ , where the hyper-parameters are tuned for different datasets. For evaluation, we leverage accuracy (ACC), Macro-F1 score and Matthew's corr (M' corr) metrics to evaluate the performance. We run 10 epochs for all the datasets<sup>1</sup> and then evaluate the models on dev set. Our implementation is based on Huggingface Transformers<sup>2</sup>.
Since LaCon is based on CL and label embedding technique, we compare with several SOTA models in language understanding including BERT-base fine-tuned with cross-entropy (CE) loss, label embedding based models, and self-supervised CL and supervised CL models.
- CE: we directly follow the instructions of original paper (Devlin et al., 2019) to finetune BERT for both English and Chinese language understanding tasks.
- LEAM: Wang et al. (2018) apply cosine similarity to get matching scores between words and labels and use CNN on the matching matrix to get the label-aware attention weighted text representation for classification.
- LSAN: Xiao et al. (2019) propose a label specific attention network that leverages label-attention and self-attention mechanism with an adaptive attention fusion strategy for multi-label classification. We use softmax instead of sigmoid for the model output due to our multi-class classification setting.
-  $\mathbf{CE} + \mathbf{CL}$ : Yan et al. (2021) propose to learn sentence representations by joint fine-tuning PtLM with InfoNCE and cross-entropy based on feature augmentation. Here we leverage the framework of ConSERT (Yan et al., 2021) and the feature augmentation in SimCSE (Gao et al., 2021) to finetune and evaluate the PtLM on classification datasets.
-  $\mathbf{CE} + \mathbf{SCL}$ : Gunel et al. (2021) propose to boost sentence representation learning by finetuning PtLM with both supervised contrastive learning loss (Khosla et al., 2020) and cross entropy loss. We follow the instructions in (Gunel et al., 2021) to set hyper-parameters.
1We split  $5\%$  of training set as validation for early stop.
Methods	YelpRev	DBPedia	Tnews	QNLI	RTE	QQP	MRPC	CoLA
CE	82.0±0.5	98.7±0.3	54.5±0.3	87.1±0.2	67.3±1.9	82.2±0.5	85.6±1.6	60.9±0.8
LEAM	82.1±0.6	98.7±0.5	54.1±0.3	87.2±0.7	67.3±1.3	81.9±0.5	85.6±1.3	60.9±1.0
LSAN	82.2±0.6	98.7±0.7	54.9±0.8	87.1±0.3	69.7±1.0	81.2±0.5	86.1±0.7	61.6±0.9
CE+CL	82.2±0.6	98.5±0.5	53.9±0.5	87.3±0.3	67.8±1.5	82.4±0.3	83.1±0.7	61.1±0.7
CE+SCL	81.4±0.8	98.5±0.6	54.6±0.2	87.7±0.1	69.1±2.2	82.5±0.6	88.1±0.9	62.3±0.6
LaCon-vanilla	82.3±0.5	98.9±0.5	56.8±0.6	88.1±0.2	71.4±0.7	82.8±0.5	87.5±0.9	62.4±1.0
LaCon-fusion	83.1±0.8	99.5±0.2	56.7±0.3	88.4±0.3	72.2±0.9	83.7±0.5	88.6±0.7	62.8±0.5
LEAM and LSAN are label embedding based methods while  $\mathbf{CE} + \mathbf{CL}$  and  $\mathbf{CE} + \mathbf{SCL}$  are contrastive learning based methods. To compare all models fairly, we use BERT-base encoder for all the baselines and our proposed model.