SlowGuess/NAACL_2022_OCR huggingface dataset:
Anton Chernyavskiy<sup>1</sup>, Dmitry Ilvovsky<sup>1</sup>, Pavel Kalinin<sup>2</sup>, Preslav Nakov<sup>3</sup>
$^{1}$ HSE University, Russia
2Yandex, Russia
$^{3}$ Qatar Computing Research Institute, HBKU, Qatar
{acherniavskii, dilvovsky}@hse.ru
kalinin.pavel.v@gmail.com pnakov@hbku.edu.qa
The use of contrastive loss for representation learning has become prominent in computer vision, and it is now getting attention in Natural Language Processing (NLP). Here, we explore the idea of using a batch-softmax contrastive loss when fine-tuning large-scale pretrained transformer models to learn better task-specific sentence embeddings for pairwise sentence scoring tasks. We introduce and study a number of variations in the calculation of the loss as well as in the overall training procedure; in particular, we find that a special data shuffling can be quite important. Our experimental results show sizable improvements on a number of datasets and pairwise sentence scoring tasks including classification, ranking, and regression. Finally, we offer detailed analysis and discussion, which should be useful for researchers aiming to explore the utility of contrastive loss in NLP.
Recent years have seen a revolution in Natural Language Processing (NLP) thanks to the advances in machine learning. A lot of attention has been paid to architectures, especially for deep learning, as well as to loss functions. Notably, loss functions based on similar ideas were proposed in unrelated papers in different machine learning fields under different names. This can cause difficulties when solving new problems or when designing new experiments based on previous results. To a greater extent, this applies to "universal" loss functions, which can be applied in different machine learning areas and tasks such as Computer Vision (CV), Recommendation Systems, and NLP. An example of such loss function is the batch-softmax contrastive (BSC) loss, which we will discuss below. For many NLP tasks, it is important to obtain representations of sentences for semantic matching problems, since they can be used for further analysis, e.g., for finding the best answer to a question.
Sentence BERT is a recent popular approach for this (Reimers and Gurevych, 2019): it can be trained with different loss functions, and we show that the choice of a loss function is important. Moreover, we show that it will not be optimal to take the "standard" batch-softmax contrastive loss, which is used for training SimCSE (Gao et al., 2021), a recent alternative to Sentence BERT, and we suggest ways to improve its efficiency. Our contributions can be summarized as follows:
- We study the use of a batch-softmax contrastive loss for fine-tuning large-scale transformers to learn better task-specific embeddings for pairwise sentence scoring tasks.
- We introduce and study a number of novel variations in the calculation of the loss such as symmetrization, incorporating labeled negatives, aligning scores on the similarity matrix diagonal, normalizing over the batch axis, as well as in the overall training procedure, e.g., shuffling, trainable temperature, and sequential pre-training.
- We demonstrate sizable improvements for a number of pairwise sentence scoring tasks such as classification, ranking, and regression.
- We offer detailed analysis and discussion, which would be useful for future research.
- We release our code at https://github.com/as Chern/BSC-loss
The contrastive loss was proposed by Hadsell et al. (2006) as metric learning that contrasts Euclidean distances between embeddings of samples from one class and between samples from different classes. Weinberger et al. (2006) suggested the triplet loss, which is based on a similar idea, but uses triplets (anchor, positive, negative), and aims for the difference between the distances for (anchor, positive) and for (anchor, negative) to be larger than a margin.
Human Language Technologies, pages 116 - 126
$N$ -pair loss was presented as a generalization of the contrastive and the triplet losses as a way to solve the problem of extensive construction of hard negative pairs and triplets (Sohn, 2016). A batch of  $N$  pairs of examples from  $N$  different classes is sampled, and the first element in each pair is considered to be an anchor. Thus, for each anchor, there are one positive and  $N - 1$  negative pairs. The loss contrasts the distances simultaneously using the softmax function over dot-product similarities. The approach was used successfully in CV tasks.
The same method of Multiple Negative Ranking for training Dot-Product Scoring Models was applied to ranking natural language responses to emails (Henderson et al., 2017), where the loss uses labeled pairs. A similar idea, called Negative Sharing, was used to reduce the computational cost when training recommender systems (Chen et al., 2017). Wu et al. (2018) presented an approach with  $N$ -pairs like logic, as a Non-Parametric Softmax Classifier, replacing the weights in the softmax with embeddings of samples from such classes. It was also proposed to use L2 normalization and temperature. Yang et al. (2018) used Multiple Negative Ranking to train general sentence representations on data from Reddit and SNLI. Logeswaran and Lee (2018) presented a Quick-Thoughts approach to learn sentence embeddings, which constructs batches of contiguous sets of sentences, and for each sentence, contrasts the next sentence in the text and all other candidates.
A lot of subsequent work focused on maximizing Mutual Information (MI). van den Oord et al. (2018) presented a loss function based on Noise-Contrastive Estimation, called InfoNCE. It models the "similarity" function that estimates the MI between the target (future) and the context (present) signals, and maximizes the MI between temporally nearby signals. If this "similarity" function expresses the dot-product between embeddings, the InfoNCE loss is equivalent to the  $N$ -pair loss up to some constants. It was also shown that InfoNCE is equivalent to the Mutual Information Neural Estimator (MINE) up to a constant (Belghazi et al., 2018), which maximizes a lower bound on MI. Deep InfoMax (DIM) (Hjelm et al., 2019) improves MINE, and can be modified to incorporate some autoregression as InfoNCE. However, the effectiveness of loss functions such as DIM and InfoNCE might be primarily connected to MI, rather than to deep metric learning (Tschannen et al., 2020).
The idea gained a lot of popularity in the field of Computer Vision with the advent of SimCLR (a Simple framework for Contrastive Learning of visual Representations), which introduced NT-Xent (normalized temperature-scaled cross-entropy loss) (Chen et al., 2020). It uses self-supervised learning, where augmentations of the same image are used as positive examples and augmentations of different images are used as negative examples. Thus, the task is as follows: for each example in a batch, find its paired positive augmentation. Here, the  $N$ -pairs loss is modified with a temperature parameter and with an L2 normalization of embeddings to the unit hypersphere. The loss was further extended for supervised learning as SupCon loss (Khosla et al., 2020), which aggregates all positive examples (from the same class) in the softmax numerator.
Subsequently, similar kinds of loss functions were also introduced to the field of Natural Language Processing (NLP). Gunel et al. (2021) combined the SupCon loss with the cross-entropy loss and obtained state-of-the-art results for several downstream NLP tasks using RoBERTa. Giorgi et al. (2021), Fang and Xie (2020) and Meng et al. (2021) used NT-Xent to pre-train Transformers, considering spans sampled from the same document, sentences augmented with back-translation as positive examples, and sequences corrupted with MLM. Luo et al. (2020) proposed to use NT-Xent in a self-supervised setting to learn noise-invariant sequence representations, where sentences augmented with masking were considered as positive examples. Finally, Gao et al. (2021) introduced the SimCLR loss to NLP under the name SimCSE (Simple Contrastive Learning of Sentence Embeddings), where sentences processed by a neural network with dropout served as augmentations of the original sentences. Here, we explore various ways to use a similar loss function for pairwise sentence scoring tasks.
While the above-described loss functions have different names, they are all based on similar ideas. Below, we will use the name Batch-Softmax Contrastive (BSC) loss, which we believe reflects the main idea best. In our experiments below, we will use the "modern" variant of the loss: with temperature, normalization, and symmetrization components (described in more detail in Section 3.1). These components were not used for NLP in combination before.
Figure 1: For the set of positives pairs  $(q_{i},a_{i})$ , e.g., question-answer, for each  $q_{i}$ , the BSC loss contrasts the scores between  $q_{i}$  and  $a_{i}$  (positive examples) vs. between  $q_{i}$  and  $a_{j}$  for all  $j\neq i$  (negative examples) using softmax. Here  $\blacksquare$  denotes the dot-product.
We further introduce a number of novel and important modifications in the definition of the loss and in the training procedure, which make it more efficient, and we show that using the resulting loss yields better task-specific sentence embeddings for pairwise sentence scoring tasks.
3.1 Batch-Softmax Contrastive (BSC) Loss
Pointwise approaches for training models for pairwise sentence scoring tasks, such as mean squared error (MSE), are problematic as the loss does not take the relative order into account. For instance, for two pairs with correct target scores (0.4, 0.5), the loss function would equally penalize answers like (0.3, 0.6) and (0.5, 0.4). However, the first pair is better, as it keeps the correct ranking, while the second one does not. This is addressed in pairwise approaches, e.g., in triplet loss, where the model directly learns an ordering. Yet, there is a problem for constructing pairs or triplets in the training set, as it is hard to find non-trivial negatives examples.
Unlike traditional pairwise loss functions, the BSC loss treats all other possible pairs of examples in the batch as "negatives." That is, only positive pairs are needed for training. Consider a batch  $X$  of pairs from a question-answering dataset. In general, let  $Q_{m\times n}$  and  $A_{m\times n}$  be the matrices of embeddings produced by a query model and an answer model. We define the loss function as follows:
$$
\begin{array}{l} \mathcal {L} _ {B S C} (X) = \mathcal {L} _ {0} (X) + \mathcal {L} _ {1} (X) \\ = - \operatorname {m e a n} \left(\log \left(d i a g \left(\operatorname {s o f t m a x} \left(\frac {Q A ^ {T}}{\tau}\right)\right)\right)\right) \\ \left. - \operatorname {m e a n} \left(\log \left(d i a g \left(\operatorname {s o f t m a x} \left(\frac {A Q ^ {T}}{\tau}\right)\right)\right)\right)\right) \tag {1} \\ \end{array}
$$
Here, softmax is applied by rows (Figure 1), and  $\tau$  is the temperature. Both components can be rewritten, e.g.,  $\mathcal{L}_0(X)$  can be written as follows:
$$
- \frac {1}{m \tau} \sum_ {i = 1} ^ {m} q _ {i} ^ {T} a _ {i} + \frac {1}{m} \sum_ {i = 1} ^ {m} \log \sum_ {j = 1} ^ {m} \exp \left(\frac {q _ {i} ^ {T} a _ {j}}{\tau}\right) \tag {2}
$$
Mathematically, this loss function is similar to the one presented in (Chen et al., 2020). The difference is that we do not use augmentations, and we do not compare  $q_{i}$  to  $q_{j}$  (or  $a_{i}$  to  $a_{j}$ ) due to their different nature: we want to compare a question to an answer, not a question to a question or an answer to an answer. Thus, we apply the symmetrization in the formula. So, the difference from SimCSE (Gao et al., 2021) is that we compare not only  $q_{i}$  to all  $a_{j}$ , but also  $a_{i}$  to all  $q_{j}$  in the batch.
Note that, although a frequent short answer may fit multiple questions in a batch, such pairs are considered as "negative" examples in the loss. However, the loss learns Mutual Information (Tschannen et al., 2020), that is  $p(q_i, a_i) / (p(q_i)p(a_i))$ , and thus it is robust to this false negatives problem.
Early research has already shown the importance of properly configuring and using some BSC loss settings. For example, low temperatures are equivalent to optimizing for hard positives/negatives (Khosla et al., 2020), while L2 normalization of vectors to the unit hypersphere along with temperature effectively weighs different examples (Chen et al., 2020). We further propose a number of important modifications that can have a major impact on the performance for a number of tasks.
3.2 Batch Construction
In computer vision, it is common to use a batch size of 5,000, which in turn would naturally be very likely to contain some hard negative examples. In NLP, fine-tuning Transformers with large batch sizes requires very large amounts of memory. Thus, much smaller batches are used in practice, and it becomes important to make sure these batches do contain some hard negative examples. We achieve this by fixing the content of the batches at each training epoch. Note that this is much simpler than mining hard negatives, as we only need to increase the likelihood that there would be a hard negative example present in the batch, but we do not need to know which particular example in the batch would be hard. Inside the batch, this would be controlled by the temperature parameter.
Example-based shuffling The key idea of this method is to batch several groups, so that within each group all pairs are similar based on their first or their second elements. In this way, each positive pair would be accompanied by hard negatives from the same group and by simpler negatives from the remaining examples inside the batch (which come from other groups). We use the  $k$ -nearest neighbors for an input example to form a group for it, and Faiss (Johnson et al., 2021) to quickly find these nearest neighbors in the embedding space. Let the pairs be grouped by their first elements  $q_{i}$ . Algorithm 1 summarizes the proposed method.
Note that in this approach we use two stages in kNN to limit the range of possible candidates and thus to reduce the computational costs (both in terms of time and memory). We first extract the top- $n$  neighbors (for some large  $n$ , e.g., 500), and then we take the top- $k$  from them, so that no duplicates appear in the final sequence (for some small  $k = 7$ ). The time complexity of such a check is O(1). If all such neighbors are already used, then only the considered example will be added to the resulting sequence. This case will often arise for the last examples, and thus batches will consist of simple 1-element groups. Therefore, we reverse the sequence to start with these simple batches, as in curriculum learning.
By default, we assume that there should be one positive example for each question/answer (on the diagonal of the matrix), and thus identical neighbors could be optionally filtered. Still, if there are the same  $q_{i}$  in the batch  $X$ , the loss definition (eq. 1) does not change. Indeed, let  $P_{q} = \{i \mid q_{i} = q, (q_{i}, a_{i}) \in X\}$ , then  $\forall i, j \in P_{q}: (q_{i}, a_{j})$  form a positive pair. According to Khosla et al. (2020), for each  $q$ , all  $\tilde{q} \in P_{q}$  should be placed in the softmax numerator and then averaging over all such  $\tilde{q}$  should be performed outside the logarithm.
Thus, in  $\mathcal{L}_0(X)$  (eq. 2) only the first sum would change as follows:
$$
\begin{array}{l} \sum_ {i \in P _ {q}} q _ {i} ^ {T} a _ {i} \rightsquigarrow \sum_ {i \in P _ {q}} \frac {1}{| P _ {q} |} \sum_ {j \in P _ {q}} q _ {i} ^ {T} a _ {j} \\ = \sum_ {j \in P _ {q}} \frac {1}{\left| P _ {q} \right|} \sum_ {i \in P _ {q}} q ^ {T} a _ {j} = \sum_ {j \in P _ {q}} q _ {j} ^ {T} a _ {j} \tag {3} \\ \end{array}
$$
In  $\mathcal{L}_1(X)$ :
$$
\begin{array}{l} \sum_ {i \in P _ {q}} q _ {i} ^ {T} a _ {i} \rightsquigarrow \sum_ {i \in P _ {q}} \frac {1}{| P _ {q} |} \sum_ {j \in P _ {q}} q _ {j} ^ {T} a _ {i} \\ = \sum_ {i \in P _ {q}} \frac {1}{\left| P _ {q} \right|} \sum_ {j \in P _ {q}} q ^ {T} a _ {i} = \sum_ {i \in P _ {q}} q _ {i} ^ {T} a _ {i} \tag {4} \\ \end{array}
$$
In order to select the groups even better, we consider task-specific embeddings. To this end, we apply the current model to encode all pairs at each epoch.
Fast shuffling For extremely large datasets, example-based shuffling is time-consuming even with Faiss; thus, we propose several effective options to perform a less-thorough shuffling. We choose some attribute by which we will group the examples, that is, we guarantee some closeness of the examples. Thus, the examples are close if they share the same words, the same cluster number or the same nearest neighbors.
First, consider the case of words and grouping by the first elements of the pairs (the case of second elements is the same). Algorithm 2 presents the shuffling process. We choose a random word or phrase (shingle) in each element and sort the dataset by these shingles. Since examples with the same shingles will be in a sequence, we assign group IDs sequentially. We split a group if it is too large. Then we shuffle (sort) the dataset by group IDs. The resulting algorithmic complexity is equal to the complexity of two sorts of the dataset.
To produce a shuffle by clusters, we apply the same algorithm, where each sentence is replaced by its cluster number. Thus, each individual shingle is of size  $t = 1$ . To make a shuffle by nearest neighbors, we create shingles by "sentences," where the words are the positions of the top- $k$  nearest neighbors in the input sequence (for some small value of  $k$ ). All these approaches, including  $k$ -means clustering, can be effectively implemented using MapReduce and parallel computations.
3.3 Labeled Negatives
Usually, when the data size is small, hard negative examples may be hard to obtain even with data shuffling, e.g., when all examples are semantically distant. Nonetheless, if the dataset contains a labeled negative pair with some anchor, then its elements are semantically close by traditional rules of dataset construction. Thus, using such a pair inside the batch, where this anchor is present, will add the necessary hard negative example.
The only change that is added in the loss function is the masking of negative examples—we have no guarantees that the selected negative example is closer to the anchor than the rest of the examples inside the batch. Let  $y_{i}$  be a binary label, where  $y_{i} = 1$  if the  $i$ -th pair is positive. Then, we have
$$
\begin{array}{l} \mathcal {L} _ {0} (X) = - \frac {1}{m \tau} \sum_ {i = 1} ^ {m} \mathbb {1} [ y _ {i} = 1 ] q _ {i} ^ {T} a _ {i} \\ + \frac {1}{m} \sum_ {i = 1} ^ {m} \mathbb {1} [ y _ {i} = 1 ] \log \sum_ {j = 1} ^ {m} \exp \left(\frac {q _ {i} ^ {T} a _ {j}}{\tau}\right) \tag {5} \\ \end{array}
$$
3.4 Combo Loss
Theoretically, it is beneficial to use several loss functions for training if they are calculated on the same batch (and thus do not require additional computations). That is, joint training of BSC and MSE losses combines the advantages of pointwise and of pairwise approaches.
This ensures that for positives examples, the values on the diagonal of the dot-product matrix are not only greater than the rest, but are also close to 1 or to some target similarity. Note that here  $L_{MSE}(X) = \frac{1}{m}\sum_{i}^{m}((q_i^T a_i) - y_i)^2$  for target positive similarities  $y_{i}$ , and thus we do not force all other similarities to zero. At the same time, the BSC loss adds new examples (i.e., "negative" pairs) to the training set.
In order to use the BSC loss when training a model for tasks with non-binary labels, we modify the indicator function in equation (5) as  $\mathbb{1}[y_i > t]$ , where  $t$  is a configurable binarization threshold. Then, we use their convex combination with a configurable hyperparameter  $\mu \in (0,1)$ :
$$
L (X) = \mu L _ {B S C} (X) + (1 - \mu) L _ {M S E} (X) \tag {6}
$$
3.5 Normalization
L2 normalization of matrices  $A$  and  $B$  means that  $a_{i}^{T}b_{j}$  will be equivalent to cosine similarity. The embeddings can also be normalized by the batch dimension (by coordinates), which can bring additional regularization. In our experiments, we confirm the importance of this, e.g., new representations can be calculated with L2 normalization by coordinates or in a min-max scale.
NLP tasks that compare pairs of sentences can be roughly divided into three general categories: regression (predicting a similarity score), classification (e.g., similar vs. dissimilar), and ranking (search for the best matches). They differ only by the quality assessment functions, and thus they all can benefit from the above losses.
Note that it is important to calculate the sentence representations for ranking tasks in advance, as when independently calculating the embeddings of the individual elements in the pairs, the inference time of the model becomes linear instead of quadratic. Therefore, we use Sentence-BERT (SBERT), which is trained as a Siamese BERT model, and offers a way to obtain state-of-the-art sentence embeddings, which have been proven useful for a number of tasks (Reimers and Gurevych, 2019; Thakur et al., 2021). At inference time, we first use SBERT to obtain independently a representation for the individual sentences, and then we calculate the cosine similarities between the corresponding embeddings as we construct pairs as part of the process of ranking.
We use a variety of public English datasets and tasks for the evaluation. In particular, we have four ranking tasks (ranking answers to non-factoid questions, ranking questions by their similarity with respect to other questions, ranking comments by their similarity to a given question, ranking fact-checked claims by their relevance with respect to an input claim), two binary classification tasks (paraphrases identification, and duplicate question identification), and one regression task (semantic sentence similarity).
Antique The dataset contains 2,626 non-factoid questions with answer choices (Hashemi et al., 2020), asked by users on Yahoo! Answers. There are a total of 34,011 question-answer pairs: 27,422 for training and 6,589 for validation. Each answer is annotated with a relevance score with respect to the question on a scale from 1 to 4 (where 4 denotes the highest relevance), and the task is to rank the answers by their relevance. To model relevance as a cosine similarity, we normalize the scores to the [0, 1] interval. We use Mean Reciprocal Rank (MRR) as the main evaluation measure.
CQA-A This dataset was used in SemEval-2017 Task 3 on Community Question Answering subtask A (Nakov et al., 2017). The goal is to rank the first ten answers in a question thread on Qatar Living, so that good answers are ranked higher than bad ones. We used the clean part of the dataset, which consists of 14,110 and 2,440 labeled question/comments pairs for training and development, respectively. The evaluation measure is Mean Average Precision (MAP). This dataset contains important metadata, e.g., the date and time of the comment, and sorting the comments by time yields a strong baseline; yet, we only use the text. To train the model with the triplet loss, we group the pairs by the first element (anchor).
CQA-B This dataset was developed for SemEval-2017 Task 3, subtask B (Nakov et al., 2017), whose goal was to rank ten potentially related questions by their similarity with respect to an input question. These questions were retrieved from the Qatar Living forum using Google and the input question as a query. We use the clean part of the dataset, which consists of 19,990 training and 5,500 development labeled question-question pairs. The main evaluation measure here is MAP. There is additional information, e.g., the rank of the retrieved question in the Google search results, which we do not use.