SlowGuess/NAACL_2021_OCR huggingface dataset:
Nandan Thakur, Nils Reimers, Johannes Daxenberger and Iryna Gurevych
There are two approaches for pairwise sentence scoring: Cross-encoders, which perform full-attention over the input pair, and Bi-encoders, which map each input independently to a dense vector space. While cross-encoders often achieve higher performance, they are too slow for many practical use cases. Bi-encoders, on the other hand, require substantial training data and fine-tuning over the target task to achieve competitive performance. We present a simple yet efficient data augmentation strategy called Augmented SBERT, where we use the cross-encoder to label a larger set of input pairs to augment the training data for the bi-encoder. We show that, in this process, selecting the sentence pairs is non-trivial and crucial for the success of the method. We evaluate our approach on multiple tasks (in-domain) as well as on a domain adaptation task. Augmented SBERT achieves an improvement of up to 6 points for in-domain and of up to 37 points for domain adaptation tasks compared to the original bi-encoder performance.
Pairwise sentence scoring tasks have wide applications in NLP. They can be used in information retrieval, question answering, duplicate question detection, or clustering. An approach that sets new state-of-the-art performance for many tasks including pairwise sentence scoring is BERT (Devlin et al., 2018). Both sentences are passed to the network and attention is applied across all tokens of the inputs. This approach, where both sentences are simultaneously passed to the network, is called cross-encoder (Humeau et al., 2020).
A downside of cross-encoders is the extreme computational overhead for many tasks. For example, clustering of 10,000 sentences has a quadratic complexity with a cross-encoder and would require
Cross vs. Bi-Encoders in STSb (English)
Figure 1: Spearman rank correlation  $(\rho)$  test scores for different STS Benchmark (English) training sizes.
about 65 hours with BERT (Reimers and Gurevych, 2019). End-to-end information retrieval is also not possible with cross-encoders, as they do not yield independent representations for the inputs that could be indexed. In contrast, bi-encoders such as Sentence BERT (SBERT) (Reimers and Gurevych, 2019) encode each sentence independently and map them to a dense vector space. This allows efficient indexing and comparison. For example, the complexity of clustering 10,000 sentences is reduced from 65 hours to about 5 seconds (Reimers and Gurevych, 2019). Many real-world applications hence depend on the quality of bi-encoders.
A drawback of the SBERT bi-encoder is usually a lower performance in comparison with the BERT cross-encoder. We depict this in Figure 1, where we compare a fine-tuned cross-encoder (BERT) and a fine-tuned bi-encoder (SBERT) over the popular English STS Benchmark dataset $^2$  (Cer et al., 2017) for different training sizes and spearman rank correlation  $(\rho)$  on the test split.
This performance gap is the largest when little
$^{1}$ Code available: www.sbert.net
<sup>2</sup>http://ixa2.si.ehu.es/stswiki/index.php/STSbenchmark
Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 296-310 June 6-11, 2021. ©2021 Association for Computational Linguistics
training data is available. The BERT cross-encoder can compare both inputs simultaneously, while the SBERT bi-encoder has to solve the much more challenging task of mapping inputs independently to a meaningful vector space which requires a sufficient amount of training examples for fine-tuning.
In this work, we present a data augmentation method, which we call Augmented SBERT (AugSBERT), that uses a BERT cross-encoder to improve the performance for the SBERT bi-encoder. We use the cross-encoder to label new input pairs, which are added to the training set for the bi-encoder. The SBERT bi-encoder is then fine-tuned on this larger augmented training set, which yields a significant performance increase. As we show, selecting the input pairs for soft-labeling with the cross-encoder is non-trivial and crucial for improving performance. Our method is easy to apply to many pair classification and regression problems, as we show in the exhaustive evaluation of our approach.
First, we evaluate the proposed AugSBERT method on four diverse tasks: Argument similarity, semantic textual similarity, duplicate question detection, and news paraphrase identification. We observe consistent performance increases of 1 to 6 percentage points over the state of the art SBERT bi-encoder's performance. Next, we demonstrate the strength of AugSBERT in a domain adaptation scenario. Since the bi-encoder is not able to map the new domain to a sensible vector space, the performance drop on the target domain for SBERT bi-encoders is much higher than for BERT cross-encoders. In this scenario, AugSBERT achieves a performance increase of up to 37 percentage points.
Sentence embeddings are a well studied area in recent literature. Earlier techniques included unsupervised methods such as Skip-thought vectors (Kiros et al., 2015) and supervised methods such as InferSent (Conneau et al., 2017) or USE (Cer et al., 2018). For pairwise scoring tasks, more recent sentence embedding techniques are also able to encode a pair of sentences jointly. Among these, BERT (Devlin et al., 2018) can be used as a cross-encoder. Both inputs are separated by a special SEP token and multi-head attention is applied over all input tokens. While the BERT cross-encoder achieves high performances for many sentence pair-tasks, a drawback is that no independent sentence representations are generated. This drawback was ad
dressed by SBERT (Reimers and Gurevych, 2019), which applies BERT independently on the inputs followed by mean pooling on the output to create fixed-sized sentence embeddings.
Humeau et al. (2020) showed that cross-encoders typically outperform bi-encoders on sentence scoring tasks. They proposed a third strategy (poly-encoders), that is in-between cross- and bi-encoders. Poly-encoders utilize two separate transformers, one for the candidate and one for the context. A given candidate is represented by one vector, while the context is jointly encoded with the candidates (similar to cross-encoders). Unlike cross-encoder's full self attention technique, poly-encoders apply attention between two inputs only at the top layer. Poly-encoders have the drawback that they are only practical for certain applications: The score function is not symmetric, i.e., they cannot be applied for tasks with a symmetric similarity relation. Further, poly-encoder representations cannot be efficiently indexed, causing issues for retrieval tasks with large corpora sizes.
Chen et al. (2020) propose the DiPair architecture which, similar to our work, also uses a cross-encoder model to annotate unlabeled pairs for finetuning a bi-encoder model. DiPair focuses on inference speed and provides a detailed ablation for optimal bi-encoder architectures for performance versus speed trade-offs. The focus of our work are sampling techniques, which we find crucial for performance boosts in the bi-encoder model while keeping its architecture constant.
Our proposed data augmentation approach is based on semi-supervision (Blum and Mitchell, 1998) for in-domain tasks, which has been applied successfully for a wide range of tasks. Uva et al. (2018) train a SVM model with few gold samples and apply semi-supervision with pre-training neural networks. Another common strategy is to generate paraphrases of existent sentences, for example, by replacing words with synonyms (Wei and Zou, 2019), by using round-trip translation (Yu et al., 2018; Xie et al., 2020), or with seq2seq-models (Kumar et al., 2019). Other approaches generate synthetic data by using generative adversarial networks (Tanaka and Aranha, 2019), by using a language model to replace certain words (Wu et al., 2019) or to generate complete sentences (AnabyTavor et al., 2019). These data augmentation approaches have in common that they were applied to single sentence classification tasks. In our work,
we focus on sentence pair tasks, for which we need to generate suitable sentence pairs. As we show, randomly combining sentences is insufficient. Sampling appropriate pairs has a decisive impact on performance which corresponds to recent findings on similar datasets (Peinelt et al., 2019).
In this section we present Augmented SBERT for diverse sentence pair in-domain tasks. We also evaluate our method for domain adaptation tasks.
3.1 Augmented SBERT
Given a pre-trained, well-performing cross-encoder, we sample sentence pairs according to a certain sampling strategy (discussed later) and label these using the cross-encoder. We call these weakly labeled examples the silver dataset and they will be merged with the gold training dataset. We then train the bi-encoder on this extended training dataset. We refer to this model as Augmented SBERT (AugSBERT). The process is illustrated in Figure 2.
Figure 2: Augmented SBERT In-domain approach
Pair Sampling Strategies The novel sentence pairs, that are to be labeled with the cross-encoder, can either be new data or we can re-use individual sentences from the gold training set and re-combine pairs. In our in-domain experiments, we re-use the sentences from the gold training set. This is of course only possible if not all combinations have
been annotated. However, this is seldom the case as there are  $n \times (n - 1) / 2$  possible combinations for  $n$  sentences. Weakly labeling all possible combinations would create an extreme computational overhead, and, as our experiments show, would likely not lead to a performance improvement. Instead, using the right sampling strategy is crucial to achieve a performance improvement.
Random Sampling (RS): We randomly sample a sentence pair and weakly label it with the cross-encoder. Randomly selecting two sentences usually leads to a dissimilar (negative) pair; positive pairs are extremely rare. This skews the label distribution of the silver dataset heavily towards negative pairs.
Kernel Density Estimation (KDE): We aim to get a similar label distribution for the silver dataset as for the gold training set. To do so, we weakly label a large set of randomly sampled pairs and then keep only certain pairs. For classification tasks, we keep all the positive pairs. Subsequently we randomly sample out negative pairs from the remaining dominant negative silver-pairs, in a ratio identical to the gold dataset training distribution (positives/negatives). For regression tasks, we use kernel density estimation (KDE) to estimate the continuous density functions  $F_{gold}(s)$  and  $F_{silver}(s)$  for scores  $s$ . We try to minimize KL Divergence (Kullback and Leibler, 1951) between distributions
using a sampling function which retains a sample with score  $s$  with probability  $Q(s)$ :
$$
Q (s) = \left\{ \begin{array}{c l} 1 & \text {i f F _ {g o l d} (s) \geq F _ {s i l v e r} (s)} \\ \frac {F _ {g o l d} (s)}{F _ {s i l v e r} (s)} & \text {i f F _ {g o l d} (s) <   F _ {s i l v e r} (s)} \end{array} \right.
$$
Note, that the KDE sampling strategy is computationally inefficient as it requires labeling many, randomly drawn samples, which are later discarded.
BM25 Sampling (BM25): In information retrieval, the Okapi BM25 (Amati, 2009) algorithm is based on lexical overlap and is commonly used as a scoring function by many search engines. We utilize ElasticSearch<sup>3</sup> for the creation of indices which helps in fast retrieval of search query results. For our experiments, we index every unique sentence, query for each sentence and retrieve the top  $k$  similar sentences. These pairs are then weakly labeled using the cross-encoder. Indexing and re
<sup>3</sup>https://www/elastic.co/
trieving similar sentences is efficient and all weakly labeled pairs will be used in the silver dataset.
Semantic Search Sampling (SS): A drawback of BM25 is that only sentences with lexical overlap can be found. Synonymous sentences with no or little lexical overlap will not be returned, and hence, not be part of the silver dataset. We train a bi-encoder (SBERT) on the gold training set as described in section 5 and use it to sample further, similar sentence pairs. We use cosine-similarity and retrieve for every sentence the top  $k$  most similar sentences in our collection. For large collections, approximate nearest neighbour search like Faiss<sup>4</sup> could be used to quickly retrieve the  $k$  most similar sentences.
BM25 + Semantic Search Sampling (BM25-S.S.): We apply both BM25 and Semantic Search (S.S.) sampling techniques simultaneously. Aggregating the strategies helps capture the lexical and semantically similar sentences but skews the label distribution towards negative pairs.
Seed Optimization Dodge et al. (2020) show a high dependence on the random seed for transformer based models like BERT, as it converges to different minima that generalize differently to unseen data (LeCun et al., 1998; Erhan et al., 2010; Reimers and Gurevych, 2017). This is especially the case for small training datasets. In our experiments, we apply seed optimization: We train with 5 random seeds and select the model that performs best on the development set. In order to speed this up, we apply early stopping at  $20\%$  of the training steps and only continue training the best performing model until the end. We empirically found that we can predict the final score with high confidence at  $20\%$  of the training steps (Appendix D).
Figure 3: Domain adaptation with AugSBERT.
3.2 Domain Adaptation with AugSBERT
Until now we discussed Augmented SBERT for indomain setups, i.e., when the training and test data are from the same domain. However, we expect an even higher performance gap of SBERT on out-of-domain data. This is because SBERT fails to map sentences with unseen terminology to a sensible vector space. Unfortunately, annotated data for new domains is rarely available.
Hence, we evaluate the proposed data augmentation strategy for domain adaptation: We first finetune a cross-encoder (BERT) over the source domain containing pairwise annotations. After finetuning, we use this fine-tuned cross-encoder to label the target domain. Once labeling is complete, we train the bi-encoder (SBERT) over the labeled target domain sentence pairs (Figure 3).
Sentence pair scoring can be differentiated in regression and classification tasks. Regression tasks assign a score to indicate the similarity between the inputs. For classification tasks, we have distinct labels, for example, paraphrase vs. non-paraphrase.
4.1 Single-Domain Datasets
In our single-domain (i.e. in-domain) experiments, we use two sentence pair regression tasks: semantic textual similarity and argument similarity. Furthermore, we use two binary sentence pair classification tasks: Duplicate question detection and news paraphrase identification. Examples for all datasets are given in Table 2.
SemEval Spanish STS: Semantic Textual Similarity (STS) $^5$  is the task of assessing the degree of similarity between two sentences over a scale ranging from [0, 5] with 0 indicating no semantic overlap and 5 indicating identical content (Agirre et al., 2016). We choose Spanish STS data to test our methods for a different language than English. For our training and development dataset, we use the datasets provided by SemEval STS 2014 (Agirre et al., 2014) and SemEval STS 2015 (Agirre et al., 2015). These consist of annotated sentence pairs from news articles and from Wikipedia. As test set, we use SemEval STS 2017 (Cer et al., 2017), which annotated image caption pairs from SNLI (Bowman et al., 2015). For all our experiments, we normalise the original similarity scores to [0, 1] by dividing the score by 5.
<sup>4</sup>https://github.com/facebookresearch/faiss
<sup>5</sup>https://ixa2.si.edu.es/stswiki
Dataset	Spanish-STS	BWS (cross-topic)	BWS (in-topic)	Quora-QP	MRPC
# training-samples	1,400	2125	2471	10,000	4,340
# development-samples	220	425	478	3,000	731
# testing-samples	250	850	451	3,000	730
# total-samples	1,870	3,400	3,400	16,000	5,801
Table 1: Summary of all datasets being used for diverse in-domain sentence pair tasks in this paper.
Dataset	Sentence 1	Sentence 2	Score
BWS	Cloning treats children as objects.	It encourages parents to regard their children as property.	0.89
Quora-QP	How does one cook broccoli?	What are the best ways to cook broccoli?	1
Spanish-STS	Dos hombres en trajes rojos practicando artes marciales.	Dos hombre en uniformes de artes marcialescomings.	0.80
MRPC	The DVD-CCA then appealed to the state Supreme Court.	DVD CCA appealed that decision to the U.S. Supreme Court.	1
Table 2: Dataset examples for our in-domain tasks. We report the normalized similarity score [0, 1] for regression tasks and the binary label  $\{0, 1\}$  for classification tasks.
BWS Argument Similarity Dataset (BWS):
Existing similarity datasets have the disadvantage that the sentence pair selection/sampling process is not always comprehensible. To overcome this limitation, we create and publicly release a novel dataset for argument similarity.
We annotate sentential arguments on controversial topics on a continuous scale. We use the dataset by Stab et al. (2018), which contains pro and con stance arguments for eight controversial topics  $(T_{1} - T_{8})$  ("cloning", "abortion", "minimum wage", "marijuana legalization", "nuclear energy", "death penalty", "gun control", "school uniforms") retrieved from heterogeneous web sources.
Previous work addressing argument similarity (Misra et al., 2016; Reimers et al., 2019) used discrete scales. However, expressing an inherently continuous property in this way is counter-intuitive and potentially unreliable due to different assumptions made when binning a range of values into a discrete class (Kingsley and Brown, 2010).
Collecting continuous annotations is complex due to selection bias and due to a lack of consistency for a single annotator (Kendall, 1948). To solve the consistency problem, we apply a comparative approach, which converts the annotation into a preference problem: the annotators stated their preference on pairs of sentential arguments. We utilized the Best-Worst Scaling (BWS) method (Kiritchenko and Mohammad, 2016) to reduce the number of required annotations. For each topic
regardless of stance, all arguments were randomly paired and for ensuring a certain proportion of similar arguments within the pairings, a distant supervision filtering strategy was implemented by labeling pairs with scores between 0 and 1 using the system proposed by Misra et al. (2016). Next, all argument pairs were sampled with a desired similarity distribution, by creating argument pair bins across three categories: top  $1\%$ , top  $2 - 50\%$  and remaining pairs. As the final step, we randomly drew pairs from the top  $1\%$  with  $50\%$  probability, and with each  $25\%$  from the two other bins.
The resulting argument pairs were annotated using crowdsourcing via the Amazon Mechanical Turk Platform. For each annotation task, workers were shown four argument pairs and had to select the most and least similar pair amongst them. Each of these tasks was assigned to four different workers. To assess the quality of the resulting annotations, we used split-half reliability measure (Callender and Osburn, 1979). Workers' votes were split by half and used to independently rank all argument pairs with the BWS method for each half on each task. Finally, the Spearman's rank correlation between the resulting rankings is calculated as a proxy for consistency. The resulting average correlation across all topics in our dataset is 0.66 (random splits are repeated 25 times and final scores averaged), which, given the small number of votes per half (two), is in an acceptable range and reflects the difficulty of this task (Kiritchenko and Mohammad, 2016). Table 3 lists the mean split-half reliability estimates for all topics (averaged
$^{6}$ Public Data Release (BWS Argument Similarity Corpus): https://tudatalib.ulb.tu-darmstadt.de/handle/tudatalib/2496
over 25 random splits) in the dataset.
Topic T	Score	Topic T	Score
Cloning	0.84	Nuclear energy	0.64
Abortion	0.79	Death penalty	0.58
Minimum wage	0.50	Gun control	0.59
Marijuana legal.	0.57	School uniforms	0.64
Whole dataset = 0.66
We use the resulting BWS Argument Similarity Dataset with different splitting strategies in our paper. In cross-topic tasks, we fix topics  $(T_{1} - T_{5})$  for training,  $T_{6}$  for development and  $(T_{7}$  and  $T_{8})$  for test sets. This is a difficult task, as models are evaluated on completely unseen topics.
Note that the cross-topic experiments on this dataset are quite different from cross-domain tasks (subsection 3.2): the model fine-tunes in-domain on fixed topics ( $T_{1} - T_{5}$  in our case) and is evaluated on unseen topics, whereas in the domain adaptation experiments we fine-tune on target domain data. For in-topic, we randomly sample fixed and disjoint pairs from each and every topic ( $T_{1} - T_{8}$ ) and create our train, development and test splits with approximately equal number of pairs from each topic.
Quora Question Pairs (Quora-QP): Duplicate question classification identifies whether two questions are duplicates. Quora released a dataset containing 404,290 question pairs. We start with the same dataset partitions from Wang et al. (2017). We remove all overlaps and ensure that a question in one split of the dataset does not appear in any other split to mitigate the transductive classification problem (Ji et al., 2010). As we observe performance differences between cross- and bi-encoders mainly for small datasets, we randomly downsample the training set to 10,000 pairs while preserving the original balance of non-duplicate to duplicate question pairs.
Microsoft Research Paraphrase Corpus (MRPC): Dolan et al. (2004) presented a paraphrase identification dataset consisting of sentence pairs automatically extracted from online news sources. Each pair was manually annotated by
Table 3: Mean split-half reliability estimate is calculated using Spearman's rank correlation  $\rho$  per topic  $T$  and over the whole BWS Argument Similarity dataset.
Dataset k	Train / Dev / Test (Total Pairs)	Train (Ratio)	Dev / Test (Ratio)
AskUbuntu	919706 / 101k / 101k	1:100	1:100
Quora	254142 / 10k / 10k	3.71:100	1:1
Sprint	919100 / 101k / 101k	1:100	1:100
SuperUser	919706 / 101k / 101k	1:100	1:100
Table 4: Summary of multi-domain datasets originally proposed by Shah et al. (2018) and used for our domain adaptation experiments. Ratio denotes the duplicate pairs (positives) vs. not duplicate pairs (negatives).
two human judges whether they describe the same news event. We use the originally provided train-test splits<sup>9</sup>. We ensured that all splits have disjoint sentences.
4.2 Multi-Domain Datasets
One of the most prominent sentence pair classification tasks with datasets from multiple domains is duplicate question detection. Since our focus is on pairwise sentence scoring, we model this task as a question vs. question (title/headline) binary classification task.
AskUbuntu, Quora, Sprint, and SuperUser: We replicate the setup of Shah et al. (2018) for domain adaptation experiments. The AskUbuntu and SuperUser data comes from Stack Exchange, which is a family of technical community support forums. Sprint FAQ is a crawled dataset from the Sprint technical forum website. We exclude Apple and Android datasets due to unavailability of labeled question pairs. The Quora dataset (originally derived from the Quora website) is artificially balanced by removing negative question pairs. The statistics for the datasets can be found in Table 4. Since negative question pairs are not explicitly labeled, Shah et al. (2018) add 100 randomly sampled (presumably) negative question pairs per duplicate question for all datasets except Quora, which has explicit negatives.