SlowGuess/NAACL_2022_OCR huggingface dataset:
PFCC-S Shaar et al. (2020) presented a dataset for detecting Previously Fact-Checked Claims on Snopes (PFCC-S), aimed at facilitating the solution of a fact-checking problem: given an input claim, it asks to rank claims that have been previously fact-checked, so that claims that can help verify the input claim are ranked as high as possible. The dataset has 800 positive input-verified claim pairs for training and 200 such positive pairs for testing, and they are to be matched against a database of 10,369 verified claims. The evaluation is performed in terms of a HasPositive@k metric, which checks whether there is a positive match among the first  $k$  results in the ranked list. There are no negative examples in the dataset, but about eight million such pairs can be created. Thus, in order to train models using MSE or triplet loss, we sampled negatives according to the following scheme. First, we encoded all sentences using SBERT (we used the model pretrained on STSb and NLI). Then, we selected the first element in each positive pair as an anchor and we sorted all other examples by their similarity to this anchor. The assumption is that positive examples will be concentrated in the beginning, e.g., among the top-100. Thus, we selected negatives starting from 101 on, logarithmically: on positions  $100 + 2^k$ ,  $k \in \mathbb{N}$ . As a result, we obtained many hard negative examples and a small number of easy ones. Finally, we oversampled the positive pairs to correct the balance of positive and negative examples.
Microsoft Research Paraphrase Corpus (MRPC) It was created by Dolan et al. (2004) and contains 5,800 pairs of sentences, extracted from online news sources. Each pair was labeled with a tag indicating whether the sentences are paraphrases (semantically equivalent). There are 3,668, 407, and 1,725 pairs in the training, development, and test subsets. As it is a binary classification task with class imbalance, it is evaluated in terms of F1.
Quora Question Pairs (QQP) Quora presented a dataset containing over 500,000 sentences with over 400,000 lines of potential duplicate questions. Each line has a binary label indicating whether the line truly contains a duplicate pair. Due to the sampling method, which returns mostly positive pairs, the authors supplemented the dataset with negative pairs composed of "related questions." Following Thakur et al. (2021), we sampled randomly 10,000 examples for training, and we used F1 score as the main evaluation measure.
Semantic Textual Similarity Benchmark (STSb) The STS benchmark comprises a selection of English datasets used in the STS tasks organized in the context of SemEval between 2012 and 2017 (Cer et al., 2017). The benchmark comprises 8,628 sentence pairs. The pairs were annotated with similarity scores on a scale from 0 to 5 (with 5 indicating complete equivalence). There are a total of 5,749, 1,500 and 1,379 pairs in the training, in the development, and in the testing split, respectively. The main evaluation measure is Spearman's rank correlation. As for the Antique dataset, here we also normalized the scores to the [0, 1] interval and then we binarized them based on a threshold of 0.6.
Below, we describe our baselines and hyperparameter settings.
We used BERT-base uncased in all our experiments to be able to perform direct comparison for tasks such as MRPC, QQP, and STS to previous work (Reimers and Gurevych, 2019; Thakur et al., 2021).
We considered SimCSE (sup-simcse-bert-base-uncased checkpoint) (Gao et al., 2021) as an unsupervised baseline as it uses the base version of the BSC loss, which we modified. Below, by BSC we will denote using optimal settings in the tables, and variants like BSC - random shuffle would mean that instead of these optimal settings, we applied random shuffling.
5.2 Hyperparameters Setting
We set the number of warm-up steps to  $10\%$  of the total steps, and we limited the input sequence length to 90 subtokens. We used a batch size of 30 in all tasks, except for Antique and QQP, where we used 50. Note that an order of magnitude larger batch sizes would probably yield better results, but they would also require much more memory. We tried learning rates from  $\{5\mathrm{e} - 6,1\mathrm{e} - 5,2\mathrm{e} - 5,3\mathrm{e} - 5\}$  and we selected (on dev) 3e-5 for CQA-B and 2e-5 for all other experiments. We used the AdamW optimizer with the bias correction for the CQA tasks, and without bias correction for the rest. We trained the model for five epochs for Antique, CQA-A and STSb, for six epochs for QQP, MRPC and PFCC-S, and for seven epochs for CQA-B, saving a checkpoint after each one, and we selected the best checkpoint on the development set.
As recommended by Thakur et al. (2021), due to instability, we did seed optimization, running each experiment five times and selecting the best one (on dev). To train with the BSC loss, we used min-max normalization by coordinates with  $\tau = 1.2$  for PFCC-S and QQP, standard L2 normalization with  $\tau = 0.055$  for CQA-A,  $\tau = 0.07$  for CQA-B, and  $\tau = 0.1$  for all other tasks (to find the optimal  $\tau$ , we made it trainable for one run). We applied example-based shuffling to train with the BSC loss. We used a group size of four in MRPC, of five in CQA-B, and of eight in all other tasks. We iterated over  $\mu$  values from the set  $\{0.1, 0.5, 0.9\}$ , and we chose  $\mu = 0.1$  to train the combo approach for CQA-A, MRPC, QQP and STSb tasks, and  $\mu = 0.9$  for the other experiments.
Our experiments demonstrated that there is only a minor impact of changing the values of these hyperparameters. For example, the maximum quality drop from  $\mu$  (0.1 vs. 0.5) is  $0.4\%$ , i.e., almost at the noise level, while we obtained gains of up to  $10\%$  when using our proposed modifications.
We trained the triplet loss variant from (Reimers and Gurevych, 2019) with margin  $= 0.6$  for PFCC-S, and margin  $= 0.5$  for all other tasks. As we have no answers for the test set in MRPC, and no test sets in Antique and PFCC-S, we split the training set into 9:1 to tune the hyper-parameters. The time for training SBERT with the BSC loss (or combo loss) was almost equal to the time for training with the standard MSE loss. The most complex example-based shuffling only adds  $8\%$  to the training time. Note that it is several times faster than triplet loss, as only positive examples are used for training, and thus the dataset becomes several times smaller. We ran all experiments on a GeForce GTX 1080 GPU Ti. Training SBERT-base took up ten minutes per epoch depending on dataset size.
Below, we describe the evaluation results.
Antique The results are shown in Table 1. Our best approach of combo-training the MSE and the BSC losses outperforms all other variants and the approach proposed in (Hashemi et al., 2020), where specific negative sampling and a triplet loss were used. Besides, the best BSC configuration achieves higher scores than MSE. We can see the importance of using predefined hand-crafted negative examples, which brings additional difficult cases and increases MRR by 0.02.
Approach / Metric	MRR	P@1	nDCG@1
MSE	0.781	0.660	0.769
BSC	0.804	0.680	0.754
BSC - positives	0.784	0.655	0.744
BSC - random shuffle	0.799	0.670	0.754
Combo BSC + MSE	0.822	0.710	0.773
SimCSE (unsup.)	0.681	0.525	0.686
Hashemi et al. (2020)	0.797	0.709	0.713
Table 1: Results for Antique.
Approach / Metric	MAP	MRR	MAP	MRR
MSE	0.869	0.911	0.471	0.513
BSC	0.801	0.867	0.495	0.534
BSC - clusters shuffle	0.787	0.859	0.493	0.534
BSC - random shuffle	0.763	0.828	0.487	0.530
BSC - w/o shuffle	0.816	0.884	0.481	0.532
Combo BSC + MSE	0.872	0.912	0.496	0.540
Triplet loss	0.857	0.917	0.475	0.529
SimCSE (unsup.)	0.684	0.735	0.439	0.478
Nakov et al. (2017)	0.884	0.928	0.472	0.501
CQA-A The results for CQA subtask A are shown in Table 2. A comparison with (Nakov et al., 2017) is not very fair, as we did not use the metadata, e.g., the comment position, which was crucial for the best systems. Besides, we use SBERT, which is inferior to a fine-tuned BERT. Nevertheless, our best approach of combo training with MSE and BSC losses yielded competitive results. We further compared different shuffling strategies. The data is ordered by questions, and keeping this order turns out to be best. That is, the model learns to distinguish positive answers for each question from manually selected negative ones and from answers to other questions. Also, note that random shuffling completely eliminates this structure, and MAP drops by  $6\%$  absolute. Fast shuffling by 300 clusters, an advanced version of shuffling by words, improves these results. Example-based shuffling finds a data order similar to the initial one, and the quality does not degrade much.
CQA-B The evaluation results for CQA-B are shown in Table 2. Again, we did not use the question position, which is a critically important feature for the best systems. We can see that the BSC loss achieved the best score, noticeably outperforming MSE and triplet losses. The experiments also demonstrate the importance of data order when training with the BSC loss. Since the dataset is small, the model overfits when the original data order is fixed.
Table 2: Results for CQA-A and CQA-B.
Approach / Metric	HP@1	HP@5	HP@50
MSE	0.362	0.508	0.709
BSC	0.673	0.844	0.899
BSC - 1-dim norm	0.588	0.764	0.899
BSC - no norm	0.608	0.744	0.884
BSC - random shuffle	0.663	0.794	0.915
Triplet loss	0.668	0.794	0.899
SimCSE (unsup.)	0.412	0.693	0.849
Shaar et al. (2020)	0.402	0.653	0.784
Table 3: Results for PFCC-S.
Approach / Metric	MRPC (F1)	QQP (F1)
MSE	89.08	74.29
BSC	86.73	73.13
Combo BSC + MSE	89.46	75.07
SimCSE (unsup.)	85.43	68.65
Thakur et al. (2021)	87.89 (88.55)	74.97 (79.77)
Table 4: Results for MRPC and QQP.
PFCC-S Table 3 shows the results for PFCC-S  $(HP@k$  stands for HasPositives  $@k)$ . Note that the scores from (Shaar et al., 2020) are for the pre-trained SBERT without fine-tuning. We observed that even when using oversampling to improve class imbalance, fine-tuning with MSE performed worse than their results. Here, we used only positives examples to train with BSC, and normalizing by the zero dimension. Overall, the approaches using BSC and triplet loss were comparable. However, the dataset size for training with the BSC loss was much smaller, which is also true for MSE. Thus, the BSC loss is faster, and preferable.
MRPC Table 4 shows the results for MRPC. MSE outperformed the BSC loss, but combo achieved a slightly higher F1 score.
QQP The results for QQP are shown in Table 4. We also show results for SBERT and augmented SBERT (in parentheses) from (Thakur et al., 2021). They trained SBERT with MSE using another random training sample, but nonetheless, the F1 score is close to ours. The combo approach outperformed separate training with BSC or MSE.
STSb Table 5 shows the results for STS. It is the only task where combo with the BSC loss was worse than MSE. This could be due to hard negatives not appearing in the batch in any of the shuffling procedures. Moreover, we observed only marginal improvement when fine-tuning with a BSC model initially trained with MSE. However, if it was pretrained with BSC up to overfitting, fine-tuning it with MSE yielded sizable improvements.
Approach / Metric	ρ × 100
MSE	84.80
BSC	83.26
Combo BSC + MSE	84.59
Fine-tuning MSE with BSC	84.95
Fine-tuning BSC with MSE	85.71
SimCSE (unsup.)	84.25
Reimers and Gurevych (2019)	84.86
Table 5: Results for STSb: Spearman rank correlation.
We highlight the following observations:
- Combo-training with BSC and MSE losses generally yields the best results (the only exception is STSb), and it outperforms the triplet loss with advanced negative sampling.
- The order in which the data is presented for training can be critical, as we have seen in the cases of CQA-A and CQA-B.
- The use of labeled negative examples generally improves the results by  $1 - 2\%$  absolute.
- Embedding normalization during training is important. Moreover, it is useful to normalize to the zero dimension (e.g., for PFCC-S).
- A temperature  $\tau$  of order 0.1 should be used with the standard normalization, and  $\tau$  of order 1-3 for coordinate normalization.
- A suboptimal setup may hurt performance by over  $10\%$ , as was demonstrated for  $(i)$  filtering out negative examples for which no positives were given in the dataset (Table 1),  $(ii)$  using poorly formed batches (highest impact in Table 2),  $(iii)$  suboptimal normalization (Table 3), and  $(iv)$  wrong temperature value.
- BSC is more suitable for ranking tasks (Antique, CQA, and PFCC-S), but it can help for other tasks (classification and regression tasks, e.g., for MRPC, QQP, and STSb) if applied as pre-training or in joint training with MSE.
Selecting a loss function is important. For instance, if the model optimizes Pearson correlation, it achieves a score of 85.57 on the STS task. Thus, it outperforms almost all considered approaches. Moreover, the combination of such a loss with BSC allows the model to achieve an F1 score of 89.88 in the MRPC task (a classification task).
Finally, we would like to draw a parallel between our work and Augmented SBERT (Thakur et al., 2021). When using the BSC loss, some negatives are implicitly added to the dataset. Augmented SBERT adds new examples too and retrieves them using BM25 or Semantic Search samplings. These methods are comparable to our fast shuffling by words ( $n$ -grams) and to example-based shuffling, respectively. Moreover, the task-specific model is used to encode the data in both cases. However, we do not need to label such pairs with another model (cross-encoder) due to the BSC loss definition.
We explored the idea of using a batch-softmax contrastive loss for fine-tuning large-scale pre-trained transformers to learn better task-specific sentence embeddings for pairwise sentence scoring tasks. We introduced and studied a number of variations in the calculation of the loss as well as in the overall training procedure. Our experimental results have shown sizable improvements on a number of datasets and pairwise sentence scoring tasks including ranking, classification, and regression.
In future work, we want to explore new variations of the loss, and to gain better understanding of when to use which variation. We further plan experiments with a larger set of NLP tasks.
Ethics and Broader Impact
We would like to warn that the use of large-scale Transformers requires a lot of computations and the use of GPUs/TPUs for training, which contributes to global warming (Strubell et al., 2019). This is a bit less of an issue in our case, as we do not train such models from scratch; rather, we fine-tune them on relatively small datasets. Moreover, running on a CPU for inference, once the model is fine-tuned, is perfectly feasible, and CPUs contribute much less to global warming.
The article was prepared within the framework of the HSE University Basic Research Program. It is also part of the Tanbih mega-project, developed at the Qatar Computing Research Institute, HBKU, which aims to limit the impact of "fake news," propaganda, and media bias by making users aware of what they are reading, thus promoting media literacy and critical thinking.
Mohamed Ishmael Belghazi, Aristide Baratin, Sai Rajeshwar, Sherjil Ozair, Yoshua Bengio, Aaron Courville, and Devon Hjelm. 2018. Mutual information neural estimation. In Proceedings of the 35th International Conference on Machine Learning, volume 80 of ICML '18, pages 531-540, Stockholm, Sweden. PMLR.
Daniel M. Cer, Mona T. Diab, Eneko Agirre, Inigo Lopez-Gazpio, and Lucia Specia. 2017. SemEval2017 task 1: Semantic textual similarity multilingual and crosslingual focused evaluation. In Proceedings of the 11th International Workshop on Semantic Evaluation, SemEval '2017, pages 1-14, Vancouver, Canada. Association for Computational Linguistics.
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. 2020. A simple framework for contrastive learning of visual representations. In Proceedings of the 37th International Conference on Machine Learning, volume 119 of ICML '20, pages 1597-1607. PMLR.
Ting Chen, Yizhou Sun, Yue Shi, and Liangjie Hong. 2017. On sampling strategies for neural network-based collaborative filtering. In Proceedings of the 23rd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, KDD '17, pages 767-776, Halifax, NS, Canada. Association for Computing Machinery.
Bill Dolan, Chris Quirk, and Chris Brockett. 2004. Unsupervised construction of large paraphrase corpora: Exploiting massively parallel news sources. In Proceedings of the 20th International Conference on Computational Linguistics, COLING '04, pages 350-356, Geneva, Switzerland. Association for Computational Linguistics.
Hongchao Fang and Pengtao Xie. 2020. CERT: Contrastive self-supervised learning for language understanding. CoRR, abs/2005.12766.
Tianyu Gao, Xingcheng Yao, and Danqi Chen. 2021. SimCSE: Simple contrastive learning of sentence embeddings. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, EMNLP '21, pages 6894-6910, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
John Giorgi, Osvald Nitski, Bo Wang, and Gary Bader. 2021. DeCLUTR: Deep contrastive learning for unsupervised textual representations. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), NAACL-HLT '21, pages 879-895, Online. Association for Computational Linguistics.
Beliz Gunel, Jingfei Du, Alexis Conneau, and Veselin Stoyanov. 2021. Supervised contrastive learning for
pre-trained language model fine-tuning. In Proceedings of the 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria.
Raia Hadsell, Sumit Chopra, and Yann LeCun. 2006. Dimensionality reduction by learning an invariant mapping. In Proceedings of the 2006 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, CVPR '06, pages 1735-1742, New York, NY, USA. IEEE Computer Society.
Helia Hashemi, Mohammad Aliannejadi, Hamed Zamani, and W. Bruce Croft. 2020. ANTIQUE: A non-factoid question answering benchmark. In Advances in Information Retrieval, pages 166-173, Cham. Springer International Publishing.
Matthew Henderson, Rami Al-Rfou, Brian Strope, Yun hsuan Sung, Laszlo Lukacs, Ruiqi Guo, Sanjiv Kumar, Balint Miklos, and Ray Kurzweil. 2017. Efficient natural language response suggestion for smart reply. ArXiv, abs/1705.00652.
Devon Hjelm, Alex Fedorov, Samuel Lavoie-Marchildon, Karan Grewal, Phil Bachman, Adam Trischler, and Yoshua Bengio. 2019. Learning deep representations by mutual information estimation and maximization. In Proceedings of the 7th International Conference on Learning Representations, ICLR '19, New Orleans, Louisiana, USA.
Jeff Johnson, Matthijs Douze, and Hervé Jégou. 2021. Billion-scale similarity search with GPUs. IEEE Transactions on Big Data, 7(3):535-547.
Prannay Khosla, Piotr Teterwak, Chen Wang, Aaron Sarna, Yonglong Tian, Phillip Isola, Aaron Maschinot, Ce Liu, and Dilip Krishnan. 2020. Supervised contrastive learning. In Advances in Neural Information Processing Systems: Annual Conference on Neural Information Processing Systems, volume 33 of NeurIPS '20, pages 18661-18673. Curran Associates, Inc.
Lajanugen Logeswaran and Honglak Lee. 2018. An efficient framework for learning sentence representations. In Proceedings of the 6th International Conference on Learning Representations, ICLR 2018, Vancouver, Canada.
Fuli Luo, Pengcheng Yang, Shicheng Li, Xuancheng Ren, and Xu Sun. 2020. CAPT: contrastive pretraining for learning denoised sequence representations. CoRR, abs/2010.06351.
Yu Meng, Chenyan Xiong, Payal Bajaj, Saurabh Tiwary, Paul Bennett, Jiawei Han, and Xia Song. 2021. COCO-LM: Correcting and contrasting text sequences for language model pretraining. In Advances in Neural Information Processing Systems: Annual Conference on Neural Information Processing Systems, NeurIPS '21, pages 23102-23114, Virtual Event.
Preslav Nakov, Doris Hoogeveen, Lluis Marquez, Alessandro Moschitti, Hamdy Mubarak, Timothy Baldwin, and Karin Verspoor. 2017. SemEval-2017 task 3: Community question answering. In Proceedings of the 11th International Workshop on Semantic Evaluation, SemEval '17, pages 27-48, Vancouver, Canada. Association for Computational Linguistics.
Nils Reimers and Iryna Gurevych. 2019. SentenceBERT: Sentence embeddings using Siamese BERTnetworks. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, EMNLP-IJCNLP '19, pages 3982-3992, Hong Kong, China. Association for Computational Linguistics.
Shaden Shaar, Nikolay Babulkov, Giovanni Da San Martino, and Preslav Nakov. 2020. That is a known lie: Detecting previously fact-checked claims. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, ACL '20, pages 3607-3618, Online. Association for Computational Linguistics.
Kihyuk Sohn. 2016. Improved deep metric learning with multi-class n-pair loss objective. In Advances in Neural Information Processing Systems: Annual Conference on Neural Information Processing Systems, volume 29, pages 1857-1865. Curran Associates, Inc.
Emma Strubell, Ananya Ganesh, and Andrew McCallum. 2019. Energy and policy considerations for deep learning in NLP. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, ACL '19, pages 3645-3650, Florence, Italy. Association for Computational Linguistics.
Nandan Thakur, Nils Reimers, Johannes Daxenberger, and Iryna Gurevych. 2021. Augmented SBERT: Data augmentation method for improving bi-encoders for pairwise sentence scoring tasks. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT '21, pages 296-310, Online. Association for Computational Linguistics.
Michael Tschannen, Josip Djolonga, Paul K. Rubenstein, Sylvain Gelly, and Mario Lucic. 2020. On mutual information maximization for representation learning. In Proceedings of the 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia.
Aäron van den Oord, Yazhe Li, and Oriol Vinyals. 2018. Representation learning with contrastive predictive coding. CoRR, abs/1807.03748.
Kilian Q Weinberger, John Blitzer, and Lawrence Saul. 2006. Distance metric learning for large margin nearest neighbor classification. In Advances in Neural Information Processing Systems: Annual Conference on Neural Information Processing Systems, volume 18 of NIPS '06, pages 1473-1480. MIT Press.
Zhirong Wu, Yuanjun Xiong, Stella X. Yu, and Dahua Lin. 2018. Unsupervised feature learning via nonparametric instance discrimination. In Proceedings of the 2018 IEEE Conference on Computer Vision and Pattern Recognition, CVPR '18, pages 3733-3742, Salt Lake City, Utah, USA. Computer Vision Foundation / IEEE Computer Society.
Yinfei Yang, Steve Yuan, Daniel Cer, Sheng-yi Kong, Noah Constant, Petr Pilar, Heming Ge, Yun-Hsuan Sung, Brian Strope, and Ray Kurzweil. 2018. Learning semantic textual similarity from conversations. In Proceedings of the Third Workshop on Representation Learning for NLP, RepL4NLP '18, pages 164-174, Melbourne, Australia. Association for Computational Linguistics.