# Extracting Sentence Embeddings from Pretrained Transformer Models
## 2 Related Work
### Token-Level Augmentations
The resulting supervised approach achieved state-of-the-art results at the time for SentEval tasks. The authors also tried to incorporate neutral pairs as less weighted negatives but did not observe improvements. The authors of [198] note that the use of NLI labels to construct positive and negative pairs can contradict apparent semantic information. Elements of the negative pair may not be negative in semantic space. To address this issue, the authors train the PairSupCon model by incorporating an instance discrimination objective, which is claimed to have an implicit grouping effect. The objective discriminates both hypothesis and premise sentences from the positive pair separately from all other sentences in the batch. The authors also incorporated importance weighting on negatives to facilitate the better effect of hard ones. In total, the overall loss consisted of two instances of discrimination (one for hypothesis and the other for premise) with negative weighting and a third cross-entropy predicting NLI labels. Significant improvements were shown for clustering tasks, while for semantic textual similarity tasks in SentEval, only moderate gains were shown. A novel use of NLI dataset labels was implemented in the PairSCL [199] model. First, the pair of hypothesis and premise sentences from the NLI dataset is passed through an encoder, and the unified representation of such pair is aggregated by a cross-attention module. Then a supervised contrastive loss is applied between positive and negative samples. What is interesting is that positives are regarded as hypothesis and premise pairs from the same class, i.e., whether contradiction, neutral, or entailment. Pairs labeled in different classes are considered negatives. In addition to such supervised contrastive loss, an additional cross-entropy loss is applied that predicts the actual class label. This setup showed improvements in both the NLI and SentEval transfer tasks. ST5 [200] adopted the two-stage training strategy. In the first, 2B mined question-answering data from Community QA sites were used, framing the question and answer into the positive pair. During the second stage, NLI data were used, similar to SimSCE [143]. Such a more data-rich training allowed the authors to outperform the previous approaches for the SentEval task. Furthermore, using larger models with up to 11B parameters brought even further gains. Supervised datasets are difficult to produce. If the domain of the task in question differs from publicly available NLI datasets, the easiest solution is to use unsupervised methods. Otherwise, labeled sources are indispensable. Exploiting multiple datasets and label information can make the benefits even more obvious. Direction of Contrastive Learning for NLP  
The classic setup of contrastive learning is becoming more nuanced. The usual setup of two positives and multiple in-batch negatives (just any other sentence) is not perfect. As a result, the exact distance to the negatives or between the positives varies, and learning produces only a coarse approximation. The distinction between soft, hard, and weighted negatives begins to arise [92; 143; 198], as well as the use of multiple positives [195]. The classic NT-Xent (also called InfoNCE) loss is thus often modified or additional losses added to incorporate finer training signals. The desire to scale models and datasets is also observed, as in [200], yet we think that more memory-efficient approaches, as in [196], should be prioritized. Nonetheless, contrastive learning currently produces state-of-the-art sentence embeddings.