# Extracting Sentence Embeddings from Pretrained Transformer Models
## 2 Related Work
### Token-Level Augmentations
This is accomplished by sampling the anchor span length to be mostly longer than the positive spans. At the time, DeCLUTR showed improvements in SentEval [194] transfer tasks in an unsupervised setting. Location exploitation is a simple approach, but it has some drawbacks. As the authors used the dataset consisting of documents of at least 2048 tokens in length, it clearly reveals that sentence placement methods have limited applications for short text domains. Moreover, random spans of DeCLUTR are subject to fragmentation in semantics. Positives by Two Networks  
The augmented version of the sample can also be constructed using two encoders. Different weight initialization and training data order are the most common causes of fluctuations in neural network performance. Although the performance is generally comparable, each model learned differently represents a different local minimum. Such a subtle difference can act as an augmentation. One of the first such setups was the CT model [99]. The simple objective required two models to retain similar representations for identical sentences while distinguishing their representations from different ones. The authors of [86] presented an approach of two BERT models, one fixed and the other tunable. The authors imposed a contrastive tension between the [CLS] token of the tunable model and a holistic representation of the fixed one. The latter is aggregated from all BERT layers, thus encouraging the tunable model to learn to concentrate all the relevant information to the [CLS] token. The authors of [195] proposed using multiple diverse positives instead of the usual two. The authors claim that this increases the probability of \"at-least-one\" effective positive during training. To implement diverse positives, the agreement of two similarity distributions of samples between two encoder models had to be maximized. Another setup incorporating two networks can be used for Knowledge Distillation (KD). In [196], after the regular KD step with the original pre-training task, an additional contrastive pre-training was added. The pair of positives for a single text segment consisted of a representation from the teacher model and another from the student. After the third stage of fine-tuning on semantic textual similarity tasks, [196] showed that the 110 M model outperformed the one with 11B parameters. MoCoSE [185] combines both feature-level augmentations and two branches based on asymmetric BERT encoders. While the online branch is updated through the loss gradient, the second, the target branch, is updated by the Exponential Moving Average (EMA). These discrepancies between the two branches prevented the model from collapsing and allowed the achievement of competitive results in SentEval. Although two-network approaches are conceptually similar, they require extra memory or time, which is a big drawback. Transformer models already require the latest state-of-the-art GPU with the largest available memory. Techniques such as gradient accumulation are often used to process large mini-batches sequentially, but in the contrastive learning approach, the requirement for large in-batch negatives gives additional complexity. Positives and Negatives from Supervised Data  
Up to now, we have described various augmentation techniques that modify existing unlabeled samples for use in contrastive learning. However, there are several labeled datasets that can also be turned into positives or negatives by reusing the label information. One of the first works to implement this idea was SimSCE [143]. The authors analyzed six candidate datasets. They found that the Natural Language Inference (NLI) datasets, SNLI [171] and MNLI [197], together performed the best. The combined dataset contains sentence pairs in the form of (premise, hypothesis, label), labeled as entailment, neutral, or contradiction, 314k for each class. The SimSCE authors constructed positive pairs from entailment samples and negatives using contradiction ones.