# Extracting Sentence Embeddings from Pretrained Transformer Models
## 2 Related Work
### Token-Level Augmentations  
Language gives us many possibilities to convey almost the same meaning through similar or very different sequences of words. This can be easily utilized for augmentation purposes. Using synonym replacement, random swapping of two consequent words, random insertion and deletion, the so-called Easy Data Augmentation (EDA) the authors of [187] managed to reduce the required dataset size by half for the same performance in multiple classification tasks. In contrastive learning experiments for the biomedical relation extraction task, the authors of [188] found that synonym replacement outperformed random swap and random deletion. Meanwhile, one of the first contrastive works for text [189] showed that back-translation [190] augmentation outperformed the four mentioned techniques. The authors took a pre-trained BERT [5] and pre-trained it further on the input text of GLUE [191] tasks in a contrastive fashion, to be later fine-tuned and evaluated in GLUE. The resulting CERT model achieved slight improvements in addition to the regular BERT. CLEAR [192] was also an attempt to improve the regular BERT and RoBERTa pre-training by jointly using MLM loss and constrastive loss. The authors found that the different combinations of different augmentation strategies (in particular, substitution, span-deletion, and reordering) favor different improvements for GLUE and various SentEval tasks. A recent work [185] tried token-level augmentations such as typos, synonym replacement, paraphrasing, and back-translation but found that they underperformed feature-level ones. As the popular BERT model is pre-trained with a masked language modeling task, i.e., predicting the true masked token behind the special input [MASK] token, the same [MASK] can also be used to induce augmentations by randomly masking tokens. The mirror-BERT model of [180] used such random span masking for input augmentation, which, together with feature augmentation, showed gains over off-the-shelf models in both lexical and sentence-level tasks, across different domains and different languages. Token-level augmentations can greatly contribute to existing vector-level ones. Feature-level augmentations do not affect the length of the text sequence and are therefore susceptible to length bias. ESimCSE [181] showed that simple word repetition augmentation can effectively counteract it. The authors also tested the insertion of the [MASK] token with small improvements, while the insertion of stop words slightly decreased the effect. A better distinction of negatives can also be obtained by token augmentations. The authors of [92] proposed Bidirectional Margin Loss (BML) to incorporate soft-negative samples that are generated using a simple rule-based method. According to its dependency syntax tree, the positive sentence is converted to its negation with correct syntax and clear semantics. The proposed setup saw improvements in the semantic textual similarity tasks from SentEval. One drawback of token-level augmentations is the chance of producing false positives. It is especially risky in stochastic modifications, such as back-translation, deletion, or insertion, where exact output cannot be controlled. Furthermore, token-level augmentations are more complicated than feature-level ones, and thus they cannot be performed on-the-fly during the training and must be prepared in advance. Positives by Relative Placement  
Similar to the older idea of skip-thoughts [168], the placement of sentences in a text can be exploited. In such a setting, text segments near or overlapping each other in the long text should also be near each other in the sentence embedding space. Therefore, text parts near the same anchor in the text can be regarded as positive, while further away as negative pairs for contrastive learning. DeCLUTR [193] for each document in a mini-batch samples multiple anchor spans. The corresponding positives are further sampled for each anchor and can be partially overlapping, adjacent, or subsumed by the anchor.