# Extracting Sentence Embeddings from Pretrained Transformer Models
## 2 Related Work
### Learning Sentence Embeddings Directly
#### 2.3.3 Contrastive Learning Approaches  
The search for better training tasks and data utilization culminated in a new area of contrastive learning. Distance-based contrastive loss [178] is more attractive for learning sentence-level embeddings than the more conventional error prediction losses. It allows for a simple construction of self-supervised learning using pairs of positive and negative examples. Embeddings of the first group of samples are encouraged to be the same (by utilizing a loss on a distance function between the text vectors), while the negative ones are encouraged to be different. In this way, the burden of expensive labeling can be relieved. Moreover, it can be used as an intermediate step between pre-training and fine-tuning to inexpensively align the model to the target task domain. These properties are especially useful for the \"data-hungry\" transformer models. The models in this class differ mainly in how they construct the positive example pairs. This can be achieved in many different ways. We will describe them below. Feature/Vector/Embedding-Level Augmentations  
These are the modifications to the sentence in its vector representation space. Dropout [179] is one of the most popular feature-level augmentation techniques. It was originally used as a regularization of neural networks to increase the robustness to noise from the vector space. However, the same can be applied to derive an augmented text sequence representation. Popular BERT and RoBERTa pre-trained transformer models already have dropout layers and require almost no modifications to employ the dropout augmentation. Two positives are acquired by passing the same sample through the dropout-enabled network twice [143; 180; 181]. This does not require additional preprocessing and can be applied on the fly during the training. Despite the effectiveness reported, dropout augmentation has the disadvantage of being biased toward sentence length. This was observed in [181]. Two augmented versions of the same sentence are of the same length and can be easily discriminated against randomly drawn negatives of varying lengths. Therefore, instead of learning the general sentence representation to match the same sentence samples, the model now just takes the shortcut by only comparing their lengths. To counteract this, the authors of [181] added token-level augmentations in addition to dropout so that the length of positives would be different. Other authors of [182] also managed to avoid this problem by using negatives produced by a much higher dropout rate than those used for positives. Another feature-level (also regarded as token-level) augmentation technique is the shuffle of tokens. Usually in pre-trained language models, positional information of each sequence element is brought about by the addition of special positional embeddings to the existing token ones. As a result, the shuffling of positions can be implemented simply as the shuffling of position IDs and the corresponding positional vectors. Similarly to dropout, there is a cutoff augmentation [183]. Dropout can also be considered as random erasing of weights in \(L\times d\) text sequence embedding matrix with \(L\) tokens of length \(d\) vectors. The token cutoff erases randomly selected rows, while the feature cutoff erases columns of the \(L\times d\) matrix. In the work of [PERSON] [63] it was found that the shuffle and token cutoff are the two most effective augmentation strategies, significantly outperforming the feature cutoff and dropout. There are more sophisticated approaches to alter embeddings than a dropout randomly replacing weights with zero values. Specifically, adversarial attacks were shown to be successful. They aim to add worst-case perturbations to the input samples. The authors of [63] used the Fast Gradient Value (FGV) [184] method, which unfortunately relies on supervised loss to compute adversarial perturbations. Other authors of [185] employed the Fast Gradient Sign Method (FGSM) [186].