# Extracting Sentence Embeddings from Pretrained Transformer Models
## 2 Related Work
### Criticism
#### 2.2.2 Post-Processing Methods  
Many post-processing methods have been proposed to improve the embeddings. Whether they improve the isotropy of the representation space or help surface-relevant information [135], multiple authors show the benefits of using them. However, these methods generally favor unsupervised tasks, such as semantic textual similarity, with limited improvements for downstream tasks. If enough data on the target task are available, supervised training will inevitably be a better choice, as the transformation relevant to the task is learned. However, post-processing may be the only option if the training samples are obscure. The simplest is the centering operation. It is especially important for the STS task that involves cosine similarities. The authors of [124] found that it restores a nearly perfectly isotropic distribution. Subtracting the mean is also the first step in several other post-processing methods. In some sense, such operations as centering can be viewed as fine-tuning to the target domain, as all target embeddings participate in providing the mean coordinates. We already mentioned some operations related to isotropy enhancement in the previous section, such as zeroing target outliers or ABTT. In the following, we will mention the remaining important ones. Z-Score Normalization  
The Z-score describes the position of a point in terms of its distance from the mean when measured in units of standard deviation (in the distribution of the target samples). Z-score normalization, also called standardization, transforms the distribution of embeddings to have a zero mean and a unit standard deviation in all dimensions. The works in [102; 103] recommend that you consider it as a post-processing step. The authors of [116] advise using normalization, such as the z-score, in supervised settings as a binary hyperparameter, because it can lead to rank changes. All-But-The-Top (ABTT)  
Many works are based on the influential ABTT algorithm [118]. It has a hyperparameter, which tells the number of dominant directions to remove. Principle components are either completely removed or left intact. As a result, the main weakness is finding the right number of top components to remove, which can either cause information loss or eliminate an insufficient amount of noise. The authors of [136] proposed post-processing through variance normalization (PVN) to normalize the variance of leading principal components to the same level instead of total component disposal. The authors of [137] performed the removal in a softer way by employing matrix conceptors [138] in an unsupervised way. A similar technique that employs conceptors can also target specific components, such as those that incorporate social biases, and diminish them, as in [139]. Other authors of [140] proposed learning weights for each dominant direction removal. All these approaches try to preserve useful information residing in the top principal components while narrowing the target noise. Authors of [141] combined ABTT and dimensionality reduction. They found that the best pipeline is to perform ABTT twice while performing dimensionality reduction in between. In another line of work, the authors of [125; 133], instead of global post-processing, remove local dominant directions in separate clusters in the representation space. In this way, the structure of the representation space is accounted for. Similarly, the authors of [65], in light of the predecessor method of ABTT [60], proposed to eliminate the sentence-dependent principal component, where they rerank the top principal vectors based on correlation with each sentence. This individual removal of dominant directions was shown to improve performance on the STSB task. Whitening  
The authors of [13] proposed the whitening approach to alleviate the anisotropy problem of sentence embedding. The whitening operation involves centering embeddings at the origin and making different dimensions have a unit variance and be uncorrelated, turning their covariance matrix into the identity matrix.