# Extracting Sentence Embeddings from Pretrained Transformer Models
## 2 Related Work
### Composing Word Vectors
#### 2.1.8 Aggregating through Layers  
One special aggregation that is possible with newer contextualized models is through deep neural network layers. For simplicity, we refer to transformer model blocks as layers. One can imagine that \"each layer will increasingly magnify small but significant differences\" [44]. Therefore, the early layers capture more fundamental and low-level information [89], which dominates the learning of shallow lexical and meaning-related knowledge [93]. The authors of [75] reported that the word embeddings of the lower layers of BERT perform better than their upper layers on a word analogy task, and other authors of [53] showed that the first quarter of the models' layers perform best in lexical semantic understanding. In the middle layers of the transformers, the hidden states are the most transferable [94] and contain the most relevant information [95]. The authors of [96; 97] report that the middle layers of the multilingual transformers are more multilingually aligned. According to the authors of [93], with the inclusion of context in the upper layers, the encoded concepts evolve into a linguistic hierarchy where the middle and upper layers have a better representation of the core linguistic and semantic concepts. Finally, as the author of [98] explains, upper-layer representations become more context-specific. A general tendency, as shown by [99], is that the STS performance increases until the middle layers before decreasing toward the final ones. The authors of [99] even report that the final layer of most transformer models produces the worst-performing representations. Most scholars agree that this is due to the overspecialization of the last layers to the pre-training task [100]. Therefore, the final layers of BERT are the most task-specific [101] (but not as much as LSTM, as analyzed in [94]). Even during the fine-tuning phase, the authors of [89] determined that the last two layers encoded the highest share of task-specific features attributed to the score gain. Therefore, the last layers are specific to the training objective and cause problems if it differs a lot from the downstream task objective. However, some work in [102; 103] found that it was beneficial to apply post-processing to the final layers. This enabled them to restore the apparent drop in representational quality in later layers and gave large performance improvements. It is thought that anisotropy, known as the narrow cone of embeddings in vector space, may be to blame, which in [98] was found to increase from the earlier to the later layers. However, successful post-processing reveals that the last layers also possess relevant knowledge, but it is somewhat obscured in the raw outputs. The question is how to compose a universal representation from the relevant pieces in multiple layers. One way is to try various combinations. The authors of [14] analyzed all possible two-layer combinations for BERT. They found that combining the top and bottom layers is better than using only the top layers. They tried to expand it to more layers but remained on a combination of three layers (L1 + L2 + L12). From the older works, we can mention the AdaSent model [104]. It operates on a hierarchy of representations derived from a pyramid-like multilayer network structure. The gating network is trained to adaptively select the most appropriate representation in the hierarchy for the given task. A similar gating system for dynamically deciding which intermediate transformer layers to use was proposed by the authors of [105]. The contribution of each layer can also be accounted for by weighted average. The authors of ELMO [106], a contextualized model based on LSTM, present this configuration with additional weight parameters for each layer. Other authors investigating representations [94; 95; 107] call this weighting a scalar mix technique. The authors of [66], instead of learning layer weights for each word, compute them using complex alignment and novelty measures.