# Extracting Sentence Embeddings from Pretrained Transformer Models
## 4 Results
### BERT + Avg. Model  
One of the ways we sought to improve BERT transformer model representations is to combine embeddings of the regular BERT with the ones averaged over multiple contexts. That is, for each token in each BERT layer, we collected many different contexts, and the averaged vector became the vector of the Avg. model. We then combined BERT and Avg. model, according to the parameter \(w\), which shows the fraction of Avg. model representations in the resulting vector \(v\):  
\[v=v_{BERT}(1-w)+v_{Avg.}w. \tag{10}\]  
We have also varied the \(w\) parameter to the negative values in (10) to see if subtracting (instead of adding) the context-average representations from the context-aware ones helps.  
The impact of the \(w\) parameter and the choice of the layer to source the representations (same layer for both BERT and Avg.) is presented in Figure 5.  
One can see that for clustering and classification tasks, the combination of both models in equal portions of \(w=0.5\) is better than these models alone (\(w=0.0\) corresponds to BERT, \(w=1.0\) is a single Avg. model). If we look at the best scores with the two techniques applied in Table 4, for the average clustering accuracy this translates to an improvement of 1.4% from the 63.4% of BERT to 64.8% of BERT + Avg.  
Meanwhile, as we see in the same table, the average accuracy of classification tasks without additional techniques applied to the combination of the first and last layers is similar between only BERT (79.9%) and BERT + Avg. (79.7%). However, as we see in Figure 5, differently from the first + last optimal layer setting for BERT, BERT + Avg. has a sweet spot in the 10 th layer with an average classification accuracy of 80.5%, surpassing that of BERT by 0.6%.  
These results suggest a conclusion that for clustering and classification tasks, combining a regular BERT token with the same one but averaged over multiple contexts is beneficial.  
As we already mentioned, STS tasks preferred the regular BERT (\(w=0\)) to the BERT + Avg. on average. However, for several individual semantic textual similarity datasets, the best weights turned out to be even negative. This is the case for STS15 (\(w=-0.25\)) and STS16 (\(w=-0.75\)). The same negative weight of \(w=-0.5\) preference was also observed for the searchsnippets clustering dataset. Although the [PERSON]  
Figure 5: BERT + Avg. model performance dependence on the weight \(w\) of Avg. model and layer, from which (for both models) representations are used. To the right of the black line on the horizontal axis, average aggregation of multiple layers is also shown. Tokens are simply averaged and no post-processing is used. The horizontal line with \(w=0.0\) corresponds to a regular Bert (B) model, \(w=0.5\) is B + Avg., and \(w=1.0\) is the Avg. model. The white \(\times\) marks the maximum value.  
correlation for three STS tasks was only up to several percent higher, for searchsnippets, the clustering accuracy increased to a staggering 80.2% from 72.2% of \(w=0.0\). This shows that the determined values of \(w=0.0\) for STS and \(w=0.5\) for clustering and classification are not universal, and for a small percentage of tasks, they can differ. For more BERT + Avg. task-wise details, see Appendix A.2 figures for STS (Figure A1), clustering (Figure A2), and classification (Figure A3) performance.