# Extracting Sentence Embeddings from Pretrained Transformer Models
## 3 Methods
### Models
#### 3.2.1 Bert  
BERT is a very popular transformer model. When first introduced in [5], it spectacularly outperformed multiple other models on a wide range of tasks. BERT is based on the original transformer architecture of [2]. The main difference is that instead of sequence-to-sequence workings and encoder-decoder structure, it is composed only of an encoder side. This allows it to solve multiple classification and regression tasks, maintaining the same pre-trained base while only changing the classifier heads on top. In this work, we use the BERT version called bert-base-uncased, a 110 M parameter model containing 12 layers (blocks) and working with lowercase text. We employ the Hugging Face implementation [202] of BERT. It was pre-trained on BooksCorpus (800 M words) [203] and English Wikipedia (2500 M words) datasets, which take 13 GB of plain text combined. The model was trained for 40 epochs (or passes through the corpus). Back then, the training time, model size, and data used were considered to be very large, yet now they are only a small fraction of what the current state-of-the-art models use. Input  
Input to the BERT encoder consists of token, positional, and token type embeddings. BERT uses WordPiece tokenization [204]. Splitting less frequent words into sub-words (e.g., \"transformer\" \(\rightarrow\) \"transform\" and \"##er\") rather than splitting everything on word boundaries allows one to reach a manageable vocabulary size of 30,000 tokens. BERT, like other transformer models, perceives input as a set; therefore, order information must be supplied in addition. To achieve this, additional positional embeddings are used, with a unique value for each position in a token sequence and feature dimension. Token type embeddings allow the model to distinguish between the two (if there are two) separate sentences (e.g., \(\langle\) Question, Answer\(\rangle\)) in one token sequence. Overall, token, positional, and token type embeddings are added, and then layer normalization [205] and dropout [179] are applied. This results in the input to the BERT model blocks (for simplicity, we will call them layers). There are 3 special tokens. [CLS] starts every token sequence, [SEP] ends token sequences and acts as a separator in a pair of sentences, and [MASK] is used during pre-training to mask some percentage of the input tokens at random for the model to predict (known as a \"Cloze\" task [206]). The [CLS] token is also used for the next sentence prediction task--predicting whether the second sentence in a pair actually follows the first one in the training dataset or is a random one--the second unsupervised pre-training task of BERT. Multi-Head Self-Attention  
An input tensor \(X\in\mathbb{R}^{b\times t\times f}\) with \(b\) samples in a batch, \(t\) tokens, and \(f\) features is first linearly projected into queries \(Q\in\mathbb{R}^{b\times t_{Q}\times h\times f_{Q}}\), keys \(K\in\mathbb{R}^{b\times t_{x}\times h\times f_{F}}\), and values \(V\in\mathbb{R}^{b\times t_{V}\times h\times f_{V}}\) tensors with \(h\) heads using learned weights. Note that in the standard case, \(t=t_{Q}=t_{K}=t_{V}\) and \(f=hf_{Q}=hf_{K}=hf_{V}\). Then, for each sample text in a batch and each head dot, products between feature vectors of every combination of query and key tokens are calculated, resulting in an attention tensor \((QK)\in\mathbb{R}^{b\times h\times t_{Q}\times t_{K}}\). It is then scaled by dividing it by \(\sqrt{f_{K}}\), and softmax is applied along \(t_{K}\) so that all the dot products for any given query token and all the key tokens would sum to 1. Now, a dot product is calculated again for the probability-like scores and values tensor, resulting in a weighted selection of \(V\), with the form of \(\mathbb{R}^{b\times h\times t_{Q}\times f_{V}}\). Finally, the heads are concatenated back and a linear transformation layer \(W^{0}\in\mathbb{R}^{hf_{V}\times f}\) is applied. The resulting tensor is the same shape as the input one: \(\text{MultiHead}(X)\in\mathbb{R}^{b\times t\times f}\).