## 3 Bert  
BERT (Bidirectional Encoder Representations from Transformers) [2] is a deep learning architecture based on Transformer [3]. At a very high level, a Transformer is composed of two components: a stack of encoders and a stack of decoders. The first has the role of extrapolating the core information from a text, whereas the latter has the task of producing the output from that piece of information. Since BERT is a model for generating the vectorial representations of words, the decoder stack is discarded and only encoders are used in BERT. The number of encoders that compose the stack is customisable, but the original paper implemented 12 encoders. As depicted in Figure 1, the model does not accept a plain text sentence in input, but rather an embedding matrix specifically constructed as follows: first, the tokenizer splits a sentence (or a couple of sentences) into tokens, which are full words or parts of words, based on the vocabulary of the model. In output, the tokenizer produces the list of indexes of the vocabulary tokens, and if two sentences are given in input, it also provides a list that specifies which sentence each token belongs to. The tokenizer also plays the key role of adding the special tokens to the sequence; the token [CLS] is always added at the beginning of each sequence, and the token [SEP] is always added between the tokens of two different sentences and at the end of the sequence. Other special tokens, not always added, are the token [MASK], used to substitute a token in special tasks, such as masked language modeling, [UNK], used when the tokenizer does not know how to separate a portion of the sentence, and [PAD], used in short sentences to fill the sequence. After the tokenization procedure, the corresponding embedding matrix is built and positional embeddings are added to this representation. These vectors follow a specific pattern that the model learns, which helps it to determine the position of each word in a sentence or the distance between different words in the sequence. It is important to note that the positional embedding depends only on the position of the token in the sequence; therefore, different tokens, in different sentences, in the same index, have the same positional embedding. The output of this operation is the input of the first encoder of the model. Each encoder, as shown in Figure 2, is made up of two main components: a multi-head self-attention and a feed-forward network. The first one is the core of the model and is composed of a certain amount of independent self-attention mechanisms called _heads_ (12 in the original paper). Each head takes as input the same matrix, \(X\), and computes three different vector representations. These are typically called Key (\(K\)), Query (\(Q\)), and Value (\(V\)), and they are computed by multiplying \(X\) by three different weight matrices, \(W_{k}\), \(W_{q}\), and \(W_{v}\):  
\[K=X\times W_{k},\ Q=X\times W_{q},\ V=X\times W_{v} \tag{1}\]  
Figure 1: Pre-processing of the textual input for the BERT model. First, the tokenizer splits the sentence into tokens and adds the special tokens required. From the embedding matrix, the embedding layer extrapolates the vector representations of the tokens, in green, and sums the positional embeddings, in red, to produce the input for the model, in yellow. Using these new representations, the self-attention mechanism calculates the attention weights (\(A\)) by applying the softmax function on the scaled dot product (i.e., the dot product is divided by the length of the input representation of each token (\(d_{k}\)) between the Query, \(Q\), and the Key, \(K\). Finally, the new representation of each token (\(Z\)) is calculated by multiplying the attention weights for \(V\).