# Knowing Knowledge: Epistemological Study of Knowledge in Transformers
## 4 Experiments
### Transformer  
Transformer-based architectures reach state-of-the-art in many downstream text classification tasks.  
In general, the Transformer model is based on the encoder-decoder architecture [6]. The encoder is responsible for scrolling through the time steps of the input and encoding the entire sequence into a fixed-length vector called the context vector. The decoder is responsible for scrolling the output time steps by reading the context vector. The fundamental mechanism of these architectures is self-attention. This mechanism allows output to focus attention on input while producing output. In contrast, the self-attention model allows inputs to interact with each other (i.e., calculate the attention of all other inputs wrt one input. The first step is multiplying each of the encoder input vectors with three weights matrices \((W(Q),W(K),W(V))\) that we trained during the training process. This matrix multiplication will give us three vectors for each input vector: the key vector, the query vector, and the value vector. The second step in calculating self-attention is multiplying the Query vector of the current input with the key vectors from other inputs. In the third step, we will divide the score by the square root of the dimensions of the key vector (dk). For example, in the paper, the dimension of the key vector is 64, so that will be 8. The reason behind that is if the dot products become large, this causes some self-attention scores to be minimal after we apply the softmax function in the future. In the fourth step, we will apply the softmax function on all self-attention scores we calculated wrt the query word (here, first word). In the fifth step, we multiply the value vector on the vector we calculated in the previous step. In the final step, we sum up the weighted value vectors we got in the previous step, giving us the self-attention output for the given the word. The above procedure is applied to all the input sequences. Mathematically, the self-attention matrix for input matrices \((Q,K,V)\) is calculated as:  
\[Attention(Q,K,V)=softmax(\frac{QK^{T}}{\sqrt{d_{k}}})V\]  
Q, K, and V are the concatenation of query, key, and value vectors. In the attention paper [6], the authors proposed another type of attention mechanism called multi-headed attention. This mechanism is based on a distributed variant of attention. The transformer architecture uses the attention model uses multi-headed attention in three steps:  
1. encoder-decoder attention layers, in this type of layer, the queries come from the previous decoder layer while the keys and values come from the encoder output. This allows each position in the decoder to give attention to all the positions of the input sequence.
2. self-attention layer contained in the encoder receives key, value, and query input from the output of the previous encoder layer. Each position in the encoder can get an attention score from every position in the previous encoder layer.
3. self-attention in the decoder, this is similar to self-attention in the encoder where all queries, keys, and values come from the previous layer. The self-attention decoder allows each position to attend each position up to and including that position. The future values are masked with (-Inf). This is known as masked-self attention.  
BERT  
In this paper, one of the most well-known Transformers-based models was tested: Bidirectional Encoder Representations from Transformers (BERT) [45]. The BERT architecture is trained on the BooksCorpus [46] and the English Wikipedia. The BERT model was implemented using the Transformers library of Huggingface [47]. The output of each encoder was decoded by a Feed-Forward Neural Network (FFNN) to a single output layer with an input size equal to 768 and output equal to the number of classes with the classifier function. For the fine-tuning phase, the number of epochs was set to 5, and the optimizer Adam [48] was used.