# Education-to-Skill Mapping Using Hierarchical Classification and Transformer Neural Network
## 2 Materials and Methods
### Mathematical Models
#### 2.1.10 Transformer Neural Networks  
Text embedding can be performed using various techniques. One example of such a technique is the use of transformer architecture. Transformers might be used in many practical settings, such as language modelling, speech recognition, image and music analysis [22].  
Transformer is created by modifying parts of an auto-encoder [23]. The auto-encoder maps input to the output, and dimensions of input and output might be different. It consists of two important blocks: encoder and decoder [24]. The transformer uses the same logic of auto-encoder just replacing fully-connected or convolutional layers with attention function that gives specific weights over the input sequence. The advantage of attention function is that it allows the encoder to proceed to longer sequences, which is very important to NLP tasks.  
More formally, the attention transformation \(A(Q,K,V):\mathbb{R}^{d_{x}}\rightarrow\mathbb{R}^{d_{v}}\) is used in the transformer neural network. First, the input is decomposed by calculating the three matrices \(Q=\theta^{Q}x\), \(K=\theta^{K}x\), and \(V=\theta^{V}x\), then the attention is calculated:  
\[A(Q,K,V)=softmax(cQK^{T})V \tag{14}\]  
where \(c=1/\sqrt{d_{k}}\) the normalization constant, \(d_{k}\) is the dimension of \(K\). The \(Q\) matrix is called the query, \(K\)--the keys, and \(V\)--the values matrices.  
Equivalent to a fully connected neural network, the attentions can be combined, which is known as multi-head attention \(MHA(Q,K,V)\), which enables us to jointly combine the information from different representations of input at different positions.  
\[MHA(Q,K,V) = (A_{1},...,A_{h})\theta^{Y} \tag{15}\] \[\text{where }A_{i} = A(Q\theta_{i}^{Q},K\theta_{i}^{K},V\theta_{i}^{V}) \tag{16}\]  
Here, the projections are parameter matrices \(\theta_{i}^{Q}\in\mathbb{R}^{d_{x}\times d_{y}}\), \(\theta_{i}^{K}\in\mathbb{R}^{d_{x}\times d_{k}}\), \(\theta_{i}^{V}\in\mathbb{R}^{d_{x}\times d_{y}}\) and \(\theta^{Y}\in\mathbb{R}^{d_{x}\times d_{y}}\).  
Each multi-head attention layer follows with fully connected transformation:  
\[Y=\sigma(\theta\cdot MHA(Q,K,V)),\quad\theta\in\mathbb{R}^{d_{y}\times d_{k}+1} \tag{17}\]  
where \(\sigma(x)=relu(x)=max(0,x)\) the ReLU activation function [11]. In this study, we use Bi-directional transformer--BERT (Bidirectional Encoder Representations from Transformers, here after abbr. BERT) [25].  
BERT is trained using a large plain text corpus [25]. BERT, being transformer type architecture, also uses attention function. Base-BERT has 12 encoders that takes a text sequence and forwards it from one encoder to the next. The output for each sequence is a 728 length vector [26]. The most innovative part of BERT is the ability to learn sequential information in both directions. In fact, conditioning each word on its previous and next words would allow the word to \"see itself\" [25]. Therefore, BERT mask some of the words in the input and then bidirectionally condition the entire sequence to predict the masked words. In order to force the model to learn despite the presence or absence of the mask, usually only a small portion of words are randomly masked [25]. The model is trained for both masked tokens and next sentence prediction together. The next sentence prediction is used to predict whether the next sentence is random [25]. BERT embeddings have been shown to be highly accurate in NLP tasks [26].