# Artificial Intelligence to Reshape the Healthcare Ecosystem
## 2 AI Techniques for Healthcare
### Generative AI
#### 2.4.4 Transformers  
Natural language processing (NLP) has proven to be one of the most promising areas in influencing the future of healthcare [90]. For this purpose, Transformer-based models, such as the Generative Pre-trained Transformer (GPT), can be used to generate  
Figure 7: Schemes of autoencoder (**a**) and variational autoencoder (**b**) GRUs. and understand medical texts, assist in medical coding, summarization, and question-answering tasks [1; 91; 92]. The Transformer model was proposed in [93]. It is the core of many popular models for NLP, including BERT [94], the already mentioned GPT [95], and T5 [96]. The main innovation of the Transformer model is the introduction of the so-called _attention_ mechanisms. Essentially, they analyze the relationships between words in a sentence without using other models such as RNNs. The added value of the introduction of Transformers includes, essentially, parallelization, capture of long-range dependencies, and scalability. This means that unlike previous approaches, such as RNNs, that process input data sequentially, Transformers allow for parallel processing for speeding up training times. Moreover, self-attention mechanisms allow for consideration of the entire input data sequence simultaneously, improving the capability to capture contextual relationships between data. Finally, the Transformer architecture scales well with increasing data volume and model size. Figure 8 shows the basic elements of Transformers, which are illustrated in what follows. The self-attention mechanism allows the model to weigh the importance of different words in a sentence when encoding a particular word. It is defined by the following elements. The scaled dot-product attention is defined as  
\[\text{Attention}(Q,K,V)=\text{softmax}\bigg{(}\frac{QK^{T}}{\sqrt{d_{k}}} \bigg{)}V \tag{45}\]  
where \(Q\) (queries), \(K\) (keys), and \(V\) (values) are the input matrices, and \(d_{k}\) is the dimension of the keys. In more detail, consider an input sequence of length \(n\). This sequence includes tokens, that could be words, subwords, characters, or even whole sentences, depending on the level of granularity needed for text processing. Each token is encoded by an _embedding_ of dimension \(d_{\text{model}}\). Data are aggregated as a matrix \(X\in\mathbb{R}^{n\times d_{\text{model}}}\), referred to as the input embedding matrix. The \(Q\), \(K\), and \(V\) matrices are obtained by linearly transforming \(X\) as follows:  
\[Q=XW_{Q},\quad K=XW_{K},\quad V=XW_{V} \tag{46}\]  
Figure 8: Transformer architecture. where \(W_{Q}\in\mathbb{R}^{d_{\text{model}}\times d_{k}}\) is the weight matrix for queries, \(W_{K}\in\mathbb{R}^{d_{\text{model}}\times d_{k}}\) is the weight matrix for keys, and \(W_{V}\in\mathbb{R}^{d_{\text{model}}\times d_{v}}\) is the weight matrix for values. Thus, the resulting matrices \(Q\), \(K\), and \(V\) have dimensions \(Q\in\mathbb{R}^{n\times d_{k}}\), \(K\in\mathbb{R}^{n\times d_{k}}\), and \(V\in\mathbb{R}^{n\times d_{v}}\). Clearly, \(d_{k}=d_{v}=d_{\text{model}}/h\), where \(h\) is the number of attention heads, which refers to the _multi-head attention_ operation of the model, which allows for focusing on different parts of the input sequence in parallel, enhancing its ability to capture different aspects of the relationships between tokens. Essentially, the multi-head attention mechanism extends the self-attention mechanism by applying it multiple times in parallel. Hence, given the input embedding \(X\), the multi-head attention mechanism is modeled as follows:  
1. First it is necessary to linearly project the input embedding \(X\) into multiple sets of keys, queries, and values by using the learned weight matrices: \[Q_{i}=XW_{Q}^{i},\quad K_{i}=XW_{K}^{i},\quad V_{i}=XW_{V}^{i},\quad\forall i \in\{1,\dots,h\}\] (47) where \(h\) is the number of attention heads. 2. Then, the self-attention mechanism is applied to each set of keys, queries, and values, as follows: \[\text{head}_{i}=\text{Attention}(Q_{i},K_{i},V_{i})\] (48)
3.