# AI Approaches in Computer-Aided Diagnosis and Recognition of Neoplastic Changes in MRI Brain Images
## 2 Materials and Methods
### Attention Mechanism and Vision Transformers  
Despite being one of the best-known and certainly the most preferred algorithms, Convolutional Neural Networks are not the only option for computer vision. An alternative presented in this work uses the Attention Mechanism. As with many of the neural networks' features, this method was designed to mimic how the human brain works. The idea is to help the network to focus on what is vital in the input data instead of treating all the inputs equally. This mechanism differs from static weights by being dynamically computed per input, which allows it to find essential features regardless of their position in the input. The attention to Deep Learning was first introduced in Natural Language Processing (NLP) related tasks, especially machine translation [20]. This method was used in encoder-decoder architectures based on Recurrent Neural Networks (RNN) or its variants, such as Long-Short Term Memory (LSTM), to remedy the problem of performance drops with long sentences as inputs. The attention measured how influential words from the input sentence were regarding the output sentence and did it for each pair of words. In some use cases, attention was also computed for pairs of words from a single statement, extracting the information about which words mattered the most [21]. Various forms of attention in combination with RNNs became popular in NLP, which led to architectures relying on it more than on recurrency until, at some point, an encoder-decoder model for machine translation based purely on attention was created [22]. This model was a Transformer, and it was meant to achieve translation accuracy comparable with RNNs, while eliminating their shortcomings. As recurrent networks work in the time domain by calculating their hidden states one at a time, it is hard to parallelize computations, thus making them slow. Getting rid of recurrency in favor of attention allows for taking advantage of modern hardware, such as Graphical Processing Units (GPUs) or Tensor Processing Units (TPUs). The Transformer architecture is also an encoder-decoder model built of multiple processing block series that can be graphically represented as in Figure 2, where the \"Nx\" sign is the number of block repetitions. Both the encoder and decoder blocks are built of sets of operations arranged in a different order: Multi-Head Attention (masked or not), residual connection with normalization [23], and simple Multilayer Perceptrons (MLP), also called as Feed-Forward Networks [12]. The difference between a decoder and an encoder is an additional Masked Multi-Head Attention and normalization operations in the former and different inputs. The encoder's blocks accept as an input the network's input (the first block) and a previous encoder layer's output. In contrast, the decoder accepts analogous information with the addition of the encoder's corresponding layer's output. The Transformer translates the input data into some form of internal representation. Each layer's output size is a fixed value shared between every block and its internal parts. The translation happens by calculating the Multi-Head Attention of inputs and then transforming them with a simple MLP network. The attention extracts essential pieces of information, and the MLP projects them into the latent representation. Thus, the Multi-Head Attention works as a generalization for the Feed-Forward with dynamically computed weights, making it the crucial part of the model. Many different forms of attention that can be used in Deep Learning have been developed and described [24]. As this work concerns Transformers, the one described in detail is the same used by the authors of the architecture [22]. Figure 2: The Transformer architecture. The Multi-Head Attention implemented in the Transformer can be described as a three-input block consisting of a few operations. In some cases, each input might be the same vector (as in the encoder) thus, the name self-attention.