# AI Approaches in Computer-Aided Diagnosis and Recognition of Neoplastic Changes in MRI Brain Images
## 2 Materials and Methods
### Attention Mechanism and Vision Transformers
The heart of the block is the Scaled Dot-Product Attention, which multiplies input vectors (or matrices in case of stacked inputs for faster computations), scales them by the square root of inputs dimensionality, and normalizes with a softmax function, as shown in the Equation (1) [22]. This operation is not performed once for a triplet of inputs. Instead, it is done repeatedly with inputs previously multiplied by different matrices with learnable parameters that cast them into a smaller dimension. Calculating attention multiple times on reformatted data allows focusing on different parts of vectors, similar to multiple filters in a single convolutional layer of a CNN. Obtained matrices are stacked together and again cast linearly to the model's inner dimensionality. As inputs are cast to lower dimensions, these multiple computations do not slow down the whole process compared to single full-dimensionality attention. Both operations-Scaled Dot-Product Attention and Multi-Head Attention, can be visualized in the form of block diagrams in Figure 3.  
\[Attention(Q,\ K,\ V)=softmax\Big{(}\frac{QK^{T}}{\sqrt{d_{k}}}\Big{)}\,V \tag{1}\]  
The described mechanism can extract global and local features from the input vectors, although the attention works globally and does not assess anything about input structure. The information about spatial relations between parts of the input does not come from the model's inductive bias, as in RNNs or CNNs, so to draw out these associations, the locality needs to be encoded into the data. In the case of the Transformer model, the sequence order is encoded into the inputs. In the NLP task, input sequences were translated into vectors using one of the well-known techniques [25], these vectors were then summed with learned position embeddings. These embeddings are vectors of the same dimensionality as inputs that indicates each element's position. There are numerous methods for obtaining such positional encoding [26]. In this case, vectors of \(\sin/\cos\) waves of exponentially changing frequency were used. The architecture described above became one of the best algorithms for NLP tasks that largely replaced RNNs and LSTMs, but the attention itself is not suited explicitly for processing language. It is a general computational method valuable in a variety of different problems; it has been used in computer vision too, but mostly in conjunction with different techniques, such as CNNs, and did not provide far better results than well-established strategies [27]. The reason for that lies in calculating attention for every pair of input pieces, which means it is a quadratic operation. It was immaterial in the case of sentences, as even very long clauses contain, at most, dozens of words, while even small pictures consist  
Figure 3: Scaled Dot-Product Attention (**left**) and Multi-Head Attention (**right**) graphical representations. of thousands of pixels; a popular dataset, ILSVRC (a subset of ImageNet) [28], contains images of mean size of approximately 400 \(\times\) 400 pixels. Trying to compute the Multi-Head Attention on such an input, in most cases, leads to a lack of memory. Some approaches tried to mitigate that problem by limiting attention pairs, for example, by calculating it only locally, for pixels in close proximity [29], which resulted in behavior very similar to convolution kernels, or by calculating it more sparsely, for only a subset of pixels [30]. The architecture used in this work, the Vision Transformer (ViT) [16], approaches the problem differently: instead of altering the attention mechanism to work with images, it adjusts the data and leaves the algorithm untouched. In the ViT architecture shown in Figure 4, the Transformer Encoder block is built identically to that used in NLP. What changes between these two models is what happens with inputs and outputs.