## 5 Model Architectures and Training
### Model Architectures
#### 5.1.3 Vision Transformers  
In addition to the more classic approaches above, modern transformer-based models for image processing, following the fundamental work of [PERSON] et al. [31], were implemented. Vision Transformers (ViTs), as introduced by [PERSON] et al. [49], basically build up on the original transformer concept, applying the concept of multihead self-attention to image data. In accordance with the application of this work, single noise images with their corresponding \(\mathrm{C}=2\) intensity and depth channel \(\mathbf{x_{n,k}}=\{\mathbf{I_{n,k}},\mathbf{L_{n,k}}\}\) of size \(\mathrm{H}\times\mathrm{W}\) are therefore divided into a sequence of \(\mathrm{S}=\mathrm{HW}/\mathrm{P}^{2}\) flattened 2D patches with patch size \(\mathrm{P}\). In order to map the patch sequence to a latent vector of length \(\mathrm{D}\), a trainable linear projection \(\mathbf{E}\) is used. Its output is, in the ViT context, referred to as patch embedding (see Figure 8). In order to provide the model a capability of encoding spatial context, a learnable 1D position embedding \(\mathbf{E}_{\mathrm{pos}}\) is added to each projected and flattened image patch. Hence, the input of the first transformer encoder can be described as follows [49]:  
\[\mathbf{z_{0}}=\left[\mathbf{x_{\mathrm{class}}};\mathbf{x_{p}}^{1}\mathbf{E}; \mathbf{x_{p}}^{2}\mathbf{E};\cdots;\mathbf{x_{p}}^{\mathrm{N}}\mathbf{E} \right]+\mathbf{E_{\mathrm{pos}}},\quad\mathbf{E}\in\mathbb{R}^{\left( \mathrm{P}^{2}\cdot\mathrm{C}\right)\times\mathrm{D}},\mathbf{E_{\mathrm{pos}} }\in\mathbb{R}^{\left(\mathrm{S}+1\right)\times\mathrm{D}} \tag{14}\]  
After passing through multiple transformer encoders (referred to as depth), information flows through an MLP head for the classification or regression, incorporating features such as layer normalization, linear transformation, GELU activation, and dropout. Figure 7: General architecture of the implemented ResNet3D, based on the work of [PERSON] et al. [46]. The transformer encoder blocks are in accordance with the architecture proposed by [PERSON] et al. [31] (see Figure 9). Thus, the patch embeddings are normalized and forwarded to a multihead attention block. The output is added to a residual connection from the patch embedding. The result is normalized and fed through an MLP to be finally added with the residual bypassing the MLP. The classic **QKV** self-attention (SA) [31] generally computes a weighted sum over all values **V** in an input sequence \(\mathbf{z}\in\mathbb{R}^{\mathrm{N}\times\mathrm{D}}\). The respective attention weights \(A_{i,j}\) are based on the similarity between two elements of the sequence and their corresponding query \(\mathbf{Q}^{\mathrm{i}}\) and key \(\mathbf{K}^{\mathrm{i}}\) representation [31]:  
\[ \begin{array}{rcl}[\mathbf{Q},\mathbf{K},\mathbf{V}]&=&\mathbf{z}\mathbf{U} _{\mathbf{QK}\mathbf{V}}&\mathbf{U}_{\mathbf{QK}\mathbf{V}}\in\mathbb{R}^{ \mathrm{D}\times 3\mathrm{D}_{\mathrm{h}}},\\ \mathbf{A}&=&\mathrm{softmax}(\mathbf{Q}\mathbf{K}^{\top}/\sqrt{\mathrm{D}_{ \mathrm{h}}})&\mathbf{A}\in\mathbb{R}^{\mathrm{N}\times\mathrm{N}},\\ \mathrm{SA}(\mathbf{z})&=&\mathbf{A}\mathbf{V}.\end{array}  \tag{15}\]  
The multihead self-attention (MSA) is, finally, an extension of SA in which several self-attention operations (heads) are run in parallel, while projecting their concatenated outputs [31]. Now, as discussed in Section 4.2.2, the input consists of an image time series TS rather than single images. Therefore, in order to consider temporal context, the original work of [PERSON] et al. [49] is extended such that a stack of \(\mathrm{M}=200\) images is processed by the ViT instead of a single frame. In order to do so, uniform frame sampling in accordance with [PERSON] [51] and [PERSON] et al. [52] is utilized to simply extract the patches over the whole TS instead of a single frame to forward \(\mathrm{N}\times\mathrm{M}\) patch embeddings to the subsequent linear projection \(\mathbf{E}\).