# Multi-Focus Microscopy Image Fusion Based on Swin Transformer Architecture
## 3 Methodology  
### Overall Architecture  
We proposed an end-to-end MFIF module for microscope images based on the Swin Transformer architecture, as presented in Figure 1. The features pyramid of the multi-focus images was encoded by the Swin-Transformer-based model Swin-S [31]. To generate the fused image, a Swin-Transformer-based decoder with patch expansion (Figure 1) was applied to decode the feature pyramid. A Swin Transformer block is composed of a LayerNorm (LN) layer, a multi-head self-attention module, a residual connection and a two-layer MLP with GELU non-linearity. The window-based multi-head self-attention (_W-MSA_) module and the shifted window-based multi-head self-attention (_SW-MSA_) module are applied in the two successive transformer blocks (Figure 2). Based on such a window partitioning mechanism, the Swin Transformer blocks are computed as:  
\[\hat{z}^{l}=W-MSA(LN(z^{l-1}))+z^{l-1} \tag{1}\]  
\[z^{l}=MLP(\hat{z}^{l})+\hat{z}^{l} \tag{2}\]  
\[\hat{z}^{l+1}=SW-MSA(LN(z^{l}))+z^{l} \tag{3}\]  
\[z^{l+1}=MLP(\hat{z}^{l+1})+\hat{z}^{l+1} \tag{4}\]where \(\hat{z}^{l}\) and \(z^{l}\) denote the ouputs of the (\(S\))\(W\)-\(M\)SA module and the MLP module of the \(l^{th}\) block, respectively. Self-attention is computed as:  
\[Attention(Q,K,V)=SoftMax(\frac{QK^{T}}{\sqrt{d}}+B)V \tag{5}\]  
where \(Q,K,V\in\mathbb{R}^{M^{2}\times d}\) represent the query, key and value matrices. \(M^{2}\) and \(d\) denote the number of patches in a window and the dimension of the query or key, respectively. And the values in B are taken from the bias matrix \(\hat{B}\in\mathbb{R}^{(2M-1)\times(2M+1)}\).  
Figure 1: The architecture of the end-to-end MFIF models based on the Swin Transformer block, U-Swin fusion model.  
Figure 2: Two successive Swin Transformer blocks.  
Initially, the input RGB images were divided into non-overlapping patches with a 4 \(\times\) 4 patch and a linear embedding layer and were applied to project raw-value features into an arbitrary dimension, similar to ViTs [25]. To generate hierarchical representations, several Swin Transformer blocks and patch-merging layers were applied to encode the transformed patch tokens. The patch-merging layer concatenates the features of each group of 2 \(\times\) 2 neighboring patches to down-sample and increase dimensions. Swin Transformer blocks are applied to learn feature representations without dimension transformation. To generate the fused image, symmetric Swin Transformer blocks and patch expansion were applied to perform up-sampling in U-Swin fusion models, like Swin-Unet [32]. The patch expansion layer reshapes the feature maps into a higher resolution feature map (2\(\times\) up-sampling). To fuse the hierarchical features of multi-input images, varieties of fusion rules (tensor max and tensor mean) have been utilized to fuse the transformed path tokens, as presented in the IFCNN [62]. Furthermore, two types of mix fusion rules (simple mix and channel mix) were used in this study, which can be expressed as follows:  
\[F=\omega\,_{1}F^{Max}+\omega_{2}\,F^{Mean}\,\left(\textit{Simple mix}\right) \tag{6}\]  
\[F_{i}=\omega_{(1,\,i)}\,F_{i}^{Max}+\omega_{(2,\,i)}\,F_{i}^{Mean}\,\left( \textit{Channel mix}\right) \tag{7}\]  
where \(F^{Max}\) and \(F^{Mean}\) denote the fused features from tensor max and tensor mean, respectively, and i represents the different channel of features. \(\omega\) is not the hyperparameter that can be optimized during training mode. In this study, the channel mix fusion rule was implemented by the depthwise convolution to avoid information exchange between different channels. Inspired by U-net [65], the skip connections were applied to concatenate the hierarchical features from the encode with the up-sample features to enhance overall performance. The shallow features are concatenated with the up-sampling features from deep features to reduce the loss of spatial information caused by down-sampling.