# ViT-Cap: A Novel Vision Transformer-Based Capsule Network Model for Finger Vein Recognition
## 3 Materials and Methods
### Transformer-Based Module  
The resulting sequence of embedded patches \(Z_{0}\) was passed to the transformer encoder. Figure 3 shows the second part of our model. The encoder consisted of \(L\) identical layers, which were composed of alternating layers of the multi-head self-attention block (MHSA, Equation (2)) and a fully connected, feed-forward dense block. Each layer was a feed-forward network after the attention, and the function of the feed-forward dense block was spatial conversion. Feed-forward networks introduce a nonlinear ReLU activation function, which transforms the space of attention output, thereby improving the performance of the model. Layer norms were applied before every block, and the residual connections were applied after every block. \[Z_{l}=MHSA(LN(Z_{L-1}))+Z_{L-1},\quad l=1,\dots,L \tag{2}\]  
Figure 2: The first part of ViT-Cap model architecture: linear embedding of finger vein images. The MHSA was the core component of the transformer encoder, and its role was to determine the relative importance of a single patch embedding relative to other embeddings in the sequence. MHSA consists of four layers: a linear layer, a self-attention layer, a concatenation layer, and a final linear layer. The operation of the MHSA is as follows. Firstly, we inputted the linear transformation of image matrix \(\ Z\) and divided it into three matrices \(\mathcal{Q}\), \(\mathrm{K}\), and \(\mathrm{V}\), which were obtained by three different weight transformation matrices, \(\mathrm{W^{Q}}\), \(\mathrm{W^{K}}\), and \(\mathrm{W^{V}}\) (Equation (3)). \[[\mathcal{Q},\mathrm{K},\mathrm{V}]=\left[\mathrm{ZW^{Q}},\mathrm{ZW^{K}}, \mathrm{ZW^{V}}\right] \tag{3}\]  
Then the matrices \(\mathcal{Q}\), \(\mathrm{K}\), and \(\mathrm{V}\), obtained above, were linearly projected into \(h\) different subspaces, \(h\) equaled 12, and the self-attention values in each subspace were calculated (Equations (4)-(7)). \[[\mathcal{Q}_{1},\cdots,\mathcal{Q}_{h}]=\left[\mathrm{QW^{Q_{1}}},\cdots, \mathrm{QW^{Q_{h}}}\right] \tag{4}\]  
\[[\mathcal{K}_{1},\cdots,\mathcal{K}_{h}]=\left[\mathrm{KW^{K_{1}}},\cdots, \mathrm{KW^{K_{h}}}\right] \tag{5}\]  
\[[\mathcal{V}_{1},\cdots,\mathcal{V}_{h}]=\left[\mathrm{VW^{V_{1}}},\cdots, \mathrm{VW^{V_{h}}}\right] \tag{6}\]  
\[Head_{i}=\mathrm{Softmax}\Bigg{(}\frac{\mathcal{Q}_{i}\mathcal{K}_{i}^{\mathrm{ T}}}{\sqrt{d}}\Bigg{)}\mathcal{V}_{i} \tag{7}\]  
Figure 3: The second part of the ViT-Cap model architecture: transformer encoder module. Finally, we concatenated all the attention heads and then projected them through a feed-forward layer with a learnable weight \(\mathrm{W}\) (Equation (8)). \[MultiHead(\mathrm{Q},\mathrm{K},\mathrm{V})=Concat(Head_{1},\ \ldots,Head_{h}) \mathrm{W}^{O} \tag{8}\]  
In the process of finger vein recognition, attention is an effective non-local information fusion technology. Attention essentially means the average weighting according to the matrix of a relation, which means different information can be fused. CNN itself has a defect, that is that each of its operations can only focus on the information near the convolution kernel and cannot fuse distant information. However, attention can realize a weighted fusion of distant information, which plays an auxiliary role. On the other hand, attention has a higher logical interpretability than CNN, and the weighted analysis of attention naturally has the property of visualization. As shown in Figure 4, we trained the model on four public datasets of finger veins and visualized the attention maps obtained from the four datasets. The more important the area in Figure 4, the brighter the pixels. Figure 4 shows the process of obtaining finger vein information by the multi-head attention mechanism. Through this module, the important feature information of finger vein images was gradually obtained. For example, in FV-USM, the attention map head-1 only showed partial attention to the vein feature information, but the corresponding attention was also given to irrelevant information outside the veins.