## 2 Method
### ViT Model  
The Transformer model achieves good performance in many computer vision tasks. If there is enough data for pre-training, the performance of ViT exceeds that of CNN, overcoming the Transformer's lack of inductive bias and obtaining a better migration effect in downstream tasks. However, when the training dataset is not large enough, ViT usually performs worse than ResNets under the same size, and this is because Transformer lacks an inductive bias, i.e., a priori knowledge or assumption made in advance, compared to CNN. CNN has two types of inductive bias: one is locality, i.e., adjacent regions on the image have similar characteristics; the other is translation invariance, \(f(g(x))=g(f(x))\), where g represents the convolution operation and f represents the translation operation. When the CNN has the above two inductive biases, there is a lot of prior information, and only relatively little data is needed to learn a better model. The flowchart of the ViT model is shown in Figure 2.  
A ViT block can be divided into the following four steps:  
1. Patch embedding: For example, if the input image size is 224 \(\times\) 224, the image is divided into fixed-size patches, and the patch size is 16 \(\times\) 16. Then, each image will generate 224 \(\times\) 224/16 \(\times\) 16 = 196 patches, that is, the length of the input sequence is 196, the dimension of each patch is 16 \(\times\) 16 \(\times\) 3 = 768, and the dimension of the linear  
Figure 2: ViT model structure diagram.  
projection layer is 768 \(\times\)\(N\) (\(N\) = 768). Therefore, the dimension after the input passes through the linear projection layer is still 196 \(\times\) 768, i.e., there are 196 tokens in total, and the dimension of each token is 768. A special character cls is also added here, so the final dimension is 197 \(\times\) 768. In this way, a vision problem is transformed into a seq2 seq problem with patch embedding.
2. Position coding: ViT also needs to add position coding, and the ViT position code has a total of N lines. Each line represents a vector, and the dimension of the vector and the embedded dimension of the input sequence is the same. So, after adding position code information, the dimension is still 197 \(\times\) 768.
3. Linear layer/multi-head attention mechanism/linear layer: The output dimension of the linear layer is still 197 \(\times\) 768. In the multi-head-attention mechanism, the input is mapped to QKV first. If there is only one head, the dimension of QKV is 197 \(\times\) 768; if there are 12 heads (768/12 = 64), the dimension of QKV is 197 \(\times\) 64, and there are 12 groups of QKV in total; finally, the output of 12 groups of QKV is spliced, the output dimension is 197 \(\times\) 768, and then through a linear layer, the dimension is still 197 \(\times\) 768.
4. MLP: Enlarge and scale down the dimension (197 \(\times\) 768 to 197 \(\times\) 3072, and then shrink to 197 \(\times\) 768).  
After a ViT block, the dimensions are still the same as the input, which is 197 \(\times\) 768, so multiple blocks can be stacked. Finally, the output corresponding to the special character cls is used as the final output of the encoder, i.e., the final image presentation (another approach is to take an average of all tokens without adding cls characters), and the image is classified through the MLP module finally.