## 3 Proposed Method
### ViT-Based Anomaly Detection  
Each transformer encoder layer consists of a multi-head self-attention mechanism followed by a feed-forward neural network. The self-attention mechanism allows the model to capture global dependencies between different patches or tokens of the input image. After each patch is passed through the stack of transformer encoder-decoder layers, normal and defective image features are refined and represented as feature vectors in high-dimensional space. The final output of the ViT model is a sequence of feature vectors, where each vector corresponds to a specific position or patch in the input image. Using the features extracted from intermediate layers of the trained vision transformer models, normal and defective regions are predicted in the test dataset to produce prediction weights. In this paper, two ViT-based anomaly detection methods are considered. The first ViT-based anomaly detection method is to calculate the dot product similarity between the extracted feature vectors and the feature vectors of input image, as well as the predicted class of the input image. By reshaping the calculated similarity scores to original image dimensions, it is possible to generate a heatmap that highlights anomalous data regions with pixel-level localisation of defects. The sensitivity of the classification outputs can be adjusted by tuning the anomaly threshold value to meet specified industry requirements. The second ViT-based anomaly method is the modified AnoViT [50], whereby the distribution of the normal data is learned from the differences between the ViT-based encoder-decoder image reconstruction and the original input image. Pixel-level 12-distances between the two are calculated, as well as the average pooling across the channels to produce a score map. As an unsupervised learning method, the model is trained on normal data only and is therefore only able to learn the distribution of non-defective images. This means that abnormal or defective images will cause an increase in reconstruction error, as anomalous regions will be reconstructed incorrectly. By integrating the ViT-based encoder into an encoder-decoder architecture, the AnoViT model [50] leverages core vision transformer characteristics to achieve an improved method of anomaly detection and localisation. An architectural diagram of the AnoViT model is shown in Figure 10,where \(E_{[n]}\) denotes embedded patches after linear projection, \(E_{pos}^{n}\) are positionally encoded patches, and \(E_{[n]}^{\prime}\) are the output patch embeddings. By applying a multi-head self-attention (MSA), the relationship between patches is learned in order to utilise global information. Additionally, by processing patch-level images, the approach creates image embeddings with rich information for each location. Typical ViT models consider image patches of a specific size as tokens in the NLP tasks. When performing classification, the 'CLS' token is added at the beginning of the resulting sequence:  
\[[x_{class}],\left[x_{p}^{1},\ldots,\ x_{p}^{N}\right], \tag{2}\]  
where \(x_{p}^{i}\) are image patches. Where the architecture has multiple layers, the state of the CLS token on the output layer is used for classification. Unlike these models, the AnoViT excludes the patch embedding's \(E_{[cls]}\) token and additional MLP head, then creates a feature map by rearranging the remaining embeddings to match the existing positions in the patch image and used as the feature map \(\Gamma\). From the reconstruction error, the activation map is generated for each input image depicting anomalous regions of interest, such as surface defects. Having been trained on normal data, the model can learn the distribution of normal data only. Owing to this, an atypical input image causes reconstruction error to increase as the model has difficulty reproducing anomalous regions.