## 4 Experiments
### Setup
#### 4.1.2 Model Variants  
The first configuration of the ViT model utilises the conventional vision transformer structure, as described in Section 3.3, with patch extraction and standard multi-head attention. It includes an MLP head layer [48] with a Softmax activation function for multi-class image classification. The second ViT model differs in the following three ways. It employs a shifted patch tokenization layer for patch extraction, with a custom multi-head attention layer which uses a linearly scaled dot product attention mechanism with a diagonal attention mask. This attention mechanism is designed to have a lower computational complexity than the original multi-head attention. The model also includes the MLP head layer; however, instead of a Softmax activation, it returns the logits directly. The anomaly detection method is a ViT-based encoder-decoder architecture modified from the AnoViT [50], which is an unsupervised method that utilises extracted patch embeddings to reflect the global context of the image from the attention-based ViT encoder. With the embeddings of each image patch, reconstruction error at the image and pixel levels is calculated to both detect and localise image anomalies by deriving the reconstructed image's activation map at the output. Comparative supervised learning methods include three transfer learning architectures, ResNet-50, Inception-V3, and EfficientNet-B0. Modifying the original ResNet-50 architecture requires adjusting the input and output shape to [224, 224, 3], to match dataset dimensions, and the last pooling layer before the 1000 class dense layer to be extracted, so that the output can be flattened for a new dense layer to be appended. Although the new layer still uses the Softmax activation function, it is modified from the 1000 output nodes or classes to accommodate only six classes. As with the ResNet-50 model, modifying the Inception-V3 model incorporates freezing all layers except the last layer, before creating a new input layer with matching dimensions to those of the reshaped input data. The extracted output from the max pooling layer (last layer before the output layer) is flattened and linked to a newly constructed dense output layer with a Softmax activation function, where the original 1000 output nodes were redefined to six nodes or classes. The EfficientNet-B0 architecture is altered by extracting the final pooling layer before the dense layer containing 1000 classes. From here, however, a slightly different approach from the previous two models is taken. Instead of flattening the output layer, a global average pooling is applied to the data instead. This technique leaves other dimensions untouched, while applying average pooling to the spatial dimensions until each spatial dimension is reduced to one. Unlike the process of flattening, the values are not retained due to being averaged; however, global spatial information is captured by summarizing the feature maps across the entire spatial extent of the image. From here, a dense layer is added with an output of 1024 and the rectified linear units (ReLU) activation function. The final layer is another dense layer, this time with an output shape of (none, 6) to match the number of categories in the dataset and a'softmax' activation function for multi-class classification. To significantly decrease training time of the EfficientNet-B0 model, the \"efficientnet_b0_weights.h5\" weights file is utilised in the 'ModelCheckpoint' callback. The modified model is depicted in Figure 3.  
Before training the modified models, the 'Early Stopping' callback is included, which monitors the validation loss metric and ends the training session when no significant improvement to validation loss is detected over a predefined 'patience' or consecutive number of epochs. After training each model, the 10 unseen test images taken from each category (60 images total) are input for classifier prediction.