# Uncertainty Quantification for MLP-Mixer Using Bayesian Deep Learning
## 3 Experiments and Results
The number of trainable parameters was around 8 million for both MC-Dropout and VI models. The last 10 layers of the ResNet, Inception, and DenseNet models were fine-tuned using the two datasets. Fine-tuning the pre-trained models on the utilized datasets was implemented as a means of comparing their performance with that of the custom model, as well as the potential benefit of fine-tuning the latter on the utilized datasets used in this study. For all the deep learning models, the Adam optimizer was used during the training process. The Adam optimizer is a well-known and efficient optimization algorithm for deep learning problems and is known for its rapid convergence. We used the ReLU activation function, which is known for its ability to introduce nonlinearity into a model and allow it to learn more complex representations of the data. To estimate the test data uncertainty, we took a total of 100 samples for each model. The objective of this study was to investigate and differentiate the uncertainty present in various deep learning models by using MC-Dropout and VI methods. To achieve this objective, a comprehensive comparison of the two methods was conducted, and the results were meticulously analyzed. The results obtained from the models studied are presented in Tables 1 and 2 for the two datasets to facilitate a clear and concise comparison. This study highlighted the significance of incorporating uncertainty into the development of machine learning models. The results displayed in Table 1 indicate that MC-Dropout achieves higher accuracy than VI. The mean and standard deviation of each class shows a higher level of confidence with MC-Dropout than with VI. The decrease in accuracy with VI also indicates a lower level of confidence in its predictions. This suggests that MC-Dropout is better at identifying the true class probabilities, while VI tends to overgeneralize and make predictions with less certainty. These findings are crucial in determining the appropriate model for different tasks and applications. Interestingly, the MLP-Mixer performed better than other models, except for DenseNet, while it was the second-worst model before sampling. This indicates that BDL can significantly improve the performance of MLP-Mixer. Additionally, the mean and standard deviation of each class provides insights into which classes the model performs better or worse on. Table 2 presents the results of the second dataset, which was a relatively small dataset. Despite the dataset's limited size, all models were able to efficiently capture the uncertainty associated with both the model and the data. The results indicate that the MC-Dropout approach tends to be more confident in its predictions compared to the VI version of the same model. This is evident in the higher mean and lower standard deviation of the predicted classes overall. Although the VI versions showed better accuracy in some cases, MC-Dropout demonstrated a higher level of confidence. As expected, the MLP-Mixers performed the worst in terms of accuracy. However, their performance improved significantly when BDL sampling was utilized, especially with the MC-Dropout approach. This finding is of particular interest as it suggests that MLP-Mixers can benefit greatly from incorporating uncertainty quantification. Therefore, this study highlights the importance of considering both accuracy and uncertainty when evaluating machine learning models and provides valuable insights into the potential of MLP-Mixers in real-world applications. Our experimental findings reveal a noticeable improvement in the performance of MLP-Mixers by utilizing BDL to estimate their uncertainty. Although the MLP-Mixer did not achieve state-of-the-art results, our findings suggest that with some modifications, it has the potential to further enhance its performance. Additionally, we used a basic architecture for the mixer, demonstrating its versatility.