# Uncertainty Quantification for MLP-Mixer Using Bayesian Deep Learning
## 2 Methodology
### Uncertainty Quantification  
BDL is a statistical approach that combines deep learning with Bayesian methods to estimate the uncertainty of model predictions. BDL estimates the probability distribution of model parameters based on observed data, allowing for the calculation of both point estimates and associated uncertainties. This method is particularly useful in classification problems, where the goal is to predict categorical labels given a set of inputs. Figure 4: Samples from the breast cancer dataset: (**A**) normal; (**B**) benign; and (**C**) malignant. There are several methods to estimate uncertainty in BDL, but two popular methods are VI and MC-Dropout. Both methods are based on [PERSON] theorem, which states that:  
\[P(A|B)=\frac{P(A)\ P(B|A)}{P(B)} \tag{3}\]  
where \(P(A|B)\) is the posterior probability of \(A\) given \(B\), \(P(B|A)\) is the likelihood of \(B\) given \(A\), \(P(A)\) is the prior probability of \(A\), and \(P(B)\) is the marginal likelihood or evidence. The predictive distribution can be calculated as:  
\[P(y^{*}|x^{*},D)=\int P(y^{*}|x^{*},\omega)P(\omega|D)d\omega\approx\int P(y^{ *}|x^{*},\omega)q(\omega)d\omega \tag{4}\]  
In VI, the assumption is that the true posterior distribution \(P(w|D)\) (also denoted as \(P(w|X,Y)\) in the context of machine learning) of the model parameter \(w\) can be approximated with a simpler distribution \(q(w)\) that belongs to a tractable family of distributions. The goal of VI is to find the best approximation \(q(w)\) by minimizing the Kullback-Leibler (KL) divergence between the true posterior distribution \(P(w|D)\) and the approximate distribution \(q(w)\) distribution:  
\[KL(q(\omega)\|P(\omega|D)=\sum_{i=1}^{n}q_{i}(\omega).(\log q_{i}(\omega)- \log P_{i}(\omega|D)) \tag{5}\]  
The KL divergence measures the distance between the two distributions and is non-negative, with a value of 0 only when the two distributions are identical. Therefore, minimizing the KL divergence is equivalent to finding the best approximation \(q(w)\) that is closest to the true posterior distribution \(P(w|D)\). However, the MC-Dropout method involves using dropout during both training and testing to estimate the model's uncertainty. During training, dropout is applied to the inputs and/or hidden units of the neural network with a certain probability. This creates an ensemble of different neural networks that share weights, with some neurons randomly dropped out. The loss function is computed for each dropout run, and the weights are updated based on the average of the losses. During testing, the model is run multiple times with dropout applied, and the predictions are averaged over the runs. The dropout rate is considered a hyperparameter that controls the model's uncertainty. The final predictions are given by:  
\[P(y^{*}|x^{*},D)=\int P(y^{*}|x^{*},\omega)P(\omega|D)d\omega\approx\frac{1}{T} \sum_{t=1}^{T}p(y^{*}|x^{*},\omega) \tag{6}\]  
where \(p(y^{*}|x^{*},\omega)\) is the predicted probability of class \(i\) for the \(t\) th dropout run, and \(T\) is the number of dropouts runs. This method can be viewed as an approximation of Bayesian model averaging, where the dropout masks are treated as different models, and the predicted probabilities are averaged to estimate the expected probability of each class given the input \(x^{*}\). After computing the predicted probabilities for each class using MC-Dropout, the final predicted class is obtained by applying the argmax function to the predicted probabilities. The argmax function returns the index of the class with the highest predicted probability. Both VI and MC-Dropout are effective approaches that can enhance the reliability and resilience of deep learning models. VI provides a probabilistic framework for estimating complex posterior distributions, allowing for more accurate uncertainty estimations for individual neurons. On the other hand, MC-Dropout provides a computationally efficient way to estimate model uncertainty through the use of dropout masks during both training and testing.