# Uncertainty Quantification for MLP-Mixer Using Bayesian Deep Learning
## 2 Methodology  
In this study, BDL was applied to quantify the uncertainty of the MLP-Mixer model. To better understand the approach, it is crucial to understand the working mechanism of MLP-Mixer models. MLP-Mixer is a modern architecture used for vision tasks that uses multi-layer perceptrons (MLPs) as the fundamental building blocks. The idea behind MLP-Mixer is to replace convolutional layers with MLPs, which enables the network to have greater expressive power and flexibility. The MLP-Mixer architecture is composed of three primary components: patch processing, mixer layers, and classification head. In patch processing, the input image is initially divided into non-overlapping patches of size (p \(\times\) p), where p represents the patch size. Each patch is then flattened into a vector. The mixer layers are the key building blocks of MLP-Mixer. The output tokens of the last mixer layer are processed by a global average pooling layer to obtain a single feature vector. This vector is then passed through a fully connected layer, which produces the final classification output. The entire architecture of MLP-Mixers, as proposed by [9], is illustrated in Figure 1.  
In the MLP-Mixer model, each mixer layer consists of two sub-layers: token mixing and channel mixing.  
* Token mixing: The token mixing layer applies a fully connected MLP to each patch vector, which generates a set of token vectors. These tokens represent the local information within each patch, and the MLP learns to transform the tokens to extract significant features, as mathematically given in Equation (1). \[T^{\prime}=(w_{2}\sigma(w_{1}(LN(X^{*}))+b_{1})+b_{2})+X^{*}\] (1) where \(T^{\prime}\) is the output of the token mixing layer, \(LN(X^{*})\) is the layer normalization for each token vector, \(w_{1}\) and \(b_{1}\) are weight and biases for the first dense layer in token mixing, \(w_{2}\) and \(b_{2}\) are weight and biases for the second dense layer in token mixing, with a \(\sigma\) (GELU) nonlinearity activation function between the two dense layers. Finally, the \(X^{*}\) at the end represents the skip connection;
* Channel mixing: The channel mixing layer applies another MLP to the set of token vectors generated by the token mixing layer. This MLP is applied across the tokens of each patch, allowing global information to be captured across the entire image. The  
Figure 1: MLP-Mixer architecture used in ref. [9]. Reprinted from ref. [9].  
output of the channel mixing layer is a new set of token vectors that are then fed to the next mixer layer. This is defined mathematically in Equation (2).  
\[C^{\prime}=\big{(}w_{4}\sigma\big{(}w_{3}\big{(}LN\big{(}T^{\prime}\big{)}\big{)} +b_{3}\big{)}+b_{4}\big{)}+T^{\prime} \tag{2}\]  
where \(C^{\prime}\) is the output of the channel mixing layer, \(LN(T^{\prime})\) is the layer normalization for each channel vector, \(w_{3}\) and \(b_{3}\) are weight and biases for the first dense layer in channel mixing, \(w_{4}\) and \(b_{4}\) are weight and biases for the second dense layer in channel mixing, with a \(\sigma\) (GELU) nonlinearity activation function between the two dense layers. Finally, the \(T^{\prime}\) at the end represents the skip connection.  
In this study, BDL was applied to the equations presented above to quantify the uncertainty in MLP-Mixer. Rather than using single point estimates for the weight and biases (\(w\) and \(b\)), the proposed method applied Bayesian distribution to both. However, in the utilized models, the GELU activation function from the original MLP-Mixer was replaced by ReLU in the utilized MLP-Mixers. The details of the materials and methods are presented in the following sections. Figure 2 depicts the process taken to quantify the uncertainty of models.