## 3 Materials and Methods
### Proposed ConvMixer Learning Model
independently through the ConvMixer architecture, allowing the model to focus on local information and efficiently capture spatial relationships within each patch. This patch-based processing is a fundamental aspect of ConvMixer's design, enabling the model to capture fine-grained features and achieve impressive performance on various tasks without the need for complex recurrent or attention mechanisms. Therefore, the first layer of ConvMixer is:  
\[Z_{0}=BNorm(\sigma Convol\to h(X,stride=p,\,kernelsize=p)) \tag{1}\]  
The second part of the model is the main ConvMixer layer, which is repeated a number of times according to the model depth. This layer consists of residual block containing a depthwise convolution. A residual block is nothing but a block where the output of a previous layer is added to the output of another later layer. In this case, the inputs are concatenated to the output of the depthwise convolution layer. This output is followed by the activation block, which is then followed by a pointwise convolution and another activation block. \[Z_{l}=BNorm(\sigma ConvolDepthwise(Z_{l-1}))+Z_{l-1} \tag{2}\]  
and  
\[Z_{l+1}=BNorm(\sigma Convolpointwise(Z_{l})) \tag{3}\]  
The third part of the ConvMixer model involves a global pooling layer to obtain a feature vector of size h from the processed patches. Global pooling reduces the spatial dimensions of each patch to a fixed size, which can then be passed to a SoftMax classifier, depending on the specific task. The activation function used in ConvMixer is GELU (Gaussian Error Linear Unit). GELU is a smooth and differentiable activation function that is known to perform well in deep neural networks. Unlike ReLU (Rectified Linear Unit), which sets all negative values to zero, GELU weighs the inputs based on their magnitude rather than gating them based on their sign. This characteristic of GELU allows it to preserve both positive and negative information in the activation, making it suitable for models like ConvMixer. \[GELU(x)=x.\,\varphi(x) \tag{4}\]  
This smooth non-linearity helps in reducing the issues of \"dying ReLU\" where neurons get stuck and stop learning due to being always inactive (zero gradient). Overall, the global pooling and GELU activation contribute to the final feature representation of the image patches, enabling the ConvMixer model to produce a compact and informative feature vector that can be used for downstream tasks such as image classification or object detection. The patch embedding in the ConvMixer model summarizes a p \(\times\) p patch from the input image into an embedded vector of dimensions e. The embedding process is achieved through a single convolutional layer with a kernel size of p, a stride of p, and h output channels. This convolutional operation takes the p \(\times\) p patch as input and transforms it into a new representation with h channels. The result of the convolutional operation is then passed through a non-linearity, which introduces non-linearity to the embedding process. The non-linearity can be the GELU activation function, which has been previously mentioned as the activation function used throughout the ConvMixer model. This patch embedding trick is used to convert the entire n \(\times\) n image into a feature map with dimensions h \(\times\) n/p \(\times\) n/p. Each h \(\times\) n/p \(\times\) n/p feature map corresponds to the embedded representation of a particular patch of size p \(\times\) p. To normalize the output of each layer and stabilize the training process, batch normalization is applied after each convolutional layer in the ConvMixer model. Batch normalization centers and scales the activations within a batch along each dimension, introducing learnable parameters for the mean and standard deviation. In this framework, BatchNorm(H) is used to apply batch normalization after the convolutional layer, where H represents the number of output channels from the convolution operation.