# MLPs Are All You Need for Human Activity Recognition
## 3 Methodology  
### MLP-Mixer  
The MLP-Mixer (Mixer) does not use convolutions or self-attention mechanisms and is instead made up entirely of MLPs. Even with a simpler architecture than CNNs and transformers, the Mixer produces competitive results in computer vision tasks against state-of-the-art models. The Mixer only uses basic matrix multiplication, changes to data layout, and scalar non-linearities, resulting in a simpler and faster model. The Mixer has a similar architecture to the ViT; however, the Mixer's structure has benefits in terms of speed by allowing linear computation scaling when increasing the number of input patches instead of quadratic scaling in the case of the ViT. Figure 1 illustrates the MLP-Mixer architecture. The input is divided into unique patches that do not overlap. The patches are linearly projected into an embedding space. In contrast to the transformer and ViT, the input does not need positional embeddings as the Mixer is sensitive to the position of the inputs in the token-mixing MLPs [8]. The Mixer consists of two types of MLP layers: the token-mixing layer and the channel-mixing layer. The inspiration behind this is that modern vision neural architectures, according to [8], (1) mix their features at a given spatial location across channels and (2) mix their features between different spatial locations. CNNs implement (1) with a convolution layer through the \(1\times 1\) convolution operation; and (2) using large kernels and by adding multiple convolution layers with pooling, which decreases the input spatially. In attention-based models, both (1) and (2) are performed within each self-attention layer. The Mixer's purpose is to separate per-location operations (1) and cross-location operations (2). These features are achieved through two layers, called \"token-mixing\" and \"channel-mixing\", representing the per-location and the cross-location operations, respectively. Each unique patch has identical dimensions. The number of patches is calculated by dividing the input dimensions \((H,W)\) by the patch resolution \((P,P)\), \(S=HW/P^{2}\). The  
Figure 1: Annotated MLP-Mixer architecture with token-mixing annotated on the left and channel-mixing annotated on the right. Image from [8]. sequence of non-overlapping patches is projected into an embedding space with dimension \(C\), resulting in a matrix of dimensions \(S\times C\). The layers in the Mixer are all the same size and are made up of two MLP blocks each. * The first block is the token-mixing MLP; the input matrix is normalised and transposed to allow the data to mix across each patch. The MLP(MLP1) will act on each column of the input matrix, sharing its weights across the columns. The matrix is transposed back into its original form. The overall context of the input is obtained by feeding each patch's data into the MLP. This token-mixing block essentially allows different patches in the same channel to communicate. * The second block is the channel-mixing MLP; this receives residual connections from its pre-normalised original input to prevent information from being lost during the training process. The result is normalised, and a different MLP(MLP2) performs the channel-mixing with a separate set of weights. The MLP acts on each input matrix row, and its weights are shared across the rows. A single patch's MLP receives data from every channel, enabling communication between the information from various channels. Each MLP block contains two feed-forward layers with a GELU [30] activation function applied to each row of the input data. The Mixer layers are calculated in Equation (1) (the layer index is not included), and the GELU function is demonstrated in Equation (2).