# A Hyper-Attentive Multimodal Transformer for Real-Time and Robust Facial Expression Recognition
## 3 Materials and Methods
### Hierarchical Transformer with Token Merging  
Transformer architectures have fundamentally transformed the landscape of visual representation learning by enabling models to capture long-range dependencies through self-attention. As shown in Figure 2, the token merging mechanism begins after the triple attention block, where feature maps are partitioned into patches and linearly projected into token representations. Each stage of the transformer computes token importance and softly merges low-importance tokens with their semantically closest neighbors using weighted aggregation. This results in a compressed token set composed of richer, context-aware embeddings. The merged tokens are then reshaped into a 2D spatial format and passed along the residual stream. This process enhances computational efficiency while preserving the semantic richness required for FER. However, traditional transformer designs, such as Vision Transformer (ViT), often exhibit quadratic computational complexity with respect to the input sequence length, rendering them inefficient for high-resolution or real-time applications such as FER. To address this challenge, FERONet employs a hierarchical transformer with token merging (HTTM) strategy that balances global context modeling with computational efficiency, ensuring scalability without compromising accuracy. The input to the transformer the attention-refined feature map from the previous triple attention block:  
\[F_{triple}\in R^{C\times H\times W} \tag{11}\]  
This tensor is divided into non-overlapping patches of size \(p\times p\), flattened, and linearly projected into tokens:  
\[X_{0}=PatchEmbed\left(F_{triple}\right)\in R^{N_{0}\times D},\ N_{0}=\frac{H \times W}{p^{2}} \tag{12}\]  
where \(D\) is the embedding dimension of each token (typically 96 or 128). Each token corresponds to a distinct spatial region of the input. Positional encodings \(P_{0}\in R^{N_{0}\times D}\) are added to retain spatial order: \(X_{0}^{\prime}=X_{0}+P_{0}\). The hierarchical transformer employed in FERONet is structured across \(L\) progressive stages, each designed to capture increasingly abstract representations of facial features while maintaining computational efficiency. At each stage \(l\), the model performs a series of core operations that together enable both local and global information processing. The first component is a window-based multi-head self-attention (W-MSA) mechanism, which limits the scope of attention to localized spatial windows rather than the entire token sequence. This approach significantly reduces the quadratic complexity traditionally associated with global self-attention, while still enabling rich intra-window interactions. Following the attention computation, a feed-forward network (FFN) is applied to refine the attended features through nonlinear transformations and channel-wise expansion, enhancing the expressive capacity of the representation. To further improve efficiency and representation compactness, a token merging (TM) operation is introduced at the end of each stage, except for the final one. This step strategically reduces the number of tokens by identifying and aggregating those with low semantic significance, effectively compressing the spatial resolution without discarding crucial contextual information. These three components--localized attention, deep transformation,and intelligent compression--are tightly integrated within each hierarchical level of the transformer. Together, they allow FERONet to maintain a fine balance between modeling expressiveness and scalability, enabling robust facial expression recognition under real-time constraints:  
\[X_{l}^{(1)}=W-MSA\left(LN\left(X_{l-1}^{\prime}\right)\right)+X_{l-1}^{\prime} \tag{13}\]  
\[X_{l}^{(2)}=FFN\left(LN\left(X_{l}^{\prime}\right)\right)+X_{l}^{\prime} \tag{14}\]  
\[X_{l}= \begin{cases}TM\left(X_{l}^{(2)}\right),&\text{if }l<L\\ X_{l}^{(2)},&\text{otherwise}\end{cases}  \tag{15}\]  
where \(LN\) denotes layer normalization.