# A Hyper-Attentive Multimodal Transformer for Real-Time and Robust Facial Expression Recognition
## 3 Materials and Methods
### Triple Attention Block  
Facial expressions are composed of subtle, spatially distributed muscle movements whose relevance varies depending on the emotion being conveyed. Consequently, accurate FER demands a dynamic focus on expressive regions, informative channels, and inter-patch dependencies. To address this, FERONet introduces a triple attention block that integrates three complementary attention mechanisms: spatial attention, channel attention, and cross-patch attention. This hierarchical design enables the model to selectively enhance salient features both within and across facial regions, achieving greater discriminability under varying pose, illumination, and occlusion conditions. Spatial attention identifies which regions of the face contribute most significantly to a given emotional expression--typically the eyes, eyebrows, mouth, and nasolabial folds. In FERONet, spatial attention is computed over the fused multimodal feature map \(F\in R^{C\times H\times W}\) using a convolutional gating mechanism that highlights key spatial regions. The formulation is as follows Equation (3) [37]:  
\[M_{\rm s}=\sigma(Conv_{3\times 3}(F))\in R^{1\times H\times W}\]  
\[F_{s}=F\odot M_{s} \tag{3}\]  
where \(\sigma\) denotes the sigmoid function, \(Conv_{3\times 3}\) is a convolution with a kernel size of 3, \(M_{\rm s}\) is the spatial attention map, and \(\odot\) indicates element-wise multiplication. The resulting tensor \(F_{s}\) retains the original shape but has increased focus on discriminative spatial regions. This process effectively suppresses background noise and uninformative facial areas, enabling the model to attend to critical zones of expression. While spatial attention refines \"where\" to look, channel attention determines \"what\" to look for by learning inter-channel dependencies. Each channel in the feature map often represents a different semantic concept Equation (4). However, not all channels contribute equally across different expressions, Equation (5). To dynamically weigh channels, we employed a modified squeeze-and-excitation (SE) block Equation (6):  
\[z=GAP(F_{s})\in R^{C} \tag{4}\]  
\[M_{c}=\sigma(W_{2}\times ReLU(W_{1}\times z))\in R^{C} \tag{5}\]  
\[F_{sc}=F_{s}\odot M_{c} \tag{6}\]  
here \(W_{1}\in R^{\frac{C}{p}\times C}\) and \(W_{2}\in R^{C\times\frac{C}{p}}\) are the weights of a two-layer multi-layer perceptron (MLP), and \(r\) is the reduction ratio (typically 8 or 16). The output \(M_{c}\) acts as a soft gating mechanism across the channel dimension. The modulated tensor \(F_{sc}\) enhances informative features (e.g., smile curvature or eyebrow lift) while diminishing irrelevant activations. Facial expressions often involve coordinated activity across distinct facial regions--such as simultaneous widening of the eyes and opening of the mouth in surprise. These long-range interactions are difficult to capture using only local convolutional filters. FERONet addresses this limitation by incorporating a cross-patch attention (CPA) module based on vision transformer (ViT) principles, adapted for efficient facial feature modeling, Equation (7). The process begins by flattening and reshaping the spatially and channel-refined tensor \(F_{sc}\in R^{C\times H\times W}\) into a sequence of tokens:  
\[X=Flatten\ (F_{sc})\in R^{N\times C},\ N=H\times W \tag{7}\]  
Positional embeddings \(P\in R^{N\times C}\) are added to encode spatial layout: \(X^{\prime}=X+P\) Self-attention is then computed as Equation (8):  
\[Q=X^{\prime}W_{Q},\ K=X^{\prime}W_{K},\ V=X^{\prime}W_{V} \tag{8}\]  
\[A=\text{Softmax}\Bigg{(}\frac{QK^{T}}{\sqrt{d_{k}}}\Bigg{)},\ F_{CPA}A\times V \tag{9}\]  
where \(W_{Q},\ W_{K},W_{V}\in R^{C\times d_{k}}\) are learned projections, and \(d_{k}\) is the dimension of the key vectors Equation (9). The attention scores \(A\in R^{N\times N}\) represent the relevance between all patch pairs, allowing the model-to-model co-dependencies between distant facial landmarks.