# A Hyper-Attentive Multimodal Transformer for Real-Time and Robust Facial Expression Recognition
## 3 Materials and Methods
### Multimodal Feature Encoder
Its use of inverted residual blocks supports the preservation of fine-grained motion vectors--essential for understanding the progression of facial expressions over time--while maintaining a shallow depth that prevents computational overhead from escalating. This design proves especially valuable for encoding facial dynamics like blinking, brow-raising, or the evolution of a smile across frames. For structural information derived from depth maps or 3D facial landmarks, the model employs a customized version of MobileNetV3-Small. This backbone is particularly adept at handling structural reasoning tasks in visually challenging conditions, such as varying illumination or partial occlusion. Its inclusion of squeeze-and-excitation (SE) layers and hard-swish activations allows the encoder to dynamically recalibrate its feature channels, emphasizing geometrically relevant patterns such as curvature changes around the nasolabial folds or eye contours--features often central to distinguishing subtle emotional states. By mapping each modality into a common embedding space after these specialized encodings, FERONet ensures that the unique contributions of appearance, motion, and geometry are preserved and aligned in a cohesive representational format. This separation of concerns, followed by informed integration, enhances the model's ability to generalize across diverse real-world conditions, enabling robust and resilient facial expression recognition even in the face of environmental variability and inter-subject differences. Each encoder maps its input to a common feature embedding space with dimensionality \(R^{\text{C}\times H\times W}\) where C = 64, and H = W = 28. This shared feature space enables seamless modality alignment and interaction. Following modality-specific encoding, features are integrated via a Gated Multimodal Fusion (GMF) module designed to emphasize salient information from each modality while suppressing noise or redundancy. The fusion process proceeds in two stages: (1) soft modality gating each modality's feature tensor \(F_{m}\) is passed through a gating function Equation (1) [22]:  
\[G_{m}=\sigma(W_{m}\cdot GAP(F_{m})+b_{m}) \tag{1}\]  
where \(\sigma\) is the sigmoid activation, \(W_{m}\) and \(b_{m}\) are learnable weights and biases, and GAP denotes global average pooling. This generates an importance score \(G_{m}\in R^{\text{C}}\) for each modality. (2) weighted summation and projection where the gated features are aggregated Equation (2):  
\[F_{fused}=\sum_{m=1}^{M}G_{m}\odot F_{m} \tag{2}\]  
followed by a projection via a 11 \(\times\) 1 convolution to unify channel dimensions. Here, \(\odot\) denotes element-wise multiplication, and MMM is the number of modalities. To further refine the fusion, a channel-wise attention layer (CWAL) is applied to the fused feature, enhancing inter-channel dependencies and highlighting modality synergies. The output is a unified feature tensor \(F_{multi}\in R^{\text{C}\times H\times W}\), which is then forwarded to the subsequent attention and transformer modules. The integration of motion and depth signals alongside appearance data is rooted in cognitive neuroscience, where human emotion recognition relies not only on visual patterns but also on movement and 3D structure perception. For instance, a raised eyebrow (surprise) may be visually subtle but geometrically distinct; similarly, smiling may involve subtle muscle contractions more evident in optical flow than static RGB.