# A Hyper-Attentive Multimodal Transformer for Real-Time and Robust Facial Expression Recognition
## 3 Materials and Methods
### Multimodal Feature Encoder  
In real-world FER scenarios, relying solely on single-modality input, particularly static RGB imagery, often proves inadequate due to the presence of complex environmental variables such as partial occlusion, inconsistent lighting conditions, and non-static, evolving expressions. These challenges hinder the robustness and adaptability of traditional FER systems, particularly in unconstrained settings. To address this, FERONet employs a com  
Figure 1: Overview of the FERONet architecture. prehensive multimodal feature encoding framework that synthesizes visual appearance, motion dynamics, and structural geometry by integrating three distinct input modalities: RGB, optical flow, and depth or 3D landmark maps. Each modality contributes unique and complementary information that enhances the model's ability to detect, differentiate, and generalize across a wide range of facial expressions. The RGB input captures the surface-level appearance of the face, encoding crucial features such as color gradients, skin texture, and morphological contours. This modality serves as the baseline for expression analysis, particularly for recognizing fine-grained variations in static imagery. However, static frames alone cannot capture the dynamic nature of facial movements. To supplement this, optical flow is introduced as a motion-sensitive modality, encoding temporal displacements of facial regions across successive video frames. Computed using the TV-L1 algorithm, these motion vectors are represented as two-channel horizontal and vertical maps and are fused with RGB data during training, allowing the model to infer subtle transitions in facial musculature that signal emotional change. Complementing both appearance and motion, structural features are extracted via depth or 3D landmark representations, which provide insight into the spatial geometry of facial expressions. These features are particularly beneficial when distinguishing between expressions that exhibit minimal visual disparity but significant geometric deviation. In scenarios lacking dedicated depth sensors, pseudo-depth maps are generated from 2D landmarks using parametric models such as 3D Morphable Models (3 DMM) or estimated through neural frameworks like Face Alignment Networks (FAN). Together, these three modalities form a cohesive input structure that empowers FERONet to navigate the ambiguities of real-world FER with heightened accuracy and resilience, significantly surpassing the limitations of unimodal approaches. Each modality is normalized independently to account for scale and distributional differences. All inputs are resized to 224 \(\times\) 224 pixels and subjected to modality-specific augmentations (e.g., random jitter for optical flow, illumination simulation for RGB, and rotation for depth maps) to improve generalization. To effectively harness the distinct informational value embedded in each input modality, FERONet adopts a thoughtfully modular architecture in which each data stream is processed through a dedicated, lightweight, yet expressively designed backbone network. This separation ensures that the semantic and statistical properties specific to appearance, motion, and structure are preserved and learned in a manner most conducive to accurate emotion recognition. For RGB imagery--the primary source of visual appearance--FERONet employs ConvNeXt-Tiny as its principal encoder. This architecture strikes an ideal balance between computational efficiency and representational depth. By incorporating depth-wise separable convolutions and layer scaling, ConvNeXt-Tiny enables the model to retain rich semantic content, capturing subtle yet critical facial cues such as micro-tensions in the skin or nuanced shifts in muscle activation, all while maintaining a low inference burden. To process motion information, FERONet integrates a compact variant of EfficientNet-B0, pretrained on dynamic action datasets to prime the network for sensitivity to temporal displacements.