# A Hyper-Attentive Multimodal Transformer for Real-Time and Robust Facial Expression Recognition
## 2 Related Works
### Multimodal and Cross-Domain FER  
Recognizing the limitations of unimodal FER systems, researchers have explored the integration of additional sensory modalities such as depth, thermal imaging, and optical flow to enrich facial representations. Optical flow captures motion vectors between frames, making it particularly useful for analyzing muscle dynamics over time. Depth and 3D landmark maps provide structural information that complements appearance features, especially under occlusion or poor lighting. Recent works like the Multimodal Relational Attention Network (MRAN) [28] and Multimodal Facial Expression Recognition (MMFER) have sought to fuse such heterogeneous inputs using attention-based mechanisms [29]. While these models reported gains in robustness and accuracy, they typically require complex fusion strategies and were computationally intensive. Moreover, many of these models lacked mechanisms to adaptively weigh modalities based on their informativeness or reliability in different scenarios. Facial expressions are inherently dynamic, unfolding over sequences of frames [30]. Capturing these temporal patterns is critical for accurate emotion recognition, particularly in spontaneous or subtle expressions. Early attempts at temporal modeling employed Recurrent Neural Networks (RNNs) [16], Long Short-Term Memory (LSTM) units [31], and Gated Recurrent Units (GRUs) [32], which enabled FER systems to learn time-series dependencies [33]. However, these architectures often modeled only unidirectional or shallow temporal relationships, leading to suboptimal performance in rapidly changing or ambiguous expression sequences. More recent efforts have incorporated spatiotemporal convolutional networks (3D CNNs) or two-stream architectures that process RGB and optical flow in parallel [34]. While these methods improve motion sensitivity, they often lack the interpretability and selective focus provided by attention mechanisms [35]. Additionally, temporal modeling is frequently uniform across frames, failing to prioritize emotionally informative segments of the sequence [36]. While substantial progress has been made in areas such as spatial attention, temporal reasoning, and multimodal fusion, existing FER models seldom manage to unify these components within a single, coherent, and real-time framework. Many architectures are designed with a focus on spatial precision, often at the expense of computational efficiency, while others introduce temporal and multimodal capabilities that are not easily scalable or interpretable in practical deployments. FERONet directly addresses these limitations by proposing a comprehensive and integrated architecture. It incorporates a triple-attention mechanism that simultaneously attends to spatial regions, semantic channels, and cross-patch relationships; a multimodal encoder that jointly processes RGB images, optical flow, and structural information derived from depth or 3D landmarks; and a hierarchical transformer design that leverages token merging to reduce computational burden while preserving representational richness. Furthermore, the architecture features a bidirectional temporal decoder equipped with cross-attention, enabling it to capture and emphasize critical emotional transitions within video sequences. By bridging these dimensions through a unified and computationally efficient design, FERONet establishes a new paradigm for FER--combining high accuracy, interpretability, and scalability for deployment in unconstrained, real-world environments.