# A Hyper-Attentive Multimodal Transformer for Real-Time and Robust Facial Expression Recognition
## 3 Materials and Methods
### Temporal Decoder with Cross-Attention  
FER in video streams requires not only the accurate decoding of spatial patterns within individual frames but also a deep understanding of temporal progression--how expressions evolve over time. Emotions such as fear, joy, or surprise often manifest in fleeting micro-sequences that are easily lost in static analysis. To address this, FERONet incorporates a temporal decoder with cross-attention, which models emotion transitions across video sequences by learning to selectively attend to temporally salient frames, capturing both progression and contextual dependencies. Traditional frame-level FER approaches, even when enhanced by spatial and channel attention, fall short in leveraging the continuity and inter-frame relational cues that naturally exist in dynamic emotional expressions. While recurrent units such as LSTMs or GRUs have been previously used, they often operate in a unidirectional or shallow temporal context, limiting their capacity to model nuanced emotion trajectories. FERONet addresses this temporal modeling challenge through a hybrid design that brings together the strengths of recurrent and attention-based architectures. At its core, the model employs a bidirectional gated recurrent unit (Bi-GRU) encoder, which is well-suited for abstracting temporal features by simultaneously capturing information from both past and future frames. This bidirectional flow ensures that the model does not overlook contextual dependencies that span forward or backward in time--an essential capability for decoding the progression and transitions of emotional expressions. To complement this, FERONet integrates a transformer-style cross-attention decoder, which further refines the temporal representation by selectively reweighting frame-level features based on their emotional salience. Rather than treating each frame uniformly, the decoder dynamically attends to those moments in the sequence that carry the most expressive significance, allowing the model to focus on the subtle but decisive cues that often define complex emotions. This combined framework enables FERONet to maintain a continuous memory of the temporal landscape while making informed, context-aware predictions, ultimately leading to more accurate and temporally coherent facial expression recognition. A video sequence \(V=\{F_{1},\,F_{2},\,\ldots,\,F_{T}\}\) of \(T\) frames can be processed independently through the multimodal encoder and hierarchical transformer modules, producing a sequence of spatially-encoded feature embeddings:  
\[\varepsilon=\{e_{1},e_{2},\,\ldots,\,e_{T}\},\,e_{t}\in R^{d} \tag{20}\]  
This embedding sequence is passed through a bidirectional GRU, which aggregates both past and future information:  
\[\overset{\rightarrow}{h}_{t}=GRU_{\textit{fwd}}\bigg{(}e_{t, \overset{\rightarrow}{h}_{t-1}}\bigg{)} \tag{21}\] \[\overset{\leftarrow}{h}_{t}=GRU_{\textit{fwd}}\bigg{(}e_{t, \overset{\leftarrow}{h}_{t+1}}\bigg{)}\] (22) \[h_{t}=\bigg{[}\overset{\rightarrow}{h}_{t};\,\overset{ \leftarrow}{h}_{t}\bigg{]}\in R^{2d} \tag{23}\]  
The resulting sequence \(H=\{h_{1},\,h_{2},\,\ldots,\,h_{T}\}\) encapsulates bidirectional temporal dependencies, forming the temporal context tensor. While the Bi-GRU efficiently summarizes inter-frame transitions, it treats each frame uniformly. However, not all frames are equally informative--some may capture transition points, while others remain neutral. To identify and leverage these temporal \"hotspots,\" we introduce a temporal cross-attention decoder, inspired by the decoder block of the transformer.