# A Hyper-Attentive Multimodal Transformer for Real-Time and Robust Facial Expression Recognition
## 5 Experimental Setup  
To validate the performance, efficiency, and robustness of FERONet, we conducted experiments across a range of FER datasets. These datasets were selected to reflect the diversity of real-world conditions, including static images, 3D facial data, and video-based sequences Table 1.  
Training was carried out using PyTorch 2.1 on an NVIDIA A100 40 GB GPU. All images were resized to 224 \(\times\) 224 resolution. The multimodal inputs (RGB, optical flow, and depth/landmark maps) were normalized independently, and augmentation techniques such as flip, affine transformations, jitter, Gaussian blur, and synthetic occlusion were applied dynamically during training Table 2.  
To ensure a fair and consistent evaluation, each dataset was divided into training, validation, and test subsets according to standard protocols or stratified random sampling when predefined splits did not exist. The distribution of samples for each dataset is summarized in Table 3. For video datasets, sequences were assigned such that there was no subject overlap between sets, minimizing identity-based leakage (Table 3).  
For model evaluation, we used accuracy as the primary metric, supported by F1-score and AUC-ROC to assess class-wise precision and overall discrimination performance. Temporal datasets were additionally analyzed using per-frame consistency and confusion matrices. Inference latency and memory usage were measured to assess real-time viability. FERONet was compared against multiple state-of-the-art FER architectures. These include traditional CNNs like VGG-Face and ResNet-50 with dual attention (DAN), transformer-based models like EMOFormer and FER-ViT, and our own prior work FERONET. All baselines were retrained under consistent protocols to ensure a fair comparison.  
\begin{table}
\begin{tabular}{p{113.8 pt} p{113.8 pt} p{113.8 pt} p{113.8 pt}} \hline
**Dataset** & **Type** & **Classes** & **Samples** & **Remarks** \\ \hline FER-2013 & Static (grayscale) & 7 & 35,887 images & Low resolution, high variability \\ \hline RAF-DB & Static (color) & 7 (basic) & 29,672 images & In-the-wild, diverse pose and lighting \\ \hline CK+ & Video & 7 & 593 sequences & Controlled, clean peak expression frames \\ \hline BU-3 DFE & 3D Static & 6 & 2500 textured images & Includes 3D landmarks, varied intensities \\ \hline AFEW & Video & 7 & 1345 video clips & Movie scenes, high diversity in motion and emotion \\ \hline \end{tabular}
\end{table}
Table 1: We used five representative datasets, each presenting distinct challenges.  
\begin{table}
\begin{tabular}{p{113.8 pt} p{113.8 pt}} \hline
**Parameter** & **Configuration** \\ \hline Input Resolution & 224 \(\times\) 224 pixels \\ \hline Optimizer & AdamW (weight decay = 10\({}^{-4}\)) \\ \hline Learning Rate & Cosine decay from 10\({}^{-4}\) with warm-up \\ \hline Batch Size & 64 (images), 16 (videos) \\ \hline Training Duration & 100 epochs (static), 150 epochs (video) \\ \hline Main Loss & Class-balanced cross-entropy with label smoothing \\ \hline Auxiliary Losses & Focal loss, triplet loss, temporal contrastive \\ \hline Regularization & Dropout, stochastic depth, mixup \\ \hline Augmentation & Flip, blur, color jitter, occlusion, pose shift \\ \hline \end{tabular}
\end{table}
Table 2: Summarizing the training settings.  
To evaluate the training dynamics and stability of FERONet, we plotted the training and validation loss curves across all training epochs in Figure 5. As shown, the model demonstrates smooth convergence with steadily decreasing loss on both the training and validation sets. The close alignment between the two curves further indicates that the model generalizes well and does not suffer from significant overfitting.