# Nemesis: Neural Mean Teacher Learning-Based Emotion-Centric Speaker
## 3 Results
### Ablation Study  
#### 3.2.1 Visual Encoder  
First, we evaluated the role of employing different models as the visual encoder to extract visual features. We experimented with detecting the object bounding boxes using a Faster R-CNN [48] model using a ResNet-101 [38] module pre-trained on the Visual Genome dataset [49]. Additionally, extracting grid-based features via CLIP [34] and region-based features via BLIP [35] was examined. In particular, the CLIP-RN50 \(\times\) 16 variant was utilized, which is based on an EfficientNet-style [50] scaling. On the other hand, we utilized the \(BLIP_{ViT-L/16}\) variant, which is based on the vision transformer [51] approach. As can be observed in Tables 1 and 2, the best performance was achieved by the teacher model of Nemesis utilizing CLIP-RN50 \(\times\) 16 as the visual encoder; hence, we will be referring to this exact configuration when mentioning the Nemesis. For the EGNemesis, the best performance was achieved by the student model utilizing \(BLIP_{ViT-L/16}\); therefore, this particular configuration will be referred to as the EGNemesis in the following sections. As mentioned previously, the student and teacher models had equal sizes. In addition, the student model experiencing a higher number of parameter updates provided the opportunity to outperform the teacher model. On the other hand, to experiment with handling the big data in various paradigms, these visual encoders have been utilized in two visual encoding modes. The CLIP and BLIP visual encoders have been incorporated in online visual encoding mode, where the visual feature extraction is carried out in real-time while training. On the other hand, the Faster R-CNN visual encoder has been utilized in offline visual encoding mode, where the features have been extracted before training the language models. These extracted features are stored on the disk and accessed during the training process. Data parallelism has been leveraged with the BLIP visual encoder in the XE training phase to reduce the training \begin{table}
\begin{tabular}{c c c c c c c c c} \hline \hline \multirow{2}{*}{**Model**} & \multirow{2}{*}{**Visual Encoder**} & \multicolumn{8}{c}{**Teacher Model**} \\ \cline{3-8}  & & **B-1** & **B-2** & **B-3** & **B-4** & **M** & **R** & **C** \\ \hline \multirow{3}{*}{Nemesis} & Faster R-CNN & 0.503 & 0.277 & 0.154 & 0.089 & 0.141 & 0.278 & 0.093 \\ \cline{2-8}  & CLIP-RN50 \(\times\) 16 & 0.539 & 0.311 & 0.178 & 0.106 & 0.141 & 0.294 & 0.130 \\ \cline{2-8}  & \(BLIP_{ViT-L/16}\) & 0.526 & 0.304 & 0.175 & 0.105 & 0.138 & 0.291 & 0.127 \\ \hline \multirow{3}{*}{EGNemesis} & Faster R-CNN & 0.458 & 0.233 & 0.121 & 0.066 & 0.118 & 0.242 & 0.070 \\ \cline{2-8}  & CLIP-RN50 \(\times\) 16 & 0.475 & 0.252 & 0.136 & 0.076 & 0.124 & 0.254 & 0.095 \\ \cline{1-1} \cline{2-8}  & \(BLIP_{ViT-L/16}\) & 0.470 & 0.252 & 0.137 & 0.077 & 0.123 & 0.255 & 0.099 \\ \hline \hline \end{tabular}
\end{table} 
Table 1: Performance of the teacher model utilizing different visual encoders for both the Nemesis and EGNemesis models. (B: BLEU, M: METEOR, R: ROUGE, C: CIDEr). \begin{table}
\begin{tabular}{c c c c c c c c} \hline \hline \multirow{2}{*}{**Model**} & \multirow{2}{*}{**Visual Encoder**} & \multicolumn{8}{c}{**Student Model**} \\ \cline{3-8}  & & **B-1** & **B-2** & **B-3** & **B-4** & **M** & **R** & **C** \\ \hline \multirow{3}{*}{Nemesis} & Faster R-CNN & 0.498 & 0.273 & 0.151 & 0.086 & 0.130 & 0.276 & 0.087 \\ \cline{2-8}  & CLIP-RN50 \(\times\) 16 & 0.532 & 0.304 & 0.172 & 0.102 & 0.137 & 0.290 & 0.120 \\ \cline{2-8}  & \(BLIP_{ViT-L/16}\) & 0.509 & 0.290 & 0.165 & 0.097 & 0.137 & 0.281 & 0.116 \\ \hline \multirow{3}{*}{EGNemesis} & Faster R-CNN & 0.455 & 0.233 & 0.122 & 0.066 & 0.114 & 0.243 & 0.066 \\ \cline{2-8}  & CLIP-RN50 \(\times\) 16 & 0.472 & 0.251 & 0.134 & 0.076 & 0.124 & 0.254 & 0.095 \\ \cline{1-1} \cline{2-8}  & \(BLIP_{ViT-L/16}\) & 0.479 & 0.260 & 0.141 & 0.080 & 0.129 & 0.262 & 0.099 \\ \hline \hline \end{tabular}
\end{table}