# HyFusER: Hybrid Multimodal Transformer for Emotion Recognition Using Dual Cross Modal Attention
## 3 Background
### Speech Feature Extraction: HuBERT Modal  
HuBERT (Hidden Unit BERT) [20] is a self-supervised learning-based model designed to extract high-dimensional and multifaceted features from speech signals. It is specifically developed to efficiently learn both linguistic and non-linguistic information from speech representations. The core idea of HuBERT is to effectively capture the inherent structure and complex acoustic characteristics of speech signals, enabling it to learn rich representations that achieve superior performance in various downstream tasks. To achieve this, HuBERT undergoes a two-stage learning process and leverages CNNs and transformer encoders to extract both low-level and high-level features from input speech. Figure 2 illustrates the architecture of the HuBERT model. \begin{table}
\begin{tabular}{c c c c c c c c c} \hline \hline  & **NSMC** & \begin{tabular}{c} **Naver** \\ **NER** \\ **(F1)** \\ \end{tabular} & \begin{tabular}{c} **PAWS** \\ **(acc)** \\ \end{tabular} & \begin{tabular}{c} **KorNLI** \\ **(acc)** \\ \end{tabular} & \begin{tabular}{c} **KorSTS** \\ **(specarman)** \\ \end{tabular} & \begin{tabular}{c} **Question** \\ **Pair** \\ **(acc)** \\ \end{tabular} & \
\begin{tabular}{c} **KorQuD** \\ **(Dev)** \\ **(EM/F1)** \\ \end{tabular} &
\begin{tabular}{c} **Korean-** \\ **Hate-Speech** \\ \end{tabular} \\ \hline KoBERT & 89.59 & 87.92 & 81.25 & 79.62 & 81.59 & 94.85 & 51.75 & 66.21 \\ XLM- & 89.03 & 86.65 & 82.80 & 80.23 & 78.45 & 93.80 & 88.94 & 64.06 \\ HanBERT & 90.06 & 87.70 & 82.95 & 80.32 & 82.73 & 94.72 & 78.74/ & 68.32 \\ KoELECTRA- & 90.33 & 87.18 & 81.70 & 80.64 & 82.00 & 93.54 & 60.86/ & 66.09 \\ KoELECTRA- & 89.56 & 87.16 & 80.70 & 80.72 & 82.30 & 94.85 & 84.01/ & 67.45 \\
**KoELECTRA-** & **90.63** & **88.11** & **84.45** & **82.24** & **85.53** & **95.25** & **84.83/** & **67.61** \\ \hline \hline \end{tabular}
\end{table} 
Table 1: KoELECTRA base model performance table. Figure 2: HuBERT model structure. The first stage, Offline Clustering, involves dividing speech data into multiple clusters based on acoustic similarity using unsupervised learning techniques. The K-Means algorithm is commonly employed for this process, assigning segments with similar acoustic features to the same cluster, thereby automatically learning representative 'acoustic units (labels)' for each segment. These cluster labels are later used as learning targets during the masked prediction stage. The second stage, Masked Prediction Learning, is the core mechanism of HuBERT training. After setting the acoustic unit labels obtained through offline clustering as the learning targets, specific regions of the input speech data are intentionally masked, making them inaccessible to the model. The model is then trained to infer the cluster labels of the masked regions using only the information from the unmasked segments. This process is mathematically represented by Equation (3). Through this learning process, the model acquires the ability to reconstruct missing parts based on partial information, ultimately gaining a deeper understanding of the structural characteristics and contextual relationships within speech signals. \[L_{m}(f;X,\ M,\ Z)=\sum_{t\in M}logp_{f}(z_{t}|\hat{X},\ t) \tag{3}\]  
In terms of feature extraction, HuBERT employs a multi-stage approach. First, raw audio signals are transformed into low-level acoustic features using a Convolutional Neural Network (CNN). The CNN effectively captures local patterns along the time and frequency axes (e.g., energy distribution, frequency variations, and intensity of articulation), generating initial feature vectors. Next, a transformer-based encoder processes these low-level features, learning to infer high-level characteristics such as utterance context, speaker intent, emotion, and style. By leveraging multi-head attention mechanisms, the transformer encoder captures long-term dependencies and reconstructs rich representation vectors that reflect the interactions between various elements within the speech data.