# GCE: An Audio-Visual Dataset for Group Cohesion and Emotion Analysis
## 4 Baseline Model Development for Group Cohesion and Group Emotion Prediction
### Multi-Task Multi-Feature Fusion Framework  
The foundational structure of the baseline model designed for the prediction of group cohesion and group emotion by means of multi-feature fusion is illustrated in Figure 7. The initial phase of this process entails the input of two facial feature sequences into a visual temporal encoder, with the aim of harnessing temporal information. The visual temporal encoder is constructed utilizing the FNet ([[https://keras.io/examples/generative/text_generation_fnet/](https://keras.io/examples/generative/text_generation_fnet/)]([https://keras.io/examples/generative/text_generation_fnet/](https://keras.io/examples/generative/text_generation_fnet/)), accessed on 1 May 2024) architecture, which is based on the Fourier Transform, offering enhanced efficiency and reliability in contrast to the conventional Transformer network for temporal sequence processing. A visual representation of the FNet module can be observed in Figure 8. A global average pooling layer is employed to condense the temporal dimension, resulting in a 100-feature vector for the facial attribute branch and a 200-feature vector for the facial expression branch. We employ a fully connected layer to reduce the dimensions of the feature representations extracted from SlowFast, OpenSMILE, and Wav2 Vec 2.0. We derive 100-feature vectors for SlowFast and Wav2 Vec 2.0, whereas the OpenSMILE representation results in a 32-feature vector due to its smaller input size. We also employ the Squeeze-and-Excitation (SE) ([[https://github.com/taki0112/SENet-Tensorflow](https://github.com/taki0112/SENet-Tensorflow)]([https://github.com/taki0112/SENet-Tensorflow](https://github.com/taki0112/SENet-Tensorflow)), accessed on 1 May 2024) network to encode log-melspectrogram images. The SE network functions as a channel-specific attention mechanism designed for Convolutional Neural Networks (CNNs) with the aim of enhancing inter-dependencies among channels. This is achieved by introducing parameters that adjust the weight of each channel, making it more responsive to the important features while disregarding the less relevant ones. Figure 9 shows a visual representation of the SE block. Following this, we utilize a global average pooling layer to condense spatial information, resulting in a 64-feature vector that effectively represents the log-melspectrogram image. Figure 7: Baseline architecture for group cohesion and group emotion prediction. We have designed a fusion module incorporating Multi-Head Attention (MHA) strategies [33]. The MHA fusion module comprises three distinct layers: Input Attention, Cross-Attention, and Self-Attention. To effectively harness MHA, a set of queries, keys, and values are provided as the input and transformed independently via learned linear projections across \(h\) (the number of heads). Prior to the amalgamation of these representations, the Input Attention layer is deployed to align various attributes within the same domain. Cross-Attention plays a pivotal role in blending features from the paired modalities, where one modality aids in generating an attended map for another. In scenarios involving \(M\) modalities, there are \(M\)(\(M\)\(-\)\(1\)) cross-attention blocks. Subsequently, Self-Attention refines the representation by amalgamating the cross-modal data for each modality. Figure 10 illustrates the MHA fusion module, which accommodates both bimodal and trimodal scenarios. In our approach, we first apply the tri-modality MHA fusion modules to each of the three visual and auditory representations, obtaining fused features for each branch. Subsequently, the bimodal MHA fusion module is employed to further process these fused features. Finally, the resultant fused feature vector is fed into multiple fully connected  
Figure 8: FNet module with N-encoder blocks. Figure 9: Squeeze and excitation module. layers to predict the emotional state and the cohesion level, thus addressing the objectives of our study.