## 1 Introduction  
In today's data-rich environment, extracting meaningful insight from diverse sources of information has become increasingly critical across fields spanning computer vision, natural language processing, human-computer interaction, and affective computing. The convergence of data from multiple modalities, including text, images, audio, and sensor data, has fostered the emergence of the expanding field known as multimodal learning. Multimodal learning plays a crucial role in emotion recognition tasks, offering a transformative approach to understanding and interpreting human emotions from myriad data sources [1]. Emotions are multifaceted and nuanced, often expressed through verbal and nonverbal cues, such as facial expressions, speech intonation, gestures, and text [2, 3]. Recent research on multimodal learning [4, 5, 6, 7] has focused on developing deep learning models that extract cross-modal correlations and perform late fusion to produce a final output. This approach is limited to pairs of modalities, posing challenges for scaling to more complex datasets involving multiple modalities. In addition, as the number of modalities increases, the trial-and-error process becomes laborious, and expertise may not be consistently available in every scenario.  
The proposed method applies the concept of multitask learning in the context of multimodal learning. Multitask learning allows predictions from one task to enhance the prediction accuracy of related tasks. Similarly, multimodal learning employs predictions generated from specific modalities to improve predictions from other modalities or late-fusion productions. To achieve this result, this work draws inspiration from the work by [PERSON] et al. [8] on the multimodal bottleneck transformer (MBT). This novel transformer-based architecture integrates audiovisual data in multiple layers using bottleneck latent units. This study introduces the extended MBT (XMBT), which is designed to handle more than two modalities. This work evaluates the effectiveness of the proposed approach using three benchmark datasets for multimodal emotion recognition: Interactive Emotional Dyadic Motion Capture (IEMOCAP) [9], Carnegie Mellon University Multimodal Opinion Sentiment and Emotion Intensity (CMU-MOSEI) [10], and the Chinese Multimodal Sentiment Analysis dataset with independent unimodal annotations (CH-SIMS) [11]. The main contributions of this work are summarized as follows:  
1. This work introduces the XMBT for mid-state fusion, where each modality-specific branch in the model learns from the rest via bottleneck tokens.
2. This work employs a dedicated representation vector for each modality and combines them using the dynamic restrained adaptive (DRA) objective function inspired by multitask learning to calculate the loss.
3. Finally, this work presents comprehensive experiments and thorough analyses to demonstrate the effectiveness of the proposed multimodal learning framework and its ability to enhance emotion recognition performance in videos.  
The remainder of this paper is organized as follows. Section 2 introduces relevant studies on multimodal learning and emotion recognition. Next, Section 3 details the proposed framework, including the XMBT model and DRA loss function. Then, Section 4 presents the datasets and discusses the experimental results. Finally, the last section suggests future directions for further research in this area.