## 3 Proposed Method
### Extended Multimodal Bottleneck Transformer  
A traditional multimodal learning framework typically comprises multiple encoders extracting modality-specific features, cross-modal attention modules for learning associations between modalities (operating as mid-fusion), late fusion to generate multimodal features, and a single loss function [22]. Conversely, a typical multitask learning model includes an encoder generating a common feature and a set of dedicated decoders producing output for individual tasks [23]. Uncertainty loss is often employed to balance the loss of each task. Based on the insight from these architectures, this work proposes a multimodal learning framework that incorporates attention mechanisms using low-cost bottleneck tokens to learn cross-modal information between each modality pair. Adaptive regularization balances losses from modality embeddings and late-fusion embeddings. Figure 1 illustrates the high-level architectures of these frameworks. The proposed framework (Figure 2) employs appropriate feature extractors for each input type to generate a sequence of tokens. Transformers [24] perform self-attention for all modalities to capture intramodal interactions. Next, multilayer perceptron networks standardize the dimensions of features across all modalities. The proposed XMBT concurrently learns cross-modal patterns using special aggregation tokens identified as bottleneck tokens. Finally, classification tokens are employed for a specific multimodal adaptive loss. This work developed the XMBT method by building on the original MBT [8] designed for audiovisual understanding and generalized it to adapt to \(M\) input modalities. Unlike previous homogeneous fusion networks that follow pairwise or cascade multimodal learning, XMBT simultaneously learns the cross-modal interactions of all modalities. The bottleneck tokens prevent one modality-specific transformer from attending to entire sequences of tokens from other modalities, which may have different noise distributions and learning dynamics. Define \(M\) sequences of features representing information from \(M\) modalities as follows:  
\[\left\{ \begin{array}{l}\{x_{i}^{1}\in\mathbb{R}^{p};i=1,\ldots,N_{1}\}\\ \{x_{i}^{2}\in\mathbb{R}^{p};i=1,\ldots,N_{2}\}\\ \cdots\\ \{x_{i}^{M}\in\mathbb{R}^{p};i=1,\ldots,N_{M}\}\end{array} \right. \tag{1}\]  
Figure 1: Proposed multimodal learning framework (**a**) developed from traditional multimodal learning (**b**) and inspired by multitask learning (**c**). Figure 2: Block diagram of the proposed framework. where \(N_{1},N_{2},\ldots,N_{M}\) represent the corresponding lengths of the modal-specific sequences of feature vectors. Learnable tokens are defined as follows: \(M\) classification tokens \(x^{1}_{\text{cls}},\ldots,x^{M}_{\text{cls}}\in\mathbb{R}^{D}\), and \(M(M-1)/2\) bottleneck tokens \(x^{m,n}_{\text{bot}}\in\mathbb{R}^{D}\), where \(m\) and \(n\) are two distinct numbers ranging from 1 to \(M\). Moreover, \(x^{m,n}_{\text{bot}}\) is identical to \(x^{n,m}_{\text{bot}}\) and transfers knowledge between modality \(m^{\text{th}}\) and \(n^{\text{th}}\). All tokens from all modalities share the same feature dimension \(D\). For each modality \(m^{\text{th}}\), an organized sequence \(X_{m}\) is constructed by concatenating the series of feature vectors \(x^{m}_{i}\) with the corresponding classification token \(x^{m}_{\text{cls}}\) and all related bottleneck tokens \(x^{m,n}_{\text{bot}}\). The concatenating step is formulated as follows:  
\[\left\{ \begin{array}{l}X_{1}=[x^{1,2}_{bot},\ldots,x^{1,M}_{bot},x^{1,}_{cls },x^{1}_{1},\ldots,x^{1}_{N_{1}}]\\ \ldots\\ X_{m}=[x^{m,1}_{bot},\ldots,x^{m,m-1}_{bot},x^{m,m+1}_{bot},\ldots,x^{m,M}_{ bot},x^{m}_{cls},x^{m}_{1},\ldots,x^{m}_{N_{m}}]\\ \ldots\\ X_{M}=[x^{M,1}_{bot},\ldots,x^{M,M-1}_{bot},x^{M}_{cls},x^{M}_{1},\ldots,x^{M}_ {N_{M}}]\end{array} \right.