## 3 Proposed Approach
### DBMM as Ensemble for Single Modality and 2L-DBMM for Multimodality
The resulting updated probability serves as new weights for each base classifier in the current frame classification. The Bayesian update is expressed as follows:  
\[w_{i}^{t}=P(w_{i}|h_{i})=\frac{P(h_{i}|w_{i})P(w_{i})}{\sum_{i=1}^{N}P(h_{i}|w _{i})P(w_{i})}, \tag{3}\]  
where \(P(h_{i}\,|\,w_{i})\) is given by (2) for each ith base classifier using their previous frames posteriors and the prior \(P(w_{i})=w_{i}^{t-1}\). While the DBMM effectively combines multiple classifiers within a single modality, our novel contribution lies in extending it to a two-layered model (2L-DBMM) for multimodal fusion. The 2L-DBMM consists of two separate DBMMs, one for SER another for SA. Each DBMM operates independently on its respective modality, combining the outputs of multiple base classifiers. The final step involves a fusion layer that merges the outputs of the two DBMMs, creating a new class of emotion that combines both speech emotion and text sentiment. This approach allows for a more nuanced and comprehensive understanding of affective communication by leveraging the strengths of both modalities. This two-layered architecture (Figure 3) not only captures the complementary strengths of multiple classifiers within each modality but also allows for a more robust and nuanced understanding of affective communication by considering the interplay between speech emotion and text sentiment. Then proposed 2L-DBMM is computed as follows:  
\[P\big{(}C^{t}\big{|}A^{t}\big{)}=\frac{1}{\beta}\times\prod_{k= 1}^{t-T}\left\{\underbrace{\overbrace{P\Big{(}C^{k}\Big{|}C^{k-1}\Big{)}}^{ \text{dynamic prior}}}_{P\Big{(}C^{k}\Big{|}C^{k-1}\Big{)}}\times\sum_{y=1}^{M} \Bigg{[}w_{2y}^{k}\times\underbrace{\left(\sum_{i=1}^{N}w_{1y_{i}}^{k}\ P_{y,i} \Big{(}A^{k}\Big{|}C^{k}\Big{)}\right)}_{\text{fusion of base classifiers}} \Bigg{)}\right\}, \tag{4}\]where \(P(C^{t}\,|\,A^{t})\) is the resulting posterior; \(C\) represents classes of emotions; \(A\) represents the set of feature models; \(1/\beta\) is a normalization factor; the dynamic prior \(P(C^{k}\,|\,C^{k-1})\) is obtained from the set of posteriors from previous time slices; \(y=\{1,\ldots,M\}\) is an index to represent the mixture models = number of modalities; \(i=\{1,\ldots,N\}\) is an index to represent the base classifiers; \(t\) is the current time instant, \(k\) is an index to represent the time instants, and \(T\) is the number of time slices used in the model (i.e., number of previous posteriors used as prior to reinforce the current classification); \(P(A\,|\,C)\) denotes the output conditional probability from a learning model (i.e., base classifier); \(w_{1}\) is the first layer's weight used for fusion; and \(w_{2}\) is the second layer's weight to merge all modalities, in this study, two. In this model, the weights \(w_{y}\) can be computed through any uncertainty measure (e.g., entropy-based). It is important to highlight that the 2L-DBMM introduces a fundamental shift in how dynamic weights are updated within the ensemble learning framework. By incorporating a dynamic weighting mechanism based on the temporal evolution of classifier performance, the 2L-DBMM captures the varying reliability of different modalities over time, leading to a more adaptive and robust fusion strategy. Initially, the model receives handcrafted features based on spectral feature statistics, which feed two modified DBMM models. These models utilize uncertainty measures to assess the confidence of each inference model, both during training and real-time classification. This on-the-fly classification is enhanced by using inverse entropy-based weighting and Bayesian updates of the weights over time. This approach incorporates learned weights as priors, updating them based on the inverse entropy of the current classification, allowing for the model to adapt to changes in classifier behavior. Our research demonstrates that updating the weights in each modality (classifiers to build the ensemble) can lead to a 3% improvement in final accuracy compared to using static weights from the training set.