## 3 Proposed Approach
### DBMM as Ensemble for Single Modality and 2L-DBMM for Multimodality  
We employ a probabilistic framework to enhance the recognition of affective communication by leveraging the complementary strengths of multiple classifiers for speech emotion and sentiment analysis, extracted from both audio and text data. We utilize the DBMM, a flexible and dynamic probabilistic framework previously showcased in our work [7]. The DBMM is an ensemble of classifiers that merges conditional probability outputs from different base classifiers to enhance overall performance. Each classifier is assigned a weight based on prior knowledge and an uncertainty measure, such as confidence levels, acquired during training. This allows for the model to dynamically update classifier weights at runtime, accounting for potential variations in individual classifier performance. The DBMM also incorporates prior information to iteratively reinforce current classifications. In this study, we further enhance the DBMM with dynamic weight updates during classification, based on historical behaviors of the base classifiers. Additionally, we implement a grid-based optimization to search for the optimal number of time slices for the DBMM, determining the extent of temporal information to incorporate as new priors. The general model of DBMM is given by computing the posterior distribution P(C I A) at time instant t as follows:  
\[P\left(C^{t}\middle|C^{t-1:t-T},A^{t}\right)=\frac{\prod_{k=t}^{t-T}P\left(C^{k} \middle|C^{k-1}\right)\times\sum_{i=1}^{N}w_{i}^{t}\times P_{i}\left(A^{t} \middle|C^{t}\right)}{\sum_{j=1}^{classes}\left[\prod_{k=t}^{t-T}P_{i,j}\left( C^{k}\middle|C^{k-1}\right)\times\sum_{i=1}^{N}w_{i}^{t}\times P_{i,j}(A^{t} \middle|C^{t})\right]}, \tag{1}\]  
where \(P(C^{k}\mid C^{k-1})\) is the prior information (previous ensemble posteriors), and \(w_{i}^{t}\) is the weight for each \(i\)th base classifier, which was learned previously using any uncertainty measure applied to a dataset or to a temporal window with previous predictions from the base classifiers. \(P_{i}\) (\(A^{t}\mid C^{t}\)) is the likelihood resulted given the posterior of each base classifier. An alternative to attain \(w_{i}\) for each base classifier based on their confidence is computing the inverse entropy to update the global probabilistic model. Weights are computed by analyzing base classifiers previous outputs (i.e., learned form a dataset or from previous time instants). We can compute the inverse entropy of the posterior probabilities previously observed as follows:  
\[w_{i}=1-\left(\frac{-\sum_{k=1}^{s}P_{i,k}(\cdot)\text{log}(P_{i,k}(\cdot))}{ \sum_{i=1}^{N}(-\sum_{k=1}^{s}P_{i,k}(\cdot)\text{log}(P_{i,k}(\cdot)))}\right), \tag{2}\]  
where \(P_{i,k}(\cdot)=P_{i,k}(C\,1\,A)\) is the class conditional probability given the model of an \(i\)th base classifier and \(s\) is the number of posteriors used. During classification tasks, individual models within an ensemble may exhibit varying performance over time. To address this and improve overall classification accuracy, we employ a strategy of locally updating weights during classification. This approach identifies classifiers showing increased fluctuations in performance (e.g., frequent accuracy drops) and reduces their confidence accordingly. We adjust weights based on past performances of base classifiers on observed frames, assigning higher weights to those with better performance. Assuming the system's memory follows the Markov property during online classification, we utilize temporal information from the set of posteriors for each classifier: \(P\Big{(}C_{i}^{t}\Big{|}C_{i}^{t-1}\Big{)}\); \(P\Big{(}C_{i}^{t-1}\Big{|}C_{i}^{t-2}\Big{)}\); \(P\Big{(}C_{i}^{t-2}\Big{|}C_{i}^{t-3}\Big{)}\ldots P\Big{(}C_{i}^{t-s}\Big{|} C_{i}^{t-(s-1)}\Big{)}\). This temporal information is integrated into the Bayesian update model to compute likelihood by recalculating entropy h, coupled with weights from the previous time instant \(w_{i}^{(t-1)}\) as prior knowledge.