## 1 Introduction  
Affective communication, encompassing verbal and non-verbal cues, is essential for understanding and connecting with others on an emotional level. While facial expressions and other non-verbal modalities have successfully been used to detect emotions [1; 2; 3; 4; 5; 6], speech is also a powerful channel for conveying emotional nuances. However, deciphering these emotions and sentiments embedded within speech remains a challenge, requiring advanced machine learning techniques. The implications of effectively recognizing emotional communication are far-reaching. In the domain of mental health, the ability to identify subtle emotional cues in speech holds promise for early detection and intervention of psychological distress [1; 2; 3; 4]. Additionally, affective communication analysis can transform human-robot interaction applications, empowering AI systems to respond to human emotions more accurately and empathetically [5; 6]. Beyond these domains, the insights gained from understanding affective communication can be applied to various fields, including healthcare, education, and customer service, where the ability to perceive and respond to human emotions is crucial. This study investigates affective communication, exploring how the fusion of SER and SA can deepen our understanding of emotional dynamics in conversations. Recognizing the need for tools to identify and interpret emotions and sentiments in speech, we present a novel framework integrating both domains for a more comprehensive analysis. Our primary objective is to create a framework that discerns and categorizes emotions and sentiments conveyed through speech. We utilize a hybrid methodology, employing traditional statistical features, deep learning techniques (1D-CNN), and classical machine learning algorithms (GNB, SVM, RF, MLP, and LR) to achieve this. Following the identification of the most effective individual classifiers for each dataset, we leverage our previously proposed dynamic Bayesian mixture model (DBMM) [2] for speech emotion recognition and sentiment analysis. This ensemble method, originally applied to activity recognition and facial expressions, is now adapted, and extended to the challenges of affective communication analysis. To further enhance our analysis, we incorporate sentiment derived from speech-to-text conversion, applying the same classification models to textual data. Our unique contribution lies in the integration of SER and SA through a novel two-layered dynamic Bayesian mixture model (2L-DBMM). This model, based on Bayesian inference, merges information from both domains, enabling the classification of more nuanced, second-level emotional states (Figure 1). Thus, our main contributions are as follows:  
* **Novel Feature Engineering**: Defining powerful hand-crafted features for speech emotion recognition (SER). * **Enhancement of DBMM**: We adapt and extend the DBMM, previously demonstrated in [2; 7], used for activity recognition, facial expressions, and semantic place categorization, to the domains of SER and SA. Our enhancement involves dynamically updating the classifier weights during test-set classification, rather than relying solely on pre-trained weights. This allows for the model to adapt to potential shifts in classifier performance over time. Furthermore, we employ a grid search optimization to determine the optimal number of time slices (previous priors) to incorporate in the model, enhancing the model's ability to leverage temporal information and further improve classification accuracy. * **Novel 2L-DBMM**: Extending the DBMM to a two-layered model to enable multimodal fusion. This allows for not only the merging of individual classifiers for each modality, but also the fusion of multiple modalities (e.g., SER and SA) to achieve a more robust and nuanced understanding of affective communication. This novel 2L-DBMM model is the main contribution of this work, facilitating the recognition of new classes of emotional patterns derived from combined SER and SA data.