###### Abstract  
Sentiment analysis is pivotal in advancing human-computer interaction (HCI) systems as it enables emotionally intelligent responses. While existing models show potential for HCI applications, current conversational datasets exhibit critical limitations in real-world deployment, particularly in capturing domain-specific emotional dynamics and context-sensitive behavioral patterns--constraints that hinder semantic comprehension and adaptive capabilities in task-driven HCI scenarios. To address these gaps, we present Multi-HM, the first multimodal emotion recognition dataset explicitly designed for human-machine consultation systems. It contains 2000 professionally annotated dialogues across 10 major HCI domains. Our dataset employs a five-dimensional annotation framework that systematically integrates textual, vocal, and visual modalities while simulating authentic HCI workflows to encode pragmatic behavioral cues and mission-critical emotional trajectories. Experiments demonstrate that Multi-HM-trained models achieve state-of-the-art performance in recognizing task-oriented affective states. This resource establishes a crucial foundation for developing human-centric AI systems that dynamically adapt to users' evolving emotional needs. emotion recognition in conversation; multimodal emotion recognition; multimodal HCI dataset; human-machine consultation +
Footnote †: journal: Applied sciences  
0000-0001-5000-0000-0000-0000-0000-0000-0000-0000-0000-0000-0000-00000-00000-  
spectrum of emotional categories, such as anger and joy. While existing Emotion Recognition in Conversation (ERC) datasets, such as MELD [4; 5] and M3 ED [6], have significantly advanced emotion understanding in human-human conversations (HHCs), they fundamentally fail to model human-machine conversational contexts (HMCs). This is a critical limitation given the proliferation of intelligent dialogue systems. This critical gap manifests in three dimensions: First, current benchmarks predominantly capture symmetrical emotional exchanges between humans, neglecting the asymmetrical interaction patterns characteristic of human-machine dialogues, where users exhibit task-contingent emotional responses. Second, multimodal data, in this context, refer to the integration of information from different modalities, such as visual information from facial expressions, auditory information from vocal intonation, and textual information from language. Existing multimodal annotations fail to account for machine-specific contextual factors, including system latency profiles, task completion trajectories, and interface affordance constraints. Third, traditional emotion taxonomies prove inadequate for capturing the attenuated affective expressions (e.g., suppressed frustration during system failures) prevalent in HMC scenarios. The methodological constraints of HHC-oriented approaches become particularly apparent when analyzing Figure 1 versus Figure 2. While hierarchical LSTMs [7] and graph neural networks [8] achieve strong performance on HHC benchmarks, their direct application to HMC contexts yields suboptimal results due to divergent emotional causality mechanisms. For instance, user affective responses in HMC exhibit stronger coupling with task progression states compared to HHC's social alignment dynamics. This limitation stems from fundamental dataset deficiencies: Current resources neither record micro-temporal emotion transitions during system response delays nor annotate multimodal disfluency patterns unique to human-machine exchanges. While methodologically rigorous, the IEMOCAP corpus [9] employs laboratory-controlled HHC scenarios that poorly generalize to real-world HMC contexts. Furthermore, ecologically valid HHC collections like M3 ED lack essential HMC metadata, including system performance metrics and user adaptation trajectories.