Figure 1: Example conversations that convey different semantic meanings depending on the emotion. In this paper, we propose EmoSDS, an Emotional Spoken Dialogue System that leverages self-supervised speech representations to unify speech and emotion recognition within a single framework. Unlike conventional approaches that rely on cascaded ASR and external emotion encoders, EmoSDS directly processes speech features within the LLM, enabling the simultaneous capture of both linguistic content and emotional cues. To achieve this, we leverage the structured representation properties of SSL features [1]. Specifically, we apply k-means clustering to SSL features to obtain discrete linguistic tokens, while utilizing the residuals as continuous paralinguistic embeddings to preserve emotional and prosodic attributes. This approach enables a speech-language multimodal architecture that effectively integrates paralinguistic features and linguistic tokens eliminating the need for external emotion encoders. By leveraging the strong representational capabilities of the LLM, EmoSDS captures fine-grained emotional expressions, enhancing the naturalness of generated responses. Furthermore, we adopt a multi-stage training strategy to enable the LLM to learn progressively, starting with fundamental tasks, such as ASR and speech emotion recognition (SER), and moving onto more complex objectives, including response emotion conditioning and text generation. Specifically, we introduce a three-stage training pipeline: in Stage 1, the model is trained on the ASR task; in Stage 2, it learns both ASR and SER; and in Stage 3, it incorporates response emotion conditioning and text generation. This structured training approach allows the LLM to gradually learn linguistic and paralinguistic features, resulting in more emotionally aligned and contextually rich responses. Finally, to mitigate the scarcity of high-quality emotional speech conversation datasets, we utilize the emotional speech database (ESD) [11] with GPT-4o-mini [12] to generate diverse emotional conversations, constructing an expanded emotional speech conversation dataset, EmoSC. This dataset not only improves emotional expressiveness but also increases the number of samples across different emotion categories, ensuring a more balanced and comprehensive dataset for training. As a result, the system can generate more emotionally nuanced and contextually appropriate responses. The experimental results demonstrate that EmoSDS effectively recognizes both linguistic and emotional features from speech inputs and outperforms baseline systems in response emotion and text generation quality. Furthermore, comparative analyses show that our newly introduced dataset improves the diversity and balance of generated response emotions, enabling the system to engage in more emotionally expressive and contextually sensitive conversations. Our contribution can be summarized as follows:  
1. We propose EmoSDS, an Emotional Spoken Dialogue System that unifies speech and emotion recognition within a single LLM, enabling an end-to-end speech understanding while effectively capturing both linguistic content and emotional cues by incorporating continuous paralinguistic features. 2. We introduce a three-stage training pipeline that progressively improves the LLM's learning of content and emotional representations for more expressive and contextually aligned responses. 3. We propose EmoSC, a balanced emotional speech dataset that enhances expressiveness and supports diverse, emotionally rich dialogue generation. The remainder of this paper is organized as follows. Section 2 reviews related work, while Section 3 presents the proposed method in detail. Section 4 describes the experimental setup and Section 5 provides the experimental results. Finally, Section 6 concludes this paper.