## 6 Conclusions  
This paper presented EmoSDS, an Emotional Spoken Dialogue System that unifies speech and emotion recognition within a single LLM-based framework. Unlike conventional SDS architectures that rely on cascaded ASR or external emotion encoders, EmoSDS directly feeds self-supervised speech representations into the LLM, enabling end-to-end speech understanding while effectively capturing both linguistic content and emotional cues. Our approach leverages structured SSL feature representations, where discrete linguistic tokens are extracted through k-means clustering, and continuous residual embeddings preserve paralinguistic attributes. This enables the seamless integration of emotional and prosodic information, eliminating the need for external emotion encoders. To further enhance performance, we introduced a three-stage training pipeline, progressively training the model on ASR, SER, and response generation, allowing for the better learning of linguistic and paralinguistic features. Through extensive evaluations, we demonstrated that EmoSDS achieves state-of-the-art performance in both emotion recognition and response generation. Our results show that residual input is essential for capturing paralinguistic cues, significantly improving emotion recognition accuracy, particularly in datasets where the same content is spoken with different emotional states. Additionally, stage-wise training enhances both transcription accuracy and emotional expressiveness, resulting in more coherent and emotionally adaptive responses. Furthermore, our findings indicate that LLMs can effectively process continuous paralinguistic features, leveraging prosodic variations to improve emotional understanding and response generation. To address the scarcity of high-quality emotional speech conversation datasets, we introduced EmoSC, a balanced, emotional speech dataset constructed using GPT-4o-mini and the ESD corpus. EmoSC significantly improves emotion recognition robustness, reduces data imbalance, and enhances the LLM's ability to generate diverse and emotionally nuanced responses. Our findings confirm that emotionally expressive datasets and structured training pipelines are critical for advancing spoken dialogue systems. By integrating continuous paralinguistic features and multi-stage learning, EmoSDS bridges the gap between linguistic understanding and emotional expressiveness, offering a more natural and engaging conversational experience. Future research can explore a fully end-to-end approach that eliminates not only the ASR front-end but also the LLM-TTS pipeline, enabling direct speech-to-speech interaction. Additionally, leveraging LLM methods capable of processing continuous paralinguistic features could extend the model's applicability to more diverse and generalizable speech-related tasks, such as speaker recognition, speaker diarization, pitch estimation, and energy prediction, which are challenging for discrete, token-based LLMs. This could lead to a highly generalized speech-language multimodal LLM capable of capturing richer acoustic and linguistic information. \begin{table}
\begin{tabular}{c c c c c c} \hline \hline
**Pipeline** & **Input Acc.\(\uparrow\)** & **WER \(\downarrow\)** & **CER \(\downarrow\)** & **BLEU \(\uparrow\)** & **BERTScore \(\uparrow\)** & **ROUGE-L \(\uparrow\)** \\ \hline w/o residual & 58.4\% & 0.6\% & 0.4\% & 9.7\% & 87.8\% & 39.7\% \\ w/o (stage 1 + 2) & 90.0\% & 7.6\% & 5.1\% & 8.5\% & 87.7\% & 38.9\% \\ w/o (stage 1) & 96.0\% & 1.9\% & 1.1\% & 8.9\% & 87.6\% & 38.8\% \\ w/o (stage 2) & 96.4\% & 0.1\% & 0.1\% & 10.1\% & 87.7\% & 40.3\% \\ EmoSDS & 98.0\% & 0.1\% & 0.1\% & 10.4\% & 88.0\% & 40.5\% \\ \hline \hline \end{tabular}
\end{table} 
Table 5: Ablation study results for the three-stage training pipeline and the impact of residual input. An upward arrow (\(\uparrow\)) indicates a higher value, signifying better performance, while a downward arrow (\(\downarrow\)) indicates the opposite. Furthermore, our current framework is constrained to five discrete emotion categories.