ParaS2S: Benchmarking and Aligning Spoken Language Models for Paralinguistic-aware Speech-to-Speech Interaction
Authors: Shu-wen Yang
Ming Tu
Andy T. Liu
Xinghua Qu
Hung-yi Lee
Lu Lu
Yuxuan Wang
Yonghui Wu
Venue: ICLR 2026 Poster (2026)
Keywords: speech-to-speech
spoken dialogues
LLM
benchmark
evaluation
judge
RL
RLAIF
GRPO
Primary area: applications to computer vision, audio, language, and other modalities
TL;DR: Enable paralinguistic-aware speech-to-speech (S2S) interaction through RL with an S2S automatic judge

Abstract: Speech-to-Speech (S2S) models have shown promising dialogue capabilities, but their ability to handle paralinguistic cues—such as emotion, tone, and speaker attributes—and to respond appropriately in both content and style remains underexplored. Progress is further hindered by the scarcity of high-quality and expressive demonstrations. To address this, we introduce a novel reinforcement learning (RL) framework for paralinguistic-aware S2S, ParaS2S, which evaluates and optimizes both response content and speaking style directly at the waveform level. We first construct ParaS2SBench, a benchmark that evaluates the naturalness of input–output pairs in terms of content and speaking style using expressive and challenging queries. For the automatic judge, we propose a PolyTone training strategy and a multi-stage framework, preventing the style hallucination of end-to-end audio LLM judging. Our judge correlates well with human preferences and is scalable, enabling the model to interact and learn from unlabeled speech via RL. Experiments show that existing S2S models fail to respond appropriately to paralinguistic attributes, performing no better than pipeline-based baselines. Our RL approach (ParaS2SAlign) achieves an 10% relative improvement in the appropriateness of response content and speaking style on ParaS2SBench over supervised fine-tuning (SFT), surpassing all prior models while requiring substantially fewer paired demonstrations than pure SFT. Our findings highlight the need for a scalable and accurate automatic evaluator for speech-to-speech interaction.

Published as a conference paper at ICLR 2026
PARAS2S: BENCHMARKING AND ALIGNING SPOKEN LANGUAGE MODELS FOR PARALINGUISTIC-AWARR SPEECH-TO-SPERCE INTERACTION
Shu-wen Ming Tu Andy T. Liu', Xinghua Qu', Hung-yi Lee', Lu Lu Yuxuan Wang', Yonghui 2u
"Grdduaee Institute of Communication Engineering, National Taiwan University *ByteDance Seed
010199912277gmaaan com,
(mingtu, lulu. 031 @by www.yyyddnnc..ooom com
ABSTRACT
Speech-to-Speech (S2S) models have shown promising dialogue capabilities, but their ability to handle paralinguistic cues-such as emotion, tone, and speaker attributes-ant to respond appropriately in both content and style remains under- explored. Progress is further hindered by the scarcity of high-quality and expres- sive demonstrations. To address this, we introduce a novel reinforcement learning (RL) framework for paralinguittic-wware S2S, ParaS2S, which evaluates and op- timizes both response content and speaking style directly at the waveform level. We first construct ParaS2SBench, benchmark that evaluates the naturalness of input-output pairs in terms of content and speaking style using expressive and challenging queries. For the automatic judge, we propose PolyTone training strategy and multi-stage framework, preventing the style hallucination of end- to-end audio L.LM judging. Our judge correlates well with human preferences and is scalable, enabling the model to interact and learn from unlabeled speech via RL. Experiments show that existing S2S models fail to respond appropriately to par- alinguistic attributes, performing approach (ParaS2SAlign) achieves an 10% relative improvement in the appropri- ateness of response content and speaking style ParaS2SBench over supervised fine-tuning (SFT), surpassing all prior models while requiring substantially fewer paired demonstrations than pure SFT. Our findings highlight the need for a scal- able and accurate automatic evaluator for speech-to-speech interaction] [
better than pipeline-basse baselines. Our RL
INTRODUCTION
Speech is the most natural medium of communication, conveying not only words but also paralin- guistic cues cues- emotion, tone, and speaker attributes- that jointly shape true intent and guide appro- priate responses (Schuller & Batliner, 2013). This interplay of linguistic and paralinguistic signals
motivates speech-to-speech (S2S) models (Xu al. 2025 Huang et al., 2025b Zeng et al. 2024)
for hikan-like, empathetic interaction beyond text-based dialogue systems (Achiam et al. Grattafiori et al. 2024).
2023
S2S models show strong dialogue abilities (Fang et al. 2025a b Zeng et al. 2024 as seen in Qwen2.5-Omn (Xu et al. 2025) and CharGPT Voice Mode Built on LLMs, they preserve reason- ing and conversational abilities while adding speech as new I/O modality, achieving high scores on benchmarks like Voice (Chen et al.] (024) and Llama Questions (Nachmani et al. 2024). Yet most benchmarks focus question answering (Nachmani et al. 2024). instruction following (Lu et al. (202)). or speech-to-text understanding tasks ( (Yang et al.
2024 Sakshi et al. (0256), over- looking paralinguistic-wvevnn dialogue. StyleTalk (Lin al. 2024a) and VoxDialogue (Cheng al.
1 Coorreppondigg authors.
Project page and demo: https//pannaashhechh.ghbbbbbbb

---

Published as a conference paper at ICLR 2026
2025 partially address the problem but remain speech-to-text benchmarks where evaluation ends the textual response, leaving no benchmark that directly evaluates S2S models' response speech for paralinguistic awareness.
Beyond the lack of benchmarks, no paralinguistie-waar S2S models currently exist. Our study shows that most S2S models fail to appropriately adjust responses according to different speaking styles (e.g., emotional tone) of users, often inferring speaker state from content alone and producing tone-deaf or awkward replies. This limitation stems from existing spoken dialogue datasets, which rarely capture the style dynamics between input and output (Ding et al. 2025 Fang et al.) 2025a b). Collecting such data is expensive, as it requires style annotation and expressive response recording, making data scarcity a critical bottleneck for developing paralinguistic-wwan S2S models (Huang et al. 2025b).
Inspired by DeepSeek-R1 (Guo et al. 2025), which incentivizes novel behaviors through RL with- out SFT demonstrations, we ask whether paralinguistic-wware dialogue capabilitiess can simi- larly emerge via RL with minimal supervision. To answer, we introduce novel framework for paralinguistic-aware S2S, ParaS2S. ParaS2S comprises a new S2S benchmark ParaS2SBench and a RL framework ParaS2SAlign Specifically, ParaSZSBench is designed to jointly evaluates both the content and speaking styles of input and output speech. It uniquely covers challenging test queries with controlled scenarios and contrastive speaking styles, properties that enable close amination of models' paralinguistic awareness. We design a data curation pipeline to automatically generate high-quality speech prompts adhering to these properties, covering key paralinguistic pects- sarcasm, age, and gender. Using this dataset, we expose the common tone-deaf is- sue in current S2S models, including state-of-the-art (SOTA) open-source models such Qwen2.5 Omni (Xu et al. (2025) and Kimi-Audio (Ding et al. 2025), as well as closed-source systems such as ChatGPT Voiced Mode (Achiam et al. 2023).
To enable a faithful proxy for human evaluation, we study automatic judges for speech-to-speech evaluation. Recent works have explored leveraging audio large language models (ALLMs) for au- tomatic evaluation Chiang et al. (2025); ZZang et al. (2026). These approaches feed the output audio directly into ALLMS (e.g., ChatGPT) and obtain a judgment score. However, in our study, we find that current ALLMs lack the ability to reliably score the naturalness of dialogue pair, often halluci- nating paralinguistic cues in both the input and output speech based on the spoken content. Instead, propose to decouple the paralinguistic analysis from the linguistic content with the PolyTone training strategy. In our multi-stage judge, decoupled captioners separately analyzes spoken con- tent and speaking style-includigg age, gender, emotion, tone-into textual representations, which is then reliably evaluated by large language models (LLMMs.. By reducing language-based style hallucination, our judge achieves higher correlation with human scoring than ALL.M-based judging.
The automatic judge further enables S2S models to explore and learn from unlabeled audio via RL. We verify the effectiveness of the automatic judge in guiding RL post-training with ParaS2SAlign. We first distill the pipeline-based judge into a smaller and faster reward model for online learn- ing, and optimize the base S2S model with Group Relative Policy Optimization (GRPO) (Shao) et al. 2024). Our results show that while supervised fine-tuning (SFT) is effective and outperforms existing m RL. surpasses SFT by more than 10% response content and speaking-style ap- propriateness on ParaS2SBench under both objective and subjective evaluations, demonstrating the effectiveness of our automatic judge. Furthermore, in cost-controlled experiments, RL requires only 10 hours of demonstrations as warm-up and achieves the same performance pure SFT with five times as many demonstrations, highlighting its learning efficiency. Our contributions are multifold:
We present a novel benchmark, ParaS2SBench, for paralinguistic-wver S2S dialogue. It evaluates the naturalness of input-output speech pairs in terms of both content and speak- ing style at the waveform level. The contrastive speaking-style design further reveals the common tone-deaf issue S2S models.
We propose a pipeline- based automatic evaluator for S2S interaction naturalness. Our sep- arate modeling of content and speaking style surpasses end-to-end ALLM-based judging.
With ParaS2SAlign, we show that our judge accurate enough to guide RL post-training, improving S2S models on ParaS2SBench under both objective and subjective evaluations and achieving state-of-the th-the- performance.
At the cost of requiring expensive and non-scalable demonstrations.
2

---

Published as a conference paper at ICLR 2026
We demonstrate that RL is more data-efficient than SFT for paralinguistic-waare S2S, ducing the required demonstrations, mitigating the data scarcity and highlighting the im- portance of scalable automatic evaluation.
We will open- data, code, and models to lower the barrier for future research.
2 RELATED WORK
2.1 SPOKEN DIALOGUE BENCHMARKS
Benchmarks have been proposed to evaluate spoken dialogue models. Table 5 compares key dif- ferences across benchmarks. Synamic-SUPERS (Huang al. 2024) tests instruction-following on 180 tasks (yu Huang et al. 2025). AudioBench (Wang et al. 2025a unifies speech/sound under- standing and QA. Bench Yang al.) (204) adds speech, sound, music tasks, and a chat cate- gory. MMAU (Sakshi et al. 2025a raises difficulty with reasoning-intensive QA. SpokenWOZ (Si al. 2023) provides large-scale human-to-human dialogue data. VoxEval (Cui al. (025) converts al. 2021) to speech to assess model intelligence. VoiceBench (Chen et al., 2024) adds more based QA datasets including AlpacaEval (Li al. 2023), OpenBookQA (Mi-
MMLU (Hendrycks
haylov et al. 2018). and MMLU-pro (Wang et al., (2024). FullDuplex:Bench (Lin et al. 2025a) eval- uates response timing for full-duplex models. Among these works, ADU-Bench (Gao et al. 2025) SD-eval (Ao al. 2024), VoxDialogue (Cheng et al. (022), and StyleTallk (Lin et al. [0224 eval- uate responses under different input speaking styles, but focus only on the dialogue models" output text In contrast, ParaS2SBench performs end-to-end evaluation both input and output speech, jointly considering content and speaking style.
2.2 SPOKEN DIALOGUE MODELS
From S2T to S2S dialogue models. Early Speech-to-Text LLMs equip LLMs with hearing capa-
bilities while leveraging textual reasoning for audio interaction (Tang et al. 2024 Hu et al. 2024
Gong et al. 2024). AudioReasoner (Xie et al. 2025) introduces Chain-of-Thought (CoT) reasoning
mitigate hallucination, while Qwen-Audio 1/2 (Chu et al.. 2023 2024) and StepAudio (Huang 2025b) further extend dialogue capabilities to enable spoken agents Recent works ex- Speech-to-Speech LLMs that learn onput speech interaction end-to-end (Zhang al.) Défossez et al., 2024). GLM-4- Voice (Zeng et al., 2024) and Step-Audio-AAAA (Huang
et al.
plore
2023
al.
2025a) rely on interleaved text and audio tokens for grounded speech generation. L.LMMa- (Fang et al. 2025a]b) b). Freeze-Omni (Wang et al.) (025b) and Mini-Omni (Xie & Wu
Omni
2024)
propose fine-tuning techniques to preserve LLM intelligence when adding speech modality.
Qwen2.5 Omni (Xu et al., 2025) proposes the thinker-talker architecture, while Kimi-Audio (Ding ct al.
2025) introduces a dual-head design for text and audio generation.
Paralinguistic-aware dialogue models. Among these models, ParalinGPT (Lin 2024b) and StyleTalk (Lin et al., 2024a) are the first to enable speech-to-text LLMs to respond differently diverse speaking styles. OmniChat (Cheng et al., (2024) extends the speech-to-tee to-teett study multi- turn, paralinguistic-wween aware dialogues. For speech-to to-speech models, GOAT-SLM (Chen et al.) 2025) the only model emphasizing paralinguistic-aware dialogue with a multi-stage SFT pipeline. These works rely on SFT with carefully curated, high-quality data, whereas we explore RL to reduce this reliance.
RL for dialogue models. RL has been applied to align spoken dialogue models. Align-SLM (Lin et al. (025b) follows RLAIF (Lee et al. 2024) and adopts DPO (Rafailov et al.) 2023) to improve long-range semantics. Qwen2.5 Omni (Xu al. 2025) uses WER as preference signal to ground speech generation. Step-Audio (Huang et al. 2025b) and Step-Audio-AQAA (Huang et al. 2025a) rely on human feedhack, which model content-stye style and input-output dynamics using scalable AI feedback.
-ectttioon-evyyyy ParaS2SAlign is the first RL. framework to
TalleeTalk predicts both response text and style in textual format, enabling style learning and evaluation. However, it limited to the few categorical styles supported hy Microsoft Azure TTS, and its format assump- tion prevents evaluation of S2S models.
"The response is usually text, and the speaking capability enabled by separate TTS module.
3

---

Published as a conference paper at ICLR 2026
ParaS2SBench ParaS2SAligg
Distillaton
Reward Model
Model Score
S25 model
Human
Score
Stylish Speech Synthesis
Transcription
Style
Correlate
Query
Response
+
Content
Style
Content
Style
Judge
Test I.E.M
Figure 1: The overall framework of ParaS2S. The bottom part illustrates the dataset construction and automatic judge in ParaS2SBench, while the top part illustrates reward model distillation in ParaS2S Align. The distilled reward model can then be used by standard RI. algorithms such as PPO and GRPO.
3 PARAS2SBENCH
ParaS2SBench is uniquely designed to evaluate paralinguistic-aware S2S abilities following three principles:
1. Contrasting speaking styles. Following StyleTaakk (Lin al.) 2024a) each test query is paired with two contrasting speaking styles that demand distinct responses. For example, "I just bumped into my ex." may be spoken in either surprised or sad tone. This design ensures the model to listen to the vocal cues instead of solely relying on the spoken content.
2. Scenario-controlledd queries. To ensure realistic scenarios, we design each test query to be paralinguistically relevant so that speaking style genuinely influences how the response should be generated To further ensure challenging scenarios, we require the test queries have neutral textual content that models cannot infer the speaker's state from words alone and must attend to vocal cues?
Speech-to-speech evaluation. Evaluation performed on output speech, assessing whether the model generates responses with both appropriate content and speaking styles.
In Section 3.1
we deseribe the curation of the benchmark dataset. Section 3.2 we present the methodology for automatically evaluating model responses given an input query.
3.1 DATASET CONSTRUCTION
In this section, we aim to curate a benchmark dataset, specifically the testing quries, following the first two principles. Please see Table | and the demo pagell for concrete examples. Table 6 shows the statistics. Appendix A.2 provides detailed steps of dataset construction, and we provide the high-level concepts here for conciseness.
Synthetic speech. We design multi-stage generation and filtering pipeline to generate synthetic test queries. First, an L.M generates ccenario-controlled and challenging queries where each spec- ifies one spoken content and two contrasting speaking styles in text. For spoken content, prompt the LLM the cover a wide range or dialogue topics including interests, work, studies, relationships, travel, health, religion, fashion, finance, and more. For speaking styles, cover various key par- alinguistic factors, including emotion, sarcasm, gender, and age.
'For example, "Who is the president of the US?" is not paralinguistically relevant, since any voicing tone should result the same response, whereas "How should dress for the upcoming wedding?" is relevant the speaker's gender.
For example, "Wow; that's surprising" " clearly expresses surprise and is not textually neutral, yielding simple test case, whereas "I just got call from my boss" is textually neutral, and the speaker's emotion difficult infer without vocal cues.
4

---

Published as a conference paper at ICLR 2026
Category
Example Dialogue
Emotion
User (happy): The city is planning build new mall near my house.
Model (cheerful): That's wonderful! A new mail will bring so many shops and restaurants to explore -oooorre--oundsss- exciting!
User (angry): The city planning to build new mall near my house.
Model (empathetie): I'm sorry to bear that you're re upset about it, Would you
like to share w what concerns you must about the new mall?
Sarcasm
User (sincere): It looks like you've ve outdone yourself with these answers!
Model (happy): Thank you! I'm glad my answers were helpful let me know if you'd like me to expand on anything.
User (sarcastic): It looks like you 've outdone yourself with these answers!
Model (apologetic): Hmm, it sounds like you might not he happy with the an- swers. Could you tell me what was missing or what you's '' like done differently?
Age
User (adult): need instructions on buying stock with my money.
Model (confident): Sure! can walk you through opening brokerage accnunt,
researching companies, and placing your first order step by step.
User (child): I need instructions on buying stock with my money.
Model (supportive): Investing exciting! Since you might be under 18, you may need help from parent guardian open custodial account before you can buy stocks. Would you like mc to explain how that works?
Gender
User (male): Help pick workout clothes for the gym. Model (professional): Absolutely! Breathable tank tops, moisture-wicking T- shirts, and flexible shorts are great choices. Do you prefer a sporty or casual style?
User (female): Help me pick workout clothes for the gym Model (professional): Sure! High-support sports bras, moisture-wicking leg- gings, and lightweight tops are excellent options. Would you like suggestions for specific brands?
Table 1: Examples from ParaS2SBench test set. The model responses in gray are for demonstration and not part of the benchmark dataset.
Next, we filter the queries by asking LLM for several checks, including neutrality, reasonability and paralinguistic relevance. Neutrality prevents models from inferring the speaker's state solely from spoken content; reasonability ensures all the content-style pairs are reasonable; paralinguistic relevance ensures that speaking style non-trivially affects the response. If any test is not passed, the query is discard.
Finally, we synthesize spoken queries given the spoken content and style. We leverage the most suitable TTS systems for different styles speech, and we apply both Emotion2vec- al (2.22 and bER-based filtering reject the incorrect generation. Human annotator check is conducted to ensure test set authenticity. We recruit three annotators to manually include only speech prompts with correct content and style from the final filtered set.
Real speech. To further examine model behavior in realistic scenarios, we construct a test set using real speech by filtering queries from existing dialogue datasets. Given the known content and style labels provided by the dataset, we apply filters to check the length R and paralinguistic relevance.
We rely on two emotion datasets, IEMOCAP (Busso et al. 2008) and MELD (Poria et al.) 2019). as they provide sufficient queries that meet constraints. In contrast, we find it challenging to source enough queries for age and gender from datasets like Common Voice due to the paralinguistic relevance constraint
3.2 AUTOMATIC JUDGE
In ParaS2SBench, subjective evaluation is the gold standard. However, as subjective evaluation is costly, we study how to approximate human judgments with an automatic judge. A straightforward approach is leverage ALLMs such as ChatGPT or Gemini to directly score naturalness given the user audio and the response audio. However, our preliminary experiments show that these ALLMS
'In real dialogues, some turns consist of only a few words (c.g., Haha or Sounds good), which are not suitable for evaluation. We therefore filter out queries with fewer than five words. For example, in Could you read the book for me? (female), the gender attribute is negligible.
5

---

Published as a conference paper at ICLR 2026
tend hallucinate paralinguistic properties based the spoken language, leading to entangled analysis and inaccurate judgments. In contrast, we decouple paralinguistic analysis from linguistic content by using specialized training technique and designing multi-stage judgment framework.
Stage 1: PolyTone Tone training. We train acoustic analysts for emotion, sarcasm, gender, and age by fine-tuning state-of-the-art open-source captioners. To encourage these analysts to attend to vocal cues rather than infer labels from the spoken content, we propose Poly Tone training, where the training set consists of utterances spoken in diverse styles but sharing the same linguistic content. This design guides the model to produce different style judgments given identical content and forces it to rely on paralinguistic cues. To curate such parallel data, we use the most suitable TTS systems, as described in Appendix A.2, See Appendix A.3 for model details.
Stage 2: Separated information extraction. We extract transcriptions and style labels, including gender, age, emotion, sarcasm, and natural-language tone description. For transcription, we use [Radford et al. 2023). For gender, age, emotion, and sareasm, we leverage the acoustic analysts trained with PolyTone. For the tone descriptor, use AudioReasoner Xie et al. (2025) , which reduces style hallucination via a structured reasoning pattern and is prompted to ignore the spoken content. By using separate, specialized components in our evaluation framework, we avoid hallucinations that are difficult to resolve via prompt engineering and mitigate the black-box nature of end-to-end ALLMs.
Step 2: LLM analysis on text. After we explicitly project various aspects of the speech informa- tion into text for both input and output speech. A text LLM is then used to analyze matching score on a 1-5 Likert scale following an expert-designed guideline. We will study its correlation to human scoring, and compare with the ALLM-based judging.
PARAS2SALIGN
To gauge the accuracy and usefulness of the automatic judge in ParaS2SBench, we study its ef- fectiveness in guiding RL post-training of S2S models. Although the pipeline in Section [3.] is accurate, it is slow in practice: it requires running several style analysts, a reasoning-basee speech- to-text LLM, and multiple LLM API calls. This makes typical online RL training impractical when rewards are computed using the automatic-judge pipeline. To address this, we aim to distill the au- tomatic judge pipeline into single reward model and enable fast online scoring. We propose a two-stage approach for the distillation. The details are illustrated in Appendix A.4
Stage 1: S2S model warmup. The reward model produces score given user speech and a re- sponse speech. We aim to collect diverse responses from S2S models given input queries and score them with the automatic judge to serve as training data for the reward model. Unfortunately, exist- ing S2S models do not possess paralinguistic-awer responses under contrasting speaking styles. Moreover, it is difficult to obtain satisfactory samples from off-the-shelf S2S models when they behave in tone-deaf manner, resulting in uniformly low scores. This issue not only hinders reward model learning but also impedes subsequent RL training, as the model cannot learn from positive examples and self-improve. To address this, a warm-up supervised fine-tuning (SFT) stage is critical. We follow similar steps to those in Section [.] to construct the training queries. For demonstrations, we use text LLM generate suitable response content and style, followed by TTS to synthesize speech and manual selection to ensure expressive- ness.
dialogue capabilities, often producing similar
Stage 2: Reward model distillation. With the warm-up model, we sample diverse responses for each training query and score naturalness using the full judge pipeline described in Section 3.2 The warm-up model exhibits preliminary paralinguistic-wware dialogue capabilities and begins to respond differently according to the input speaking styles, albeit unstably. As a result, the resulting responses include both high- and low-scoring samples. Using the resulting preference dataset of (query speech, response speech, score) triples, we fine-tune Qwen2.5-Omni Xu et al. (2025) with LoRA Hu et al. (2022) to obtain the reward model.

---

Published as a conference paper at ICLR 2026
With the reward model ready, we can apply standard RL algorithms to validate the effectiveness of the benchmark judge. We use GRPO Shao et al. (2024) because it is simple and widely adopted. S2S model under the judge, verified by
We will show that GRPO significantly improves both objective and subjective evaluations. With our pipeline open-sourced, other methods such as PPO SShuumann al., (2017) DPO Rafailov et al. (2023) can also be used for post-training: leave this to future work.
5
EXPERIMENTS
Data. We report benchmark data statistics in Table 6 For PolyTone data, we follow Section 3.1 to create queries with multiple speaking styles. For each category in gender, age, emotion and sarcasm, 10k speech prompts alone with their style lables are generated for fine-tuning captioners. For SFT data, follow Section 4 to build prompt-demonstration pairs from 10k speech prompts, totaling 100 hours of paired data. We use this SFT data for both the RL warmup (small-scale) setting and the pure SFT (large-scale) setting. Following Section [.1]] we curate up 100k prompts, each with speech and text annotations for content and style. For reward model distillation, we use up to 10k completions, yielding 320k pairs. For RL training, we use all speech prompts; however, we use only the speech and discard all labels during training. For text LLM, We use ChatGPT 4.1 across the paper.
speech prompts. For each prompt, the warmup model generates
Training. For PolyTone training, single H100 GPU is used. For SFT, use 8 NVIDIA H100 GPUs with FSDP (Zhao et al., (023). a learning rate of 1e-5, and a global batch size of 64. For reward model LoRA fine-tuning, of 10. For RL, we again use 8 H100 GPUs with FSDP, a learning rate of 5e-4, a global query batch size B of 32, and a group size G of 8. Each batch includes 256 scored completions for learning.
a single H100 with learning rate of 1e-6 and a batch size
Baselines. For our multi-stage automatic judge, we ablate the effective of the linguistically decou- pled methods such as PolyTone. We use gpt-audio as the end-to-end ALLM baseline.
5.1 COMPARING AUTOMATIC JUDGES WITH HUMAN SCORING
We study how to make the automatic judge align with human scoring. For this study, sampled 200 prompts per paralinguistic category from the benchmark for human annotation. For each speech prompt, we obtain two types of responses:
baSe responses: Text LLM generates the response content and style, and di- verse TTS systems synthesize speech to simulate different speaking styles. We include YourTTS (Casanova et al., 2022). Cosy Voice (Du et al. 2024), Sesame [0] and gpi--o-mini- Us.. These systems range from flat and neutral to expressive, spontaneous, and fine-grained controlled styles. We also add baseline, gpi-4o-mini-tt (bad), where we instruct LLM to produce suboptimal content or style such as tone-deaf content and inappropriate speaking style.
S2S model-based responses: End-to-end S2S models directly generate speech responses. We include GPT-4o Voice mode, Qwen2.5 Omni, and LLLM--VVoics
TTS-based responses isolate the effect of response tone under identical gold content, while S2S responses reflect real model behavior. Each prompt-response pair is scored by three human experts on a Likert scale"] [I] Each input query is then paired with several TTS- responses and several S2S model responses. For each query-response pair, we have human final score and several automatic judgment scores from different methods. We compute the Pearson correlation between the judges and human scores in Table 2
Multi-stage automatic judges correlate better with human scoring, In Table our multi-stage judgment framework clearly outperforms end-to-end ALLMS. When provided with ground-truth
10 hhtpps:/www.ssssee www.amewww.e....m.m....m.......m www.:/eswaee.eeceeeeeeeeeeeeeeee com//recarcc/croosingg.ht uunanny_vvlleye loflleyyoo_viccco
We first conducted preliminary annotations to align guidelines and maximize agreement, and discarded official annotations where all three experts disagreed. We use the major as the final score.
7

---

Published as a conference paper at ICLR 2026
Table 2: Correlation comparison. Correlation between automatic judges and human scores. We ablate different information provided to the text-LLM judge. The transcriptions of both the input and the output are always included. "-" indicates inclusion of ground-trut labels from the dataset; Xin- dicates that the information is excluded; /indicates inclusion via linguistically decoupled methods (e.g., PolyTone Tone& & structural reasoning); /indicates inclusion without such methods.
Input
Output
Pearson Correlation
ID
age
gender
emotion
sareasm
emotion
tone
age
gender
emotion
sarcasm
average
Audio LLMs (baseline)
B1
gpt-audio
0.682
0.637
0.612
0.541
0.618
stlti-stage Automatic Judge (ours)
O1
X
X
0.851
0.705
0.742
0.731
0.757
02
X
0.862
0.702
0.760
0.779
0.776
0.821
0.668
0.712
0.731
04
V
0.835
0,662
0.738
0.735
0.743
05
/
X
0.821
0,683
0,684
0.703
0.723
06
/
0.817
0.672
0.645
0.669
0.701
paralinguistic labels for the input speech, our pipeline consistently outperforms ALLMs by more than 15% in correlation. When using dedicated captioners to infer the input style, the performance gap remains above 10%. By analyzing the ALLM's reasoning steps, we find that it heavily relies on the spoken content infer plausible context for the final judgment, frequently neglecting or hallucinating the relevant paralinguistic cues in contrasting utterances. These results suggest that ALLMS may hallucinate paralinguistic cues and are less stable than the pipeline-based approach.
Natural-language tone descriptions better capture response style. In (O1-044 natural- language tone descriptions (O2) yield substantially higher correlations with the human judgments than emotion-tag conditioning (03, O4), while emotion-tag conditioning can be comparable worse than leaving the output style unspecified (01). By inspecting the predicted style labels, plau- sible explanation is that dialogue tone (e.g., understanding, reassuring, apologetic, professional) does not map cleanly to discrete emotion categories: emotions primarily describe speaker's 's affec- tive state Ekman 1992). whereas tone reflects pragmatic/interactiona voice functions (e.g., stance or communicative attitude) Vandepitte| (1989); Allan et al. (1997). In line with this, an emotion cap- tioner often produces labels that are weakly aligned with the intended attitude 2 which can introduce noise when such labels are used by a text-based LLM judge and thus reduce correlation.
Pone training helps most in capturing true emotion and attitude. In Table (O5), (OS), replac- ing the ground-truth labels with model predictions substantially degrades performance, especially in the emotion and sarcasm categories. By contrast, in the age and gender categories, the degrada- tion minor, as predicting child/adult and male/female may be easier for state-of-the-an captioners. In Table 2 (06), the training data construction for the captioners is similar to that of the PonyTone dataset, but the spoken content is always unique. As shown 05 and 06, the PolyTone training technique yields significant improvements for emotion and sarcasm but smaller gains for age and gender, suggesting the benefit of content-invariant, style-augmented training data.
Automatic judge preserves the ranking of human scores. In Table 3 S2S responses lag signifi- cantly behind TTS responses due to the tone-deaf content, where the latter benefit from ground-truth style labels. The scores of S2S responses hover around 3, indicating models fail to adapt to con- trasting speaking styles Second, across all models, the rankings with benchmark and human
1his issue may be exacerhated because three of Ekman's six basic emotions and disgust) are often undesirable for spoken agents. For prompts with two contrasting styles, models often score 5 for one response and for the other tone-deaf response, averaging 3.
Ekman 1992 (anger, fear,
8

---

Published as a conference paper at ICLR 2026
Table 3: Comparison of automatic judge and human scoring response ranking.
The default
automatic judge uses the input ground-truth style labels to avoid error propagation.
Age
Emotion
Gender
Sarcasm
Avg
Auto
Human
Auto
Human
Auto
Human
Auto
Human
Auto
Human
TTS-based
gpt-4o-mini-tt (good)
4,420
4.380
4.646
4.654
4.739
4.506
4.790
4.337
4.649 (1)
4.469 (1)
gpt-4u-mini-ts (bad)
1.215
1.159
1.041
1.325
1.590
1.251
1.227
1.265
Sesume
4,412
4.216
4.512
4.324
4.701
4.332
4.71
4.182
4.583
4.263 (2)
CosyVoice Voice
4.380
3,994
4,417
4,012
4.612
4.201
4,680
3.864
4.522 (3)
4.018 (3)
YourTTS
4,410
4.037
4,302
3.801
4.534
4.230
4.580
3.804
4.457 (4)
3.968 (4)
S2S models
spt-to-audio-previer
2.685
2.630
3,711
2.713
3.096
3,682
2.815
2.611
3.077 (6)
2.909 (5)
Owen2.5 Omni
2.930
2.728
3.680
2.522
2.933
3.493
2.509
2.863
GoM-4-Voice
2.570
2,493
3.489
2.384
2.821
3.521
2.720
2.301
2.900 (7)
2.675
Table 4: Comparing paralinguistic-awar dialogue capabilities with ParaS2SBench score.
Synthetic
Avg
Age
Emotion
Gender
Sarcasm
Avg.
IEMOCAP
MELD
Avg
Baseline
Thisper-GPT-TEE
3,050
3.121
2.916
3.005
3,022
3.562
3,412
3.487
3.176
Closed Source
gpt-to-audio-prvvine
3,205
3,633
3,342
2.957
3.284
3.770
3.508
3.639
3.403
Gemini
3.301
3.811
3.413
3.263
3.447
3.813
3.712
3.762
3.552
Open Source
Qwen2.5 Ommi
3.170
3.653
3.236
2.935
3.248
3.626
3.599
3.612
3.369
GLM 4
2,885
3.447
2.976
2,803
3.033
2.934
3.141
3.037
3.034
LLaMa-Ommi
3.123
3,512
3.064
3.164
3.215
3.425
3.462
3.443
3.291
Freeze-Orie Omni
2.819
2.316
2.884
2.701
2.680
2.835
3.061
2.948
2.769
Kimi-Audio
3.141
2.673
3.091
2.665
2,892
1.365
1.166
1.265
2.350
Ours
Kimi-Auda SFT
4,393
4,090
3.530
4.291
4.076
4,121
3.307
3.714
3.955
Kimi-Audio GRPO
4.496
4,490
4.239
4.538
4.441
4,394
3.927
4.161
4.382
Topline
GPT-TTS
4.525
4.691
4.812
4.791
4.705
4.710
4.824
4.766
4.725
scores nearly identical, with only switch. The rankings of TTS systems also consistent: 8pt-4o-mini-trs
Sesame > Voice > YourTTS
5.2 VERIFYING THE EFFECTIVENESS AUTOMATIC JUDGE FOR RL
Settings. We study the effectiveness of the automatic judge guide RL post-training. Note that in Table O2 stands for the highest correlation with the ground-trume truth style labels. We use this pipeline the default benchmarking judge to serve conduct human evaluation in Appendix A.6 On the other hand, for the reward model distillation in Section we use 05 which enables the pipeline judge unlabeled input-output pairs, more realistic scenario. We use Kimi-Audis Ding et al. (2025) as the base S2S model as is possess the highest intelligence among the open-source S2S models.
proxy for the human evaluation. Note that also
In Table 4 the Whisper-GPT-TTS pipeline uses Whisper-v3 to transcribe the input query without considering the speaking style, generates the response text with LLM, and synthesizes speech with gpt-4o-minii-ts. This pipeline serves as a bascline where speaking style is ignored. The topline, on the other hand, leverages the ground-truth transcription and style label of the query to generate both the response content and style with LLM, and then synthesizes expressive speech using gpt-4o-mini- nts.
"Since the four TTS systems share the same response content, their scoring differences stem from the speaking style. We observe that AudioReasoner tends to classify Voice and YouTTTS outputs calm, neutral, flat, which is less empathetic.

---

Published as a conference paper at ICLR 2026
Paralingulstic Awareness
Badiee Doylic: Choesearce
Oyee
4.2
4.0
ParnSgSHennh
3.8-
3.6-
Sles-Anth theae mockll fo calor
3.4
3.2
Kodl cave
connt danm
Norr Crad
ppttper
Eoss Aasi
Llatt
0
20
60
60
80
100
SFT Labeling Hours
Mice
(b)
Figure 2: (a) Studying the effectiveness of RL across different labeled data regimes. (b) Comparing the paralinguistic-an aware dialogue capabilities and original capabilities across models.
Automatic Judge help guiding RL post-training. In Table 4 GRPO elearly helps improve the performance upon SFT on the paralinguistic-awar dialogue, with over 10% overall improvement, including both synthetic and real speech.
Existing S2S models fail to adjust responses upon different speaking styles. Table 4 shows that most existing S2S models perform similarly to the pipeline baseline, suggesting that they do not account for the input speaking style and produce similar responses even for contrasting queries. In contrast, models achieve more than a 68% improvement over the base model and surpasses all existing models. Human evaluation in Appendix A.6 further corroborates the findings.
RL with a judge helps mitigate SFT data scarcity. As shown in Figure 2 a). RL continues to benefit the SFT model across most data regimes, from 10 to 100 hours of SFT data, suggesting the effectiveness of the automatic judge and RI. post-training Notably, Figure 2 trained model with only 20 hours of warmup data surpasses the SFT model trained with the full 100 hours of SFT data. This suggests that RL mitigates data scarcity for paralinguistie-wwarr dialogue and may be more suitable approach for this problem. These results further highlight the importance of a scalable and accurate automatic judge for speech-to-spese to--eeechhphhhece interaction, which serves as a key component of RL.
shows that the post-
The original capabilities of the base model are preserved. Finally, we benchmark the original capabilities of the warmup and post-trained models on VoiceBench Chen et al. (2024). Figure 2 shows that these original capabilities are largely preserved. We attribute this to lightweight SFT warmup (only epochs), along with KL-regularized GRPO to prevent the policy from drifting too far from the base model (Appendix A.7.. In contrast, when experimented with pure SFT for more than 10 epochs, we observed degradation in the original capabilities, especially in response length: the model starts to favor short responses resembling those in our SFT data even for queries that ask for detailed information, suggesting overfitting.
6 CONCLUSION
We present ParaS2S, framework designed for paralinguistic-waan speech-to-speech interaction. We formulate the problem and construct a benchmark dataset covering diverse scenarios and multi- ple paralinguistic aspects, including both synthetic and real speech. We provide automatic judge that correlates well with human preferences to enable model scoring. We demonstrate the effective- ness and efficiency of exploring on unlabeled speech and learning from the judge's signal. With our judge and GRPO, we enable state-of-the- art paralinguistic-wwan dialogue capabilities using only 10 hours of warm-up demonstrations, consistently demonstrating superior label efficiency compared to pure SFT. We will release the data, models, and code to lower the barrier for future research.
With only five hours of SFT data, the warmup model cannot sample sufficiently good responses to learn reliable reward model, resulting in improvement over SFT after post-training.
10

---

Published as a conference paper at ICLR 2026
REFERENCES
Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Tige Akkaya, Florencia Leoni Ale- man, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, al. Gpt-4 technical report. arXiv preprint arXiv:2303. 087744 2023.
K Allan, P Lamarque, and R Asher. Speech act theory: Overview. Concise encyclopedia of philos- ophy of language. pp. 454-66, 1997.
Junyi Ao, Yuancheng Wang. Xiaohai Tian, Dekun Chen, Jun Zhang, Lu Lu, Yuxuan Wang, Haizhou
Li, and Zhizheng Wu. Sd-eval: A benchmark dataset for spoken dialogue understanding beyond words. Advances in Neural Information Processing Systems, 37:56898-5609 -69888888661,,, 2024.
Rosana Ardila, Megan Branson, Kelly Davis, Michael Kohler, Josh Meyer, Michael Henretty, Reuben Morais, Lindsay Saunders, Francis Tyers, and Gregor Weber. Common voice: A speech corpus. In Proceedings of the Twelfth Language Resources and Evaluation Conference, pp. 4218-4222, 2020.
Carlos Busso, Murtaza Bulut, Chi-Chun Lee, Abe Kazemzadeh, Emily Mower, Samuel Kim, Jean- nette N Chang, Sungbok Lee, and Shrikanth S Narayanan. lemocap: Interactive emotional dyadic motion capture database. Language resources and evaluation, 42(4):335 359, 2008.
Edresson Casanova, Julian Weber, Christopher D Shulby, Arnaldo Candido Junior, Eren Gölge, and Moacir A Ponti. Yourtts: Towards zero-shot multi-speaker tts and zero-shot voice conversion for everyone. In International conference on machine learning, pp. 1709-27201 PMLR, 2022.
Hongjie Chen, Zehan Li, Yaodong Song, Wenming Deng, Yitong Yao, Yuxin Zhang, Hang Lv, Xuechao Zhu, Jian Kang, Jie Lian, et al. Goat-slm: A spoken language model with paralinguistic and speaker characteristic awareness. arXiv preprint arXiv:2507.18119 2025.
Yiming Chen, Xianghu Yue, Chen Zhang, Xiaoxue Gao, Robby T Tan, and Haizhou Li. Voicebench: Benchmarking Ilmebased voice assistants, arXiv preprint ar Arxiv:2410.1 17196, 2024.
Xize Cheng. Dongjie Fu, Xiaoda Yang, Minghui Fang, Ruofan Hu, Jingyu Lu, Bai Jionghao, Zehan Wang, Shengpeng Ji, Rongjie Huang, Linjun Li, Yu Chen, Tao Jin, and Zhou Zhao. Omnichat: Enhancing spoken dialogue systems with scalable synthetic data for diverse scenarios, 2024. URL
httts:://ppenrevill
net forum d dVcoo
Xize Cheng, Ruofan Hu, Xiaoda Yang, Jingyu Lu, Dongjie Fu, Zehan Wang, Shengpeng Ji, Rongjie Huang, Boyang Zhang, Tao Jin, and Zhou Zhao. Voxdialogue: Can spoken dialogue systems understand information beyond words? In The Thirteenth International Conference on Learning Representations, 2025. URL h net. fwr.mmnumm.ommm www.uummm11dbbmSSSSSSsm
Cheng-Han Chiang, Xiaofei Wang, Chung-Ching Lin, Kevin Lin, Linjie Li, Radu Kopetz, Yao Qian, Zhendong Wang, Zhengyuan Yang, Hung-yi Lee, and Lijuan Wang. Audio-aware large language models as judges for speaking styles. In Christos Christodoulopoulls, Tanmoy Chakraborty, Carolyn Rose, and Violet Peng (eds.), Findings of the Association for Computational Linguistics: EMNLP 2025, pp. 467-480, Suzhou, China, November 2025. Association for Computational Linguistics. ISBN 879-8-89176--3577 doi: 10.18553/w1/2055 http:////222...insssseeee seeiiing--emll.22.. URI. https: 7/aclantho wtw.tthhooooyy.or/222222222 org/2025. fi findi ngs-
257
Yunfei Chu, Jin Xu, Xiaohuan Zhou, Qian Yang, Shiliang Zhang, Zhijie Yan, Chang Zhou, and Jingren Zhou Qwen-audio: Advancing universal audio understanding via unified large-scale audio-language models. arXiv preprint arXiv:2311 1..0999, 2023.
Yunfei Chu, Jin Xu, Qian Yang, Haojie Wei, Xipin Wei, Zhifang Guo, Yichong Leng, Yuanjun Lv, Jinzheng He, Junyang Lin, et al. Qwen2-audhe technical report. arXiv preprint 1rXiv:2407.10759, 2024.
Wenqian Cui, Xiaoqi Jiao, Ziqiao Meng, and Irwin King. VoxEval: Benchmarking the knowl- edge understanding capabilities of end-to-end spoken language models. In Wanxiang Che, Joyce Nabende, Ekaterina Shutova, and Mohammad Taher Pilehvar (eds.), Proceedings of the 63rd An- nual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp-

---

Published as a conference paper at ICLR 2026
16735 16753, Vienna, Austria, July 2025. Association for Computational Linguistics. ISBN 979-8-99176-251-0. doi: 10.18653/w12022..c aac.oo......--...8........ URL bttpps://aclannhoooogyy. org/2025.acl-long loo.oonnnn.......
Alexandre Défossez, Laurent Mazaré, Manu Orsini, Amélie Royer, Patrick Pérez, Hervé Jégou, Edouard Grave, and Neil Zeghidour. Moshi: speech-text foundation model for real-time dia- logue. arXiv preprint arXiv:2410.00037, 2024.
Ding Ding, Zeqian Ju, Yichong Leng. Songxiang Liu, Tong Liu, Zeyu Shang, Kai Shen, Wei Song, Xu Tan, Heyi Tang, et al. Kimi-audio technical report. arXiv preprint arXiv:2504 18425, 2025.
Zhihao Du, Qian Chen, Shiliang Zhang, Kai Hu, Heng Lu, Yexin Yang, Hangrui Hu, Siqi Zheng, Yue Gu, Ziyang Ma, et al. Cosyvoice: A scalable multilingual zero-shot text-to-speech synthesizer based on supervised semantic tokens. arXiv preprint 0rXiv:2407.05407 2024.
Paul Ekman. An argument for basic emotions. Cognition & emotion, 6(3-4):16920000 200, 1992.
Qingkai Fang, Shoutao Guo, Yan Zhou, Zhengrui Ma, Shaolei Zhang, and Yang Feng. Llama- omni: Seamless speech interaction with large language models. In The Thirteenth International Conference Learning Representations, 2025a.
Qingkai Fang, Yan Zhou, Shoutao Guo, Shaolei Zhang, and Yang Feng. Llama-ommi2: Llm- based real-time spoken chatbot with autoregressive streaming speech synthesis. arXivv preprint arXiv:2505.02623. 2025b.
Kuofeng Gao, Shu-Tao Xia, Ke Xu, Philip Torr, and Jindong Gu. Benchmarking open-ended au- dio dialogue understanding for large audio-language models. In Wanxiang Che, Joyce Nabende, Ekaterina Shutova, and Mohammad Taher Pilehvar (eds.), Proceedings of the 63rd Annual Meet- ing of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 4763-4784 Vienna, Austria, July 2025. Association for Computational Linguistics. ISBN 979-8-89176- 251-0. doi: 10. 20.18653/v1/2025..ae acc- long. URL BRLLhtttp///cclannolloogo ac acl-long.22 2377
Yuan Gong, Hongyin Luo, Alexander H Liu, Leonid Karlinsky, and James Glass. Listen, think, and understand. In The Twelfth International Conference on Learning Representations, 2024.
Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, et al. The llama 3 herd of models. arXiv preprint arXiv:2407.21788 2024.
Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. Deepseek-rl: Incentivizing reasoning capability in 1lms via reinforcement learning. arXiv preprint arXiv:2501.12944 2025.
Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. International Conference on Learning Representations, 2021.
Edward J Hu, yelong shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. LoRA: Low-rank adaptation of large language models. In International Con- ference on Learning Representations, 2022. URL
net/ forum?
id=nZevKeeFyff
Shujie Hu, Long Zhou, Shujie Liu, Sanyuan Chen, Lingwei Meng, Hongkun Hao, Jing Pan, Xunying Liu, Jinyu Li, Sunit Sivasankaran, et al. Wavllm: Towards robust and adaptive speech large language model. In Findings of the Association for Computational Linguistics: EMNLP 2024, pp. 4552-4572, 2024.
Ailin Huang, Bingxin Li, Bruce Wang. Boyong Wu, Chao Yan, Chengli Feng, Heng Wang, Hongyu Zhou, Hongyuan Wang. Jingbei Li, et al. Step-audio-aqual a fully end-to-end expressive large audio language model. arXiv preprint arXiv:2506.08967 2025a.
12

---

Published as a conference paper at ICLR 2026
Ailin Huang, Boyong Wu, Bruce Wang, Chao Yan, Chen Hu, Chengli Feng, Fei Tian, Feiyu Shen, Jingbei Li, Mingrui Chen, et al. Step-audios Unified understanding and generation in intelligent speech interaction. arXiv preprint arXiv:2502.1194: 2025b.
Chien-yu Huang, Ke-Han Lu, Shih-Heng Wang, Chi- Yuan Hsiao, Chun-Yi Kuan, Haibin Wu, Siddhant Arora, Kai-Wei Chang, Jiatong Shi, Yifan Peng, et al. Dynamic-superb:: Towards dynamic, collaborative, and comprehensive instruction-tuning benchmark for speech. In ICASSP 2024-2024 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 12136-12140. IEEE, 2024.
Harrison Lee, Samrat Phatale, Hassan Mansoor, Thomas Mesnard, Johan Ferret, Kellie Ren Lu, Colton Bishop, Ethan Hall, Victor Carbune, Abhinav Rastogi, al. Rlaif vs. rlhf: Scaling re- inforcement learning from human feedback with ai feedback. In International Conference Machine Learning. pp.
PMLR, 2024.
Xuechen Li, Tianyi Zhang, Yann Dubois, Rohan Taori, Ishaan Gulrajani, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. Alpacaeval: An automatic evaluator of instruction-following models.
eval _eaal 2023.
Guan-Ting Lin, Cheng-Han Chiang, and Hung-yi Lee. Advancing large language models to capture varied speaking styles and respond properly in spoken conversations. In Lun-Wet Ku, Andre Martins, and Vivek Srikumar (eds.), Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 6626-6642, Bangkok, Thailand, August 2024a. Association for Computational Linguisties. doi: 10. 100116663/11/222.ac--ooogg URL https: htttpa://aclanttooooy.ooss
acl long 358/
Guan-Time Ting Lin, Prashanth Gurunath Shivakumar, Ankur Gandhe, Chao-Han Huck Yang. Yile Gu, Shalini Ghosh, Andreas Stolcke, Hung-yi Lee, and Ivan Bulyko. Paralinguistie--ennnned large language modeling of spoken dialogue. In ICASSP 2024-2024 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 10316-10320. IEEE, 2024b.
Guan-Ting Lin, Jiachen Lian, Tingle Li, Qirui Wang, Gopala Anumanchipalli, Alexander H Liu, and Hung-yi Lee. Full-duplex-bence bench: A benchmark to evaluate full-duplex spoken dialogue models on tur--taking capabilities. arXiv preprint 25t.:::20200042... 2025a.
Guan-Ting Lin, Prashanth Gurunath Shivakumar, Aditya Gourav, Yile Gu, Ankur Gandhe, Hung-yi Lee, and Ivan Bulyko. Align-SLM: Textless spoken language models with reinforcement learning from AI feedback. In Wanxiang Che, Joyce Nabende, Ekaterina Shutova, and Mohammad Taher Pilehvar (eds.), Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics ( (Volume 1: Long Papers), pp. 20395 (0895-204111 Vienna, Austria, July 2025b. Association for Computational Linguistics. ISBN 979-9 89176-251-0, doi: 10. long. 997/
URL
Ke-Han Lu, Chun-Yi Kuan, and Hung yi Lee. Speech-IFEval: Evaluating Instruction-Follloiing and Quantifying Catastrophic Forgetting in Speech-Aware Language Models. In Interspeech 2025, pp- 2078 2082, 2025. doi: 10.21437//mterspeecch.o 2025-619
Ziyang Ma, Zhisheng Zheng. Jiaxin Ye, Jinchao Li, Zhifu Gao, ShiLiang Zhang, and Xie Chen. emotion2vec: Self-supervised pre-training for speech emotion representation. In Findings of the Association for Computational Linguistics ACL 2024, pp. 15747-15760, 2024.
Todor Mihaylov, Peter Clark, Tushar Khot, and Ashish Sabharwal. Can suit of armor conduct elec- tricity? a new dataset for open book question answering. In Proceedings of the 2018 Conference Empirical Methods in Natural Language Processing, pp. 2381 12911, 2018
Eliya Nachmani, Alon Levkovitch, Roy Hirsch, Julian Salazar, Chulayuth Asawaroengchai. Soroosh Mariooryad, Ehud Rivlin, RJ Skerry- Ryan, and Michelle Tadmor Ramanovich. Spoken question answering and speech continuation using ppectrogram-powerne LLM. In The Twelfth Interna- tional Conference on Learning Representations, 2024. URL https: attps://cpeereviiet net/ forum? d
13

---

Published as a conference paper at ICLR 2026
Vassil Panayotov, Guoguo Chen, Daniel Povey, and Sanjeev Khudanpur. Librispeech: an corpus based on public domain audio books. In 2015 JEEE international conference on acoustics, speech and signal processing (ICASSP), pp.
IEEE, 2015.
Soujanya Poria, Devamanyu Hazarika, Navonil Majumder, Gautam Naik, Erik Cambria, and Rada Mihalcea. MELD: A multimodal multi-party dataset for emotion recognition in conversations. In Anna Korhonen, David Traum, and Lluís Märquez (eds.). Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pp. 527-536, Florence, Italy, July 2019. Association for Computational Linguisties. doi: 10. 10.18833/11PP11 1050. URL https: 7/ //claanthooogyc.ort.. wtt.://yyy.rr/11111111110
Sameer Pradhan, Ronald Cole, and Wayne Ward. My science tutor (myst)-a a large corpus of chil- dren's 's conversational speech. In Proceedings the 2024 Joint International Conference Com- putational Linguisties, Language Resources and Evaluation (LREC-COLING 2024), pp. 12040- 12045, 2024.
Alec Radford, Jong Wook Kim, Tao Xu, Greg Brockman, Christine McLeavey, and Ilya Sutskever. Robust speech recognition via large-scale weak supervision. In International conference on ma- chine learning, pp. 28492-2851 PMLR, 2023.
Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language model is secretly a reward model. In Thirty-seventh Conference Neural Information Processing Systems, 2023. URL https:
/ //openreview
Sakshi, Utkarsh Tyagi, Sonal Kumar, Ashish Seth, Ramaneswaran Selvakumar, Oriol Nieto, Ra- mani Duraiswami, Sreyan Ghosh, and Dinesh Manocha. MMAU: A massive multi-task audio un- derstanding and reasoning benchmark. The Thirteenth International Conference on Learning Representations, 2025a. URL htttps://openreviiee http://ww....teoommme//aeetataa.
S Sakshi, Utkarsh Tyagi Sonal Kumar, Ashish Seth, Ramaneswaran Selvakumar, Oriol Nieto, Ra- mani Duraiswami, Sreyan Ghosh, and Dinesh Manocha. Mmau: A massive multi-task audio understanding and reasoning benchmark. In The Thirteenth International Conference on Learn- ing Representations, 2025b.
Bjorn Schuller and Anton Batliner. Computational Paralinguistics: Emotion, Affect and Personality in Speech and Language Processing. Wiley Publishing, 1st edition, 2013. ISBN 111997 1365.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint 1707v0770.063444 2017.
Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, YK Li, et al. Deepseekmath: Pushing the limits of mathematical reasoning in open language models. arXiv preprint arXiv:2402.03300 030000 2024.
Shuzheng Si, Wentao Ma, Haoyu Gao, Yuchuan Wu, Ting-En Lin, Yinpei Dai, Hangyu Li, Rui Yan, Fei Huang, and Yongbin Li. Spokenwoz: A large-scale speech-text benchmark for spoken task- oriented dialogue agents. Advances in Neural Information Processing Systems, 36:39088-39111 2023.
Changli Tang, Wenyi Yu, Guangzhi Sun, Xianzhao Chen, Tian Tan, Wei Li, Lu Lu, Zejun MA, and Chao Zhang. Salmonn: Towards generic hearing abilities for large language models. In The Twelfth International Conference on Learning Representations, 2024.
Gemini Team, Petko Georgiev, Ving Ian Lei, Ryan Burnell, Libin Bai, Anmol Gulati, Garrett Tanzer, Damien Vincent, Zhufeng Pan, Shibo Wang, et al. Gemini 1.5: Unlocking multimodal under- standing across millions of tokens of context. arXiv preprint arXiv:2403.055.3 2024.
Sonia Vandepitte. A pragmatic function of intonation: Tone and cognitive environment. Lingua, 79 (4):265-297, 1989.
14

---

Published as a conference paper at ICLR 2026
Bin Wang. Xunlong Zou, Geyu Lin, Shuo Sun, Zhuohan Liu, Wenyu Zhang, Zhengyuan Liu, Aiti Aw, and Nancy Chen. Audiobench: A universal benchmark for audio large language models. In Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguisties: Human Language Technologies CerggisssVVlllael (Volume I: Long Papers), pp-
4297-4316, 2025a.
Xiong Wang. Yangze Li, Chaoyou Fu, Yike Zhang, Yunhang Shen, Lei Xie, Ke Li, Xing Sun, and Long MA. Freeze-omni: A smart and low latency speech-to dialogue model with frozen L.M. In Forty-second International Conference on Machine Learning, 2025b. URL.
fttts://cppereeve iew. net. forum ficiummmmsss1.. 1I1z155mm
Yubo Wang, Xueguang Ma, Ge Zhang, Yuansheng Ni, Abhranil Chandra, Shiguang Guo, Weiming Ren, Aaran Arulraj, Xuan He, Ziyan Jiang, et al. Mmlu-pro: A more robust and challenging multi- task language understanding benchmark. Advances in Neural Information Processing Systems, 37:95266-95225 95290, 2024.
Zhifei Xie and Changqiao Wu. Mini-omni: Language models can hear, talk while thinking in streaming. arXiv preprint arXiv:2408. 16725, 2024.
Zhifei Xie, Mingbao Lin, Zihang Liu, Pengcheng Wu, Shuicheng Yan, and Chunyan Miao. Audio- reasoner: Improving reasoning capability in large audio language models. arXiv preprint arXiv:2503.02318 2025.
Jin Xu, Zhifang Guo, Jinzheng He, Hangrui Hu, Ting He, Shuai Bai, Keqin Chen, Jialin Wang. Yang Fan, Kai Dang, et al. Qwen2. 5-omni technical report. arXiv preprint arXiv:2503.20215. 2025.
Qian Yang, Jin Xu, Wenrui Liu, Yunfei Chu, Ziyue Jiang, Xiaohuan Zhou, Yichong Leng, Yuanjun Lv. Zhou Zhao, Chang Zhou, et al. Air-bench: Benchmarking large audio-language models via generative comprehension. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics ( (Volume 1: Long Papers), pp. 1979-1998, 2024.
Chien yu Huang, Wei-Chih Chen, Shu wen Yang, Andy T. Liu, Chen-An Li, Yu-Xiang Lin, Wei- Cheng Tseng, Anuj Diwan, Yi-Jen Shih, Jiatong Shi, William Chen, Xuanjun Chen, Chi-Yuan Hsiao, Puyuan Peng. Shih-Heng Wang, Chun-Yi Kuan, Ke-Han Lu, Kai-Wei Chang, Chih- Kai Yang. Fabian Alejandro Ritter Gutierrez, Huang Kuan-Po, Siddhant Arora, You-Kuan Lin, CHUANG Ming To, Eunjung Yeo, Kalvin Chang, Chung-Ming Chien, Kwanghee Choi, Cheng- Hsiu Hsieh Yi-Cheng Lin, Chee-En Yu, I-Hsiang Chiu, Heitor Guimarães, Jionghao Han, Tzu- Quan Lin, Tzu-Yuan Lin, Homu Chang, Ting-Wu Chang, Chun Wei Chen, Shou-Jen Chen, Yu- Hua Chen, Hsi-Chun Cheng, Kunal Dhawan, Jia-Lin Fang, Shi-Xin Fang, KUAN YU FANG CHIANG, Chi An Fu, Hsien-Fu Hsiao, Ching Yu Hsu, Shao-Syuan Huang, Lee Chen Wei, Hsi- Che Lin, Hsuan-Hao Lin, Hsuan-Ting Ting Lin, Jian-Ren Lin, Ting-Chun Liu, Li-Chun Lu, Tsung-Min Pai, Ankita Pasad, Shih- Yun Shan Kuan, Suwon Shon, Yuxun Tang, Yun-Shao Tsai, Wei Jui Chi- ang, Tzu-Chieh Wei, Chengxi Wu, Dien-Ruei Wu, Chao-Han Huck Yang, Chieh-Chi Yang, Jia Qi Yip. Shao-Xiang Yuan, Haibin Wu, Karen Livescu, David Harwath, Shinji Watanabe, and Hung yi Lee. Dynamic-SUPERB phase-2: A collaboratively expanding benchmark for measuring the capabilities of spoken language models with 180 tasks. In The Thirteenth International Confer- ence Learning Representations, 2025. URL nttts::/openzeviiee net/ forum?id S 87122pAN7T
Aohan Zeng, Zhengxiao Du, Mingdao Liu, Kedong Wang. Shengmin Jiang, Lei Zhao, Yuxiao Dong, and Jie Tang. Glmc4-voice: Towards intelligent and human-like end-to-end spoken chatbot. arXiv preprint arXiv:2412.02611. 2024.
Dong Zhang, Shimin Li, Xin Zhang, Jun Zhan, Pengyu Wang, Yaqian Zhou, and Xipeng Qiu. Speechgpt: Empowering large language models with intrinsic cross-modal conversational abil- ities. In Findings of the Association for Computational Linguistics: EMNLP 2023, pp. 15757- 15773, 2023.
Xueyao Zhang, Chaoren Wang, Huan Liao, Ziniu Li, Yuancheng Wang, Li Wang, Dongya Jia Yuanzhe Chen, Xiulin LI, Zhuo Chen, and Zhizheng Wu. Speechjudge: Towards human-level judgment for speech naturalness. In The Fourteenth International Conference on Learning Rep- resentations, 2026. URL /ttp/opprreee revi ew net ftttt/ffuummp d TED.
15

---

Published as a conference paper at ICLR 2026
Yanli Zhao, Andrew Gu, Rohan Varma, Liang Luo, Chien-Chin Huang, Min Xu, Less Wright, Hamid Shojanazeri, Myle Ott, Sam Shleifer, et al. Pytorch fsdp: Experiences on scaling fully sharded data parallel. Proceedings of the VLDB Endowment, 16(12):3848-338)) 2023.
A APPENDIX
A.1 RELATION TO PRIOR S2T & S2S BENCHMARKS
We compare ParaS2SBench to existing S2T & S2S Benchmarks in Table Note that ParaS2SBench is the first benchmark to directly assess the speaking style of the response speech the waveform level.
Task type
Evaluate Input
Evaluate Output
Style Dimension
Benchmarks
Und.
Dia.
Content
Style
Content
Style
Para.
Speaker
Speech-to-Tesse Evaluation
Dynamic-SUPERB SUPERB
x
AudioBench
AIR-Ben Be-Bencch
/
/
/
V
/
X
4
MMAU
/
X
V
VoiceBench
/
/
X
V
X
X
X
ADU-Bench
X
X
SD-Eval
X
VoxDialogue
X
X
StyleTalk
X
V
X
Speech-to-Speeet Evaluation
VoxEval
ParaS2SBench (Ours)
X
Table 5: Comparison of spoken dialogue benchmarks. Und. stands for Understanding; Dia. stands for Dialogue; Para. stands for Paralinguistic information.
A.2 STATISTICS AND DETAILS OF DATASET CONSTRUCTION
Table 6: Statistics of the ParaS2SBench test set.
Prompts
Utterance
Avg Duration
Hours
Labels
Synthetic Speech
Emotion
300
600
4.59
0.77
Happy. Surprised, Sad, Angry. Fear, Disgust
Sarcasm
300
600
6,23
1.04
Sincere, Sarcastic
Age
300
600
4.72
0.79
Adult, Child
Gender
300
600
4.48
0.74
Male, Female
Real Speech
IEMOCAP
709
709
10.21
2.01
Happy. Surprised, Sad, Angry. Fear, Disgust Happy. Surprised, Sad, Angry. Fear, Disgust
MELD
781
781
11.31
2.45
Total
2690
3890
6.92
7.8
We present the benchmark dataset statistics in Table We detail the steps involved in constructing the benchmark dataset. Note that the training speech prompts for SFT, the reward model, and RI. fol- low the same construction pipeline; however, the training data does not undergo human verification due to the high cost.
1. Candidate Generation. We first generate large corpus of queries with LLM, each sisting of a input spoken sentence c; Σ" followed by two contrasting speaking styles, s E", that demand different responses. In the generation prompt, instruct I.I.M to
16

---

Published as a conference paper at ICLR 2026
cover diverse topics and scenarios, including interests, work, studies, relationships, travel, health, religion, fashion, finance, and more 16
2. Script Quality Filtering. For each spoken content C₁ construct two queries, (ci, st) and (c,, s)). For each query (c),8;), we control the scenario by asking LLM for several checks, including neutrality, reasonability and paralinguistic relevance. Neutrality prevents models from inferring the speaker's 's state solely from text Cú; reasonability ensures the content e; and style s; is a reasonable pair; paralinguistic relevance ensures that speaking style non-trivially affects the response. If any test is not passed, the query is discard, In the following, we provide the detailed illustration for the quality tests:
(a) Paralinguistic Relevance Test. To ensure that the speaking style is non-trivial to the dialogue scenario and meaningfully affects the response, we test whether differ- ent speaking styles lead to different responses. We ask an LLM to generate two re- sponses both content and style-based on the same input content but different speak- ing styles, once for each style. We then use the LLM to determine whether the two responses exhibit significant difference. If the responses are similar, implying that the speaking style does not meaningfully affect the outcome, we discard the test case. This test is essential because many spoken queries and their appropriate responses irrelevant to how they are voiced, such as "Who is the president of the US?".
(b) Reasonability Test. Due to hallucinations, the LLM sometimes generates queries in which the content and speaking style are mismatched. For example, I want to get screened for cervical cancer. (male/female) is reasonable for a female speaker but sounds odd for male speaker. We ask the LLM evaluate the reasonability of both speaking styles and discard queries that contain one or more unreasonable cases. After the first two tests, we obtain realistic test cases for evaluating paralinguistic-wwan dialogue capabilities.
(c) Neutrality Test. We frequently observe that S2S models respond empathetically by inferring from the spoken content rather than relying on paralinguistic cues in the speech signal. For example, Wow! That's 's big news! is almost always associated with a surprised emotion, and Oh.. got my period most likely to be spoken by a fe- male speaker in sad tone. To make our test cases more challenging and to examine whether S2S models truly attend to the audio, we design test cases using paralinguis- tically neutral gender, or age from text alone. In this way, the model must rely on the audio signal to respond appropriately. In practice, for each query, we ask the LLM whether the spoken sentence is more likely to be voiced in one speaking style, another speaking style, or whether it is neutral and difficult to determine. We then discard queries for which the answer is not neutral.
for which it is difficult to infer emotion, attitude,
3. Speech Synthesis. We synthesize input waveform w; E R given the (c,, sx) pair. For emo- tion and sarcasm, we rely on the instruction basdd TTS system gpt-4o-mini-tzg TM This sys- tem requires a style deseription, which we generate with LLM based on the style label 8;- Since gpt-4o-minni-ts supports only limited number of speakers, we Cosy Voice (Du) jet al.] 2024) for in-context zero-shot synthesis of gender and age. The voice samples for gender are drawn from LibriSpeech (Panayotov et al. 2015) and Common Voice (Ardila et al. 2020), while the samples for age are drawn from NÑCEŞ We discard samples whose WER with the ground truth exceeds threshold. For emotion, we further discard samples whose Emotion2vec (Ma et al. 2024) classifier scores are too low (Cheng et al.) 2025).
4. Train/Test Split. To avoid overlap between training and testing, we use disjoint query topics and TTS speakers.
5. Human Check. To ensure test set authenticity, we recruit three annotators to manually include only speech prompts with correct content and style from the filtered set.
The prompt also includes explicit instructions for our quality criteria. However, we find that I.IM some- times hallucinates and struggles to follow instructions and generate high-quality queries in a single pass, making subsequent quality filtering necessary. http://www.opeeni.i fm/
18 "htps://www.. wwtp://wwwwww................ corpus. - We do use MyST (Pradhan et al. 2024) since the data link is unavailable.
17

---

Published as a conference paper at ICLR 2026
Finally, we construct testing set Dest {(c,, St, W))} where G; E E* is the input spoken content, 8t E Σ" is the imput speaking style, and w/ E R is the imput audio prompt.
A.3 DETAILS OF AUTOMATIC JUDGE
Given an input query (c;, 8;, wi) ~ Diest the S2S model M samples a response speech w, ~ (O/,),,,). We project both the content and the speaking style of w, and w" into text. We denote ca and 9, as the content and style of the input; Co and so as the content and style of the output.
For content, Whisper-V3 V3 (Radford et al., 2023) is used to get the transcription. The style descrip- tion is composed of emotion, sarcasm, gender, age and tone descriptions. We fine-tune state-of- the-art captioners for different paralinguistic properties with the PolyTone strategy. We fine-tune Emotion2vec Ma et al. (2024) for emotion and sarcasm classifiers. We use a wav2vec 2.0-based gender classifie on Huggingface. We leverage Au- 2025) to extract output speaking tone. AudioReasoner equips Qwen-Audio 2 (Chu et 2024) with reasoning capabilities by distilling Sfain-of-Thhughe (CoT) paths from Gemini (Team et al. 2024) to reduce hallucination.
age classifie and E SCAPA-TDNN--aae
dioReasoner (Xie et al.
Finally, given the input content G; and style 8;, along with the extracted output content c, and style
So. we use LLM to score the naturalness following the guideline r designed by human experts, described in the Appendix A.8
fan = LLM St, Cos r)
(1)
We show that this scoring pipeline can align with human judgments feapert in Section 5.1 Both fauto and feepert on a 1-5 Likert scale.
A.4 DETAILED FORMULATION OF PARAS2SALIGN
We design a three-stage online RL framework that uses reward model to approximate the automatic judge and employs GRPO (Shao et al. 2024). We use Kimi-Audio (Ding et al.) (025)) as the base model Obas 21 while the framework can be applied to any LM-based S2S model. For Kimi-Audio, the audio input wi, text input G, audio output Wa, and text output Co are preprocessed and organized into four token streams: a, Z and to Zh. The input streams (a,, t2)) are padded to the same length L; E Z. and the output streams to L, E Z. The input embeddings of the audio and text streams are summed before being fed into the Transformer, and from the middle of the model, two prediction heads predict the next token for each stream.
lawl
To(aos to a., 44)
II
To( do, to ais aisti)
For inference, output audio and text tokens are sampled auto-regressively (a,, (o) ~ To(O ai li). Audio tokens are decoded into the sampled waveform with a flow-matching decoder: wi = pla.).
Stage 1. Warm-up. SFT serves as crucial warm-up stage for RL, as we observe that existing S2S models lack paralinguistic-wwan dialogue capabilities. Consequently, they fail to sample high- quality responses and cannot provide a useful learning signal for RL. To construct the SFT dataset Dan, we follow Section[.]]pp generate training set of speech queries with both input content c; and style labels 8;. For each query (ci, 8:), we use LLM to produce the most suitable response (Co. 8n), including both textual transcription and tone description. We then synthesize the expressive response wo using ept-4o-miniitts Because gpt-4o-mini-tts can be unstable, we synthesize 10 can- didates, apply WER-based filtering, and perform manual selection to obtain high-quality warm-up
"httss//huggingface http:://wwggnn.aa...ccc///annnna ccw.aadderrnng/wwawveeeeaaaeeeee
https:/hugiingface. coll halllaasuuuuh//iiceeeennete saaer//oooeceeemmeeereassseeeeee
21 Since it exhibits high intelligence and strong dialogue capabilities (Chen et al. 2024) and is fully open- sourced. We do not use Qwen2.5-Ommi (Xu et al. 2025) hecause its speech tokenizer not released, making
S2S fine-tuning infeasible.
18

---

Published as a conference paper at ICLR 2026
demonstrations wo- With the outptt mapping Dan {(w, Wos Ca; Co)), we train next-token prediction on both the preprocessed audio stream a; ajl|a, and the text stream t. by optimizing for higher likelihood ED (aoatt to (g, tt)], initializing from Brsee and obtaining Asa.
To distill our benchmark pipeline into a reward model, we construct a preference dataset prefer- We first prepare Q speech queries {(c, sg w follow- ing Section 3.1 The SFT model now possesses preliminary paralinguistic-waar dialogue capabil- ities and begins to respond differently according to the input speaking styles, but unstably. Each query (c,, S2; w,) is preprocessed into input token streams (a,, t.). We sample K diverse speech responses with high sampling temperature, (a,, (a) ~ To (O di 42), wa p(aa). We then score the resulting Q K query-response pairs following Equation [ to construct a preference dataset Precfer = [w,, w/o, [o. where fgp is the fitness score of wy and w. depending on content label C, style label S₁, extracted content Co C(w,) and extracted style so S(w)). Finally, we use (Xu et al. (025) as the reward model, which employed as a Speech-to-Text LLM. The model takes the query speech, response speech, and scoring guideline r as input, and outputs a single score on Likert scale. We denote the reward model as The score is treated as a single character and optimized with the cross entropy loss:
Stage 2. Distilling Reward Model.
LoRA (Hu et al. 2022) to fine-tune Qwen2.5-Onn
E ( gnt Wi, r..
Stage 3. Post-training. Using the warm-up model Osn and the reward model we enable the model to explore the search space for higher scores via GRPO (Shao et al. (2024) on the large set of unlabeled speech. We do not use PPO (Schulman et al., 2017) due to its substantial memory and computational burden of the value function. Moreover, in our case, only the last token of the response is assigned a final reward, which complicates the training of the value function that needs to be accurate at every token (Shao et al.] (024).. Given the unlabeled speech prompt dataset Da = [w;}. we obtain the transcription with Whisper-v3 and construct input token streams Dí = {(w,, 0; 4:)). We optimize Os to maximize the objective:
JGRPO = = ai, t.) ~ D'r llag
(g) ~ Temu (0 a, ti]
1
lag1
To(ag 18,
ai li, 0g eu, 10 (o,5n)
min
Å9,
G
Tona (no,n, 12,n
as Osg li, 0g an 19 pg,en))
g
To (ag.n: 12.m ass lis (nsenn Te (aā, tg.n ai to og <n tn)
clip
c,1 1
- Dxa lo Tral
(3)
Tag (ag on on
19 a², 1,,,,,,, Cg 19 12,en)
DKI[T Kroel
To (a ty
a' t¹, 0 o tg tg
(4)
Tie (ur onn 18, a² 1 1 Co 19 (n)
log (a to a² og <nz tg.cn)
We sample prompts from D/r generate G responses, decode tokens into waveforms with p. score them with d. compute the normalized advantage A (d(w,, p(ag), - w)/a. and update the policy 0 for higher rewards. l and are the mean and standard deviation of the raw scores within group. is the clipping threshold.
A.5 ABLATION FOR GRPO TRAINING
Global batch size. The global query batch size defines the total number of distinct speech prompts across devices. Figure 3[a) shows that the ParaS2SBench score continues to improve with larger global batch sizes, while exhibiting diminishing returns the computing requirement (more GPUs) increases. We use a batch size of 32 as the default, where 8 NVIDIA H100s are sufficient for single run.
19

---

Published as a conference paper at ICLR 2026
4.45
79
Paras2SBench Score
4.4
Para52SBench Score
4.40
4.35
PeraS25Bench
4.435
VoiceBench
4.30
4.430
73
4.25
3.8
76
4.20
4.420
16
0.0
0.2
0.4
Global Query Batch Size
GRPO Group Size
GRPO KL weight (bela)
(N)
(h)
(c)
Figure 3: Ablating the effect of global batch size and GRPO's group size and KL penalty weight. For all experiments, we optimize for the same number of steps.
GRPO group size. GRPO group size defines how many samples drawn for each speech prompt. Since GRPO relies on differences between samples for the learning signal, it is crucial to have large enough group size ensure diversity. Figure shows that when the group size is smaller than 8, performance drops significantly. For example, when the group size is 2, the two samples often receive the
score, providing no learning signal. Interestingly, group size of 8 is sufficient for effective learning, and increasing the group size further does not provide additional gains.
find that
KL weight is critical for preserving original capabilities. Finally, we study the effect of the KL penalty weight S. During GRPO, we aim to enable paralinguistic aware dialogue capabilities with- out degrading the original dialogue capabilities, as training might otherwise overfit to the training set. We leverage VoiceBench (Chen et al., 2024) to quantify changes in the original dialogue ca- tasks
pabilities. The benchmark includes daily QA. knowledge-intensive QA. cestruction-followwer in both close-ended and open-ended scenarios. Higher VoiceBench scores indicate stronger gen- eral dialogue capabilities, while higher ParaS2SBench scores indicate stronger paralinguistie-aware dialogue capabilities. Figure Bic) shows that: (1) without a KL penalty, the model suffers from catastrophic forgetting and VoiceBench performance drops significantly; (2) with too high KL penalty, the model is overly constrained by the original parameters and cannot freely explore the search space, leading drop in ParaS2SBench score. We therefore set the default to 0.2, which achieves both capabilities without one degrading severely.
A.6 HUMAN EVALUATION
Table 7: Comparing paralinguistic-aware dialogue capabilities with human evaluation.
Synthetic
Real
Avg
Age
Emotion
Gender
Sarcasm
Avg
IRMOCAP
MELD
Avg
Baseline
GPi-TTs-GPT-TTS
3.212
3.041
3.042
3.112
3.102
3.601
3.552
3.487
3.230
Closed Source
GPT-4o voice mode
3.375
3.833
3.542
3.078
3.457
3.862
3.694
3.778
3.564
Open Source Qwen2.5 Omni
3.352
3.953
3.496
3.131
3,483
3.713
3.581
3.647
3.538
GL. 4
3.012
3.514
3.228
2,781
3.134
3.521
3.325
3.423
3.230
Kimi-Audio
3.278
2.382
3.121
2.912
2.924
2.231
2.272
2.252
2.699
Ours
Kimi-Audio SFT
4.192
4.223
3.812
4.131
4.089
4.212
3.407
3.810
3.996
Kimi-Audio GRPO
4.316
4.510
4.381
4.422
4.407
4.336
3.859
4.098
4.303
Topline GPT-TTS
4.752
4.889
4.923
4.813
4.844
4.911
4.925
4.918
4,922
In the main article, we present the objective evaluation using the automatic ParaS2SBench score. Although the ParaS2SBench score shows high correlation with human judgments in Section 5.1
20

---

Published as a conference paper at ICLR 2026
the correlation remains below 0.9, leaving for inconsistencies. We therefore study the ef- fectiveness of our approach under human subjective evaluation. Specifically, we crowd-source 10 participants outside expert annotation group. which designed the scoring guideline and anno- tated the preference scores in Section These participants have minimal knowledge of the project, including the guideline r, avoid inductive bias. They given pairs of input and response audio clips and asked to assign 1-5 mean opinion score based on how naturally the two clips fit together in dialogue. Due annotation costs, sample a subset from the ParaS2SBench test set, with 30 prompts per category. For each prompt-respons. pair, 10 human scores are collected and averaged as the final score.
Consistency between human evaluation and automatic judgment. Table shows that the over- all trend is consistent with Table SFT on Kimi-Audio provides significant boost over the base model and surpasses existing models. Kimi-Audio with GRPO further outperforms SFT by 7.6%. Similar to Table the existing models struggle on paralinguistic-an dialogue.
Differences between crowd-sourced and expert annotators. One notable difference between the objective and subjective evaluations is that our crowd-sourced participants tend to assign higher scores than both the benchmark pipeline and our expert annotators. This may be because the par- ticipants are not trained to recognize fine-grained paralinguistic labels in speect 22 and often assign high scores when stylistic deviations are subtle This suggests that in everyday use, typical users may be more tolerant of paralinguistic unawareness or tone-deaf responses than our benchmark criteria imply, which explains the smaller relative improvement compared to objective evalua- tion. Nevertheless, the 7.6% relative improvement in the subjective evaluation remains substantial, underscoring the importance paralinguistic awareness for improving user satisfaction.
A.7 ANALYSIS ON THE ORIGINAL CAPABILITIES PRESERVATION
Baseline Topline
Close
Open-source
100
Ours
4.5
Para5258ench Score (EQ)
80
Volce Score (10)
3.0
2.5
40
1.0
Whisper-GPEATTS
GPTTTS
GPT-40 vaice mode
Gemini
Kiml Audio GRPO
Kimi-Audio SFT
Kimi-Audio
Owen2.5 5 Omni
LLaMa-Ommiz 2
GLM
Freeze-Ome Omni
Model
Figure 4: Comparing the overall intelligence and paralinguistic-wwarr dialogue capabilities across models.
maintain the base model's intelligence by carefully tuning the KL penalty. We leverage VoiceBench (Chen et al. (2024) to quantify changes in general intelligence. The benchmark includes daily QA,
As discussed in Appendix A.5
QA, and instruction-followiii tasks in both
22 They are only instructed pay attention speaking style, age, and gender, but given detailed style labels to avoid inductive bias. For example, a slightly sad expression may be perceived as neutral, and an otherwise normal response may still receive high score.
21

---

Published as a conference paper at ICLR 2026
close-ended and open-ended scenarios. Higher VoiceBench scores indicate stronger general intelli- gence, while higher ParaS2SBench scores indicate better paralinguistie-awe -ware dialogue capabilities.
Our post-trained model preserves its top-tier intelligence. Figure shows that the pipeline- based baseline and topline achieve the highest intelligence, followed by closed-source models. Among all open-source models, ours demonstrates the highest intelligence. This stems from our choice of Kimi-Audio the base model-sinee it already possesses the strongest intelligence among existing open-source models-and and from our training procedure, which preserves its original capa- bilities.
State-of-the-art paralinguissic-awarr dialogue capabilities without sacrificing intelligence through RI. For paralinguistic-aware dialogue capabilities, our SFT model surpasses all exist- ing models except the topline, which leverages ground-truth style labels. Our GRPO model further improves upon the SFT model without compromising intelligence. Overall, our models achieve state-of-the-art performance in both general intelligence and paralinguistic-awarr dialogue capabil- ities.
A.8 PROMPTS
We include all the prompts used for data generation and automatic scoring.
A.8.1 QUERY GENERATION FOR AGE
Prompt Template:
Give some kids' requests that should not be directly processed due to adult-level permission quali- fication. Also, these requests should age-ambiguous such that is hard tell age from the content. The request should oral suitable text-to-speech. want specific user chats instead of category or topic names. Give me a DIVERSE dataset 100 requests related to [topic} in the following format:
1. user_chatl
2. user_chat?
3.
You should give ALL the requests without ending carly.
Topics:
Location Travel
Privacy & Security
Interpersonal Social
Mental Health & Emotions
Money& Online Transactions
Technology & Content
Physical Health Safety
Sensitive & Risk-flagged
A.8.2 QUERY GENERATION FOR GENDER
Prompt Template:
want to test whether the spoken agent can really understand gender from voice. want to design user requests such that the request itself is sender-ambiguous, but require the agent to find out the gender to avoid inappropriate response. Avoid using the word: 'or', 'and'. 'this'. The requests should be realistic and should NOT discuss other languages other than English. The request should be in oral form suitable for text-to-speech. Give 100 requests (specific user request, topic or category names) related (ropic} in the following format:
22

---

Published as a conference paper at ICLR 2026
1. user.chatl
2. user.chat2
3.
You should give ALL the requests without ending early.
Topies:
Cultural and Religious Advice
Medical and Health Advice
Gender-specific Activity
Fashion, Beauty, and Grooming
A.8.3 QUERY GENERATION FOR EMOTION
Prompt Template:
Can you give me some user chats with that the chat's content itself is emotion-angiiuuus and can be normally voiced by OPPOSITE emotions. Avoid the word: ChatGPT, Al. The request should be realistic, detailed and specific. The request should also the specific user chats instead the category or topic names. Give me the user request and the two OPPOSITE emotions. Give me 100 requests that can be voiced by both {emotion!) and {emotion2), related to (topic), in the following format (do not repeat the demo):
oral form suitable text-to-speech. want
1. The city is planning to build new mall near my house. (emotionl, emotion2)
2.
3.
You should give ALL the requests without ending carly.
Topics:
Personal Life
Current Events & News
Entertainment
Hobbies & Interests
Work Studies
Internet Culture
Travel
Food & Drinks
Relationships
Technology & Gadgets
Health & Fitness
Education
Finance & Money
Fashion & Style
Life Advice
Cultural Differences
Dreams & Sleep
Holidays & Celebrations
Childhood Mcmories
Future Plans
23

---

Published as a conference paper at ICLR 2026
A.8.4 QUERY GENERATION FOR SARCASM
Prompt Template:
You are generating user chat requests that, in plain text, read as entirely positive, cheerful, or supportive, yet could plausibly be sarcastic depending tone of voice context. They must be realistic for human-Al interactions and slightly longer (1-2 sentences), with natural variety.
Requirements:
DO NOT use obvious sarcasm markers: "yeah right". "sureee", "as if". "totally...., ellipses for irony, scare quotes, ALL CAPS emphasis. exaggerated punctuation, or emoji/emoticons.
Avoid overly negative words (e.g., "hate", "awful", "broken", "slow", "crash").
Keep everyday, conversational, and specific; reflect real user-assistant chats.
All items must relate to the topic: {topic}.
Output EXACTLY 100 items numbered list starting One line per item.
Bad (too obvious) examples (DO NOT imitate):
Oh GREAT, another eroor just what needed!!!
Yeah right, your 'amazing" update totally helped.
Good (subdy ambiguous, still positive) examples (DO imitate the vibe, not the content):
I'm so glad you're 're handling can you walk me through your plan so don't mess up anything on my end?
That's exactly what I expected from you; love the confidence- should hit "apply" check the details first?
double-
Wow, you'r re really on top of things today! Could you also sync what you did with the previous settings everything stays consistent?
Now produce list this format:
1. <item 1>
2.
<item
3.
You should give ALL the requests without ending carly.
Topics:
Personal Life
Current Events News
Entertainment
Hobbies & Interests
Work & Studies
Internet Culture
Travel
Food & Drinks
Relationships
Technology Gadgets
Health & Fitmess
Education
Finance & Money
Fashion & Style
Life Advice
Cultural Differences
24

---

Published as a conference paper at ICLR 2026
Dreams & Sleep
Holidays Celebrations
Childhood Memories
Future Plans
A.8.5 SCORING GUIDELINE
This is the scoring guideline designed by the human annotation expert, in Section 3.2 The user information includes the transcription as well as the emotion, sarcasm, gender, and age labels. The agent information includes the response transcription and a description of the speaking tone.
You are a human dialogue expert. You will be given a pair of user request audio and an agent response audio. Please focus the user's speech attributes, including content, emotion, age, gender, sarcasm, and decide whether the agent's response good fit in natural dialogue. Here guideline defining the good response of an agent. You need to score the response's 's quality for Continuation Fitness.
General guideline of Continuation Fitness:
MOS on how well the response speech fits the user speech.
Listen carefully to spoken content and speaking style the user speech and the response speech. Decide whether the response speech, including how and what the agent says, is natural continuation to the user's spoken content and speaking style (emotion, age, gender, sarcasm). Here are some general rules and examples:
The agent should be friendly, helpful, considerate, with high EQ
2. Besides the replying content, please also listen carefully the response's speaking style:
tion, tone, volume and speed, and decide whether the speaking style appropriate.
3. The agent should be aware of the user's emotions provide emotional companionship. For example, when the
happy, the agent share that happiness; when the sad, the agent should be sad and empathic.
4. If the user is child, the agent should respond with caution and safety. For example, should redirect the user if he or she attempts to purchase alcohol online.
5. The agent should be aware of the user's gender and personalize responses. For example, it may offer different suggestions for swimwear.
6. Suggesting items more related to male for female. or vice versa, is considered very ackward. For example, suggesting masculinised perfume for female is ackward.
7. The agent should recognize sarcasm and understand the user's true intent. For example, when gives sarcastic compliment, the agent should recognize that the underlying sentiment is negative. Hence, it would be weird if the agent feels happy for the compliment.
You should rate the response's quality in 5 points.
Points Perfect (Enhanced):
The user's age, gender, speaking style characteristics recognized and reflected in the response with appropriate enhancements. The user's query contains clear emotional cues, and the response provides empathetic feedback. The user's query has a clear sarcastic tone, and the response offers high-EQ reassurance or clarification. The user's query is a sincere compliment, and the response is thankful.
Examples: When the user is happy, the response shares the joy; when the user is sad, the re- sponse offers appropriate comfort. If minor attempts to purchase alcoholic beverages online, the model provides correct guidance. For young user, the response uses trendy slang popular among young people. Provides gender-suitable response (i.e. different swimwear suggestions) based on the user's ' gender. When receiving a sarcastic comment, the model identifies the un- derlying negative sentiment and responds accordingly.
Points Excellent (No Enhancement):
The user's paralinguistic cues are addressed the replying content is good, but the response's vocal tone does not enhance the user's experience.
Examples: A neutral-tone response to a female user inquiring about cancer screening. A neutral-tone response to neutral question. The response's content picks the user's sarcastic comment, but the not appropriate.
25

---

Please give the S-point score VERY brief reason format: The reason 4; The score Published as a conference paper at ICLR 2026
Points Average:
The user's paralinguistic cues or other speaking style features are considered, but the response does not provide correct personalized content, though it is not jarring: Mechanical empathy, awkward praise,
Examples: A happy or sad response neutral question.
Points Poor:
The user's paralinguistic cues or other speaking style features are considered but poorly ad- dressed. Emotion mismatch: if the agent identifies the wrong user emotion. e.g. Reply to mismatched.
fearful user as if he/she is sad; Reply angry user if he/she is fearful. Style partially
Examples: A flat response sad question. Using slang when responding elderly user.
Point Very Poor:
The user's paralinguistic cues or other speaking style features are considered but addressed incorrectly. Reverse empathy, condescending Reply to happy user he/she sad. Completely mismatched style, e.g., responding an adult completely childish tone. Misinterpret a sincere compliment as negatvie comment, and give apologetic, clarifying, or reassuring comment. Misinterpret sarcastic compliement as sincere, and give positive thankful comment.
c.g. Reply to sad user as if hc/she is happy:
Examples: A cheerful response to sad user. Using language that is too complex for child. Giving male-specifir recommendation female, vise versa.
The information of the user:
(transcription) ) (emotion) (sarcasm) {age) (gender)
Here is the information of the agent:
(transcription))(nn (taniiioonnnneee {tone}
26


Reviews:
[{"review_id": "kNpiJrvTYZ", "reviewer": "Reviewer_zJW8", "summary": "This paper introduces an innovative benchmark and reinforcement learning framework for paralinguistic-aware speech-to-speech (S2S) models, effectively addressing the current limitations in handling paralinguistic information such as emotion, tone, gender, and age. The authors design an automated data curation and speech synthesis pipeline, and leverage a reward model for efficient training and evaluation. Experimental results demonstrate that the RL approach achieves significantly better content and style appropriateness than conventional supervised fine-tuning, with much lower data and annotation costs. Overall, this work is forward-looking and practical, providing valuable tools and references for the development of paralinguistic-aware S2S models.", "questions": "1. In your data curation and speech synthesis process, how do you ensure that the generated paralinguistic styles (such as emotion, age, gender, etc.) are sufficiently authentic and diverse?\n2. After SFT and GRPO, does the model’s original capability decrease compared to the results reported for Kimi-Audio?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The proposed paralinguistic-aware S2S reinforcement learning framework is highly practical, effectively enhancing the model's ability to understand and generate paralinguistic information such as emotion and tone, which provides valuable tools and methods for the advancement of speech dialogue systems.\n2. The experiments are thoroughly designed, covering various paralinguistic factors and realistic scenarios. The results comprehensively validate the significant improvements in content and style appropriateness achieved by the proposed method, making the findings highly convincing.\n3. The paper is well-structured and clearly articulated, with a rigorous logical flow. It progresses coherently from problem background, method design, to experimental validation and result analysis, making it easy for readers to understand and follow.", "weaknesses": "1. The methodological innovation of the paper is limited, as it merely applies GRPO in a straightforward manner.\n2. The presentation lacks intuitiveness; it is difficult to fully convey the paralinguistic features of audio through text alone. It would be better if there were a demo page or web-based showcase.\n3. Some references are missing, such as [1]: \n[1] Omnichat: Enhancing spoken dialogue systems with scalable synthetic data for diverse scenarios. arXiv preprint arXiv:2501.01384.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "CxZBSEGs6x", "reviewer": "Reviewer_hX74", "summary": "The study introduces a methodology for enabling Speech-to-Speech (S2S) language models to recognize and respond to critical paralinguistic elements, such as emotion, intonation, and speaker characteristics, which extends far beyond simple content transmission.\n\nThe authors present two core components:\n\n* **ParaS2SBench:** An automated benchmark designed to evaluate how effectively S2S models align with both the **content** and **style** of an utterance.\n* **ParaS2SAlign:** A learning framework that utilizes Reinforcement Learning (RL) to achieve model alignment directly at the waveform level.\n\nThe benchmark scores show a high correlation (>0.7) with human evaluations and the RL approach achieves a notable 11% performance improvement over Supervised Fine-Tuning (SFT), alongside a five-fold enhancement in label efficiency.", "questions": "1.  **Readability:** I noticed a minor formatting point where the inconsistent use of parentheses for citations occasionally impacted readability. Clarifying this convention throughout the manuscript might be beneficial for readers.\n2.  **Data Composition:** Could the authors please specify the total number of distinct speakers represented in the training data and the ParaS2SBench benchmark, respectively?\n3.  **Confidence Intervals:** I would find it very helpful to see the 95% confidence intervals for the reported GPT and human evaluation scores, as this would further strengthen the statistical significance of the findings.\n4.  **Performance on Existing Capabilities:** Finally, a point of great interest is the trade-off with existing abilities. I would be grateful if the authors could provide an analysis of any performance degradation on foundational capabilities (e.g., as measured by VoiceBench) after the application of SFT and, subsequently, the full RL alignment process.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The paper puts forward a novel benchmark and dataset, with a welcome commitment to their public release. \n- The authors provide a valuable analysis of the respective impacts of RL and SFT on the modeling of non-verbal conversational aspects within the proposed framework.", "weaknesses": "* **On the Reward Model:** A point of consideration emerged regarding the reward model. I'm respectfully curious about the potential for it to be somewhat overfitted to the specific synthesis engines used for evaluation, namely the GPT-based TTS and CosyVoice. I would be interested to hear the authors' perspective on its generalization capabilities to other speech styles.\n* **On Data Synthesis:** Additionally, as the audio corresponding to the evaluated scenarios appears to be entirely synthetic, a slight query arises regarding potential constraints on the diversity and complexity of the model's expressive output. I wonder if this might impact the model's ability to capture the full spectrum of nuances present in organic, human-to-human interaction.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "dcDPndiJFN", "reviewer": "Reviewer_qCT1", "summary": "This paper introduces ParaS2S, a reinforcement learning framework that enables speech-to-speech models to generate responses with appropriate content and speaking style. Using the new ParaS2SBench for automatic content and style evaluation, ParaS2S improves paralinguistic awareness by 11% over supervised fine-tuning while requiring far fewer annotations.", "questions": "(1) What is the primary contribution of this paper — the new benchmark or the proposed alignment strategy? The authors should clarify which aspect represents the core focus of the work.\n\n(2) Why is the proposed alignment strategy not compared with SpeechAlign? The two methods appear quite similar, except for the adoption of the GRPO technique. Since SpeechAlign also employs DPO and evaluates comparable aspects of speech language models, a direct comparison is necessary to highlight the distinction and advantage of the proposed approach.", "limitations": "", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "The motivation to evaluate paralinguistic responses in speech language models is both natural and important. This reviewer appreciates the authors’ effort in advancing research on this topic. Moreover, the overall workload presented in this paper appears substantial.", "weaknesses": "(1) The core contributions of this paper are somewhat unclear. It mainly includes two parts: a new benchmark for evaluating speech response style and content, and an alignment technique for tuning speech language models. However, each contribution appears incomplete on its own. The benchmarking part omits many relevant speech and speech-to-speech models, while the proposed alignment method lacks sufficient novelty and empirical validation.\n\n(2) The citation format should follow the ICLR template by using \\citep instead of \\cite, as the current style blends citations into the text and reduces readability.\n\n(3) Table 1 reports several numerical results to show that the evaluation aligns with human judgments, but the justification is not rigorous. The paper should clarify what criteria define “closeness” to human evaluation and why they are reasonable. For instance, in the Emotion S2S model, the gap between GPT and human evaluations does not appear negligible.\n\n(4) The novelty of the alignment approach is limited. If the paper argues for the necessity of GRPO, this claim should be empirically supported, and comparisons with existing methods such as SpeechAlign are essential.\n\n(5) The GRPO alignment is evaluated only on the Kimi-Audio base model. A more comprehensive study should include multiple base models to demonstrate that the proposed strategy generalizes beyond a specific setup.\n\nIn conclusion, while the paper presents a decent amount of work, it remains incomplete by publication standards. The authors are encouraged to focus on a single, well-developed contribution—either the benchmarking framework or the alignment technique.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "X9MD9frjh6", "reviewer": "Reviewer_cPKU", "summary": "This paper introduces a novel reinforcement learning (RL) framework to enhance the capability of paralinguistic cues for speech-to-speech (S2S) dialogue model. It clearly introduce the topic of natural speech communication which conveying more than just words but also paralinguistic cues such as emotions, tone, speaker attributes and how it impacts the speech response. To address this, the author constructs a benchmark which can automatically evaluates the S2S model as ParaS2SBench for content and style appropriateness, then with the benchmark system, it can use RL approach like GRPO to improve the response content and style fitness. \n\nThe major contributions lies on the concrete description of this topic, construct appropriate S2S evaluation as benchmark, demonstration of RL and SFT effectiveness and cost via experiments. The benchmark is characterized by imaginative design and rigorous focus on its objectives, while the exploration of the experiments are also inspiring.", "questions": "It would be much friendly if the author could select the most crucial experiments to introduce more profoundly, analysis/conclude them clearly in the main paper session 5 and put the other experiments details in the appendix paper. Anyway, there are some part which is not fully introduced due to the length to the article and need to read the appendix to get the full results. \nFor the generalization to real speech experiment, is there any human evaluation to compare with the bench score? This is quite curial to this RL framework.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1). The construction of this benchmark system is quite clear and scientific. It designs various of domain for different query, contrasting speaking style, and the most key part is the scenario-controlled queries which is designed to control the neutral text content filtering those doesn't convey too much additional paralinguistic information. According to the appendix table for query example and prompt, the data curation is full of authors' thought on this topic and looks very interesting.  \n\n2）In the experiments part, besides the validation of the effective of RL framework and SFT analysis, it propose so many questions with experiments which are very instructive. Such as the amount of annotations can RL save? should we invest more costs on SFT or reword model, how is the generalization to real speech? These are realistic and serious problems for this S2S model need to handle and could be continuous and broad interest for the research problem.", "weaknesses": "The most weakness of this paper from my perspective is the depth of the experiments part. The author provides rich and many good inspects questions here which is good, however, it seems there isn't any experiment or analysis which shows the most important and critical point view from the author. It may weaken the paper's persuasiveness and confuse reader about the core information obtained from the experiments.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)