VowelPrompt: Hearing Speech Emotions from Text via Vowel-level Prosodic Augmentation
Authors: Yancheng Wang
Osama Hanna
Ruiming Xie
Xianfeng Rui
Maohao Shen
Xuedong Zhang
Christian Fuegen
Jilong Wu
Debjyoti Paul
Arthur Guo
Zhihong Lei
Ozlem Kalinli
Qing He
Yingzhen Yang
Venue: ICLR 2026 Poster (2026)
Keywords: Reasoning
Large Language Models
Emotion Recognition
Vowel
Primary area: applications to computer vision, audio, language, and other modalities
TL;DR: We propose VowelPrompt, a linguistically grounded framework that augments language models with vowel-level prosodic features to enhance emotion recognition performance and generalization across diverse inference settings.

Abstract: Emotion recognition in speech presents a complex multimodal challenge, requiring comprehension of both linguistic content and vocal expressivity, particularly prosodic features such as fundamental frequency, intensity, and temporal dynamics. Although large language models (LLMs) have shown promise in reasoning over textual transcriptions for emotion recognition, they typically neglect fine-grained prosodic information, limiting their effectiveness and interpretability. In this work, we propose VowelPrompt, a linguistically grounded framework that augments LLM-based emotion recognition with interpretable, fine-grained vowel-level prosodic cues. Drawing on phonetic evidence that vowels serve as primary carriers of affective prosody, VowelPrompt extracts pitch-, energy-, and duration-based descriptors from time-aligned vowel segments, and converts these features into natural language descriptions for better interpretability.
Such a design enables LLMs to jointly reason over lexical semantics and fine-grained prosodic variation. Moreover, we adopt a two-stage adaptation procedure comprising supervised fine-tuning (SFT) followed by Reinforcement Learning with Verifiable Reward (RLVR), implemented via Group Relative Policy Optimization (GRPO), to enhance reasoning capability, enforce structured output adherence, and improve generalization across domains and speaker variations. Extensive evaluations across diverse benchmark datasets demonstrate that VowelPrompt consistently outperforms state-of-the-art emotion recognition methods under zero-shot, fine-tuned, cross-domain, and cross-linguistic conditions, while enabling the generation of interpretable explanations that are jointly grounded in contextual semantics and fine-grained prosodic structure.

Published as a conference paper at ICLR 2026
VOWELPROMPT: HEARING SPEECH EMOTIONS FROM TEXT VIA VOWEL-LEVEL PROSODIC AUGMENTATION
Yancheng Wang 1.2 Osama Hanna'
Xianfeng Rui' Debjyoti Paul Zhihong Lei Ozlem Kalinli' Qing He' Yingzhen Yang²
Ruiming Xie
Maohao Shen 1.3+
Xuedong Zhang' Christian Fuegen' Jilong Wu
Arthur Guo
Meta Superintelligence Labs, Ariizna State University, Massachuseetts Institute of Technology yancheng. wangeasu.cc
ohanna&meta: com
ABSTRACT
Emotion recognition in speech presents a complex multimodal challenge, requir- ing comprehension of both linguistic content and vocal expressivity, particularly prosodic features such as fundamental frequency, intensity, and temporal dynam- ies. Although large language models (L.LMs) have shown promise in reasoning over textual transcriptions for emotion recognition, they typically neglect fine- grained prosodic information, limiting their effectiveness and interpretability. In this work, we propose VowelPrompt, a linguistically grounded framework that augments LLM- emotion recognition with interpretable, fine- grained vowel- level prosodic cues. Drawing on phonetic evidence that vowels serve as primary carriers of affective prosody, VowelPrommt extracts pitch-, energy-, and duration- based descriptors from time-aligned vowel segments, and converts these features into natural language descriptions for better interpretability. Such a design enables LLMs jointly reason over lexical semantics and fine-grained prosodic varia- tion. Moreover, we adopt a two-stage adaptation procedure comprising supervised fine-tuning (SFT) followed by Reinforcement Learning with Verifiable Reward (RLVR), implemented via Group Relative Policy Optimization (GRPO), to en- hance reasoning capability, enforce structured output adherence. and improve gen- eralization across domains and speaker variations. Extensive evaluations across diverse benchmark datasets demonstrate that VowelPrommpt consistently outper- forms oft--h--ttt-ttttt- emotion recognition methods under zero-shot, fine-tuned, cross-domain, and linguiiin conditions, while enabling the generation of interpretable explanations that are jointly grounded in contextual semantics and fine- ene-ggained prosodic structure.
INTRODUCTION
Paralinguistic speech understanding requires modeling not only what said but how is said with prosodic patterns in fundamental frequency (Fo), intensity (RMS energy), timing (duration, rhythm, pause), and voice quality. Speech emotion recognition (SER) is commonly framed either with discrete categories, such as, angry, sad, happy, neutral, with dimensional labels in the lence-arousal suse-arroosa--ddmiaancee space, and evaluated on acted and naturalistic corpora (Busso et al., 2008; Poria al., 2019; Cao al., 2014; Livingstone & Russo, 2018: Russell, 1980; Bradley & Lang, 1994). Classic SER pipelines extract engineered low-level descriptors (LLDs) and functionals via OPENSMILE (Eyben et al., 2010) and standardized sets (GeMAPS/eGeMMAPS) (Eyben al., 2015), chosen specifically for interpretability in paralinguistics. Recent advances (Pepino et al., 2021; Yang al., 1)2 are driven by self supervisd speech representation learning methods, such as wav2vec 2.0 (Chen et al., 2022), HuBERT (Hsu et al., 2021). and WavI.M (Chen et al., 2022), which provide robust utterance representations and often set strong baselines on SER and SUPERB-style evalua- tions. While effective, these embeddings are opaque and require audio encoder at inference time, which complicates interpretability and deployment in text-only settings.
Work done during internship Meta Superintelligence Labs.

---

Published as a conference paper at ICLR 2026
Large language models (LLMs) have introduced two complementary paths for spoken affect. Audio language LLMs such as AudioPaLM (Rubenstein et al., 2023), SALMONN (Tang et al., 2024), Qwen2-Audio (Chu et al., 2024), and task-specific AmmotionLLLMMe (Cheng et al., 2024), integrate continuous acoustic encoders with L.LM backbones to reason directly over speech (Rubenstein et al,, 2023). In parallel, text-only prompting augments ASR transcripts with hatural-language descriptions of prosody (e.g., "spoken loudly with rising intonation"), enabling LLMs to exploit affective cues without consuming raw audio (Wu et al., 2025), motivated by the strong reasoning capabilities of text LLM DeepSeek-Al Al et al. (2025); Wang et al. (2024); Zhao et al. (2025); Wei et al. (2022). The latter is lightweight and interpretable but typically uses coarse, utterance-level descriptors that can blur fine-grained cues.
Phonetic evidence suggests that phoneme classes contribute unequally to affective cues. Vowels, voiced nuclei with relatively stable Fo and energy, often carry salient intonation patterns; syllable nuclei have also been used to localize prosodic variation (Ringeval & Chetouani, 2008). At the same time, class-aggregated analyses indicate that consonantal regions can encode complementary or even stronger spectral evidence for emotion in some settings (Bitouk et al., 2010). This moti- vates segment-centric representation that emphasizes vowel nuclei to capture fine-grained prosodic structure, while preserving the full lexical context.
We propose VowelPrompt, simple yet effective interpretable augmentation method for LLM-based speech emotion recognition. Given an utterance and its transcript, the method first obtains time- aligned vowel segments through a standard forced-alignment pipeline. It then extracts vowel-level low-level descriptors, including Fo level and slope, Fo variability, intensity level and variability, and segment duration, applying both speaker and vowel-type normalization. These values are dis- cretized via quantile binning and converted into concise natural-languns prosodic descriptors such as "high Fo. rising. loud, lengthened." " The resulting descriptors are appended to the transcript so that a text-only LLM can jointly reason over lexical content and segment-level prosody. Model adapta- tion follows two-stage regimen, beginning with supervised fine-tuning (SFT) and continuing with Reinforcement I Learning with Verifiable Reward (RLVR) using Group Relative Policy Optimization (GRPO) to improve reasoning quality, output-format adherence, and robustness while maintaining proximity to the SFT reference (McAuliffe et al., 2017; DeepSeek-Al et al., 2025).
Contributions. The contributions this paper are summarized as follows.
First, leveraging well-established phonetic evidence, VowelPrompt extracts vowel-level prosodic descriptors, including pitch level and contour, intensity, and temporal duration, from time-aligned segments obtained via forced alignment, applies both speaker- and vowel-type normalization, and discretizes these features into natural language descriptions. These interpretable descriptors are ap- pended to transcripts, enabling LLMs jointly reason over lexical semantics and localized prosodic variation, in contrast to opaque acoustic embeddings.
Second, to adapt LLMs this enriched input, we design two-stage training pipeline that begins with supervised fine-tuning (SFT) for cold-start alignment and continues with Reinforcement Learn- ing with Verifiable Rewards (RLVR) using Group Relative Policy Optimization (GRPO), which improves structural adherence, robustness, and reasoning quality.
Third, extensive experiments on five benchmark datasets, including IEMOCAP (Busso al., 2008), MELD (Poria et al., 2019), CaFE (Gournay et al., 2018), EmoDB (Burkhardt et al., 2005), and ASVP-ESD (Tientcheu Touko et al., 2021), demonstrate that VowelPrompt consistently surpasses competitive baselines across zero-shot, few-shot, fine-tuned, cross- doos-, and multilingual condi- tions, while enabling interpretable and verifiable emotion reasoning grounded both linguistic and prosodic information.
2
RELATED WORKS
2.1 SPEECH EMOTION RECOGNITION AND PARALINGUISTIC ANALYSIS
Speech emotion recognition (SER) aims to infer a speaker's affective state from acoustic signals, often leveraging prosodic, spectral, and linguistic features. Early SER systems relied heavily on low-level descriptors such as fundamental frequency (FO), energy, and temporal statistics, extracted via toolkits like openSMILE (Byben et al., 2010). The INTERSPEECH Computational Paralinguis-
2

---

Published as a conference paper at ICLR 2026
ties Challenge series established standardized feature sets such as the Geneva Minimalistic Acoustic Parameter Set (GeMAPS) (Eyben et al., 2015), which provide interpretable descriptors covering pitch, loudness, and voice quality. Deep learning methods have since outperformed handcrafted fea- tures in performance, with wav2vec 2.0-based embeddings (Pepino et al., 2021) and contextualized transformer encoders such EmoBERTa (Kim & Provost, 2021) achieving state-of-the- results. However, these high-dimensional representations are difficult to interpret, making it challenging to explain or control model predictions in sensitive applications. Recent advances integrate language models with acoustic or visual modalities to improve emotion reasoning. Prompt-based augmenta- tion has been explored, where prosodic descriptions (e.g., "spoken loudly with rising intonation") are prepended to transcripts to guide large language models (Wu et al., 2025). This approach yields measurable improvements in zero-shot emotion recognition, particularly in clean speech settings. At the architectural level, multimodal models such as AudioPal.M (Rubenstein et al., 2023) and EimotionLLLMMe (Cheng et al., 2024) fuse audio embeddings directly into transformer-bass LLMs, enabling joint reasoning over text and audio inputs. While effective, these systems typically rely on audio embeddings learned through black-box models, which limit their interpretabilltyy Our work bridges this gap by combining interpretable vowel-level acoustic features with textual prompting, enabling accuracy gains while preserving human-readable intermediate representations.
2.2 VOWEL-CENTRIC PROSODY IN EMOTIONAL SPEECH
Phonetic studies consistently highlight vowels as primary carriers of emotional prosody. Vowels, being voiced and acoustically stable, exhibit clear correlates of affect such as pitch level, contour, intensity, and duration (Ringeval & Chetouani, 2008). Ringeval & Chetouani (2008) has demon- strated that bowel-based acoustic features improve emotion classification compared to utterance- level statistics, while Schuller (2009) has found that class-level spectral features for vowels and consonants can capture complementary emotional cues. Subsequent work in articulatory phonetics found that emotional states systematically shift vowel articulation and formant positions, influenc- ing both perceived tone and loudness (Shah & Busso, 2019). Despite these findings, most modern SER pipelines extract features uniformly across all phonemes, potentially diluting the discriminative power of vowel-specific prosodic cues.
Vowels, as voiced phonemes characterized by a relatively open vocal tract configuration, domi- nate both the acoustic energy and temporal duration of spoken utterances. They convey substantial portion of prosodic information, including pitch (fundamental frequency), intensity (perceived loud- ness). and temporal patterns (duration and rhythm), which are critical to paralinguistic expression and emotional communication (Crystal, 1969; Mozziconacci, 2002). Extensive phonetic research has shown that vowels function as primary carriers of intonation contours and emotional coloration, owing to their sustained voicing and spectral stability. Building on these insights, we construct a structured, interpretable intermediate representation that focuses on vowel-centric acoustic features as bridge between raw audio signals and downstream language models.
3 METHODS
We propose VowelPrompt, unified framework that enriches LLMs with interpretable vowel-level prosodic cues for enhanced speech emotion recognition. The central premise is that vowels, which carry the majority of the energy and prosodic variation of the speech signal, serve fine-grained indicators. To exploit this property, VowelPrompt first derives discrete, human-interpretable descrip- tors of pitch, intensity. and temporal dynamics from individual vowel segments, isolating localized acoustic events that often correspond to emotionally salient moments in speech. These descriptors are converted into natural language and integrated directly into the input prompts alongside the tex- tual transcript, enabling the LLM to reason jointly over lexical semantics and prosodic structure. The model is then adapted to the emotion recognition task via a two-phase training regime. The su- pervised fine-tuning phase aligns the LLM's generation behavior to produce accurate, well-reasoned emotion predictions conditioned on both textual and prosodic information, while the reinforcement learning phase refines the LLM's reasoning guality, adherence to output format, and robustness to speaker and context variability. This design bridges the interpretability of phonetic-level anal- ysis with the reasoning capabilities of modern LLMs, yielding system that can explicitly link
3

---

Published as a conference paper at ICLR 2026
[transcript]
[allgniment]
But true, babe. You've get be more posilive about things. some yoursal
babe
got
berb
gazt
vowel:
Vowel:
vowel: /a/
vowel: /a/
[phonemes]
slope: medium
slope: low
slope: high
siope: medium
both juav rgaer tabi mou pazzatir
pitchT variance
pitch variance-
pitchT variance volume varianced duration (very low)
pttch variance
abact Barz. lzak hare swm foc8 jocaself
volured variance duration (high)
volume variance duration (high)
volume variancel duration (medium)
[vowel level acoustic features]
[Input prompt]
Question: [EMOTION QUERY PROMPT TEMPLATE]
Transcripts: true, You've got
posttive trings.
Like Have some faith yourself.
The vowel in Yit'!" has medium pitch slope, high pitch with very low variation, low volume with high variation, high duration. The \'aabel" low pitch slope, very pitch with
Acoustic Features:
veriation, very low volume with high variation, high duration. The vowel /b/ in Ygoff very pitch slope, high pitch with moderate variation, moderate volume very variation, very low duration. vowel MMol" medium pitch slope, high pitch with moderato variation, low volume with very high variation, moderate duration.
Finetuning SFT or GRPO
LLM
[response] Angry.
The prediction supported by the prosodic features of several stressed vowels that exhibit hallmark slopes, high very high pitch with strong variation, elevated volume, and extanded durations. For example, la/ in shows very high pitch slope and high pitch with high variation, while in 'stil' and las/ in "cant" maintain very high pitch and loudness, with las/ also negative
anger, namely very high pitch
fluctuation. These petterns indicate heightened arousal vocal tension, which, along semantic content of the utterance, collectively justily the angry label.
showing high
Figure 1: An example of the proposed VowelPrompt framework for the emotion recognition task.
acoustic-pree prossodcc patterns to emotion categories in an interpretable manner. Figure illustrates an example of the proposed VowelPrompt for the emotion recognition task.
3.1 VOWEL-LEVEL ACOUSTIC FEATURE EXTRACTION
Forced Alignment and Vowel Selection. Given an utterance and its orthographic transcript, we employ phoneme-level forced alignment to obtain precise temporal boundaries for each phoneme. Vowel segments are then extracted based on a predefined inventory derived from the International Phonetic Alphabet (IPA), encompassing both monophthongs and diphthongs. This selective filtering excludes consonantal segments, isolating the voiced, resonance-rich nuclei that are most informa- tive for prosodic and affective analysis. By anchoring our vowel selection to IPA standards, ensure cross cross-linguistic consistency and compatibility with multilingual phonetic analysis pipelines for languages including English, German, and French.
Low-Level Descriptor Extraction. For each vowel segment, we compute a compact set low-level descriptors (LLDs) that are both human-interpretable and suitable for integration into large language models, as presented in Table 1. The LLDs used as the acoustic features include (1) average pitch (F)) and pitch slope, which jointly capture the segment's intonation level and rising/falling trends; (2) pitch variation, defined as the within- aithin-segment standard deviation of Fb, indicating the degree of dynamic modulation; (3) average intensity and intensity variation, which reflect loudness and its fluctuation; and (4) duration, representing the temporal extent of the vowel and conveying infor- mation about speech rate and emphasis. Pitch and intensity features are computed using Praat-style signal processing algorithms (Boersma & Weenink, 2001), configured with speaker-adaptive floor and ceiling parameters to account for individual vocal characteristics, while segment durations are derived directly from the phoneme-level foreed alignment boundaries.
To ensure comparability across speakers and vowel categories, we employ a two-stage normal- ization. First, apply speaker-level 2 z-normalization to control for individual voice characteris- tics. Second, vowel-type normalization is applied to mitigate systematic differences among vowel classes. The normalized continuous values then discretized via quantile-based binning into K
4

---

Published as a conference paper at ICLR 2026
Table 1: Vowel-level low-level descriptors (LLDs) used VoweiPrompt for prosodic augmentation.
Category
Feature
Interpretation
Pitch Level (Mean Fo)
Average fundamental frequency of the vowel Rising or falling trend in pitch across the segment Standard deviation of Fo. indicating dynamic range
Pitch
Pitch Slope
Pitch Variation
Intensity Level
Average loudness (RMS energy) the vowel Fluctuation in loudness during vowel segment
Intensity
Intensity Variation
Temporal
Duration
Length of the vowel segment in seconds
You started complaining about not being able to find job the first day you went out didn't find u job. I'm saying like and trying to you know put you down anything. it's just that-i"t Speaker 1: That's really hard to hear!"i Speaker BBt it's true, babe. You've got to more positive about things. Like- Have some faith in yourself."
Transcripts:
Speaker
The vowel // in Vita"" has medium pitch slope, high pitch with very low variation, low volume with high variation, high duration. The vowel /e/ in \"babel" has low pitch slope, very low pitch with moderate variation, very low volume with high variation, high duration. The vowel /ol in ""got" has vay high pitch slope, high pitch with moderate variation, moderate volume with very low variation, very low duration. The vowel /al in ''tol" has medium pitch slope, high pitch with moderate variation, low volume with very high variation, moderate duration.
Acoustic Features:
emotional label <Speaker B:u true, babe. You've got more positive about things. Like- Have some faith yourself)" based both the conversation transcript and the acoustic features. Please output ONLY ONE label from <happy, sad, neutral, angry. excited, frustrated>.. Return the reasoning process in <think> </think> and final emotion sanswer> s/answer* lags
Question: Please
<hhnnk-The repeated urging to "be more positive" and "have some faith" implies an emotional buildup and suggests that Speaker 0 is not simply offering support, but is also reacting to prolonged negativity with a degree of emotional strain. This is mirrored in the acoustic features: vowels such as /o/ in "got" and /v/ in "some" exhibit very high pitch slopes and variation, signaling emotional intensity and urgency; the vowel /n/ in "positive" shows high pitch variation and very long duration, suggesting strong, emotionally changed emphasis. The ovemil vocal pattem, marked by intensity, uneven dynamics. deliberate stress on emotionally salient words, aligns fusrrecsersstreccck///sseeer
Response:
frustrated emotional state, </think
Example of a prompt of VowelPrompt combining conversational context, target utterance, and vowel-level prosodic descriptors. The transcript provides lexical content, while each vowel in the target utterance is annotated with interpretable acoustic features, including pitch slope, pitch level and variation, intensity level and variation, and duration. These features are expressed in natural language and integrated into the input to guide the emotion inference by LLM. The example illustrates a frustration- case from IEMOCAP, where prosodic patterns such high pitch slope and extended vowel duration convey heightened emotional intensity.
Figure 2:
ordinal categories (e.g., "very low." "low," "moderate," "high." "very high"), with K selected to balance interpretability and resolution.
Natural Language Conversion. The discretized features are deterministically mapped into concise textual descriptors for each vowel segment. This process is parameter-free, ensuring transparency and reprodueibility. The resulting descriptors can be appended to transcripts. Compared to sentence- level acoustic summaries, vowel-level descriptors capture fine-grained, localized prosodic variation that often aligns with emotionally salient or emphasized words. This representation provides higher temporal resolution, direct interpretability for human analysts, and flexibility to serve as controllable units in expressive speech generation or style transfer.
3.2 FINE-TUNING LLM FOR EMOTION RECOGNITION WITH VOWEL-LEVEL ACOUSTIC FEATURES
We adopt a two-stage fine-tuning pipeline to adapt a Large Language Model (LLLM) for emotion recognition using the extracted vowel-level acoustic features described in Section 3.1. The first stage, supervised fine-tuning (SFT), serves as a cold-start adaptation, while the second stage, rein- forcement learning with verifiable rewards (RLVR), further refines reasoning accuracy and output structure. Figure 2 illustrates an example of VowelPrompt fine-tuned by SFT and RL for better reasoning over the context and acoustic features for emotion recognition.
Supervised Fine- Tone-Tunigg (SFT). In the SFT stage, we augment each utterance's 's textual transcript with its corresponding vowel-level prosodic descriptors in natural language form, following fixed
5

---

Published as a conference paper at ICLR 2026
prompt template. This augmentation explicitly grounds the LLM in acoustic cues, enabling it to reason over both lexical semantics and prosodic dynamics. To establish a cold-start alignment with the target task,
only small portion of the available training data, paired with gold reason- ing traces automatically generated by a high-capacity text-only LLM such as GPT-4o (Hurst et al., 2024). These reasoning traces serve as reference outputs, allowing the target L.M to learn both the correct label and an interpretable reasoning process. We initialize from a pretrained instruction- tuned LLM and fine-tune with cross-entropy loss maximize the likelihood of generating the ref- erence reasoning and correct emotion label.
Reinforcement Learning with Verifiable Reward (RLVR). Following SFT, we finetune the LLM using Reinforcement Learning with Verifiable Reward (RLVR) (DeepSeck-Al et al., 2025), which jointly optimizes reasoning accuracy and adherence to prescribed output format. Given an in- put prompt containing both the transcript and its aligned prosodic feature descriptions, the policy model Ty produces an output o consisting of two distinct components, including an explicit rea- soning trace enclosed within <think> seiinnkk/ttinkk tags, and final predicted emotion enclosed within <answer>< < answer> tags. Such an explicit separation enables independent, rule-based verification of both the reasoning process and the final prediction.
To perform RLVR, we define a composite reward that integrates an accuracy- based term Race and a format-based term Riormat:
R(o, v) = Roce(o, v) Riormat (o),
where y denotes the ground-truth emotion label. The accuracy reward Race is assigned a value of if the predicted emotion in exactly matches y, and 0 otherwise. The format re- ward Riormat is assigned value of if o contains both a syntactically valid reasoming block (<think>. </think>) and a final answer block (<answer>, wwaeeer..../aaeeeeeee </answer>); otherwise, it is set to 0. Both components are deterministic and require no learned parameters, ensuring the verifiability of the reward signal.
Group Relative Policy Optimization. We optimize the response generation policy using Group Relative Policy Optimization (GRPO), which encourages each candidate response to outperform the group average while maintaining diversity (DeepSeek-AI et al., 2025). To stabilize training and prevent drift from the supervised initialization, we add a KL penalty that constrains updates relative to the SFT reference model. This lightweight formulation enables verifiable reward optimization without requiring complex learned reward models.
3.3 MULTILINGUAL EXTENSION WITH IPA- BASED VOWEL MAPPING
To extend VowelPrompt to multilingual SER, we adopt a language-agnostic framework grounded in the International Phonetic Alphabet (IPA) to unify vowel representations across languages. Such adaptation enables consistent extraction of vowel-level prosodic descriptors regardless of language- specific phoneme inventories or orthographic conventions.
Phoneme Alignment and IPA Normalization. For each language, we employ a phoneme-level forced alignment tool capable of aligning speech phonemic transcriptions in the target language. In our experiments, we use Montreal Forced Aligner (MFA) (McAuliffe et al., 2017), which sup- ports over 20 languages with pretrained acoustic and prapheme-to-phonemee (G2P) models. Aligned phonemes are then mapped into shared set of IPA symbols to ensure phonetic comparability across languages. To control for rinss-lingual variation in prosodic realization, we further perform normal- ization at the language level. For each of the languages considered in this paper, including English, German, and French, we compute global means and standard deviations for each prosodic feature and apply s-score normalization within that language.
Prompt Construction and Adaptation. Once normalized and discretized, the resulting vowel- level descriptors are converted into natural language descriptions in English. The generated acoustic features are appended to the transcript. We use multilingual LLMs, such as GPT-4o (Hurst et al., 2024) and Qwen2-7B Instrucct (Yang al., 2024), that natively support the input language. For SFT, we finetune these models using multilingual emotion datasets, preserving the same prompt structure and training objectives as described in Section 3.2.

---

Published as a conference paper at ICLR 2026
4 EXPERIMENTS
This section presents rigorous empirical evaluation of VowelPrompt across five widely-used speech emotion recognition benchmarks under range of experimental configurations. The dataset charac- teristies are summarized Section 4.1. Section 4.2 examines zero-shot emotion recognition per- formance relative to existing prompting-basee based baselines, while Section 4.3 investigates the effec- tiveness of SFT and GRPO. The generalizability of VoweiPrompt under domain shift is assessed in Section 4.4, and its applicability to multilingual emotion recognition is explored in Section 4.5. In the appendix, Section A.1 presents feature-level ablation study to assess the individual contribu- tions of vowel-level prosodic descriptors, and Section A.2 analyzes the comparative performance of zero-shot and few-shot prompting. Section A.3 provides a direct comparison between VowelPrompt and a projection- based baseline incorporating the audio embeddings for emotion recognition with LLMs. In Section A.4, we perform a study on the number of quantization bins K used for dis- cretizing continuous vowel-level acoustic features. Section A.5 analyzes the influence of utterance duration on zero-shot recognition performance in MELD.. In Section A.6, we conduct a series of ab- lation and counterfactual experiments verify that VowelPrompt'' predictions are driven by aligned vowel-level prosodic descriptors rather than spurious lexical or formatting cues, including transcript shuffle, prosody permutation, matched-margiing placebo, and cross-swap analysis. In Section A.7, we study the impact of label verbalizer tokenization by analyzing the tokenization behavior of emo- tion labels under different tokenizers and by replacing the original labels with synthetic single-token verbalizers under randomized mappings. In Section A.8, we compare vowel-level prosodic descrip- tors with consonant-level prosodic descriptors and further evaluate variant that incorporates both types of cues. In Section A.9, compare VowelPrompt with non-LLM acoustic bascline methods based on self-supervised speech representations, including HuBERT-large and wav2vec-large under both in-domain and cross cross-domain evaluation settings. In Section A0 perform human evalu- ation of the reasoning traces generated by VowelPrompt, where annotators assess prosodic ground- ing, causal coherence, and internal consistency. In Section A.11, analyze the balance between in-domain performance and cross-domain generalization by varying the KL regularization weight in GRPO. In Section 1. we compare VowelPrompt with classifiers trained directly on vowel-level prosodic features, including MLP, tree-based models, and transformer operating on phoneme-level tokens. In Section A.13, we study the impact of incorrect vowel aligmment by perturbing a con- trolled fraction of vowel boundary timestamps in the forced alignment results. In Section A.14, we analyze the effect of speech rate by grouping utterances according to phones-per-secome statisties and evaluating performance across different speech-rate regimes. In Section A.15, we perform an ablation study on the incorporation of explicit reasoning in both SFT and SFT combined with GRPO by enabling and disabling reasoning during training.
4.1 DATASETS
We evaluate method on five widely used speech emotion recognition (SER) benchmarks that span acted, semi-acted, and naturalistic speech across multiple languages. The IEMOCAP cor- pus (Busso et al., 2008) contains dyadic interactions between ten actors (five male, five female), with utterances annotated for emotions including angry. happy, sad, neutral, and excited. The MELD dataset (Poria et al., 2019) is derived from the TV series Friends, consisting of multi- party conversations annotated with seven emotion categories in multimodal setting. To assess further include three public benchmarks, including CaFE (Gournay et al.. 2018) in French, EmoDB (Burkhardt et al., 2005) in German, and the multilingual ASVP- ESD (Tientcheu Touko al., 2021), which covers 12 emotions across diverse speakers and recording conditions. The statistics of all the datasets used are summarized in Table 2.
cross-lingual generalization,
Table 2: Summary of emotion recognition datasets used in our experiments.
Dataset
Source
Language
#limotions
#Speakers
#Utterances
#Hours
IEMOCAP (Busso et al., 2008)
Act
English
10
5531
7.0
MELD (Poria et al., 2019)
English
304
13706
12.1
CaFE (Gournay et al., 2018)
Act
French
12
936
1.2
EmoDB (Burkhardt et al., 2005)
Act
German
10
535
0.5
ASVP-ESD (Tientcheu Touko et al., 2021)
Media
Mix
12
131
13964
18.0
7

---

Published as a conference paper at ICLR 2026
4.2 ZERO-SHI SHOT EMOTION RECOGNITION
We evaluate the proposed VowelPrompt approach in zero-shot setting the IEMOCAP and MELD datasets, comparing it against two baselines, including a vanilla zero-shot prompt using only transcripts, denoted as Zero-Shot Baseline, and SpeechCuel..MM (Wu al., 2025), which augments transcripts with sentence-level prosodic descriptions. For each method, we evaluate two input con- figurations: (i) Transcript, which utilizes solely the target utterance, and (ii) Transcript & Context, which additionally incorporates preceding conversational turns to provide discourse-level informa- tion. Performance assessed using Unweighted Accuracy (UACC) and Weighted F1 (WF1), which respectively quantify mass-balanced recognition capability and overall classification effectiveness.
Table 3: Zero-shot performance on IEMOCAP and MELD, Results are reported as Unweighted
Accuracy / Weighted Fl (%). "Context" indicates inclusion of preceding conversational turns.
JEMOCAP
MELD
Method
Input
L.MM
UACC
WFL
UACC
WFI
Zero-Shot Baseline
Trunscript
43.38
41.03
61.15
60.92
SpeechCueLLM (Wu et al., 2025)
Transcript
GPT-4o
49.97
48.54
52.44
53.59
VowelPrompt (Ours)
Transcript
51.18
50.15
63.61
61.76
Zero-Shot Baseline
Transcript Context
55.51
53.63
62.76
63.57
SpeechCueLLMM (Wu al., 2025)
Transcript & Context
GPT-4o
60.07
58.52
56.74
57.90
VowelPrompt (Ours)
Transeript & Context
62.26
60.74
64.34
64.17
Zero-Shot Baseline
Transcript
40.60
40.44
47.55
48.74
SpeechCueLLM (Wu al., 2025)
Transcript
LLaMA-3-8B-Instee
44.18
43.88
44.41
44.62
VowelPrompt (Ours)
Transcript
46.57
44.96
49.21
49.99
Zero-Shot Baseline
Transcript & Context
50.40
49.47
42.30
42.09
SpeechCueLLM (Wu et al., 2025)
Transcript & Context
LLaMA-3-8B-Inste
52.63
53.85
43,49
42.59
VowelPrompt (Ours)
Transcript & Context
53.82
54.10
46.45
46.26
As shown in Table 3, VowelPrompt consistently outperforms both baselines across models and datasets. On GPT-4o, VowelPrompt improves over the Zero-Shot Baseline by up to 7.80% UACC and 7.11% WF1 on IEMOCAP, and by up to 2.19% UACC and 3.25% WFI on MELD. Compared to LpLechCueLLLM, our method achieves gains in all settings, indicating that fine-grained vowel-level prosodic cues are more effective than coarse sentence-level descriptions for emotion recognition in large language models. The trend holds for LLLMA---8BBInnttee Instruct, despite its weaker overall LLM, VowelPrompt yields
performance compared to GPT-4o. Even in this resource-constraing consistent improvements over both baselines, with gains of up to 3. 3.64% UACC and 3. 3.63% WF1. These results demonstrate that VowelPrompt is portable, model-agnostic prompting strategy that can enhance zero-shot emotion recognition without saec-fpe fine-tuning.
4.3 LLM FINE-TUNING FOR EMOTION RECOGNITION
We further evaluate VowelPrompt in a supervised adaptation setting to examine whether vowel-level prosodic augmentation yields benefits beyond zero-shot prompting. Experiments are conducted on IEMOCAP and MELD with two instruction-tuner L.M backbones, which are LLaMA-3-8B. Instruct (Dubey et al., 2024) and 4LaMAA--Scuut1117 1-7----6--Innrrue (Meta AI, 2025). For the SFT setting, the reasoning is not incorporated into the training and inference processes for VowelPrompt and the baseline methdos. For the SFT & GRPO setting, both models are adapted using LoRA- based parameter-efficient fine-tuning on 20% of the training data, followed by GRPO as described in Section 3.2. We use the official vrain/validation/tes splits for each dataset, and all methods are trained and evaluated on identical utterance-labe label pairs to ensure fair comparison. Similar to the set- tings for the zero-shot experiments, we conduct comparisons across multiple input configurations. The Baseline leverages only the transcript and preceding conversational turns without incorporat- ing any prosodic information. InstructERC (Lei et al., 2023) applies instruction tuning to enhance context-sensitive emotion recognition. SALMONN (Tang et al., 2024) integrates speech and lan- guage modalities through multimodal aligmment. SpeechCueLLM (Wu et al., 2025) augments the transcript with sentence-level prosodic summaries. Finally, VowelPrompt enriches the input with fine-grained, interpretable prosodic descriptors for each vowel segment, as described in Section 3.1. Each method is evaluated under both SFT and SFT & GRPO regimes, enabling systematic
8

---

Published as a conference paper at ICLR 2026
sessment of the benefits of prosodic granularity, multimodal integration, and reinforcement-baat based refinement.
Table 4: Weighted F1 (%) IEMOCAP and MELD under SFT and SFT & GRPO settings with different LLMS.
LLaMA-3-8B-In-a -n--rstect
LLaMA-4
7B-16EEInstruct
Method
SFT
SFT GRPO
SFT
SFT & GRPO
IEMOCAP
MELD
IEMOCAP
MELD
IEMOCAP
MELD
IEMOCAP
MELD
Baseline
70.32
67.44
70.82
67.90
InstruciERC (Lei et al., 2023)
71.65
67.25
71.32
66.96
71.75
68.15
71.52
67.35
SALMONN (Tang 2024)
67.25
66.85
67.96
67.10
SpeechCuel LM (Wa et al,, 2025)
71.74
67.07
71.55
67.10
72.02
68.02
72.18
67.96
VowelPrompt (Ours)
73.46
69.61
73.02
68.98
73.85
70.12
74.02
69.79
As shown in Table VowelPrompt consistently outperforms all competing baselines across both datasets and model scales. Under SFT. vowel-level augmentation yields absolute Weighted FI improvements of up 3.14% on IEMOCAP and 2. 2.17% on MELD with LLaMA-3-8B--T Instrucct, with comparable gains observed for the larger LLaMA---Scour model. The advantage remains af- ter RLVR refinement, where VowelPrompt outperforms sentence-level prosodic descriptions by as much as 1.47% on IEMOCAP and 1.88% on MELD. These results demonstrate that fine- sre-ggaaidd, interpretable vowel-centric features encode richer emotional cues than coarse prosodic summaries, and that RLVR refinement can further capitalize these cues to improve classification performance.
4.4
EMOTION RECOGNITION
We further assess the robustness of VowelPrompt under domain shift through cross-domain evalu- ations, where models are trained on one dataset and directly tested on another without additional adaptation. Specifically, and from MELD to IEMOCAP. The study evaluates whether VowelPrompt can capture emotional cues that generalize across variations in speaker identity, conversational style, and recording condi- tions. Following the protocol in Section 4.3, we compare VowelPrompt against SpeechCueLLM (Wu al., 2025), which augments transcripts with sentence-level prosodic descriptions. Both methods are tested under three regimes: zero-shot prompting, supervised fine-tuning (SFT), and SFT fol- lowed by GRPO (SFT GRPO). All experiments employ the -LLMA-3-8B-IIntte backbone, with training performed on the full source-domain dataset.
examine transfer scenarios, which are from IEMOCAP MELD,
Table 5:
Cross-domain results for IEMOCAP + MELD and MELD - IEMOCAP. Models are trained on the source dataset and evaluated on the target dataset without adaptation.
IEMOCAP MELD
MELD JEMOCAP
Method
Shro-Shot
SFT
SFT GRPO
Zero-Shot
SFT
SFT & GRPO
SALMONN (Tang et al., 2024)
40.25
51.48
23.65
40.85
InstructERC (Lei al., 2023)
51.42
43.15
50.18
42.68
25.49
43.36
SpeechCueLLM (Wu et al., 2025)
53.85
42.36
55.16
42.59
25.10
44,79
VowelPrompt (Ours)
54.10
46.26
60.28
46.26
28.71
51.75
As shown Table 5, VowelPrompt consistently outperforms basclines across transfer settings. Gains are modest in the zero-shot condition but increase substantially with supervised adaptation. Under SFT & GRPO, VowelPrommpt improves by 5.12% in the IEMOCAP > MELD transfer and by 6.96% in the MELD IEMOCAP transfer compared to SpeechCueLLM, These findings indi- cate that fine-grained vowel-level acoustic features provide more domain- inmarinnnnvaiiatt emotional cues than coarse sentence-level summaries, and that RL-based refinement further enhances cross-domain generalization.
4.5 EXTRACTING VOWEL-LEVEL ACOUSTIC FEATURES FROM MULTILINGUAL SPEECH
To evaluate cross-lingual generalization, we extend VowelPrompt to three additional benchmarks: the French CaFE corpus (Gournay al., 2018), the German EmoDB corpus (Burkhardt et al.,

---

Published as a conference paper at ICLR 2026
2005), and the mixed- -ingual ASVP-ESD corpus (Tientcheu Touko al., 2021). Phoneme-level forced alignment is performed using the Montreal Forced Aligner (MFA), after which vowel seg- ments mapped into shared IPA-based inventory. Prosodic features, including pitch, intensity, and duration, are normalized at both the speaker and language level before being converted into natural-language descriptors. Moreover, with GPT-4o, comparing against transcript-only baselines, InstructERC (Lei et al., 2023), and SpeechCueLLM (Wu al., 2025). For ASVP-ESD, which inherently multilingual, we perform supervised adaptation using Bren2-7B-Instruue chosen for its stronger multilingual capabilities. The evaluation compares VowelPrommtt against InstructERC, SALMONN (Tang et al., 2024), and LLL under both SFT and SFT & GRPO training regimes.
conduct zero-shot evaluations on CaFE and EmoDB
Table 6: Zero-shot results on CaFE (French) and
Table 7: Fine-tuning results on ASVP-ESD
EmoDB (German) using GPT-4o. Performance is reported as Weighted F1 (%).
(Mixlingual) using Qwen2-7B-Instruct. Per- formance is reported as Weighted FI (%).
Method
CaFE (Fr)
EmoDB (De)
Method
SFT
SFT & GRPO
Transcript Only
45.10
64.86
InstrucEERC (Lei et al., 2023)
67.25
67.96
InstructERC (Lei al., 2023)
48.35
66.74
SALMONN (Tang al., 2024)
67.10
67.85
SpeechCuel.LMM (Wu et al., 2025)
49.16
67.32
SpeechCueLLM (Wu et al., 2025)
67.85
68.12
NowelPrompt (Ours)
51.42
69.85
NowelIPropp (Ours)
70.54
71.36
As shown in Tables 6 and 7, VowelPrompt achieves consistent improvements over all baselines across languages and evaluation settings. In the zero-shot scenario, it delivers the best Fl scores on both CaFE and EmoDB, outperforming transcript-only prompts, InstructERC, and SpeechCuel.LMM thereby demonstrating effective transferability without language-specine supervision. On the mixed-lingual ASVP-ESD corpus, supervised adaptation with SFT & GRPO further improves per- formance, where VowelPrompt outperforms InstructERC, SALMONN, and SpeechCueLLLM un- derscoring the effectiveness of vowel-level prosodic augmentation multilingual contexts.
CONCLUSION
In this work, we introduced VowelPrompt, a unified and interpretable framework that augments large language models with fine-grained, vowel-level prosodic cues for speech emotion recognition. Grounded in phonetic theory, VowelPrompt extracts prosodic descriptors of pitch, intensity, and du- ration from time-aligned vowel segments, discretizes them through quantile- based binning, and con- verts them into natural language descriptions appended to transcripts. This design enables language models reason reasonjointly over lexical and prosodic information without requiring direct access raw audio at inference. To enhance task adaptation, we developed a two-stage training pipeline com- bining supervised fine-tuning with Reinforcement Learning using Verifiable Reward (RLVR) via Group Relative Policy Optimization (GRPO), which improves predictive accuracy, structural sistency, and robustness. Comprehensive experiments across zero-shot, fine-tuned, cross-domain, and multilingual settings demonstrate that VowelPrompt consistently outperforms transcript-only and sentence-level prosody baselines. Beyond improved performance, the framework offers inter- pretable intermediate representations that explicitly connect acoustic-prosodic patterns emotional categories, providing both practical effectiveness and scientific transparency for prosody-aware emo- tion recognition with language models.
ACKNOWLEDGMENTS
Y. Wang and Y. Yang supported by the 2023 Mayo Clinic and Arizona State University Alliance for Health Care Collaborative Research Seed Grant Program under Award No. AWD00038846 and by the NIH under Award No. 1OT20D037955-01.
REFERENCES
Dmitry Bitouk, Ragini Verma, and Ani Nenkova. Class-level spectral features for emotion recogni- tion. Speech Communication, 52(7-8):613-62) 2010. doi: (0016/.specom.2000.2222
10

---

Published as a conference paper at ICLR 2026
Paul Boersma and David Weenink. Praat, system for doing phonetics by computer. Glot Interna- tional, 5(9/10) 345, 2001.
Margaret M. Bradley and Peter J. Lang. Measuring emotion: The self-assessment manikin and the semantic differential. Journal of Behavior Therapy and Experimental Psychiatry, 25(1):49-59,
1994. doi: 10. (11666000077999999990000
Felix Burkhardt, Astrid Paeschke, Miriam Rolfes, Walter F Sendlmeier, Benjamin Weiss, et al. A database of german emotional speech. In Interspeech, volume pp. 1517-1520, 2005.
Carlos Busso, Murtaza Bulut, Chi-Chun Lee, Abe Kazemzadeh, Emily Mower. Samuel Kim, Jean- nette N Chang, Sungbok Lee, and Shrikanth S Narayanan. lemocap: Interactive emotional dyadic motion capture database. In Language resources and evaluation, volume 42, pp. 335-359, 2008,
Huan Cao, David G Cooper, Michael K Keutmann, Ruben C Gur, Ani Nenkova, and Ragini Verma. Crema-d: Crowd-sourced emotional multimodal actors dataset. In IEEE Transactions on Affective Computing, volume 5, pp. 377-390, 2014.
Sanyuan Chen, Chengyi Wang. Zhengyang Chen, Yu Wu, Shujie Liu, Zhuo Chen, Jinyu Li, Naoyuki Kanda, Takuya Yoshioka, Xiong Xiao, Jian Wu, Long Zhou, Shuo Ren, Yanmin Qian, Yao Qian, Jian Wu, Michael Zeng. Xiangzhan Yu, and Furu Wei. WavLM: Large-scale self-supervised pre- training for full stack speech processing, 2022.
Zebang Cheng. Zhi-Qi Cheng, Yan He, Kai Wang, Yuxiang Lin, Zheng Lian, Xiao- jiang Peng, and Alexander G. Hauptmann. Emotion-llama: Multimodal emotion recogni- tion and reasoning with instruction tuning. In Amir Globersons, Lester Mackey, Danielle Belgrave, Angela Fan, Ulrich Paquet, Jakub M. Tomezak, and Cheng Zhang (eds.), Ad- vances in Neural Information Processing Systems 38: Annual Conference on Neural Infor- mation Processing Systems 2024, NeurIPS 2024, Vancouver, BC, Canada, December 10 15, 2024, 2024. URL http:/ hwwprs/papppppppaaaaaeeeeea nips wwtpppppppappaeeeeeeeee wwwp:/tttteeeeeeeeeeeaeaaaaaeeee httpp/pppp////// 2004 2224//222//2//2///// /hash/ (aA.aaa.777c233356666666622222
html.
Yunfei Chu, Jin Xu, Qian Yang, Haojie Wei, Xipin Wei, Zhifang Guo, Yichong Leng, Yuanjun Lv, Jinzheng He, Junyang Lin, Chang Zhou, and Jingren Zhou. Qwen2-Audio technical report, 2024.
David Crystal. Prosodic Systems and Intonation in English. Cambridge University Press, Cam- bridge, UK, 1969.
Dee,Seek-AI, Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang. Xiao Bi, Xiaokang Zhang, Xingkai Yu, Yu Wu, Z. F. Wu, Zhibin Gou, Zhihong Shao, Zhuoshu Li, Ziyi Gao, Aixin Liu, Bing Xue, Bingxuan Wang, Bochao Wu, Bei Feng, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, Damai Dai, Deli Chen, Dongjie Ji, Erhang Li, Fangyun Lin, Fucong Dai, Fuli Luo, Guangbo Hao, Guanting Chen, Guowei Li, H. Zhang. Han Bao, Hanwei Xu, Haocheng Wang, Honghui Ding, Huajian Xin, Huazuo Gao. Hui Qu, Hui Li, Jianzhong Guo. Jiashi Li, Jiawei Wang, Jingchang Chen, Jingyang Yuan, Junjie Qiu, Junlong Li, J. L. Cai, Jiaqi Ni, Jian Liang, Jin Chen, Kai Dong, Kai Hu, Kaige Gao, Kang Guan, Kexin Huang, Kuai Yu, Lean Wang. Lecong Zhang, Liang Zhao, Litong Wang, Liyue Zhang, Lei Xu, Leyi Xia, Mingchuan Zhang, Minghua Zhang, Minghui Tang, Meng Li, Miaojun Wang, Mingming Li, Ning Tian, Panpan Huang. Peng Zhang, Qiancheng Wang. Qinyu Chen, Qiushi Du, Ruiqi Ge, Ruisong Zhang, Ruizhe Pan, Runji Wang, J. Chen, R. Jin, Ruyi Chen, Shanghao Lu, Shangyan Zhou, Shanhuang Chen, Shengfeng Ye, Shiyu Wang, Shuiping Yu, Shunfeng Zhou, Shuting Pan, and S. S. Li. Deepseek-rl: Incentivizing reasoning capability in I1ms via reinforcement learning. CoRR, abs/2501. 2025, doi: 10. 48550/ARXIV.201..11
URL https: /doii.ioo//// org/10...//. 48550/ arxiv. 2501. 12948.
Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, Anirudh Goyal, Anthony Hartshom, Aobo Yang, Archi Mitra, Archie Sravankumar, Artem Korenev, Arthur Hinsvark, Arun Rao, Aston Zhang, Aurélien Rodriguez, Austen Gregerson, Ava Spataru, Baptiste Rozière, Bethany Biron, Binh Tang, Bobbie Chern, Charlotte Caucheteux. Chaya Nayak, Chloe Bi, Chris Marra, Chris McConnell, Christian Keller, Christophe Touret, Chunyang Wu, Corinne Wong, Cristian Canton Ferrer, Cyrus Nikolaidis, Damien Allonsius, Daniel Song, Danielle Pintz, Danny

---

Published as a conference paper at ICLR 2026
Livshits, David Esiobu, Dhruv Choudhary, Dhruv Mahajan, Diego Garcia- Diego Perino, Dieuwke Hupkes, Egor Lakomkin, Ehab AlBadawy, Elina Lobanova, Emily Dinan, Eric Michael Smith, Filip Radenovic, Frank Zhang, Gabriel Synnaeve, Gabrielle Lee, Georgia Lewis Ander- son, Graeme Nail, Grégoire Mialon, Guan Pang, Guillem Cucurell, Hailey Nguyen, Hannah Ko- revaar, Hu Xu, Hugo Touvron, Iliyan Zarov, Imanol Arrieta Ibarra, Isabel M. Kloumann, Ishan Misra, Ivan Evtimov, Jade Copet, Jaewon Lee, Jan Geffert, Jana Vranes, Jason Park, Jay Ma- hadeokar, Jeet Shah, Jelmer van der Linde, Jennifer Billock, Jenny Hong, Jenya Lee, Jeremy Fu, Jianfeng Chi, Jianyu Huang, Jiawen Liu, Jie Wang, Jiecao Yu, Joanna Bitton, Joe Spisak, Jongsoo Park, Joseph Rocca, Joshua Johnstun, Joshua Saxe, Junteng Jia, Kalyan Vasuden Al- wala, Kartikeya Upasani, Kate Plawiak, Ke Li, Kenneth Heafield. Kevin Stone, and et al. The Ilama herd of models. CoRR, 0tss/2407.2178,, 2024, doi: 10445500AAX..22402227555 URL https: /doi. wrgg /0///11..00 48550/ a xx x 2407. .27883.
Florian Eyben, Martin Wöllmer, and Bjöm Schuller. Opensmile: The munich versatile and fast open-source audio feature extractor. In Proc. ACM Multimedia, pp. 1459-1462, 2010.
Florian Eyben, Klaus R Scherer, Bjöm W Schuller, Johan Sundberg, Elisabeth Andre, Carlos Busso, Laurence Y Devillers, Julien Epps, Petri Laukka, Shrikanth S Narayanan, and Khiet P Truong. The geneva minimalistic acoustic parameter set (gemaps) for voice research and affective com- puting. IEEE Transactions on Affective Computing, 7(2):190-202, 2015.
Philippe Gournay, Olivier Lahaie, and Roch Lefebvre. A canadian french emotional speech dataset. In Proceedings of the 9th ACM multimedia systems conference, pp. 399-402, 2018
Wei-Ning Hsu, Benjamin Bolte, Yao-Hung Hubert Tsai, Kushal Lakhotia, Ruslan Salakhutdi- nov, and Abdelrahman Mohamed. HuBERT: Self-supervised speech representation learning by masked prediction of hidden units, 2021.
Aaron Hurst, Adam Lerer, Adam Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, AJ Os- trow, Akila Welihinda, Alan Hayes, Alec Radford, Aleksander Madry, Alex Baker-Whitcome Alex Beutel, Alex Borzunov, Alex Carney, Alex Chow, Alex Kirillov, Alex Nichol, Alex Paino, Alex Renzin, Alex Tachard Passos, Alexander Kirillov, Alexi Christakis, Alexis Conneau, Ali Ka- mali, Allan Jabri, Allison Moyer, Allison Tam, Amadou Crookes, Amin Tootoonchian, Ananya Kumar, Andrea Vallone, Andrej Karpathy, Andrew Braunstein, Andrew Cann, Andrew Codis- poti, Andrew Galu, Andrew Kondrich, Andrew Tulloch, Andrey Mishchenko, Angela Baek, An- gela Jiang, Antoine Pelisse, Antonia Woodford, Anuj Gosalia, Arka Dhar, Ashley Pantuliano, Avi Nayak, Avital Oliver, Barret Zoph. Behrooz Ghorbani, Ben Leimberger, Ben Rossen, Ben Sokolowsky, Ben Wang, Benjamin Zweig, Beth Hoover, Blake Samic, Bob McGrew, Bobby Spero, Bogo Giertler, Bowen Cheng, Brad Lightcap. Brandon Walkin, Brendan Quinn, Brian Guarraci, Brian Hsu, Bright Kellogg. Brydon Eastman, Camillo Lugaresi, Carroll L. Wainwright, Cary Bassin, Cary Hudson, Casey Chu, Chad Nelson, Chak Li, Chan Jun Shern, Channing Con- ger, Charlotte Barette, Chelsea Voss, Chen Ding, Cheng Lu, Chong Zhang, Chris Beaumont, Chris Hallacy, Chris Koch, Christian Gibson, Christina Kim, Christine Choi, Christine McLeavey, Christopher Hesse, Claudia Fischer, Clemens Winter, Coley Czamecki, Colin Jarvis, Colin Wei, Constantin Koumouzelis, and Dane Sherburn. Gpt-4o system card. CoRR, abs/2410.21276 2024.
Youngjae Kim and Emily M. Provost. Emoberta: Speaker- emotion recognition in conversa- tion with roberta. In Proc. Interspeech, pp. 897-901, 2021.
Shanglin Lei, Guanting Dong, Xiaoping Wang, Keheng Wang, and Sirui Wang. Instructerc: Re- forming emotion recognition in conversation with a retrieval multi-task llms framework. CORR, abs/2309.11911, 2023.
Steven R Livingstone and Frank A Russo. The ryerson audio-visual database of emotional speech and song (ravdess): A dynamic. multimodal set of facial and vocal expressions in north american english. PloS one, 13(5)ee016331,, 2018.
Michael McAuliffe, Michaela Socolof, Sarah Mihuc, Michael Wagner, and Morgan Sonderegger. Montreal forced aligner: Trainable text-speech alignment using kaldi. In Proc. Interspeech, 2017.
Meta AI. The laaaa 4 herd: The beginning of new era of natively multimodal ai in- novation. https:/ //ai meta. con/blog//////ooccoooooooo.oooco hwwp//ooolllaaaa amama- - 4 - -mullte t imodal -in--nneelll intelligence, April 2025. Blog post.
12

---

Published as a conference paper at ICLR 2026
Sylvie Mozziconacci. Prosody and emotions. In Proceedings of Speech Prosody 2002, pp. 1-9, Aix-en-Provenee France, 2002. International Speech Communication Association (ISCA).
Leonardo Pepino, Paula Riera, and Lucia Ferrer. Emotion recognition from speech using wav2vec 2.0 embeddings. In Proc. Interspeech, pp. 3400-3404, 2021.
Soujanya Poria, Devamanyu Hazarika, Navonil Majumder, Gautam Naik, Erik Cambria, and Rada Mihalcea. Meld: A multimodal multi-part dataset for emotion recognition in conversations. In Proc. ACL. pp. 527-536, 2019.
Fabien Ringeval and Mohamed Chetouani. A vowel based approach for acted emotion recognition. In Proc. Interspeech, pp. 276-279, 2008.
Paul K. Rubenstein, Chulayuth A sawarrengchaii, Duc Dung Nguyen, Ankur Bapna, Zalán Borsos, Félix de Chaumont Quitry, Peter Chen, Dalia Badawy, Wei Han, Eugene Kharitonov, Hannah Muckenhirn, Dirk Padfield, James Qin, Danny Rozenberg, Tara N. Sainath, Johan Schalkwyk, Matthew Sharifi, Michelle Tadmor Ramanovich, Marco Tagliasacchi, Alexandru Tudor, Miha- jlo Velimirovic, Damien Vincent, Jiahui Yu, Yongqiang Wang, Vicky Zayats, Neil Zeghidour, Yu Zhang, Zhishuai Zhang, Lukas Zilka, and Christian Havnø Frank. Audiopalm: A large lan- guage model that can speak and listen. CoRR, abs/2306. 12925, 2023.
James A. Russell. A circumplex model of affect. Journal of Personality and Social Psychology, 39 (6):1161-1178, 1980. doi: 10. 10370//00077714.
Björ Schuller, Stefan Steidl, Anton Batliner, Felix Burkhardt, Laurence Devillers, Christian Müller, and Shrikanth Narayanan. The interspeech 2009 emotion challenge. In Pruc. Interspeech, pp- 312-315, 2009.
Saurabh Shah and Carlos Busso. Articulation constrained learning with application to speech emo- tion recognition. In EURASIP Journal on Audio, Speech, and Music Processing, volume 2019,
pp. 1-15, 2019.
Changli Tang, Wenyi Yu, Guangzhi Sun, Xianzhao Chen, Tian Tan, Wei Li, Lu Lu, Zejun MA, and Chao Zhang. SALMONN: Towards generic hearing abilities for large language models. In The Twelfth International Conference Learning Representations, 2024. URL https: / wwwp:reeiiww..et/ooronoo.
Landry Dejoli Tientcheu Touko, Qianhua He, and Wei Xie. Audio, speech and vision processing lab emotional sound database (asvp-esd), Dataset, May 2021. Audio, Speech and Vision Processing
Lab.
Yancheng Wang, Ziyan Jiang, Zheng Chen, Fan Yang, Yingxue Zhou, Eunah Cho, Xing Fan, Yanbin Lu, Xiaojiang Huang, and Yingzhen Yang. Recmind: Large language model powered agent for recommendation. In Kevin Duh, Helena Gómez-Adorno, and Steven Bethard (eds.), Findings of the Association for Computational Linguistics: NAACL 2024, Mexico City, Mexico, June 16-21, 2024, volume NAACL 2024 of Findings of ACL, pp. 4351 4364. Association for Computational Linguistics, 2024.
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed H. Chi, Quoc V. Le, and Denny Zhou. Chain-of-thought prompting elicits reasoning in large language models. In Sanmi Koyejo, S. Mohamed, A. Agarwal, Danielle Belgrave, K. Cho, and A. Oh (eds.), Advances in Neural Information Processing Systems 35: Annual Conference on Neural Information Processing Systems 2022, NeurIPS 2022, New Orleans, LA, USA, November 28 December 9, 2022, 2022.
Zehui Wu, Ziwei Gong, Lin Ai, Pengyuan Shi, Kaan Donbekci, and Julia Hirschberg. Beyond silent letters: Amplifying Ilms in emotion recognition with vocal nuances. In Luis Chiruzzo, Alan Ritter, and Lu Wang (eds.), Findings of the Association for Computational Linguistics: NAACL 2025, Albuquerque, New Mexico, USA, April 29 May 4, 2025, pp. 2202-2218 Association for Computational Linguistics, 2025. doi: in find
URL 117.
https:: //oof .org/10 10. 10w.0.11633/111//
13

---

Published as a conference paper at ICLR 2026
An Yang, Baosong Yang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Zhou, Chengpeng Li, Chengyuan Li, Dayiheng Liu, Fei Huang. Guanting Dong, Haoran Wei, Huan Lin, Jialong Tang, Jialin Wang, Jian Yang. Jianhong Tu, Jianwei Zhang, Jianxin Ma, Jianxin Yang, Jin Xu, Jingren Zhou, Jinze Bai, Jinzheng He, Junyang Lin, Kai Dang, Keming Lu, Keqin Chen, Kexin Yang, Mei Li, Mingfeng Xue, Na Ni, Pei Zhang, Peng Wang, Ru Peng, Rui Men, Ruize Gao, Runji Lin, Shijie Wang. Shuai Bai, Sinan Tan, Tianhang Zhu, Tianhao Li, Tianyu Liu, Wenbin Ge, Xiaodong Deng, Xiaohuan Zhou, Xingzhang Ren, Xinyu Zhang, Xipin Wei, Xuancheng Ren, Xuejing Liu, Yang Fan, Yang Yao, Yichang Zhang, Yu Wan, Yunfei Chu, Yugiong Liu, Zeyu Cui, Zhenru Zhang, Zhifang Guo, and Zhihao Fan. Qwen2 technical report. CoRR, abs/2407.10671. 2024. doi: 1044550/ARXIV.2407..00 10671. URL https: /tt////oiiii http:/11..855//raa/./..... 2407. 10671.
Shu-wen Yang, Po-Han Chi, Yung-Sung Chuang, Cheng-I Jeff Lai, Kushal Lakhotia, Yist Y. Lin, Andy T. Liu, Jiatong Shi, Xuankai Chang, Guan-Ting Lin, Tzu-Hsien Huang, Wei-Cheng Tseng, Shang-Wen
Ko-tik Lee, Da-Rong Liu, Zili Huang, Shuyan Dong, Shang- Wen Li, Shinji Watanabe, Abdelrah- man Mohamed, and Hung-yi Lee. SUPERB: Speech processing universal PERformance bench- mark. Proc. Interspeech, pp. 3161-3165, 2021.
Chengshuai Zhao, Zhen Tan, Pingchuan Ma, Dawei Li, Bohan Jiang, Yancheng Wang, Yingzhen Yang, and Huan Liu. Is chain-of-thought reasoning of llms mirage? data distribution lens. arXiv preprint arXiv:2508.011990 2025.
A ADDITIONAL EXPERIMENT RESULTS
A.1 ABLATION STUDY ON INDIVIDUAL ACOUSTIC FEATURES
To disentangle the contributions of each vowel-level descriptor, perform a fine- fine-grained ablation study by selectively removing one feature at a time from the six categories listed Table 1. In particular, evaluate the impact of excluding pitch level, pitch slope, pitch variation, intensity level, intensity variation, and duration while keeping all other descriptors intact. Each ablation model is trained under the same supervised fine-tuning (SFT) protocol with LLaMA-3-8B-Inttee on IEMOCAP and MELD to ensure comparability. This design allows us to assess the relative importance of each feature type for emotion recognition. As shown in Table 8, the removal of any single descriptor results in modest but consistent decreases in performance relative to the full model. All ablation settings preserve competitive results, with scores above 72.5% on IEMOCAP and 69.05% on MELD, confirming that VowelPrompt does not rely disproportionately on a single cue. Among the six descriptors, pitch-related features (level, slope, variation) exhibit the most noticeable impact, reflecting their well-established role primary carriers of prosodic information. Intensity and duration features also contribute measurable improvements, as their exclusion reduces recognition accuracy despite more subtle effects. Taken together, these findings demonstrate that each vowel-level descriptor contributes complementary information to the framework, and that the integration of all six is necessary to achieve optimal performance.
Table 8: Ablation of individual vowel-level features under SFT with LLaMA-3-8B-Instt
Model Variant
IEMOCAP
MELD
Full VowelPrompt (all features)
73.46
69.61
w/o Pitch Level
72.91
69.18
w/o Pitch Slope
73.02
69.27
w/o Pitch Variation
72.87
69.12
w/o Intensity Level
73.15
69.25
w/o Intensity Variation
72.94
69.09
w/o Duration
73.25
69.22
A.2 FEW-SHOT EMOTION RECOGNITION
We assess the performance of VowelPrompt in both zero-shot and few-shot scenarios on the IEMO- CAP and MELD datasets, focusing the Transcript & Context configuration. In the few-shot
14

---

Published as a conference paper at ICLR 2026
setting, each prompt is augmented with three labeled in-context exemplars drawn from the train- ing data, enabling the models to leverage limited supervision in addition to their inherent zero-shot reasoning capability. All results reported in terms of Weighted Fl (WF1), which provides balanced measure of classification performance under label imbalance.
The results in Table 9 show that all methods obtain consistent improvements in the few-shot regime, with WFI gains ranging from approximately 0.8% to 1.2% relative to zero-shot performance. Across both model backbones, VowelPrompt achieves the best results, outperforming the baseline and SpeechCueLLMN in both evaluation settings. These findings indicate that vowel-level prosodic descriptors not only strengthen zero-shot emotion recognition but also enhance few-shot general- ization, demonstrating their effectiveness as interpretable and transferable cues for prosody-awer aware large language models.
Table 9: Zero-shot vs. few-shot performance IEMOCAP and MELD with Transcript & Context inputs. Method
IEMOCAP
MELD
Zero-Shot
Few-Shot
A
Zero-Shot
Few-Shot
A
GPT-4o
Baseline
53.63
54.42
+0.79
63.57
64.51
+0,94
SPM (Wu et al., 2025)
58.52
59.41
+0.89
57.90
58.95
+1.05
VowelPrompt (Ours)
60.74
61.72
+0.98
64.17
65.20
+1.03
L.LaMA---8BBIIsrruc
Baseline
49.47
50.26
+0.79
42.09
43.05
+0.96
SpeechCueLLLM (Wa et al., 2025)
53.85
54.71
+0.86
42.59
43.66
+1 +1.07
VowelPrompt (Ours)
54.10
55.12
+1.02
46.26
47.42
+1.16
A.3 COMPARISON WITH PROJECTION-BASE AUDIO INCORPORATION METHOD
To further assess the impact of vowel-level augmentation, we compare VowelPrompt with two ab- lation models, including transcript-only baseline, where the LLM is fine-tuned textual tran- scripts and conversational context without any prosodic cues, and projection-based audio encoder baseline, where continuous acoustic embeddings from Whisper are temporally pooled and passed through a learned projection module into the LLaMA token space. Both approaches are evaluated under supervised fine-tuning (SFT) on IEMOCAP and MELD, using LLaMA-3-BB-Insrrue LLaMA---Scout (ecout-17B.. 17B-16E-Instruc backbones. This comparison highlights the trade-offs between purely textual inputs, continuous projection- augmentation, and discrete interpretable vowel- level descriptors.
and
Comparison of transcript-only, projection-based audio encoders (Whisper + projec- tor), and VowelPrompp (VowelPrompt) under supervised fine-tuning (SFT). Results reported as Weighted Fl (%).
Table 10:
IRMOCAP
MELD
Model
Transcript-Only
Projection
VowelPrompt
Transcript-Only
Projection
VowelPrompt
LLaMA-3-8B-Lnsttine
70.32
72.65
73.46
67.44
68.85
69.61
Scou--44
70.82
73.05
73.85
67.90
69.32
70.12
As shown in Table 10, both projection based augmentation and VowelPrompt yield clear gains over the transcript-only baseline, underscoring the value of incorporating prosodic information. Among the augmentation strategies, VowelPrompt achieves the best results across all settings, outperforming the projection-basee baseline method on both IEMOCAP and MELDD, and across both L.LaMA-3 and LLaMA-4 backbones.
A.4 ABLATION STUDY ON THE NUMBER OF BINS K
The number of quantization bins used for discretizing continuous vowel-level acoustic features determines the balance between interpretability and granularity. With very small K (e.g., K 2),
15

---

Published as a conference paper at ICLR 2026
Table 11: Ablation the number of bins K for quantile-based discretization of vowel-level features.
Results are reported as Weighted Fl (%)
Zero-Sho Shot
SFT
K
IEMOCAP
MELD
IEMOCAP
MELD
2
57.45
61.32
71.12
67.28
58.72
62.18
72.04
68.01
59.86
63.47
73.02
69.05
60.74
64.17
73.46
69.61
60.22
63.89
73.12
69.18
59.74
63.41
72.78
68,92
59.15
62.95
72.33
68.40
descriptors are overly coarse and fail to capture fine prosodic variation. Increasing K improves resolution, but excessively large values, such as K > 7, introduce sparsity and noisy distinctions. reducing model generalization. To assess this effect, we perform an ablation study on IEMOCAP and MELD under both zero-shot prompting and supervised fine- tine-tuning (SFT). Results in Table 11. Performance improves stcadily as K increases from 2 to 5, with K = consistently achieving the best results across all datasets and training regimes. Beyond this point, performance declines slightly, indicating that excessive discretization is detrimental. These findings support = 5 the optimal setting, striking a balance between interpretability and discriminative power in VowelPrompt.
presented
A.5 ANALYSIS BY UTTERANCE DURATION ON MELD
To further examine how utterance duration influences model performance, we analyze zero- shot results on the MELD dataset by grouping test utterances into short (<ls), medium (Is- 3s), and long (>3s) categories. Table 12 reports both Unweighted Accuracy (UACC) and Weighted Fl (WF1) scores for GPT-4o and LLaMA-3-8B-Instee ing, SpeechCueLLM (Wu et al., 2025), and VowelProompt. As shown in Table 12, performance declines as utterances grow longer, reflecting the increased variability and contextual complexity of extended speech. Despite this trend, VowelPrompt consistently provides improvements over both baselines across all duration categories. The gains especially pronounced for short and long utterances, where vowel-level cues help disambiguate emotions that may otherwise be blurred by brevity diluted extended discourse. This demonstrates that VowelPrompt remains robust across diverse temporal scales of spoken dialogue.
under transcript-only prompt-
Table 12: Zero-shot performance on MELD under different utterance durations. Results are reported Unweighted Accuracy / Weighted Fl (%).
Target Utterance Duration
Method
LLM
<s
Is-3s
>>s
Transcript Only
67.03 / 66.92
65.17
64.28
54.34 / 55. 20
SPeechCueLLM (Wu et al., 2025)
GPT-4o
59.50 / 60.39
55.11
55.96
47.04 48.84
VowelPrompt (Ours)
69.53 / 68.26
65.62
63.04
59.37 / 58.46
Transcript Only
59.50 / 60.47
49.77
551,43
41.32 /11.58
SpeechCueLLM (Wu et al., 2025)
LLaMA-3-BB-Imae Instruct
53,41 / 54.41
47.10/ 47.77
38.46/ 37.69
VowelPrompt (Ours)
59.14 / 59.67
52.29 53.49
42.50 / 42.64
A.6 ABLATION STUDY ON THE PROSODY-DRIVER EMOTION PREDICTION IN VOWELPROMPT
To demonstrate that VowelPrompt relies on vowel-level prosodic descriptors instead of spurious lex- ical or formatting heuristics inherited from the oracle reasoning traces, we conducted a series of ab- lation studies transcript shuffle control, prosody permutation control, matched matched-marginal placebo, and cross-swap. The ablation study is performed on MELD using LLaMA-3 1-MA----B-. A-AA--88B-Insttee 8---nnstruct trained with GRPO. In the study transcript shuffle control, randomly permute the word order while
16

---

Published as a conference paper at ICLR 2026
keeping the vowel-level prosodic descriptors intact. It observed in Table 13 that the performance of VowelProompt only marginally decreases under this perturbation, indicating that lexical ordering or content identity is not the dominant predictive signal, and the prediction of VowelPrompt heavily relies on the vowel-level prosodic information coupled with the lexical information of the vow- els. In the study prosody permutation control, permute the vowel-prosody descriptors across utterances within each training mini-batch while leaving transcripts unchanged. It is observed in Table 13 that the prosody permutation leads to a significant performance degradation, which demon- strates that VowelPromppt significantly depends on the alignment between vowel-level prosodic cues and the corresponding utterances, instead of relying on the transcript alone. We further perform a aatched-margina placebo experiment, where prosody tokens are replaced with random draws from their empirical per-vowel distributions. This preserves the marginal statisties, token frequencies, and style patterns but destroys semantic grounding. It is observed in Table 13 that the performance of the ablation model decreases significantly, which demonstrates that VowelPrompp does not rely on superficial token regularities and instead requires aligned prosodic descriptors to make accurate predictions.
Table 13: Ablation study on the prosody-driven emotion prediction in VowelPrompt. The study is performed on MELD using LLaMA-3-8B-Instnun trained with GRPO.
Methods
Weighted (%)
VowelPromipt (Prosody Permutation)
41.72
VowelPrompt (Matched-Marginal Placebo)
44.10
VowelPrompt (Transcript Shuffle) VowelPrompt
67.00
68.90
Finally, we perform a cross-swap counterfactual consistency experiment, where we preserve the transcript but attach prosodic descriptors extracted from utterances belonging to different emotion category. The study is performed on the happy and the sad emotions in IEMOCAP. It is observed that the predicted emotion systematically follows the swapped prosodic profile rather than the lexical content. As shown in Table 14, when happy utterances of predictions labeled as sad increases from 18.7% to 45.8%, while retaining the original transcript. Conversely, when sad utterances are paired with happy prosody, the proportion of happy predictions increases from 27.5% to 51 51.0% The above results demonstrate that VowelPrompt does not merely memorize lexical patterns but actively attributes emotional prediction to vowel-level prosodic cues, which provides direct causal evidence that the prosodic descriptors, rather than text alone, drive the model's decision-making
paired with sad prosody, the proportion
Table 14: Counterfactual cross-swap analysis on the happy and the sad emotions. Predicted Sad (%)
Ground-Trun Emotion
Prosody Source
Predicted Happy (%)
Happy
Happy
81.3
18.7
Happy
Sad
$4.2
45.8
Sad
Sad
27.5
72.5
Sad
Happy
51.0
49.0
A.7 STUDY ON THE IMPACT OF THE TOKENIZATION OF THE LABEL VERBALIZER
To study the impact of the tokenization behavior of the labels, we first perform a study on the tok- enization behavior of the IEMOCAP verbalizers, including angry, excited, happy, neutral, and sad, under both the LLaMA-3-88 and Qwen-2-7B tokenizers. In both models, happy, neutral, and sad are each encoded as single-token verbalizers, while angry (l'ang", :ry'') 1) and excited (l'exc', 'ited'I) are consistently split into two subword units. To study the impact of the decoding bias arising from such variation, we replaced all emotion labels with synthetic two-letter tokens (happy-ha, sad- angry-an, neutral-nne excited-tex) that are uniformly represented as single tokens across both tokenizers. We then randomly permuted the emotion verbalizer mapping 10 times, thereby elimi- nating any lexical or semantic prior that the tokenizer could exploit. It is observed in Table 15 that replacing the original emotion verbalizers with synthetic two-letter tokens leads to only a marginal performance drop on VowelPrompt, which demonstrates that VowelPrompt is marginally impacted
17

---

Published as a conference paper at ICLR 2026
by the decoding bias. Notably, the impact is significantly smaller for VowelPrompt compared to SpeechCueLLMM which demonstrates that VowelPrompt is significantly more robust to label per- turbations because the predictions are grounded in detailed vowel-level prosodic cues rather than lexical or tokenization-based priors associated with the verbalizers.
Table 15: Impact of label tokenization and permutation on emotion recognition performance of VoweIPrompt.
Methods
Weighted Fl (%)
VowelPrompt
73.0
SpeechCueLLM
71.5
VowelPrompt (Two-Letter)
71.7
SpeechCueLLLM (Two-Letter)
67.8
VowelPrompt (Two-Letter Permutated)
71.0
L.M (Two-Letter Permutated)
65.2
A.8 COMPARISON BETWEEN VOWEL-LEVEL, AND CONSONANT-LEVEL PROSODIC DESCRIPTORS
To demonstrate the effectiveness of the vowel-level prosodic descriptors, we perform an ablation study replacing vowel-level descriptors with consonant-level descriptors, including segment dura- tion, voice onset time (VOT), frication energy, and nasal intensity. We have also tested the per- formance of variant of VowelPrompt, which incorporates both the vowel-level descriptors and the consonant-level descriptors. It is observed in Table 16 that vowel-level descriptors consistently achieve the highest performance across three corpora, while consonant-level descriptors alone yield significantly worse performance. Incorporating both vowel- and consonant-level cues im- proves the performance of VowelPrompt on German, which is attributed to the richer stop and frica- tive contrasts in its phonological system, but does not surpass the vowel-only setting French or English. These results indicate that consonantal information does not provide universally comple- mentary emotional cues and further substantiate the sufficiency of vowel-level descriptors for the languages evaluated in this study.
Table 16: Comparison between vowel-level with consonant-level prosodic descriptors.
Methods
CaFE (French)
EmoDB (German)
MELD (English)
VowelPrompt (Vowel-Level Cues)
51.42
64.17
VowelPrompt (Consonant-Level Cues)
48.73
67.80
62.95
VowelPrompt (Vowel + Consonant Cues)
51.04
70.21
64.08
A.9 COMPARISON WITH NON-LLM ACOUSTIC BASELINE METHODS
To demonstrate the advantages of VowelPrompt over existing non-LLM deep learning models, we have compared VowelPrompt with strong non-LLM speech emotion recognition baselines using state-of-the-arr self-supervised speech models, including HuBERTT-argge and way2vec-large. In par- ticular, we extract HuBERT-larg and wav2vec-large embeddings and train ML.P classifiers top of the embeddings. The results below include both in-domain and cross-domain evaluations on IEMO- CAP and MELD. It is observed in Table 17 that VowelPrompt consistently outperforms HuBERT- large and wav2vec-large across all settings.
A. HUMAN EVALUATION ON THE REASONING TRACES BY VOWELPROMPT
To evaluate the quality of the reasoning traces VowelPrompt trained with GRPO, we conducted a human evaluation study with four annotators who rated 200 randomly sampled reasoning traces from each of the models trained on IEMOCAP and MELD. Each trace was evaluated on prosodic ground- ing, causal coherence, and internal consistency using 1-5 Likert scale. It is observed in Table 18 that VowelPrompt demonstrates significantly higher reasoning faithfulness than SpeechCueLLM across all four annotators. L.M receives average of 3.14, while VowelPrompt
18

---

Published as a conference paper at ICLR 2026
Comparison of VowelPrompt with strong non-IL.M acoustic baselines using self- supervised speech representations calculated from HuBERT-lage and way2vec-large.
Table 17:
Dutasets
HuBERT-large wav2vec-large
VowelPrompt
In-Domain Evaluations
IEMOCAP
67.6
65.6
73.4
MELD
56.8
55.1
69.6
Cooss-Domain Evaluations
IEMOCAP MELD
45.0
43.5
60.2
MELD IEMOCAP
44.2
41.7
51.7
receives an average score of 3.77, reflecting more accurate grounding in prosodic cues and greater internal coherence. These results confirm that VowelPrompt's reasoning traces are not only more interpretable but also more consistently aligned with the prosodic evidence that drives its final pre- dictions.
Table 18: Human evaluation results across four evaluators.
Methods
Evaluator
Evaluator 2
Evaluator 3
Evaluator
Average
SPeechCueLLMM
3.12
3.55
2.82
3.08
3.14
Vowe/Prompt (Ours)
4.05
3.96
3.42
3.65
3.77
A.11 STUDY ON THE BALANCE BETWEEN THE IN-DOMAIN PERFORMANCE AND THE CROSS-DOMAIN PERFORMANCE
The Knowledge Distillation (KL) weight in GRPO controls how strongly the policy is regularized toward the supervised SFT model, which effectively limits how far reinforcement learning can devi- ate from the source-domain distribution. To better understand the trade-off between the in-domain performance and the cross-domain performance, KL weight and measuring both in-domain and cross-domain performance. As shown in Table 19, decreasing the KL. weight relaxes the constraint the policy, resulting in slightly improved cross- domain robustness, indicating that the model relies less on dataset-specific lexical patterns and more on
conducted sensitivity analysis by varying the
prosodic cues. On the other hand, increasing the KL weight leads higher in- domain performance. Notably, both the in-domain and cross-domain performance of VowelPrompt vary only marginally across different values of the KL weight, which demonstrates that the GRPO- trained model is largely insensitive to the value of the KL weight. In addition, in the cross-domain setting, the GRPO data is from the source domain alone.
Table 19: Effect of KL weight on VowelPrompt performance under GRPO for both in-domain and cross-domain evaluation.
Weight
IEMOCAP
MELD
IRMOCAP-MMEIN
MEL.D-IFMOCAP
0.1
71.9
68.1
60.5
$1.3
0.25
73.4
69.6
60.2
$1.7
73.4
58.9
49.6
73.6
70.0
58.4
49.2
A.12 COMPARISON WITH CLASSIFIERS TRAINED DIRECTLY THE VOWEL-LEVEL PROSODIC FEATURES
To demonstrate the necessity an LLM-based architecture, we have conducted an ablation study comparing VowelPrompt against classifiers trained directly on the vowel-level prosodic features. In particular, the classifiers are a multilayer perceptron (MLP), a random forest (RF), and a transformer. In the transformer baseline, each phoneme is treated token, and its corresponding prosodic fea- tures are treated as the features of the token. is observed Table 20 that VowelPrompt significantly outperforms all baseline classifiers, which demonstrates that access the same attributes alone is
19

---

Published as a conference paper at ICLR 2026
insufficient. Vanilla classifiers fail capture the contextual and linguistic dependencies that govern how vowel-level prosody conveys affect. contrast, VowelPrompt leverages the LLM's pretrained linguistic priors integrate prosodic cues with lexical semantics, discourse context, and phonotactic patterns.
Table 20: Comparison with traditional classifiers trained directly the vowel-level prosodic fea- tures.
Datasets
XGBoost
MLP
Transformer
VowelPrompt
IEMOCAP
40.2
39.6
48.5
73.4
MELD
45.1
44.5
51.2
69.6
A.13 STUDY ON THE IMPACT OF THE INCORRECT VOWEL ALIGNMENT
To study the impact of incorrect vowel alignment, we have performed an ablation study that per- turbed 5%, 10%, and 15% of the boundaries of the vowels in the alignment results. The study is performed on MELD using LLaMA-3-8B-Instrue ment, randomly shifted its start end times by 50% of its original duration. It is observed in Table 21 that the performance of VowelPrompt is robust to the perturbation of the boundaries of the vowels the aligmment results and consistently achieves significantly better performance than SpeechCuel.LMM. For example, even with 15% of the vowel boundaries perturbed, VowelPrompt still achieves a Weighted F1 of 69.11% 11%, which outperforms SpeechCuel.LM by 2.04%.
In particular, for each selected vowel seg-
Table 21: Robustness of VowelPrompt under phenom alignment perturbations.
Method
Perturbation Ratio
Weighted Fl (%)
SpeechCueLLMM
0
67.07
VowelPrompt
0
69.61
VowelPrompt
5%
69.50
VowelPrompt
10%
69.23
VowelPrompt 15%
69.11
A. STUDY ON THE IMPACT OF SPEECH RATE ON THE PERFORMANCE OF VOWELPROMPT
To study the impact of speech rate on the performance of VowelPrompt, we conducted ablation study on the MELD dataset by categorizing testing utterances according their phone-per-seeee (PPS) rate. In particular, we segmented the test set into three categories based on PPS statistics computed from Montreal Forced Aligner alignments, including slow (PPS S 6.0), normal (6.0 < PPS < 8.5), and fast (PPS > 8.5). is observed in Table 22 that although the performance of Vow- elPrompt and the baseline method SpeechCueLLMM degrade as speech rate increases, VowelPrommpt consistently outperforms SpeechCuel.LMM across all PPS ranges.
Table 22: Impact of speech rate (phones-per-seoee PPS) on the performance of VowelPrompt.
Method
PPS E 6.0
6.0 < PPS < 8.5
PPS >> 8.5
Overall
SpeechCueLLM
68.21
64.19
67.07
VowelPrompt
71.08
69.61
67.25
69.61
A.15 ABLATION STUDY ON INCORPORATING THE REASONING SFT AND SFT COMBINED WITH GRPO
In this section, we perform an ablation study by enabling and disabling reasoning/thinking both the SFT and the SFT & GRPO settings. The study is performed MELD using LLaMA-3-8B. Instruct. observed in Table 23 that our VowelPrompt consistently outperforms the baseline methods under different settings.
20

---

Published as a conference paper at ICLR 2026
Table 23: Comparison of different training strategies with and without reasoning.
Method
SFT (wio Reasoning)
SFT (with Reasoning)
SFT & GRPO (w/ Reasening)
SFT & GRPO (with Reasoning)
InstriciERC
67.25
67.02
67.51
66.96
SALMONN
67.25
67.25
67.43
66.85
SpccelCueLLMM
67.07
67.15
67.29
67.10
VowelPrompt (Ours)
69.61
69.82
69,88
68
B PROMPT TEMPLATES
We present representative prompt templates used in experiments fine-tuning regimes. Each prompt includes three main components: the conversational context, the target utterance, and the prosodic descriptors (either sentence-level or vowel-level). Descriptors are expressed in natural language and inserted into prompts using a consistent format to guide emotion reasoning.
zero-shot, few-shot, and
Supervised Fine- Tune-Tuning Prompt (with Reasoning):
Now you are an expert sentiment and emotional analysis. The following conversation noted between #### ###" involves several speakers. ### Speaker 0:...
Speaker.1 : <wwrgttttta.... <target.speech> #illl
Vowel-level Speech Descriptions of Spexker.1:
sewel_descrile ponsss>>>>
Please select the emotional label of spoaker. based the context and the
vowel-level acoustic features.
Output the thinking process <think> </think> and emotion label prediction <answer>
</answer> tags.
Supervised Fine- Fine-Tuning Prompt (without Reasoning):
Now are an expert in sentiment and emotional analysis. The following conversation noted between '### ###* involves several speakers. ### Speaker.O:...
Speaker_1 sttarget_sppeettt ### Vowel-level Speech Descriptions of Speaker 1: <target.speechh::
<vowel descriptions:
Please select emotional label Speaker listarget speech> on the context and the vowel-level acoustic features.
Output the emotion label prediction in <answer> </ </answer> tags.
Zero-Shot Prompt (Transcript Only):
Now expert in sentiment and emotional analysis.
The following conversation noted between "### ###' involves several speakers. ### Speaker .....
Speaker.1 wwwrrttttt.... .spergetsspeehh>>> ###l Please select the emotional label of stpaaker..taargettppeehhh based the context.
Please output ONLY ONE label from <available.om waw.eetttonnnnelenm.....m the first word, and then explain your choice.
Zero-Shot Prompt (Transcript Vowel-Level Prosody):
21

---

Published as a conference paper at ICLR 2026
Now you are an expert in sentiment and emotional analysis. The following conversation noted between ### ### involves several speakers. ### Speaker_0:...
Speaker_1 www.gge..sppeeeeeeeeeeeeee ###
Vowel-level Speech Descriptions of 1 www.el_descrip....
Please select emotional label
based on the context and the
vowel-level acoustic features. Please output ONLY ONE lahel from <avai lable emot on 1abel 8/> as the first word, and then explain your choice.
Few-Shot Prompt (3 Examples + Target Query):
Now you are an expert sentiment and emotional analysis.
<Rxample e 1>
<Example 2>
<Example_3>
The following conversation noted between '### ###*' involves several speakers. ### Speaker.O:...
Speaker.1: s sttrrttttetteteeeetiitete ###l
Vowel-level Speech Descriptions of S www.el.descrpt....
Please select emotional label $peaker.1
based on the context and the
vowel-level acoustic features. Please output ONLY ONE lahel from <avai 1ab² e_emot on_oo_____ Jabelaeell.... s/> as the first word, and then explain your choice.
22


Reviews:
[{"review_id": "HAOcNzRkGp", "reviewer": "Reviewer_ksVz", "summary": "The paper proposes a novel Speech Emotion Recognition approach that augments speech transcripts with vowel-level prosodic descriptors (F0 level/slope/variation, intensity level/variation, duration) extracted via forced alignment. These descriptors are then used in an LLM-as-classifier framework, fine-tuned via supervised fine-tuning (SFT) and reinforcement learning (RLVR) from GPT-4o (LLM as oracle) for reasoning traces.\n\nTo account for cross-speaker and vowel variability, the authors first perform speaker- and vowel-type normalization, then quantile-bin the prosodic statistics and convert them into natural-language tokens, which are appended to the transcript so the LLM can reason over lexical and localized prosody. The method is evaluated on IEMOCAP, MELD, CaFE, EmoDB, and ASVP-ESD (mixed-lingual), showing consistent gains over transcript-only and sentence-level prosody prompts, with interpretable rationales.", "questions": "Some follow-up questions:\n\n- Are `Speaker_{%d}` tokens used only as placeholders to distinguish multiple speakers within an utterance, or do they carry any stable identity across utterances and speakers?\n\n- Related work organization: The “Vowel-Centric Prosody in Emotional Speech” paragraph and the first paragraph of Section 3.1 can be merged. This would reduce redundancy and improve flow by introducing the vowel-centric rationale once, then immediately presenting the operational relevance.\n\n- Cross-linguistic normalization and scope: the manuscript states: ``We ensure cross-linguistic consistency and compatibility with multilingual phonetic analysis pipelines …`` and later ``To control for cross-lingual variation in prosodic realization, we further perform normalization at the language level. For each language, we compute global means and standard deviations for each prosodic feature and apply z-score normalization within that language``...  Please specify the language set to which these claims apply. Cross-linguistic prosody-emotion mappings differ across languages, especially in tone languages such as Vietnamese and Chinese where pitch contours are lexical. It would be helpful to limit claims to the languages actually evaluated (for example, English, French, German).\n\n- Determinism of “parameter-free” processing and oracle traces: The manuscript states that the process is ``parameter-free, ensuring transparency and reproducibility,`` and that ``gold reasoning traces [are] automatically generated by a high-capacity text-only LLM such as GPT-4o.``... Are these oracle traces generated deterministically. Please report the decoding settings used for GPT-4o, including temperature, top-p, and any seed control. If temperature was set to zero, state this clearly. If not, either provide information about variance across repeated generations and whether multiple samples were filtered or selected, or elaborate on the \"reproducibility\" claim.\n\n- RLVR objective and the role of reasoning traces: on the RLVR stage, how are the reasoning traces used. Are rationales part of the reward signal beyond format verification? Please clarify whether any rationale consistency checks are part of the verifiable objectives.", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "The proposed method has several strengths, which can be grouped into two categories:\n\n**Conceptual**: the method provides a privacy-oriented, interpretable formulation of SER by augmenting transcripts with symbolic, vowel-level prosody tokens (F0 level, intensity, duration), allowing emotion inference without raw audio at inference. This design is linguistically grounded (vowels are typically stable prosodic carriers), separates lexical content from paralinguistic cues, and supports closed-set classification, addressing both explainability and data-minimization goals.\n\n**Implementation strengths**: the end-to-end pipeline is clear and modular: forced alignment, followed by speaker and vowel normalization, then quantile binning, conversion to natural-language tokens, and an LLM-as-classifier stage. Each step is inspectable and easily ablated. Training uses a practical two-stage recipe (supervised fine-tuning followed by RLVR/GRPO with verifiable rewards) and performs consistently across datasets and model families, improving over transcript-only and sentence-level prosody prompts. At deployment, the approach seems easy to integrate with existing text infrastructures.", "weaknesses": "Because the oracle used for RLVR traces (GPT-4o) is itself an LLM, there is a material risk that fine-tuned student LLMs learn spurious lexical or formatting heuristics unrelated to the intended prosodic mechanism. To establish that the model uses vowel-level prosodic tokens rather than incidental cues, please include controlled counterfactual ablations that preserve input statistics while breaking the hypothesized channel. Some examples to test:\n- Transcript shuffle control: randomly permute word order, or replace content words with frequency-matched synonyms, while keeping the vowel-prosody tokens intact. Performance should remain near the original if prosody carries the signal and should drop sharply if lexical priors dominate.\n- Prosody permutation control: permute the vowel-prosody descriptors across utterances within a mini-batch while keeping transcripts fixed. Performance should drop to near chance if the model relies on the prosodic channel.\n- Matched-marginal placebo: replace prosody tokens with random draws from their empirical per-vowel distribution so that marginals are preserved but alignment is broken. This controls for token frequency or style leakage.\n- Cross-swap (counterfactual consistency): for the same transcript, attach prosody from an utterance of a different emotion. Predicted labels should flip in the direction implied by the swapped prosody if the mechanism is causal.\n\nAlso, the paper does not analyze how the discrete labels are tokenized by each model’s tokenizer. If some labels map to a single token while others split into multi-token sequences, the model may exhibit unequal calibration and decoding bias unrelated to prosodic evidence. An ablation study can include:\n\n- Tokenization audit per model: for each model used (for example, LLaMA-3-8B, Qwen-2-7B), report (i) tokenization of each label verbalizer (angry, sad, neutral, happy, excited), including the number of tokens and subword pieces, and (ii) empirical prior token likelihoods for these verbalizers under pseudo-neutral prompts, for example, a transcript without prosody information.\n- Label set distribution: compute normalized label probabilities via the product of subtoken log-probabilities for multi-token labels and report calibration. This assesses whether the decoding process favors specific verbalizers independent of evidence.\n- Label name permutation: assign the emotion categories to some random verbalizer strings to test sensitivity to label names. If the model captures the underlying categories rather than specific strings, performance should remain stable.\n\nAnswering some of these questions would strengthen the conceptual foundation of the paper and the interpretability of the suggested method.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "QaVHM3V8ET", "reviewer": "Reviewer_RfwA", "summary": "The paper targets a key limitation in speech emotion recognition (SER) with large language models (LLMs): difficulty leveraging fine-grained prosodic cues and lack of interpretability. The authors propose VowelPrompt, a framework that annotates and conditions LLMs on multi-dimensional, phoneme-level prosodic attributes with a focus on vowels. Experiments across five datasets (including IEMOCAP, MELD, CaFE) show consistent gains over text-only baselines and sentence-level prosody augmentation methods (e.g., SpeechCueLLM) in zero-shot, few-shot, supervised fine-tuning, cross-domain , and multilingual settings, while providing interpretable emotion reasoning.", "questions": "1. Vowel vs. consonant contributions: Can you provide ablations comparing vowel-only, consonant-only, and all-phoneme prompting across languages and conditions? Are there language-specific effects where consonant cues (e.g., voicing contrasts, place/manner-dependent energy) help more?\n2. Non-LLM baselines: How does VowelPrompt compare against a strong acoustic SER pipeline using phoneme-aligned prosody with CNN/Conformer or self-supervised speech features (HuBERT/w2v2) plus a classifier? Please include both in-domain and cross-domain comparisons.\n3. Reasoning faithfulness: Do you measure the consistency between intermediate prosodic rationales and final predictions? For example, human judgments of faithfulness, agreement with attributions  or causal tests (masking/removing cited cues).\n4. GRPO dynamics:  Can you explain or give learning curves and sensitivity analyses demonstrating when GRPO helps harm in-domain performance but helps cross-domain transfer? Also, if I understand correctly, the GRPO data is also from the source domain, right?\n5. Value of LLMs: If the same phoneme-level attributes are available, what performance and interpretability gains remain when replacing the LLM with a standard classifier? Conversely, can the LLM operate without explicit phoneme labels (i.e., text + raw audio features) and still retain its advantages?\n6. How robust is VowelPrompt to alignment errors and background noise? What is the performance degradation under realistic ASR/aligner noise or accented speech?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "1. Fine-grained modeling unit: The vowel-centric, phoneme-level prosodic prompting is a clear step beyond sentence-level prosody features, improving both recognition performance and interpretability. The focus on vowels is linguistically motivated and empirically supported.\n2. Two-stage training design: The two-stage framework is logically structured, and the RL-based targeted optimization appears to contribute to robust cross-domain generalization.\n3. Breadth of evaluation: The empirical study is comprehensive, covering zero-shot, few-shot, supervised fine-tuning, cross-domain transfer (IEMOCAP↔MELD), and multilingual scenarios (EN/FR/DE/mixed). The baselines include strong text-only and prosody-augmented LLM variants (e.g., SpeechCueLLM), making the comparisons meaningful.", "weaknesses": "1. The paper emphasizes the primacy of vowels for prosody, while citing evidence that consonants can convey complementary emotional cues (e.g., Bitouk et al., 2010). However, the method entirely excludes consonant segments. This design choice risks discarding potentially informative signals (e.g., frication intensity, voicing onsets, burst characteristics) that may be emotion-sensitive in certain languages and speaking styles. A controlled analysis is needed to justify the exclusion.\n2. The baseline suite lacks strong non-LLM SER systems, despite a long line of work using acoustic-prosodic features and modern architectures (e.g., CNN/TDNN/Conformer or HuBERT/w2v2 features with phoneme-aligned prosody). Without such baselines, it is difficult to disentangle the benefit of VowelPrompt from the general advantage conferred by LLM priors and instruction tuning.\n3. The framework assumes fine-grained (phoneme/vowel-level) labels and intermediate “reasoning” steps align with the final emotion decision. In practice, manual annotations and model-generated rationales may not be fully consistent. This raises concern that the LLM could optimize for correct final labels while producing intermediate explanations that are partially contradictory or post hoc. The paper should explicitly evaluate the faithfulness and consistency of the reasoning traces.\n4. Table 4 suggests GRPO leads to consistent performance drops (negative optimization) under certain settings, while Table 5 shows clear cross-domain improvements. The manuscript needs a unified explanation reconciling these outcomes. For instance, is GRPO trading in-domain accuracy for distributional robustness, regularizing prosody usage, or mitigating spurious correlations? What hyperparameters or curricula drive this trade-off?\n5. At inference time, the system still relies on MFA and explicitly labeled phoneme attributes before passing prompts to the LLM. It is therefore unclear why an LLM is preferable to a purpose-built classifier over the same discrete features (e.g., a gradient-boosted tree, MLP, or Transformer). If the LLM mainly converts discrete features to natural-language prompts, the incremental contribution appears limited. The paper should quantify the benefit of the LLM beyond a feature-to-label classifier and clarify where language priors or compositional reasoning matter.\n5. The framework depends on accurate phoneme segmentation and prosodic feature extraction. It is unclear how errors in forced alignment or noisy conditions propagate to final performance. Robustness analyses (e.g., perturbing boundaries, adding noise, cross-accent conditions) are limited.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "IoHwKaxv9a", "reviewer": "Reviewer_W37J", "summary": "The paper presents an approach to introduce an explainable and interpretable approach to perform emotion recognition from speech using  LLMs. The work introduces pitch, energy, and duration based descriptors from time-aligned vowel segments and converts these into natural language descriptors to fine tune existing LLMs. The authors hypothesize that the proposed descriptors would enable an LLM to reason using both semantic information in lexical representations as well as prosodic variations in acoustic information. \n\nEvaluation on multiple datasets demonstrate that the proposed approach helped to improve performance compared to prior art.\n\nThe motivation of the paper is well outlined, clearly citing relevant prior art and outlining the key contributions made in this paper. \nThe proposed approach introduces prosodic descriptors based on their proposed approach, that are generated from standard benchmark datasets, it is not clear whether the authors intend to share the information with the community as that would help in both replication of the reported results and foster future research directions.", "questions": "(1) By vowel level, does it mean at the individual distinct vowel level or by vowel-groups, such as front and back vowels?\n\n(2) For the cross-domain evaluations, the datasets had different number of emotion categories, how were the difference in categories accounted during the evaluation?\n\n(3) When reporting the emotion recognition performance across the different datasets, were all emotion categories specified in those data sets (Table 2) used to obtain the evaluation metric?\n\n(4) The descriptors presented in this work rely on the forced alignment information, in spontaneous speech due to coarticulation vowel boundaries may be uncertain, any thoughts on how such conditions may impact the proposed descriptors?", "limitations": "", "rating": 4, "confidence": 5, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "The paper introduces pitch, energy, and duration based descriptors from time-aligned vowel segments to generate emotion-salient prompts to improve emotion recognition performance using LLMs.\n\nThe process of generating the descriptors is well described and results demonstrate the promise of the proposed work. Evaluation from multiple datasets demonstrate the generalization of the findings.", "weaknesses": "The proposed approach introduces prosodic descriptors based on their proposed approach, that are generated from standard benchmark datasets, it is not clear whether the authors intend to share the information with the community as that would help in both replication of the reported results and foster future research directions. It is not clear whether the setup used to generate the descriptors, or the descriptors obtained from the five datasets (used in the paper) will be publicly shared.\n\nThere are some open questions regarding the introduction of the descriptors that need to be addressed: coarticulation and lenition in spontaneous speech can alter vowel durations where vowels can be deleted or influenced by neighboring phonemes. It is not clear how the proposed approach would address such situations, or such instances were ignored in the current work? It will also be interesting to explore how speech rate impacts the efficacy of using the proposed descriptors?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "cdGecSjivB", "reviewer": "Reviewer_b43y", "summary": "This paper described a system that augments vowel-level prosody descriptors as additional information to the LLMs. It shows that with the vowel-level speech information, the LLMs performs better on emotion recognition in both zero-shot and supervised fine-tuning setup.", "questions": "See the above weaknesses.", "limitations": "", "rating": 4, "confidence": 5, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "The paper is well written with good clarity, and easy to understand. It could contribute to the SER field as a baseline of fine-tuning LLMs with natural language descriptors of speech. I am personally interested in what granularity of speech information is needed for SER. This paper seems to suggests that going toward vowel-level will provide some improvement.", "weaknesses": "On the other hand, the idea of appending prosody descriptions itself is not particularly novel, e.g., SpeechCueLLM, which is also cited as a baseline by the authors. The novelty of the work lies in expanding it to the vowel-level.\n\nThere is one flaw that undermines the validity of the experiments that the work needs to revise before publication.\nIn Table 4, it is unclear to me how the other models are trained and tested. I assume only your model is fine-tuned with the thinking trajectory but not the other baselines, as I don't find how you generate the thinking trajectories for the baselines. In this case, I don't know if the other models will still output thinking trajectory after fine-tuning. And the performance gain you get, may simply be a difference of inference time scaling. And it can be possible that if you make the baseline models think/reason, they can also improve their performance to be potentially on par with your model.\n\nThe authors should add experiments either comparing the proposed model v.s. the baselines in either (both thinking enabled, or both thinking disabled). I am willing to raise the score if this issue is resolved.\n\nAdditionally, I don't see adding GRPO helps in Table 4, the authors should explain this. \n\nTypo: in the abstract: neglect \"or\"", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)