# EmoSLLM: Parameter-Efficient Adaptation of LLMs for Speech Emotion Recognition

Hugo Thimonier, Antony Perzo, Renaud Seguier

► To cite this version:

Hugo Thimonier, Antony Perzo, Renaud Seguier. EmoSLLM: Parameter-Efficient Adaptation of LLMs for Speech Emotion Recognition. 2025. hal-05442659

HAL Id: hal-05442659

https://hal.science/hal-05442659v1

Preprint submitted on 5 Jan 2026

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_1.png" /></figure>

Distributed under a Creative Commons CC BY 4.0 - Attribution - International License
# EmSLLM: Parameter-Efficient Adaptation of LLMs for Speech Emotion Recognition

Hugo Thimonier¹, * Antony Perzo¹ Renaud Seguier¹, 2

¹ Emobot, Paris, France

`{name}. {surname} @emobot.fr`

² CentraleSupélec, IETR (UMR CNRS 6164), Rennes, France

* Corresponding author.

## Abstract

Emotion recognition from speech is a challenging task that requires capturing both linguistic and paralinguistic cues, with critical applications in human-computer interaction and mental health monitoring. Recent works have highlighted the ability of Large Language Models (LLMs) to perform tasks outside of the sole natural language area. In particular, recent approaches have investigated coupling LLMs with other data modalities by using pre-trained backbones and different fusion mechanisms. This work proposes a novel approach that fine-tunes an LLM with audio and text representations for emotion prediction. Our method first extracts audio features using an audio feature extractor, which are then mapped into the LLM's representation space via a learnable interfacing module. The LLM takes as input (1) the transformed audio features, (2) additional features in the form of natural language (e.g., the transcript), and (3) a textual prompt describing the emotion prediction task. To efficiently adapt the LLM to this multimodal task, we employ Low-Rank Adaptation (LoRA), enabling parameter-efficient fine-tuning. Experimental results on standard emotion recognition benchmarks demonstrate that our model outperforms all but one existing Speech-Text LLMs in the literature, while requiring less than half the parameters of competing approaches. This highlights our approach's effectiveness in integrating multi-modal inputs for speech-based emotion understanding while maintaining significant computational efficiency.

## 1 Introduction

Predicting the emotion conveyed in audio is a critical task with many healthcare applications. For instance, tracking a patient's emotional fluctuations throughout the day can offer psychiatrists valuable insights into conditions such as depression—a disorder characterized by persistent sadness, irritability, and apathy [1]. As a result, continuous and non-invasive emotion monitoring could significantly improve diagnostic accuracy and treatment personalization.

The widespread adoption of smartphones among both minors and adults [21, 42] has enabled scalable, real-time monitoring of behavioral and emotional health. Among the modalities accessible through smartphones, speech is particularly informative due to its rich linguistic and paralinguistic content [46, 17]. These cues have been linked to various mental health conditions, and numerous studies have explored speech emotion recognition (SER) as a proxy for psychological well-being [65, 23].

SER has been addressed lately by leveraging feature representations coming from models trained for different tasks [15, 35, 36, 3, 10, 4, 24]. For instance, [60] fine-tune HuBERT [24] and Wav2Vec2.0 [4] for the task of SER [59]. Other approaches consider the use of frozen self-supervised models as feature extractors to train a supervised classifier [45, 28] by solely adding a linear layer on top of the
self-supervised model. While promising, these approaches are quite simple and often rely exclusively
on speech-related information.

Given the recent discoveries on the strong capacities of LLMs for multimodal tasks, research has been oriented towards leveraging LLMs for other modalities, including audio. In particular, different overlapping lines of works have been considered: LLMs that *speak*, LLMs that *listen*, and LLMs that can do both. Relevant to the present work is LLMs that *listen*, which describe LLMs that can take as input both natural language and audio features [62, 27, 29, 16, 53, 13, 44].

Current state-of-the-art LLM-based approaches for Speech Emotion Recognition, such as SIFT-LLM [44] and SALOMONN [53], demonstrate impressive performance but rely on models with over 7 billion parameters. This makes them impractical for privacy-sensitive, on-device deployment—an essential consideration when handling highly personal data like a user's emotional state over time.

In the present work, we propose a parameter-efficient approach LLM-based approach for speech
emotion recognition. We build on [57] and use as a downsampling module an attention-based model
that selects learnable queries to represent the audios to be fed to the LLM. We rely on their feature
mapping mechanism, QPMapper, as it is lightweight and has shown strong performance for visual
and audio data inclusion in LLMs. We rely on Robust wav2vec 2.0 [25] and WavLM as the audio
feature extractors and experiment using Llama3.2-3B-Instruct [39]. We train our model using a 3-step
learning curriculum. In the first phase, we treat automatic speech recognition (ASR) as a proxy task
to align the audio representations with the LLM embedding space. During this phase, the audio
encoder and LLM are frozen, and only the QPMapper is updated. In the second phase, we continue
training on the ASR task but enable fine-tuning of the LLM via Low-Rank Adaptation (LoRA) [26],
allowing the language model to begin adapting to audio-conditioned tasks. Finally, in the third phase,
we introduce the SER task to specialize the model for emotion recognition, further fine-tuning the
LLM with LoRA while continuing to update the weights of the downsampling module.

We compare our model, coined **Emotion Speech Large Language Model (EmSLLM)**, to existing text-audio language models for the task of speech emotion recognition. EmSLLM achieves competitive SER performance, outperforming all but one existing text-audio model while maintaining a substantially smaller parameter footprint. This demonstrates its potential for privacy-preserving, on-device emotion recognition. In addition, we carefully design prompts to guide the language model's reasoning over the audio representations, which we find to be essential for improving emotion recognition accuracy in low-resource settings.

The remainder of the paper is organized as follows: Section 2 presents the related works on self-supervised learning and multimodal LLMs. Section 3 presents the approach used to address the SER problem by using LLMs. Section 4 shows the experimental results of our proposed methodology. In Section 5 we perform some ablations to assess the relevance of our approaches key characteristics. Finally, in 6 we conclude and discuss the limitations of the present work.

## 2 Related Works

We review three key areas of prior work that underpin our approach: self-supervised learning for
audio representations, the emergence of multimodal large language models, and recent advances in
combining speech and text within LLM frameworks.

**Self-Supervised Learning** Self-supervised learning (SSL) for representation learning consists in solving a well-chosen pretext task to generate a meaningful representation of the data that can serve for downstream tasks. It has garnered increasing attention in recent years due to its success in fostering performance on downstream tasks on different data modalities. In the case of visual modality, approaches have been developed for images and video sequences. For images methods such as I-JEPA [2], SwAV [8], VICReg [5] or Barlow Twins [68] have been explored. In the case of video sequences, approaches such as V-JEPA [6], LatentMIM [61] or GTCC [18] have been studied. These learned representations have been shown to improve the performance of supervised downstream tasks. Similarly, methods for tabular data modality such as XTab [69], BinRecon [32], SwitchTab [63], or T-JEPA [55] enable better training of deep methods on tabular data. For audio data, self-supervised methods have also emerged as a powerful approach to learning representations without labeled supervision. Some seminal approaches include Wav2Vec [51], which was later improved with Wav2Vec2.0 [4]. The latter approach involves mapping the original audio signal
<figure><img src="image_2.png" /><figcaption>Figure 1: **EmSLLM Pipeline.** In step (a), the audio signal is fed to a pretrained audio encoder to obtain a vectorized embedded representation **h**<sub>AE</sub> of dimension **R**<sup>*n*×*d*<sub>AE</sub></sup>. In step (b) a text prompt is fed to the embedding module of an LLM to output a vectorized embedded representation **p** of dimension **R**<sup>*n*<sub>p</sub>×*d*<sub>LLM</sub></sup>. In step (c) **h**<sub>AE</sub> is fed to a downsampling module and the obtained sequence **h**<sub>ds</sub> is of dimension **R**<sup>*n*<sub>q</sub>×*d*<sub>LLM</sub></sup>. In step (d) **h**<sub>ds</sub> and **p** are concatenated and fed to the LLM, *f*<sub>LLM</sub> that predicts the target given a task (ASR or SER).</figcaption></figure>

to a latent space with a convolutional model, which is then mapped to a context space using a
transformer model. Their approach also includes a quantization module following previous work
[58, 50, 3]. This quantization involves masking part of the audio representation and selecting the
masked elements among some distractors. Another seminal work includes HuBERT [24], in which
the self-supervised task predicts hidden cluster assignments of the masked frames. In addition to this
approaches, WavLM [10] involves a transformer model and the mask reconstruction task and displays
strong performance; BYOL-A [40, 41] that adapts the BYOL approach [22] originally proposed
for images to audio. While subsequent works have proposed various enhancements, Wav2Vec2.0,
HuBERT, and WavLM continue to dominate as the primary self-supervised models for audio.

**Multimodal Large Language Modeling** Prior works have tried to leverage LLM to include modalities other than natural language. The main emphasis has been on including images in LLMs, coined as Vision Language Models (VLMs). VLMs could be described as LLMs that are also able to receive images as inputs. [7] distinguish several (possibly overlapping) approaches to VLMs: mask-based approaches [52, 30], contrastive-based approaches [48, 31], generative approaches [54, 66], and approaches based on pretrained backbones [56, 70].

**Speech-Text LLMs** Efforts have also been put on enabling LLMs to handle audio and speech modalities. Due to their lighter training cost, most approaches have been oriented toward including pretrained backbones to fuse modalities in LLMs rather than training models from scratch. The first approaches focused on specific tasks and enabled LLMs to perform one audio task at a time, such as ASR [20, 67]. Later work has focused on multi-task models that can perform several tasks. For instance, [53] propose SALMONN in which they combine Whisper [49] and BEATS [11] with an LLM by using a window-level Q-Former [33], and train their model to perform tasks ranging from ASR, music captioning or emotion recognition. Similarly, Qwen2-audio [13] also relies on Whisper as the audio encoder. It combines the audio encoder's output representation with the tokenized representation of the text before feeding it to an LLM. They then proceed to train the LLM on more than 30 tasks, including ASR, SER, Speech-to-Text Translation or Vocal Sound Classification. Other examples of multi-task approaches include methods such as Speechverse [16], which relies on both WavLM [10] and Flan-T5 [14]; or WavLLM [27] that relies on WavLM and Llama-2 7B-Instruct as the backbone models.

3 Method

We put forward EmSLLM, an LLM that integrates both audio and textual modalities to enhance
emotion prediction by fine-tuning a large language model (LLM) with Low-Rank Adaptation (LoRA)
(see Figure 1). We first extract audio features using a pretrained audio encoder, then project them into the LLM's representation space using a downsampling module. The resulting audio representation is concatenated with the embedded instruction prompt, and the combined input is passed to the LLM for causal generation.

## 3.1 Architecture

**Audio encoder** To extract semantically useful features from an audio signal, we rely on a pretrained audio feature extractor. Let $\mathbf{x}$ denote an audio signal, and $f_{AE}$ denote the audio feature extractor.

$$ \mathbf{h}_{AE} = f_{AE}(\mathbf{x}; \theta_{AE}) \in \mathbb{R}^{n \times d_{AE}}, \qquad (1) $$

where $d_{AE}$ is the hidden dimension of the audio encoder, and $n$ the sequence length of the model's output.

**Downsampling module** As highlighted in prior work [16, 19], audio signals typically yield much less compact latent representations compared to other modalities such as natural language. For example, the tokenized representation of a spoken sentence, when encoded by an audio foundation model, results in a substantially longer sequence than the corresponding textual representation. Consequently, directly concatenating the audio and text embeddings would introduce a disproportionate bias toward the audio modality, potentially skewing downstream tasks. To alleviate this issue, we downsample the audio representation $h_{AE}$ obtained in Eq. (1) to shorten its sequence length $n$.

We adopt a Query Pooling Mapper (QPMapper) module, previously shown to perform well on image modalities [57]. In a nutshell, this module adds $n_q$ learnable queries, $\mathbf{q} \in \mathbb{R}^{n_q \times d_{AE}}$, to the original sequence $\mathbf{h}_{AE}$. This concatenated sequence is then passed through a transformer encoder, and the output queries' representations are kept as the downsampled audio representation. Additionally, this downsampling module $g$ serves to project the audio features into the dimensional space of the language model's representations. Thus,

$$ \mathbf{h}_{ds} = g(\mathbf{h}_{AE}; \theta_{ds}) \in \mathbb{R}^{n_q \times d_{LLM}}, $$

where $n_q$ is a hyperparameter.

**Large language model** Let $f_{LLM}$ denote the large language model that will serve for the causal generation. It inputs a concatenated sequence comprised of: (i) the output of the downsampling module, $\mathbf{h}_{ds} \in \mathbb{R}^{n_q \times d_{LLM}}$, (ii) an embedded vectorized text prompt describing the task $\mathbf{p} \in \mathbb{R}^{n_p \times d_{LLM}}$ and (iii) possibly some textual information extracted from the audio signal also embedded and vectorized, e.g. a transcript, $\mathbf{z} \in \mathbb{R}^{n_z \times d_{LLM}}$. We further discuss in section 5.2 the information contained in $\mathbf{z}$ and its impact on the overall performance. Thus, the probability distribution over the output from EmQSLLM can be expressed as:

$$ \mathrm{EmQSLLM}(\mathbf{x}, \mathbf{p}, \mathbf{z}) = f_{LLM} ([\mathbf{h}_{ds}, \mathbf{p}, \mathbf{z}]; \theta_{LLM}). \qquad (2) $$

## 3.2 Optimization objective

For each task, we provide a set of 10 prompts, selected randomly for each sample in a batch. We display in sections A.2 and A.3 examples of each prompt for both SER and ASR. For each task $t \in [\mathrm{ASR}, \mathrm{SER}]$, we train the LLM for next-token prediction in an auto-regressive manner as in standard LLM training schedules.

Formally, let $\mathbf{p}^t$ denote a prompt for task $t$ uniformly sampled on $\mathcal{P}^t$ the set of prompts for task $t$, and $\mathcal{D}^t$ the set of datasets used for task $t$. Each sample can be represented as a tuple $(\mathbf{x}^t, \mathbf{p}^t, \mathbf{z}^t, \mathbf{y}^t)$, where $\mathbf{x}^t$ is the audio waveform, $\mathbf{p}^t$ the sampled prompt for the corresponding task, $\mathbf{z}^t$ some additional information relevant for task $t$ and $\mathbf{y}^t$ the label to be predicted. The probability of predicting the label $\mathbf{y}^t$ is modeled as

$$ p(\mathbf{y}^t | \mathbf{x}^t, \mathbf{p}^t, \mathbf{z}^t; \Theta) = \mathrm{EmQSLLM}(\mathbf{x}^t, \mathbf{p}^t, \mathbf{z}^t), \qquad (3) $$

where $\Theta = \{\theta_{AE}, \theta_{ds}, \theta_{LLM}\}$. The LLM can attend to all tokens in the concatenated sequence $[\mathbf{x}^t, \mathbf{p}^t, \mathbf{z}^t]$ and is trained to leverage the audio tokens to minimize the negative likelihood given the probability modeled in Eq. (3). The negative likelihood for target $\mathbf{y}^t$ for each sample in the training set is thus expressed as

$$ \mathcal{L}(\Theta) = - \log p(\mathbf{y}^t | \mathbf{x}^t, \mathbf{p}^t, \mathbf{z}^t; \Theta), \qquad (4) $$

and is minimized using gradient descent.
## 3.3 Data concatenation

We observed experimentally that *directly* concatenating the downsampled audio representation $h_{ds}$ with the textual prompt tokens $p$ that guides the LLM, i.e. forming the input as $[h_{ds}, p]$, can hinder the LLM's ability to effectively attend to the audio tokens during training. To address this, we prepend a natural language cue, Here are some audio tokens: , to the audio token sequence. This guiding phrase helps the model identify and contextualize the incoming audio tokens. An alternative strategy would be to enclose the audio tokens within special markers, such as `
`, `<|/audio|>`; however, this approach would necessitate modifying the LLM's embedding matrix, thereby introducing significant additional computational cost.

## 3.4 Task prompt

**Prompt selection** When prompting the LLM, we carefully describe the task to be performed and construct a pool of 20 distinct prompts per task. During training, each sample is randomly assigned a prompt uniformly sampled from the corresponding task-specific pool. For the Speech Emotion Recognition (SER) task, we use a closed-form prompt that explicitly enumerates the emotion categories from which the model should select its answer. To reduce positional bias and encourage generalization, the order of these categories is randomized for each sample. Empirically, we found that this randomization helps mitigate overfitting during training.

**Prompt format** We explicitly detail in the system prompt the expected format of the answer that the LLM should provide. Following previous work [16], for single task prediction, we ask for the answer to be formatted as `| TASK: <answer> |` where TASK can either be ASR or Emotion. For joint prediction, latter discussed in section 5.3, we ask for a similar formatting of the answer, in the format `| ASR: <answer> | Emotion: <answer> |`.

## 3.5 Training curriculum

We rely on a three-stage curriculum learning framework. First, we train the model only on the ASR task following previous work [27, 16]. In this first phase (P1), the audio feature extractor and LLM are frozen, while the downsampling module's weights are the only components updated. This phase aims to learn an effective mapping from the audio representation space to the LLM's embedding space, allowing the model to leverage the LLM's semantic capabilities [16, 44]. In the second phase (P2), we introduce Low-Rank Adaptation (LoRA) adapters to the LLM and continue training on the ASR task. Here, both the downsampling module and the LLM (via LoRA) are fine-tuned jointly, enabling the model to begin adapting to audio-conditioned language tasks. Finally, in the last phase (P3) we introduce the SER objective and train the model to perform both ASR and emotion recognition simultaneously.

# 4 Experiments

## 4.1 Experimental settings

**Dataset** For ASR training, we rely on the Librispeech dataset [43] as well as MSP-Podcast [38] since the transcript is also provided. Regarding SER, we only rely on the MSP-Podcast dataset for training. We evaluate the ability of EmSLLM on the SER task on the test1 share of MSP-Podcast.

**Training settings** We use AdamW [37] as the optimizer with learning rate $5 \cdot 10^{-4}$ and weight decay 0.01. We also rely on linear scheduling with a warm-up on 10% of the phase's training steps. After the downsampling module warm-up phase, the scheduling is re-started when the LoRA adapters are added to the LLM. We use WavLM [9] as the audio feature extractor and Llama 3.2-3B-Instruct [39] as the foundation language model. The LoRA adapters are added to the following layers of the LLM: `[q proj, v proj, k proj, o proj, gate proj, up proj, down proj]`, while contrary to [16] we do not add LoRA adapters to the audio encoder and keep it frozen. LoRA adapter's rank is set to 8, with dropout 0.1 and $\alpha$ equal to 16. For the downsampling module, implemented as a QPMapper, we use 32 learnable queries, 2 transformer layers with 8 attention heads each, and an embedding dimension of 768. The output of the downsampling module is then mapped to the
<figure><img src="image_3.png" /><figcaption>Figure 2: **Performance Comparison.** Performance comparison with existing Audio-Language Models that perform speech emotion recognition, Qwen2-Audio-Instruct [12] (Qwen2-A), OASQA-LLM [44] (OASQAL), and SIFT-LLM [44]. We compare EmSLLM with existing methods on MSP-Podcast test1 and observe that with fewer parameters and significantly less training time, our model displays strong SER performance as it outperforms all methods except SIFT-LLM.</figcaption></figure>

dimension of the LLM using a learned linear layer. We set the effective batch size to 512 for all three phases. Epochs for phase P1 and P2 are set to 10, while phase P3 lasts for a maximum of 20 epochs. We rely on early stopping on the validation SER loss, and stop after two consecutive epochs without improvement. To enhance the performance of EmSLLM for emotion prediction, we progressively decrease the weight of the ASR task in the optimized loss. For that purpose, we rely on a simple linear scheduler that attributes an equal weight to both ASR and Emotion Recognition losses for the first epoch of P3 while progressively decreasing the weight of the ASR loss to 0 for the last epoch.

**Prompt setting** As further discussed in Section 5.2, we refer to EmSLLM as the variant trained with additional paralinguistic features and 1-shot example hinting provided in z. In contrast, EmSLLM-base denotes the baseline architecture trained with standard prompts and without any auxiliary information in z, as detailed in Sections A.2, A.3, and A.4. Both models are trained using identical hyperparameters. Also, for both EmSLLM and EmSLLM-base, inference is performed using joint decoding as detailed in section 5.3, where the model has access to the transcript when forming its prediction. We discuss the added value of including in z additional paralinguistic features and n-shot example hinting in section 5.

**Compute** Overall, including LoRA layers and the downsampling module, our model has 12.5M trainable parameters and 3.2B parameters overall. Our model is trained on 16 H100 Nvidia GPUs with PyTorch Lightning<fnref n="1" />.

**Benchmark** To ensure a rigorous evaluation, we benchmark our approach against existing Speech-Text LLMs that incorporate SER capabilities. For instance, we compare to SALMONN-7B [53], Qwen2-Audio-7B-Instruct [12], OASQA-LLM [44] and SIFT-LLM [44] using the test1 share of MSP-Podcast and the unweighted accuracy metric following previous work [12, 44, 53, 44]. Although related to our work, we are unable to include Speechverse [16] in our benchmark as their model is not publicly available, and they only provide their model’s performance for SER on the 4-class classification problem on MSP-Podcast. Regarding the remaining models to which we compare EmSLLM, we provide details on training hours in table 5 and observe that all models requires significantly more training hours than our proposed model.

## 4.2 Results

Figure 2 (left) illustrates the performance of EmSLLM in relation to existing methods, plotted against their respective parameter counts. Our results demonstrate that EmSLLM achieves no-

<footnote n="1">¹https://lightning.ai/</footnote>
table performance despite its smaller number of parameters. EmSLLM significantly outperforms SALMONN [53], Qwen2-Audio-Instruct [12], and OASQAL [44], while having less than half (3.2B) the number of parameters than competing methods (7B+).

While SIFT-LLM exhibits superior performance in emotion prediction, this advantage comes with substantially higher computational requirements, as highlighted in Table 5. We attribute the performance gap between EmSLLM and SIFT-LLM to two primary factors. First, the backbone LLM used in their approach, Qwen2.5-7B-instruct [47], has a significantly larger number of parameters than ours. This parameter advantage likely provides SIFT-LLM with greater representational capacity and more robust contextual understanding. Second, SIFT-LLM’s multi-task training regime exposes it to significantly more diverse data, enabling better cross-modal feature learning and more generalizable representations. This multi-task approach may create synergy where emotion recognition benefits from related speech understanding tasks. Despite these advantages of SIFT-LLM, it is noteworthy that EmSLLM achieves competitive performance while maintaining a significantly smaller parameter footprint, suggesting greater computational efficiency as shown on Figure 2 (right).

# 5 Discussion

## 5.1 Large Language Model

To investigate the impact of both the LLM’s architecture and training strategy, as well as parameter count, we compare the performance obtained by EmSLLM when using Qwen3-4B [64] and Llama 3.2-3B-Instruct [39] as $f_{LLM}$.

## 5.2 Additional features

Previous work [34] has demonstrated that adding paralinguistic audio features may boost emotion recognition using text-only LLMs. [34] only rely on the audio transcript and curated prompts for emotion recognition, in which case including additional paralinguistic information in the audio logically boosts the performance for emotion recognition. In our case, some paralinguistic information is likely already contained in the audio tokens.

Nevertheless, we investigate whether adding this information in the form of textual tokens in z might enhance our model’s performance. We include the following paralinguistic features: loudness, average pitch, pitch range, jitter and shimmer. Also, following [34] we include in z the gender of the speaker.

Rather than directly providing the value of the features, we provide the binned paralinguistic features in three classes ['low', 'medium', 'high'] that each represent a third of the values based on the training set.

We include those tokens by sampling among 5 introductory

sentences and randomizing the order in which the paralinguistic features are provided. We display in table 1, the performance of EmSLLM-base when trained with z including the additional features and with an empty z. We observe that including paralinguistic features in natural language inputs improves the performance of emotion prediction, increasing accuracy from 45.8% to 46.9%, a gain of 1.1 percentage points.

<table><caption>Table 1: Performance comparison between performance with and without paralinguistic features in natural languages on test1 from MSP-Podcast [38].</caption><thead><tr><td>Add. features</td><td>Accuracy (↑)</td></tr></thead><tbody><tr><td>×</td><td>0.458</td></tr><tr><td>✓</td><td><b>0.469</b></td></tr></tbody></table>

<table><caption>Table 2: Performance comparison between n-shot hinting on test1 from MSP-Podcast [38].</caption><thead><tr><td>Hint</td><td>Accuracy (↑)</td></tr></thead><tbody><tr><td>0-shot</td><td>0.458</td></tr><tr><td>1-shot</td><td>0.473</td></tr><tr><td>2-shot</td><td><b>0.474</b></td></tr></tbody></table>

**Few-shot format hinting** We also investigate whether providing during phase P3, n examples of the expected output structure in the user prompt, might help enhance the performance of EmSLLM-base. See appendix A.6 for an example of such prompt strategy. We provide in table 2 the performance of EmSLLM-base trained with z including n examples in the user prompt for n ∈ [0, 1, 2]. We chose to keep n small as we expect the marginal gain to be quite small for higher values while increasing the computational cost. We observe a significant difference between EmSLLM-base without any hint (0-shot) and its performance when enhanced with the 1-shot

and 2-shot hinting strategies as they display a respective gain of 1.5 and 1.6 percentage points over
the 0-shot approach. Since 1-shot and 2-shot hinting provide similar performance, we chose to keep 1-shot hinting in our main approach as it involves a lower computational cost.

## 5.3 Joint prediction

**Training** During phase P3 training, when the emotion is available for a sample, we provide a prompt that asks for joint prediction. In other words, the model is asked to perform simultaneously the ASR and SER task. We believe that this could only be beneficial as it ensures that the model uses both semantic, linguistic and paralinguistic features to form its emotion prediction. See appendix A.4 for an illustrative example.

<table><caption>Table 3: **Prompt Strategies.** Performance comparison of different prompting strategies during inference on the test1 split of MSP-Podcast [38].</caption><thead><tr><td>Prompt Strategy</td><td>Accuracy (↑)</td></tr></thead><tbody><tr><td>SER-only</td><td>0.417</td></tr><tr><td>Prompt-hint</td><td>0.431</td></tr><tr><td>EmQSLLM</td><td>0.497</td></tr></tbody></table>

**Inference** The first approach, referred to as SER-only, involves prompting the LLM exclusively for the SER task without any auxiliary information. To assess the utility of providing transcript information as contextual hints, we explore two additional approaches. First, we consider providing the transcript in the user prompt, introduced by "Use the following transcript to help you predict the emotion:" , we refer to this approach as Prompt-hint. Note that this approach was never used during the training phase. Second, we consider providing the same user prompt as the ones seen during training, but

we provide the beginning of the answer to the LLM and ask it to complete it. In other words, we ask the LLM to perform auto-regressive generation where its context contains the user prompt followed by the beginning of the assistant's answer, "| ASR: <transcript> | Emotion:" . We refer to this last approach as EmQSLLM. See section A.7 for examples of such prompts. Comparison of the performance between these approaches is displayed in Table 3.

Overall, we find that incorporating the transcript into the LLM's input significantly enhances SER accuracy. Both Prompt-hint and EmQSLLM outperform the SER-only baseline. However, providing the transcript within the assistant's response, as done in EmQSLLM, proves more effective than embedding it in the user prompt. Specifically, EmQSLLM achieves an accuracy of 0.497, compared to 0.431 for Prompt-hint. The LLM's unfamiliarity with user prompts containing transcripts, since it was not exposed to such prompts during training, likely contributes to this performance gap.

## 5.4 Audio Encoder

We assess the impact of the choice of backbone audio encoder by replacing WavLM [9] with Robust wav2vec 2.0 [25]. The alternative model is trained using the same hyperparameters, training curriculum, and prompting strategy as the original configuration. Table 4 compares the performance of EmQSLLM when using WavLM versus Robust wav2vec 2.0 as the pretrained audio encoder. While both encoders yield strong results, WavLM consistently outperforms Robust wav2vec 2.0 in this setup. However, it is important to note that the hyperparameters were optimized for WavLM and may not be ideal for wav2vec 2.0, potentially limiting the latter's performance.

<table><caption>Table 4: **Audio Encoder.** Performance comparison between pretrained audio encoders in EmQSLLM on test1 from MSP-Podcast [38].</caption><thead><tr><td>Audio Encoder</td><td>Accuracy (↑)</td></tr></thead><tbody><tr><td>wav2vec 2.0</td><td>0.471</td></tr><tr><td>wavLM</td><td>0.497</td></tr></tbody></table>

# 6 Conclusion

This paper introduced EmQSLLM, a novel and computationally efficient approach for speech emotion recognition (SER) that effectively integrates audio and text modalities using LLMs. Our experimental results on standard SER benchmarks demonstrate that EmQSLLM outperforms most existing Speech-Text LLMs in the literature while requiring significantly fewer parameters and less training time. This highlights EmQSLLM's effectiveness and paves the way for more efficient and privacy-preserving applications in areas like human-computer interaction and mental health monitoring.
**Limitations and future work** While EmQSLLM shows strong performance and efficiency, it is still surpassed by SIFT-LLM. This is likely due to SIFT-LLM benefiting from a larger backbone LLM and exposure to a significantly greater volume of multi-task training data. This suggests that even with parameter efficiency, the scale of the base LLM and training data diversity remain crucial. Furthermore, achieving true on-device deployment for multimodal LLMs still requires substantially reducing the overall parameter count. Future work could explore the integration of smaller backbone LLMs, model compression techniques such as quantization, or extending EmQSLLM to handle a broader range of multimodal inputs.

## Acknowledgement

This work was granted access to the HPC resources of IDRIS under the allocation 2025-AD010616049 made by GENCI.

## References

[1] Ali J. Alsaad, Yusra Azhar, and Yasser Al Nasser. *Depression in Children*. StatPearls Publishing, Treasure Island (FL), 2023.

[2] Mahmoud Assran, Quentin Duval, Ishan Misra, Piotr Bojanowski, Pascal Vincent, Michael Rabbat, Yann LeCun, and Nicolas Ballas. Self-supervised learning from images with a joint-embedding predictive architecture. In *Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)*, pages 15619–15629, 6 2023.

[3] Alexei Baevski, Steffen Schneider, and Michael Auli. vq-wav2vec: Self-supervised learning of discrete speech representations. In *International Conference on Learning Representations*, 2020.

[4] Alexei Baevski, Henry Zhou, Abdel rahman Mohamed, and Michael Auli. wav2vec 2.0: A framework for self-supervised learning of speech representations. *ArXiv*, abs/2006.11477, 2020.

[5] Adrien Bardes, Jean Ponce, and Yann LeCun. VICReg: Variance-invariance-covariance regularization for self-supervised learning. In *International Conference on Learning Representations*, 2022.

[6] Adrien Bardes, Quentin Garrido, Jean Ponce, Xinlei Chen, Michael Rabbat, Yann LeCun, Mido Assran, and Nicolas Ballas. Revisiting feature prediction for learning visual representations from video. *Transactions on Machine Learning Research*, 2024.

[7] Florian Bordes, Richard Yuanzhe Pang, Anurag Ajay, Alexander C. Li, Adrien Bardes, Suzanne Petryk, Oscar Mañas, Zhiqiu Lin, Anas Mahmoud, Bargav Jayaraman, Mark Ibrahim, Melissa Hall, Yunyang Xiong, Jonathan Lebensold, Candace Ross, Srihari Jayakumar, Chuan Guo, Diane Bouchacourt, Haider Al-Tahan, Karthik Padthe, Vasu Sharma, Hu Xu, Xiaoqing Ellen Tan, Megan Richards, Samuel Lavoie, Pietro Astolfi, Reyhane Askari Hemmat, Jun Chen, Kushal Tirumala, Rim Assouel, Mazda Moayeri, Arjang Talattof, Kamalika Chaudhuri, Zechun Liu, Xilun Chen, Quentin Garrido, Karen Ullrich, Aishwarya Agrawal, Kate Saenko, Asli Celikyilmaz, and Vikas Chandra. An introduction to vision-language modeling, 2024.

[8] Mathilde Caron, Ishan Misra, Julien Mairal, Priya Goyal, Piotr Bojanowski, and Armand Joulin. Unsupervised learning of visual features by contrasting cluster assignments. *Advances in neural information processing systems*, 33:9912–9924, 2020.

[9] Sanyuan Chen, Chengyi Wang, Zhengyang Chen, Yu Wu, Shujie Liu, Zhuo Chen, Jinyu Li, Naoyuki Kanda, Takuya Yoshioka, Xiong Xiao, Jian Wu, Long Zhou, Shuo Ren, Yanmin Qian, Yao Qian, Jian Wu, Michael Zeng, and Furu Wei. Wavlm: Large-scale self-supervised pre-training for full stack speech processing. *CoRR*, abs/2110.13900, 2021.

[10] Sanyuan Chen, Chengyi Wang, Zhengyang Chen, Yu Wu, Shujie Liu, Zhuo Chen, Jinyu Li, Naoyuki Kanda, Takuya Yoshioka, Xiong Xiao, Jian Wu, Long Zhou, Shuo Ren, Yanmin Qian, Yao Qian, Micheal Zeng, and Furu Wei. Wavlm: Large-scale self-supervised pre-training for full stack speech processing. *IEEE Journal of Selected Topics in Signal Processing*, 16:1505–1518, 2021.
[11] Sanyuan Chen, Yu Wu, Chengyi Wang, Shujie Liu, Daniel Tompkins, Zhuo Chen, Wanxiang Che, Xiangzhan Yu, and Furu Wei. BEATs: Audio pre-training with acoustic tokenizers. In Andreas Krause, Emma Brunskill, Kyunghyun Cho, Barbara Engelhardt, Sivan Sabato, and Jonathan Scarlett, editors, *Proceedings of the 40th International Conference on Machine Learning*, volume 202 of *Proceedings of Machine Learning Research*, pages 5178–5193. PMLR, 2023.

[12] Yunfei Chu, Jin Xu, Xiaohuan Zhou, Qian Yang, Shiliang Zhang, Zhijie Yan, Chang Zhou, and Jingren Zhou. Qwen-audio: Advancing universal audio understanding via unified large-scale audio-language models. *arXiv preprint arXiv:2311.07919*, 2023.

[13] Yunfei Chu, Jin Xu, Qian Yang, Haojie Wei, Xipin Wei, Zhifang Guo, Yichong Leng, Yuanjun Lv, Jinzheng He, Junyang Lin, Chang Zhou, and Jingren Zhou. Qwen2-audio technical report, 2024.

[14] Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Yi Tay, William Fedus, Yunxuan Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, Albert Webson, Shixiang Shane Gu, Zhuyun Dai, Mirac Suzgun, Xinyun Chen, Aakanksha Chowdhery, Alex Castro-Ros, Marie Pellat, Kevin Robinson, Dasha Valter, Sharan Narang, Gaurav Mishra, Adams Yu, Vincent Zhao, Yanping Huang, Andrew Dai, Hongkun Yu, Slav Petrov, Ed H. Chi, Jeff Dean, Jacob Devlin, Adam Roberts, Denny Zhou, Quoc V. Le, and Jason Wei. Scaling instruction-finetuned language models, 2022.

[15] Yu-An Chung, Wei-Ning Hsu, Hao Tang, and James Glass. An unsupervised autoregressive model for speech representation learning. In *Interspeech*, 2019.

[16] Nilaksh Das, Saket Dingliwal, S. Ronanki, Rohit Paturi, David Huang, Prashant Mathur, Jie Yuan, Dhanush Bekal, Xing Niu, Sai Muralidhar Jayanthi, Xilai Li, Karel Mundnich, Monica Sunkara, Sundararajan Srinivasan, Kyu J Han, and Katrin Kirchhoff. Speechverse: A large-scale generalizable audio language model. *ArXiv*, abs/2405.08295, 2024.

[17] Hongwei Ding and Yang Zhang. Speech prosody in mental disorders. *Annual Review of Linguistics*, 9(Volume 9, 2023):335–355, 2023. ISSN 2333-9691.

[18] Gerard Donahue and Ehsan Elhamifar. Learning to predict activity progress by self-supervised video alignment. In *Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)*, pages 18667–18677, June 2024.

[19] Alexandre Défossez, Laurent Mazaré, Manu Orsini, Amélie Royer, Patrick Pérez, Hervé Jégou, Edouard Grave, and Neil Zeghidour. Moshi: a speech-text foundation model for real-time dialogue, 2024.

[20] Yassir Fathullah, Chunyang Wu, Egor Lakomkin, Junteng Jia, Yuan Shangguan, Ke Li, Jinxi Guo, Wenhan Xiong, Jay Mahadeokar, Ozlem Kalinli, Christian Fuegen, and Mike Seltzer. Prompting large language models with speech recognition abilities. In *ICASSP 2024 - 2024 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)*, pages 13351–13355, 2024.

[21] Linda Fischer-Grote, Oswald D Kothgassner, and Anna Felnhofer. Risk factors for problematic smartphone use in children and adolescents: a review of existing literature. *neuropsychiatrie*, 33(4):179, 2019.

[22] Jean-Bastien Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre H. Richmond, Elena Buchatskaya, Carl Doersch, Bernardo Avila Pires, Zhaohan Daniel Guo, Mohammad Gheshlaghi Azar, Bilal Piot, Koray Kavukcuoglu, Rémi Munos, and Michal Valko. Bootstrap your own latent a new approach to self-supervised learning. In *Proceedings of the 34th International Conference on Neural Information Processing Systems*, NIPS '20, Red Hook, NY, USA, 2020. Curran Associates Inc.

[23] Lasse Hansen, Yan-Ping Zhang, Detlef Wolf, Konstantinos Sechidis, Nicolai Ladegaard, and Riccardo Fusaroli. A generalizable speech emotion recognition model reveals depression and remission. *Acta Psychiatrica Scandinavica*, 145(2):186–199, 2022.
[24] Wei-Ning Hsu, Benjamin Bolte, Yao-Hung Hubert Tsai, Kushal Lakhotia, Ruslan Salakhutdinov, and Abdelrahman Mohamed. Hubert: Self-supervised speech representation learning by masked prediction of hidden units. *IEEE/ACM Trans. Audio, Speech and Lang. Proc.*, 29:3451–3460, October 2021.

[25] Wei-Ning Hsu, Anuroop Sriram, Alexei Baevski, Tatiana Likhomanenko, Qiantong Xu, Vineel Pratap, Jacob Kahn, Ann Lee, Ronan Collobert, Gabriel Synnaeve, and Michael Auli. Robust wav2vec 2.0: Analyzing domain shift in self-supervised pre-training. In *Interspeech 2021*, pages 721–725, 2021.

[26] Edward J Hu, yelong shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. LoRA: Low-rank adaptation of large language models. In *International Conference on Learning Representations*, 2022.

[27] Shujie Hu, Long Zhou, Shujie Liu, Sanyuan Chen, Lingwei Meng, Hongkun Hao, Jing Pan, Xunying Liu, Jinyu Li, Sunit Sivasankaran, Linquan Liu, and Furu Wei. WavLLM: Towards robust and adaptive speech large language model. In *Findings of the Association for Computational Linguistics: EMNLP 2024*, pages 4552–4572, Miami, Florida, USA, November 2024. Association for Computational Linguistics.

[28] Karim M. Ibrahim, Antony Perzo, and Simon Leglaive. Towards improving speech emotion recognition using synthetic data augmentation from emotion conversion. In *ICASSP 2024 - 2024 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)*, pages 10636–10640, 2024.

[29] Zhifeng Kong, Arushi Goel, Rohan Badlani, Wei Ping, Rafael Valle, and Bryan Catanzaro. Audio flamingo: A novel audio language model with few-shot learning and dialogue abilities. In *Forty-first International Conference on Machine Learning*, 2024.

[30] Gukyeong Kwon, Zhaowei Cai, Avinash Ravichandran, Erhan Bas, Rahul Bhotika, and Stefano Soatto. Masked vision and language modeling for multi-modal representation learning. In *The Eleventh International Conference on Learning Representations*, 2023.

[31] Samuel Lavoie, Polina Kirichenko, Mark Ibrahim, Mido Assran, Andrew Gordon Wilson, Aaron Courville, and Nicolas Ballas. Modeling caption diversity in contrastive vision-language pretraining. In Ruslan Salakhutdinov, Zico Kolter, Katherine Heller, Adrian Weller, Nuria Oliver, Jonathan Scarlett, and Felix Berkenkamp, editors, *Proceedings of the 41st International Conference on Machine Learning*, volume 235 of *Proceedings of Machine Learning Research*, pages 26070–26084. PMLR, 21–27 Jul 2024.

[32] Kyungeun Lee, Ye Seul Sim, Hyeseung Cho, Moonjung Eo, Suhee Yoon, Sanghyu Yoon, and Woohyung Lim. Binning as a pretext task: Improving self-supervised learning in tabular domains. In *Proceedings of the 41st International Conference on Machine Learning*, volume 235 of *Proceedings of Machine Learning Research*, pages 26929–26947. PMLR, 21–27 Jul 2024.

[33] Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. Blip-2: bootstrapping language-image pre-training with frozen image encoders and large language models. In *Proceedings of the 40th International Conference on Machine Learning*, ICML'23. JMLR.org, 2023.

[34] Yuanchao Li, Yuan Gong, Chao-Han Huck Yang, Peter Bell, and Catherine Lai. Revise, reason, and recognize: Llm-based emotion recognition via emotion-specific prompts and asr error correction. In *ICASSP 2025 - 2025 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)*, pages 1–5, 2025.

[35] Alexander H. Liu, Yu-An Chung, and James R. Glass. Non-autoregressive predictive coding for learning speech representations from local dependencies. In *Interspeech*, 2020.

[36] Andy T. Liu, Shang-Wen Li, and Hung yi Lee. Tera: Self-supervised learning of transformer encoder representation for speech. *IEEE/ACM Transactions on Audio, Speech, and Language Processing*, 29:2351–2366, 2020.
[37] Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In *International Conference on Learning Representations*, 2019.

[38] Reza Lotfian and Carlos Busso. Building naturalistic emotionally balanced speech corpus by retrieving emotional speech from existing podcast recordings. *IEEE Transactions on Affective Computing*, 10(4):471–483, 2019.

[39] AI @Meta. The llama 3 herd of models, 2024.

[40] Daisuke Niizumi, Daiki Takeuchi, Yasunori Ohishi, Noboru Harada, and Kunio Kashino. Byol for audio: Self-supervised learning for general-purpose audio representation. In *2021 International Joint Conference on Neural Networks (IJCNN)*, pages 1–8, 2021.

[41] Daisuke Niizumi, Daiki Takeuchi, Yasunori Ohishi, Noboru Harada, and Kunio Kashino. Byol for audio: Exploring pre-trained general-purpose audio representations. *IEEE/ACM Transactions on Audio, Speech, and Language Processing*, 31:137–151, 2023.

[42] Jay A. Olson, Dasha A. Sandra, Élissa S. Colucci, Alain Al Bikaii, Denis Chmoulevitch, Johnny Nahas, Amir Raz, and Samuel P.L. Veissière. Smartphone addiction is increasing across the world: A meta-analysis of 24 countries. *Computers in Human Behavior*, 129:107138, 2022. ISSN 0747-5632.

[43] Vassil Panayotov, Guoguo Chen, Daniel Povey, and Sanjeev Khudanpur. Librispeech: An asr corpus based on public domain audio books. In *2015 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)*, pages 5206–5210, 2015.

[44] Prabhat Pandey, Rupak Vignesh Swaminathan, K V Vijay Girish, Arunasish Sen, Jian Xie, Grant P. Strimel, and Andreas Schwarz. Sift-50m: A large-scale multilingual dataset for speech instruction fine-tuning, 2025.

[45] Leonardo Pepino, Pablo Riera, and Luciana Ferrer. Emotion Recognition from Speech Using wav2vec 2.0 Embeddings. In *Interspeech*, pages 3400–3404, 2021.

[46] Katarzyna Pisanski, Valentina Cartei, Carolyn McGettigan, Jordan Raine, and David Reby. Voice modulation: A window into the origins of human vocal control? *Trends in Cognitive Sciences*, 20(4):304–318, 2016. ISSN 1364-6613.

[47] Qwen, .; An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, Huan Lin, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxi Yang, Jingren Zhou, Junyang Lin, Kai Dang, Keming Lu, Keqin Bao, Kexin Yang, Le Yu, Mei Li, Mingfeng Xue, Pei Zhang, Qin Zhu, Rui Men, Runji Lin, Tianhao Li, Tianyi Tang, Tingyu Xia, Xingzhang Ren, Xuancheng Ren, Yang Fan, Yang Su, Yichang Zhang, Yu Wan, Yuqiong Liu, Zeyu Cui, Zhenru Zhang, and Zihan Qiu. Qwen2.5 technical report, 2025.

[48] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning transferable visual models from natural language supervision. In *Marina Meila and Tong Zhang, editors, Proceedings of the 38th International Conference on Machine Learning*, volume 139 of *Proceedings of Machine Learning Research*, pages 8748–8763. PMLR, 18–24 Jul 2021.

[49] Alec Radford, Jong Wook Kim, Tao Xu, Greg Brockman, Christine Mcleavey, and Ilya Sutskever. Robust speech recognition via large-scale weak supervision. In *Andreas Krause, Emma Brunskill, Kyunghyun Cho, Barbara Engelhardt, Sivan Sabato, and Jonathan Scarlett, editors, Proceedings of the 40th International Conference on Machine Learning*, volume 202 of *Proceedings of Machine Learning Research*, pages 28492–28518. PMLR, 23–29 Jul 2023.

[50] Ali Razavi, Aaron van den Oord, and Oriol Vinyals. Generating diverse high-fidelity images with vq-vae-2. In *H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Buc, E. Fox, and R. Garnett, editors, Advances in Neural Information Processing Systems*, volume 32. Curran Associates, Inc., 2019.
[51] Steffen Schneider, Alexei Baevski, Ronan Collobert, and Michael Auli. wav2vec: Unsupervised pre-training for speech recognition. CoRR, abs/1904.05862, 2019.

[52] Amanpreet Singh, Ronghang Hu, Vedanuj Goswami, Guillaume Couairon, Wojciech Galuba, Marcus Rohrbach, and Douwe Kiela. Flava: A foundational language and vision alignment model. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 15638–15650, June 2022.

[53] Changli Tang, Wenyi Yu, Guangzhi Sun, Xianzhao Chen, Tian Tan, Wei Li, Lu Lu, Zejun MA, and Chao Zhang. SALMONN: Towards generic hearing abilities for large language models. In The Twelfth International Conference on Learning Representations, 2024.

[54] Chameleon Team. Chameleon: Mixed-modal early-fusion foundation models, 2024.

[55] Hugo Thimonier, José Lucas De Melo Costa, Fabrice Popineau, Arpad Rimmel, and Bich-Liên Doan. T-JEPA: Augmentation-free self-supervised learning for tabular data. In The Thirteenth International Conference on Learning Representations, 2025.

[56] Maria Tsimpoukelli, Jacob Menick, Serkan Cabi, S. M. Ali Eslami, Oriol Vinyals, and Felix Hill. Multimodal few-shot learning with frozen language models. In A. Beygelzimer, Y. Dauphin, P. Liang, and J. Wortman Vaughan, editors, Advances in Neural Information Processing Systems, 2021.

[57] Théophane Vallaeyaş, Mustafa Shukor, Matthieu Cord, and Jakob Verbeek. Improved baselines for data-efficient perceptual augmentation of LLMs, 2024.

[58] Aaron van den Oord, Oriol Vinyals, and koray kavukcuoglu. Neural discrete representation learning. In I. Guyon, U. Von Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 30. Curran Associates, Inc., 2017.

[59] Johannes Wagner, Andreas Triantafyllopoulos, Hagen Wierstorf, Maximilian Schmitt, Felix Burkhardt, Florian Eyben, and Björn W. Schuller. Dawn of the transformer era in speech emotion recognition: Closing the valence gap. IEEE Transactions on Pattern Analysis and Machine Intelligence, 45(9):10745–10759, 2023.

[60] Yingzhi Wang, Abdelmoumene Boumadane, and Abdelwahab Heba. A fine-tuned wav2vec 2.0/hubert benchmark for speech emotion recognition, speaker verification and spoken language understanding. CoRR, abs/2111.02735, 2021.

[61] Yibing Wei, Abhinav Gupta, and Pedro Morgado. Towards latent masked image modeling for self-supervised visual representation learning. In Aleš Leonardis, Elisa Ricci, Stefan Roth, Olga Russakovsky, Torsten Sattler, and Gül Varol, editors, Computer Vision – ECCV 2024, pages 1–17, 2025.

[62] Jian Wu, Yashesh Gaur, Zhuo Chen, Long Zhou, Yimeng Zhu, Tianrui Wang, Jinyu Li, Shujie Liu, Bo Ren, Linquan Liu, and Yu Wu. On decoder-only architecture for speech-to-text and large language model integration, 2023.

[63] Jing Wu, Suiyao Chen, Qi Zhao, Renat Sergazinov, Chen Li, Shengjie Liu, Chongchao Zhao, Tianpei Xie, Hanqing Guo, Cheng Ji, Daniel Cociorva, and Hakan Brunzell. Switchtab: Switched autoencoders are effective tabular learners. Proceedings of the AAAI Conference on Artificial Intelligence, 38(14):15924–15933, Mar. 2024.

[64] An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, Chujie Zheng, Dayiheng Liu, Fan Zhou, Fei Huang, Feng Hu, Hao Ge, Haoran Wei, Huan Lin, Jialong Tang, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxi Yang, Jing Zhou, Jingren Zhou, Junyang Lin, Kai Dang, Keqin Bao, Kexin Yang, Le Yu, Lianghao Deng, Mei Li, Mingfeng Xue, Mingze Li, Pei Zhang, Peng Wang, Qin Zhu, Rui Men, Ruize Gao, Shixuan Liu, Shuang Luo, Tianhao Li, Tianyi Tang, Wenbiao Yin, Xingzhang Ren, Xinyu Wang, Xinyu Zhang, Xuancheng Ren, Yang Fan, Yang Su, Yichang Zhang, Yinger Zhang, Yu Wan, Yuqiong Liu, Zekun Wang, Zeyu Cui, Zhenru Zhang, Zhipeng Zhou, and Zihan Qiu. Qwen3 technical report. arXiv preprint arXiv:2505.09388, 2025.
[65] Tsung-Hsien Yang, Chung-Hsien Wu, Kun-Yi Huang, and Ming-Hsiang Su. Detection of mood disorder using speech emotion profiles and lstm. In *2016 10th International Symposium on Chinese Spoken Language Processing (ISCSLP)*, pages 1–5, 2016.

[66] Jiahui Yu, Zirui Wang, Vijay Vasudevan, Legg Yeung, Mojtaba Seyedhosseini, and Yonghui Wu. Coca: Contrastive captioners are image-text foundation models. *Transactions on Machine Learning Research*, 2022.

[67] Wenyi Yu, Changli Tang, Guangzhi Sun, Xianzhao Chen, Tian Tan, Wei Li, Lu Lu, Zejun Ma, and Chao Zhang. Connecting speech encoder and large language model for asr. In *ICASSP 2024 - 2024 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)*, pages 12637–12641, 2024.

[68] Jure Zbontar, Li Jing, Ishan Misra, Yann LeCun, and Stéphane Deny. Barlow twins: Self-supervised learning via redundancy reduction. In *International conference on machine learning*, pages 12310–12320. PMLR, 2021.

[69] Bingzhao Zhu, Xingjian Shi, Nick Erickson, Mu Li, George Karypis, and Mahsa Shoaran. XTab: cross-table pretraining for tabular transformers. In *Proceedings of the 40th International Conference on Machine Learning*, ICML’23, 2023.

[70] Deyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li, and Mohamed Elhoseiny. Minigpt-4: Enhancing vision-language understanding with advanced large language models, 2023.
## A Prompts

We provide in this section a more comprehensive description of the prompt structures used to train EmSLLM.

### A.1 System prompt

We carefully design a system that details to the LLM the task at hand, while providing useful information about the expected input and output structures. We provide hereafter a snippet of the curated system prompt.

System prompt

```json
{
  "role":
    "system",
  "content":
```

"You are a highly capable assistant specialized in audio processing tasks.

You receive inputs containing audio token representations followed by text instructions, and return structured answer.

You may be asked to perform:

1. **Automatic Speech Recognition (ASR)** — transcribe the spoken content.

2. **Speech Emotion Recognition (SER)** — identify the emotion expressed in the audio.

Follow one of the two output formats:

- For ASR-only tasks:

`' | ASR: <transcription> |'`

- For SER-only tasks:

`' | Emotion: <emotion code> |'`

For tasks involving both ASR and SER, use the following format:

`' | ASR: <transcription> | Emotion: <emotion code> |'`

Emotion must be provided as a single letter chosen from the following emotion

codes:

- A: Angry

- S: Sad

- H: Happy

- U: Surprise

- F: Fear

- D: Disgust

- C: Contempt

- N: Neutral

- O: Other

(...)"

```json
}
```

### A.2 Automatic Speech Recognition (ASR) prompt

As previously discussed in the main section of the paper, for each sample we select a prompt among a curated selection of prompts detailing the expected task at hand. We provide hereafter an example an ASR prompt used during training.
### ASR prompt

```json
{
  "role":
    "user",
  "content":
    "You will now perform the following audio-based task.
    Task: **Automatic Speech Recognition (ASR)**.
    Transcribe the preceding audio into written text."
}
```

## A.3 Speech Emotion Recognition (SER) prompt

We provide hereafter an example of a vanilla SER prompt used during training.

### SER prompt

```json
{
  "role":
    "user",
  "content":
    "You will now perform the following audio-based task.
    Task: **Speech Emotion Recognition**.
    Classify the tone of the speaker in the preceding audio."
}
```

## A.4 Joint decoding prompt

We provide hereafter an example of a vanilla joint decoding prompt used during training.

### Joint decoding prompt

```json
{
  "role":
    "user",
  "content":
    "Perform the following audio-based tasks in the order as described.
    1. Task: **Automatic Speech Recognition (ASR)**.
       Identify and write down the words spoken in the preceding audio.
    2. Task: **Speech Emotion Recognition**.
       Analyze the audio and determine the emotional state of the speaker."
}
```

## A.5 Supplementary features hinting

We provide a variety of supplementary features to guide the LLM in its prediction. We include features ranging from the gender of the person speaking, to some paralinguistic features. Supplementary feature can be combined or taken in isolation. We hereafter provide an example of how the supplementary features are included in the prompts as shown in sections A.2, A.3 and A.4.
## Supplementary feature prompt

```json
{
  "role":
    "user",
  "content":
    "(...)"
```

The speaker in this audio is $gender.

Here's a breakdown of paralinguistic cues in the audio:

- loudness: 'low'

- pitch: 'medium'

- pitch range: 'high'

- jitter: 'low'

- shimmer: 'high'

(...)"

}

## A.6 Example hinting

We also include the possibility of including *n* examples of answers in the format we are expecting. We include hereafter an example for example hinting in the case of joint decoding. Note that we also include this for SER-only and ASR-only prompts.

### Few shot example prompt

```json
{
  "role":
    "user",
  "content":
    "(...)"
```

Here are some examples of the expected output:

- 'I ASR: You're such a disappointment. | Emotion: C |'

- 'I ASR: I'm speechless... didn't expect that. | Emotion: U |',

- 'I ASR: I can't shake this feeling of dread. | Emotion: F |'"

(...)"

}

## A.7 Transcript hinting

As mentioned in section 5.3, in inference we consider two alternatives to include the transcript as a hint to guide the LLM to make its prediction on the emotion class.

**Prompt-hint** First, we consider including the transcript in the user prompt with an introductory sentence. This approach is never used in training.

### Prompt-hint prompt

```json
{
  "role":
    "user",
  "content":
    "(...)"
```

Use the following transcript to help you classify the emotion:

<transcript>
(...)

}
**EmoSLLM** For our main pipeline, we consider an alternative that most resembles the task that the LLM performs in the training phase. We use as a user prompt the joint-decoding prompt as detailed in section A.4, and ask the LLM to auto-regressively generate tokens given the first part of the answer that contains the true transcript.

EmoSLLM prompt

```json
{
  "role":
    "user",
  "content":
    <Joint decoding prompt>
  },
  {
    "role":
      "assistant",
    "content":
      "| ASR: <transcript> | Emotion:"
  }
}
```

## B Compute

We display in table 5 the compute hours required to train the different models to which we compare our method. The displayed hours are directly extracted from the original papers.

<table><caption>Table 5: Comparison of the total training hours and training hours specific to SER.</caption><thead><tr><th>Model</th><th>Total training hours</th><th>SER training hours</th></tr></thead><tbody><tr><td>SALMONN [53]</td><td>~ 4400</td><td>5</td></tr><tr><td>Qwen2-Audio-7B-Instruct [12]</td><td>~ 146500</td><td>~ 1000</td></tr><tr><td>SIFT-LLM [44]</td><td>173483</td><td>237</td></tr><tr><td>EmoSLLM (Ours)</td><td>~ 320</td><td>~ 180</td></tr></tbody></table>