OmniCharacter: Towards Immersive Role-Playing Agents with Seamless Speech-Language Personality Interaction
Authors: Haonan Zhang, Run Luo, Xiong Liu, Yuchuan Wu, Ting-En Lin, Pengpeng Zeng, QIANG QU, Feiteng Fang, Min Yang, Lianli Gao, Jingkuan Song, Fei Huang, Yongbin Li
Accepted at: acl25

OmniCharacter: Towards Immersive Role-Playing Agents with Seamless Speech-Language Personality Interaction

Haonan Zhang1,2*, Run Luo2,4,5*, Xiong Liu2*, Yuchuan Wu2, Ting-En Lin2, Pengpeng Zeng1, Qiang Qu4,5, Feiteng Fang2,5, Min Yang4,5, Lianli Gao3, Jingkuan Song1‡, Fei Huang2, Yongbin Li2‡
1Tongji University 2Tongyi Laboratory 3Independent Researcher 4University of Chinese Academy of Sciences 5Shenzhen Institute of Advanced Technology, Chinese Academy of Sciences {zchiowal, jingkuan.song}@gmail.com shuide.lyb@alibaba-inc.com

Abstract

Role-Playing Agents (RPAs), benefiting from large language models, is an emerging interactive AI system that simulates roles or characters with diverse personalities. However, existing methods primarily focus on mimicking dialogues among roles in textual form, neglecting the role’s voice traits (e.g., voice style and emotions) as playing a crucial effect in interaction, which tends to be more immersive experiences in realistic scenarios. Towards this goal, we propose OmniCharacter, a first seamless speech-language personality interaction model to achieve immersive RPAs with low latency. Specifically, OmniCharacter enables agents to consistently exhibit role-specific personality traits and vocal traits throughout the interaction, enabling a mixture of speech and language responses. To align the model with speech-language scenarios, we construct a dataset named OmniCharacter-10K, which involves more distinctive characters (20), richly contextualized multi-round dialogue (10K), and dynamic speech response (135K). Experimental results showcase that our method yields better responses in terms of both content and style compared to existing RPAs and mainstream speech-language models, with a response latency as low as 289ms.1

1 Introduction

The rapid advancement of Large language models (LLMs) (Wang et al., 2024b; Park et al., 2023; OpenAI, 2022; Li et al., 2022) has demonstrated strong potential in interactive AI, enabling more natural and engaging human-computer interactions (Luo et al., 2024b; Zhang et al., 2023b). One of the most promising and widely adopted research directions is the development of role-playing agents (RPAs) (Wang et al., 2024a; Tu et al., 2024; Shao et al., 2023), which have been applied in various domains, such as virtual assistants, AI-driven storytelling, and intelligent non-player characters (NPCs) in video games. The focus of RPAs allows LLMs to simulate diverse personas and mimic human-like behavior via predefined role profiles.

Despite significant advances in this field, most existing studies primarily concentrate on replicating dialogues solely in textual form, while often overlooking the role’s crucial voice traits such as tone, style, emotions, and pauses. The voice traits are important paralinguistic concepts that deeply reveal individual differences in conversations, enabling a more immersive and personalized role-playing interaction. As shown in Figure 1 (a), for the same user query, although both agents pro-

<table>
  <tr>
    <th>Agent A</th>
    <th>Agent B</th>
    <th>Agent C</th>
  </tr>
  <tr>
    <td>Name: Zhenghui<br>Age: 25<br>Gender: Male<br>Personality: Calm, Majestic.<br>Relationships: Raiden Shogun and his loyal partner who once ruled your nation.<br>Past Experiences: Emotions: Serious, Patient.</td>
    <td>Name: Kazuhira<br>Age: 25<br>Gender: Male<br>Personality: Smooth, Fast.<br>Relationships: Raiden Shogun and your brother died by her word.<br>Past Experiences: Emotions: Tired, Sad.</td>
    <td>Name: Reiner Shogun<br>Age: 25<br>Gender: Male<br>Personality: Cold, Indifferent.<br>Relationships: It is your true friend... You Miko is your friend.<br>Past Experiences: ...<br>Emotions: Cold, Indifferent.</td>
  </tr>
  <tr>
    <td colspan="3">Topics</td>
  </tr>
  <tr>
    <td colspan="3">Learning skills for effective hunting.</td>
  </tr>
  <tr>
    <td colspan="3">
      <b>(a) Text-based RPAs</b><br>
      I'm feeling stressed. Can't we just take a break?<br>
      (serious) Patience. (pause) Only with diligent training can your skills gradually improve.<br>
      I'm feeling stressed. Can't we just take a break?<br>
      (serious) Relax? If you fail to prepare, you are preparing to fail. One more time! (commanding)<br>
      Hard to explicitly express vocal traits, especially emotions.
    </td>
  </tr>
  <tr>
    <td colspan="3">
      <b>(b) Our Method</b><br>
      I'm feeling stressed...<br>
      (serious) Patience. (pause) Only with diligent training...<br>
      I'm feeling stressed...<br>
      (serious) Relax? One more time! (commanding)<br>
      Nonverbal Features | Emotion Perception | Dialog Immersion
    </td>
  </tr>
</table>

Figure 1: (a) Existing role-playing agents (RPAs) concentrate on engaging dialogue in textual form, whereas the role’s voice traits are often ignored. (b) Our method considers the importance of the characters’ vocal persona, e.g., voice style and emotions, delivering a more seamless and immersive speech-language interaction.
vide reasonable responses, their unique tones, i.e., Agent A’s serious yet calmed and Agent B’s angry and commanding, are difficult to convey through text alone, limiting their role-specific expression. Therefore, incorporating personalized voice traits into the current RPAs can help create more realistic and immersive interactions.

To bridge this gap, we introduce OmniCharacter, a first step in realizing speech-language collaborative RPAs for immersive and seamless interaction. Specifically, a Speech-Language Collaborative Model is first built as a base model to perceive both language inputs (i.e., role profile, dialogue contexts, and user text input) and speech input (i.e., user speech input) for semantic alignment. Based on this, we propose a Role Speech Decoder to generate role-specific speech responses containing their unique voice traits. In particular, it mainly consists of two major components: 1) Role-context Guided Speech Token Prediction, which leverages the textual representations from LLM as prior, ensuring the alignment of discrete speech token generation with the corresponding textual contexts. 2) Role-aware Speech Synthesis, which generates waveform containing rich role-related voice traits from speech tokens specific to the given conditions (i.e., profile, dialogue contexts, current query, and speaker embedding). Note that the proposed model can auto-regressively generate text and speech responses with 289ms latency, ensuring a seamless dialogue interaction.

To facilitate the development of OmniCharacter, we further present the OmniCharacter-10K, a speech-language RPAs dataset with detailed profile annotations, diverse dialogues, and vivid audio responses tailored to each role’s characteristic. This dataset presents several appealing properties: 1) Large Vocabulary: It includes a total of 20 characters along with 10, 072 multi-turn dialogues, along with 135K audio responses. 2) Rich Annotations: Each character is accompanied by a rich profile, highlighting aspects such as personality, voice style, relationships, and past experiences. Further, each dialogue turn is annotated with corresponding speech responses for both users and characters via TTS models (Du et al., 2024; Kim et al., 2021). 3) Dynamic Curation: The dialogue data is generated through interactive conversations between two chatbots, guided by the predefined profiles. The main contributions are as follows:

• We introduce OmniCharacter, a pioneering step in realizing speech-language collaborative RPAs.

• We construct OmniCharacter-10K, a multi-turn dialogue dataset with detailed role profiles, conversion data, and high-quality audio annotations for advancing RPAs development.

• Experimental results demonstrate that our method achieves superior performance on the role-playing benchmark, i.e., CharacterEval and SocialBench as well as providing robust results on general speech benchmarks, i.e., LibriSpeech, AISHELL-2, and LibriTTS.

2 Methodology

In this section, we present an in-depth presentation of OmniCharacter, the first RPAs that perceive both speech and language understanding while generating character-specific audio-text responses with low latency. The framework is illustrated in Figure 2.

2.1 Overview

The traditional RPAs are designed to make LLMs interact with users or other agents by emulating specific characters. To achieve this, RPAs utilize a profile denoted as \( P \), along with the ongoing dialogue contexts \( C_n = [q_1, r_1, ..., q_n] \) to generate a response \( r_n \) consistently with character sets:

\[
r_n = \text{RPAs}(C_n, P),
\]

where \( q_i \) represent the \( i \)-th input query and \( r_i \) indicates the corresponding response. However, this paradigm relies on LLMs’ behavior cloning ability and focuses mainly on textual form, neglecting essential vocal traits like voice style, tone, and emotions of different characters.

To bridge this gap, we introduce OmniCharacter, the first model considering the role-related vocal persona by collaborating speech and language for immersive interaction. Specifically, OmniCharacter mainly contains two components: (1) a Speech-Language Collaborative model to align semantics by processing both language inputs (role profile, dialogue contexts, and text query) and speech input, (2) a Role Speech Decoder that aims to generate role-specific speech responses that reflect their unique voice traits. Thereby, the dialogues contexts \( C_n \) can be redefined as \( U_n = [X_1, Y_1, \ldots, X_n] \), where \( X_i \in \{X_i^S, X_i^T, X_i^{S+T}\} \) represents the \( i \)-th input query, supporting three types: audio-only
Figure 2: The overview of our OmniCharacter framework. We first build a speech-language collaborative model, a large language model that receives both speech and language inputs for unified modeling. Furthermore, we propose a role speech decoder to synthesize speech responses containing vocal traits of different characters by devising two innovative modules: i) Role-context Guided Speech Token Prediction, which aims to enhance the generation of speech tokens by leveraging the textual representations of the base model as a contextual prior. ii) Role-aware Speech Synthesis, which generates speech responses from speech tokens maintaining unique character personality and voice traits based on several conditions.

\((X_i^S)\), text-only \((X_i^T)\), or audio-text \((X_i^{S+T})\), resulting in flexible interaction during dialogue. Similarly, \(Y_n \in \{Y_n^S, Y_n^T\}\) indicates the response, owning of two types, *i.e.*, audio response \((Y_n^S)\) and text response \((Y_n^T)\), which maintains the consistent and distinct personality of characters. Next, we elaborate on the details of two components in the subsequent subsections.

2.2 Speech-Language Collaborative Model

To fully align speech and language semantics, we first devise a unified speech-language collaborative model. Specifically, it comprises a speech encoder, a speech adaptor, and an LLM.

Speech Encoder. To tackle user or character speech input, we adopt a speech encoder to encode the speech input \(X_n^S\) of the \(n\)-th dialogue turn to speech sequence \(M_n^S = [M_{n,1}^S, ..., M_{n,N_f}^S]\), where \(N_f\) indicate the number of audio frames.

Speech Adaptor. To address the high temporal redundancy of the encoded speech sequence above, a down-sampling strategy is further applied by grouping every \(k\) consecutive frame into compact sequence \(Z_n^S = [Z_{n,1}^S, ..., Z_{n,[N_s/k]}^S]\), where

\[
Z_{n,i}^S = M_{n,k*i}^S \oplus M_{n,k*i+1}^S \oplus ... \oplus M_{n,k*i+k-1}^S.
\]

This sequence is then encoded by a speech adapter \(\tau\) to make it compatible with the LLM embedding:

\[
E_n^S = \tau(Z_n^S).
\]

Large Language Model. We utilize Qwen2.5.7B-Instruct (Yang et al., 2024) as the LLM, a state-of-the-art large language model renowned for its advanced reasoning capabilities to align text and audio sequences for unified modeling. In particular, we process the text inputs, including the role profile \(P\), dialogue contexts \(U_n\), and the text query \(X_n^T\) to form an overall sequence \(E_n^T = \{E_{n,i}^T\}_{i=1}^{N_t}\) via a text encoder. This sequence is then concatenated with the speech sequence \(E_n^S = \{E_{n,i}^S\}_{i=1}^{N_s}\) and sent into LLM. The \(N_t\) and \(N_s\) indicate the length of text and speech sequences respectively. The prompt template for organizing the overall inputs is showcased in the Appendix A.2. Finally, the LLM is trained via an auto-regressive manner:

\[
\mathcal{L}_{language} = - \sum_{i=1}^{N_t} \log P(O_i^T \mid O_{<i}^T),
\]

where \(O_i^T\) represents the textual token output from the LLM.

2.3 Role Speech Decoder

Role-context Guided Speech Token Prediction. Intuitively, it is straightforward to use the LLM
to predict both text tokens \( O^T \) and speech tokens \( O^S \). However, we find this strategy makes the model difficult to train, causing duplicate speech token outputs and hallucinations that are inconsistent with the semantics of text tokens. To facilitate stable and accurate speech token prediction, we propose an innovative module known as role-context guided speech token prediction built upon the speech-language collaborative model. Specifically, it employs a lightweight language model (denoted as SpeechLLM) that effectively utilizes the context representations \( H = [H_1, ..., H_{N_s+N_t}] \) output from LLM to generate speech tokens:

\[
O^S = \text{SpeechLLM}(O^S_{m}|O^S_{1:m-1}, \phi(H)),
\]

where \( \phi \) is a linear projection layer to map the context representations into the SpeechLLM space. We find that this design not only stabilized the training but also ensured that the generated speech tokens are highly aligned with the contextual semantics, effectively preventing the speech token repetitive output problem. Note that the SpeechLLM also follows an auto-regressive training objective:

\[
\mathcal{L}_{speech} = - \sum_{i=1}^{N_s} \log P(O^S_i | O^S_{<i}).
\]

Role-aware Speech Synthesis. A high-quality speech response that reflects the character’s voice traits and persona is important for immersive RPAs. To achieve this, we propose a role-aware speech synthesis module, which generates the audio response containing voice traits of characters from the speech tokens. Following (Zeng et al., 2024), we first decode the speech tokens into the Mel spectrogram and then synthesize the waveform with the generated Mel spectrogram as input. To achieve this, we adopt a conditional flow matching (CFM) model to sample the Mel spectrogram specified by the context representations \( H \), speech tokens \( O^S \), and speaker embedding \( v \) (Wang et al., 2023b) extracted from character voice as conditions. The sampling process can be defined by a time-dependent vector field \( T = \{T_1, ..., T_n\} \). To improve efficiency, the optimal-transport (OT) flow is used. We summarize the above process as:

\[
Y^S_n = \text{OT-CFM}(p_0|H, v, O^S),
\]

where \( p_0 \) is the initial Mel spectrogram. Ultimately, after obtaining the generated Mel spectrogram, a HiFi-GAN vocoder serves to synthesize a continuous waveform from the Mel spectrogram, facilitating the conversion of spectral representations into high-fidelity audio signals \( Y^S_n \) containing characters’ voice traits such as voice style and emotions.

2.4 Training Strategy

Following (Fang et al., 2024), we adopt a two-stage training strategy for OmniCharacter. In the first stage, we train the speech adapter and LLM to generate text responses based on the text and speech inputs using the objective \( \mathcal{L}_{language} \) in Equation (4). In the second stage, we only train the linear layer and speech LLM in role-context guided speech token prediction module for speech token prediction by using the objective \( \mathcal{L}_{speech} \) in Equation (6). For role-aware speech synthesis, we utilize the pre-trained conditional flow matching and HiFi-GAN weights from GLM-4-Voice (Zeng et al., 2024) and fine-tune them on high-quality role speech data2.

3 The OmniCharacter-10K Dataset

To facilitate the model with speech-language scenarios, we construct OmniCharacter-10K, a multi-turn dialogue dataset with distinctive characters with rich and expressive text and speech annotations. In this section, we describe the data collection, annotation, and verification of OmniCharacter-10K. We also introduce the statistics and distribution of it. The illustration of OmniCharacter-10K is showcased in Figure 3.

3.1 Data Collection, Annotation, and Verification

Step-1: Character Profile Creation. To make sure the collected characters have distinct traits as well as contain high-quality audio, we select from games since (1) the availability of rich information (e.g., personality, background, and vocal tone), and (2) clear, noise-free character audio recorded by professional voice actors. We end up with 20 foundational characters (10 Chinese and 10 English) from Genshin Impact to construct our dataset. Subsequently, we use a large language model to summarize the meta information for each character, and then instruct this model to expand the information to ensure the diversity and complexity. At last, all character profiles are subjected to rigorous human checks to ensure accuracy and reliability. Moreover, given the diversity of humans

2https://genshin.hoyoverse.com/en/
Figure 3: Illustration of OmniCharacter-10K. (a) Dataset Construction Pipeline, which consists of four steps: (1) Characters Profile Creation, (2) Dialogue Generation, (3) Speech Synthesis, and (4) Quality Verification. (b) Dataset Properties, which has three appealing properties of our dataset: (1) Large Vocabulary, the dataset includes 20 characters, 10K multi-turn dialogues, and 135K audio responses, (2) Rich Annotations, each character is equipped with rich text and speech annotations, and (3) Dynamic Curation, we generate dialogue data through chatbot interactions based on predefined character profiles.

<table>
  <tr>
    <th>Splits</th>
    <th># Characters</th>
    <th>Avg. Turns/Conv.</th>
    <th># Samples</th>
    <th># Speech Hours. (user/character)</th>
  </tr>
  <tr>
    <td>Training</td>
    <td>20</td>
    <td>15</td>
    <td>9,672</td>
    <td>360.3 (176.4/18.3.9)</td>
  </tr>
  <tr>
    <td>Test</td>
    <td>20</td>
    <td>15</td>
    <td>400</td>
    <td>14.84 (7.15/7.69)</td>
  </tr>
</table>

Table 1: The statistic of OmniCharacter-10K dataset.

and the open-domain nature of the question, we do not consider the profile for the ‘user’ side.

Step-2: Dialogue Generation. After selecting the characters and defining their profiles, we generate multi-turn dialogues to evaluate role-playing consistency and conversational quality. Specifically, we deploy two large language models in a simulated interaction: one assumes the role of the target character, while the other acts as a user or a different character. Each model is conditioned on its respective profile, which includes key personality traits, background information, emotions, etc. The dialogue is guided by prompts that encourage natural and persona-consistent responses across multiple turns. This setup enables us to systematically assess how well the models embody their assigned roles, maintain character coherence, and engage in meaningful, contextually appropriate exchanges.

Step-3: Speech Synthesis. The generated multi-turn dialogues consist of open-domain text without corresponding audio annotations. To handle this, we first collected 40K high-quality audio samples for 20 characters, which are professionally recorded by voice actors and extracted from the game’s asset files. Then, we utilize these audios as seed data and train a multi-speaker text-to-speech (TTS) model, i.e., VITS (Kim et al., 2021) for the audio synthesis of open-text in dialogue. For the user side, we use the advanced TTS model CosyVoice (Du et al., 2024) to synthesize audio. Each dialogue sample has a 50% ratio of being assigned a generic male or female voice, ensuring its balance and diverse distribution.

Step-4: Quality Verification. To ensure the quality of the constructed dataset, we manually filter samples based on several criteria. For text data, we performed data filtering by retaining the samples following ABAB (user-role or role1-role2) pattern. Besides, we preserved dialogues longer than three turns, filtering out shorter ones. Furthermore, we remove data that mimics the style of general AI assistance such as ‘I am a helpful AI assistant...’, as well as unnecessary explanatory prefaces and postfaces. For speech data, we first use the Whisper-large-v3 (Radford et al., 2023) to convert the synthesized audio into text and compute the WER metrics with

![Distribution of dialogue turns across samples in the OmniCharacter-10K dataset. Right: Distribution of audio duration for user speech and character speech respectively.](page_1012_1012_420_246.png)

Figure 4: Left: Distribution of dialogue turns across samples in the OmniCharacter-10K dataset. Right: Distribution of audio duration for user speech and character speech respectively.
<table>
  <tr>
    <th rowspan="2">Models</th>
    <th colspan="5">Character Consistency</th>
    <th colspan="4">Conversational Ability</th>
    <th colspan="3">Role-playing Attractiveness</th>
    <th rowspan="2">Avg.</th>
  </tr>
  <tr>
    <th>KE</th><th>KA</th><th>KH</th><th>PB</th><th>PU</th>
    <th>Avg.</th>
    <th>Flu.</th><th>Coh.</th><th>Cons.</th><th>Avg.</th>
    <th>HL</th><th>CS</th><th>ED</th><th>Emp.</th><th>Avg.</th>
  </tr>
  <tr>
    <th colspan="18">Proprietary Models</th>
  </tr>
  <tr>
    <td>BC-NPC-Turbo</td><td>1.802</td><td>2.964</td><td>2.993</td><td>2.910</td><td>3.151</td><td>2.764</td><td>3.578</td><td>3.898</td><td>3.916</td><td>3.798</td><td>3.836</td><td>2.643</td><td>2.336</td><td>2.971</td><td>2.946</td><td>3.169</td>
  </tr>
  <tr>
    <td>MiniMax</td><td>1.835</td><td>2.910</td><td>2.944</td><td>2.774</td><td>3.125</td><td>2.718</td><td>3.609</td><td>3.932</td><td>3.811</td><td>3.784</td><td>3.768</td><td>2.672</td><td>2.150</td><td>3.017</td><td>2.902</td><td>3.134</td>
  </tr>
  <tr>
    <td>GPT-3.5</td><td>1.716</td><td>2.339</td><td>2.212</td><td>1.921</td><td>2.316</td><td>2.101</td><td>2.629</td><td>2.917</td><td>2.700</td><td>2.749</td><td>2.565</td><td>2.422</td><td>1.660</td><td>2.526</td><td>2.293</td><td>2.381</td>
  </tr>
  <tr>
    <td>GPT-4</td><td>2.250</td><td>2.855</td><td>2.785</td><td>2.721</td><td>2.873</td><td>2.697</td><td>3.332</td><td>3.669</td><td>3.343</td><td>3.448</td><td>3.143</td><td>3.184</td><td>2.153</td><td>3.010</td><td>2.873</td><td>3.006</td>
  </tr>
  <tr>
    <th colspan="18">Open-sourced Models</th>
  </tr>
  <tr>
    <td>StarGLM-6b</td><td>2.016</td><td>2.792</td><td>2.704</td><td>2.455</td><td>2.812</td><td>2.556</td><td>3.269</td><td>3.647</td><td>3.283</td><td>3.399</td><td>3.064</td><td>2.932</td><td>1.969</td><td>2.993</td><td>2.739</td><td>2.898</td>
  </tr>
  <tr>
    <td>Baichuan2-7B</td><td>1.813</td><td>2.849</td><td>2.929</td><td>2.830</td><td>3.081</td><td>2.700</td><td>3.551</td><td>3.894</td><td>3.827</td><td>3.757</td><td>3.670</td><td>2.728</td><td>2.115</td><td>2.984</td><td>2.874</td><td>3.110</td>
  </tr>
  <tr>
    <td>Baichuan2-13B</td><td>1.802</td><td>2.869</td><td>2.946</td><td>2.808</td><td>3.081</td><td>2.701</td><td>3.596</td><td>3.924</td><td>3.864</td><td>3.759</td><td>3.700</td><td>2.703</td><td>2.136</td><td>3.021</td><td>2.890</td><td>3.116</td>
  </tr>
  <tr>
    <td>InternLM-7B</td><td>1.782</td><td>2.800</td><td>2.781</td><td>2.719</td><td>3.016</td><td>2.620</td><td>3.527</td><td>3.823</td><td>3.744</td><td>3.699</td><td>3.546</td><td>2.622</td><td>2.070</td><td>2.897</td><td>2.784</td><td>2.983</td>
  </tr>
  <tr>
    <td>InternLM-20B</td><td>1.945</td><td>2.916</td><td>2.920</td><td>2.753</td><td>3.041</td><td>2.715</td><td>3.576</td><td>3.943</td><td>3.717</td><td>3.745</td><td>3.582</td><td>2.885</td><td>2.132</td><td>3.047</td><td>2.911</td><td>3.123</td>
  </tr>
  <tr>
    <td>CharacterGLM</td><td>1.640</td><td>2.819</td><td>2.738</td><td>2.301</td><td>2.969</td><td>2.493</td><td>3.414</td><td>3.717</td><td>3.737</td><td>3.623</td><td>3.738</td><td>2.265</td><td>1.966</td><td>2.812</td><td>2.695</td><td>2.937</td>
  </tr>
  <tr>
    <td>Llama-3.1-8B</td><td>2.197</td><td>2.701</td><td>2.615</td><td>3.130</td><td>2.704</td><td>2.669</td><td>3.059</td><td>3.477</td><td>3.071</td><td>3.202</td><td>2.922</td><td>2.934</td><td>2.634</td><td>2.759</td><td>2.812</td><td>2.804</td>
  </tr>
  <tr>
    <td>Qwen-7B</td><td>1.956</td><td>2.728</td><td>2.633</td><td>2.605</td><td>2.780</td><td>2.540</td><td>3.187</td><td>3.564</td><td>3.229</td><td>3.327</td><td>3.036</td><td>2.791</td><td>2.052</td><td>2.838</td><td>2.679</td><td>2.848</td>
  </tr>
  <tr>
    <td>Qwen-14B</td><td>1.988</td><td>2.800</td><td>2.811</td><td>2.744</td><td>2.900</td><td>2.649</td><td>3.351</td><td>3.765</td><td>3.510</td><td>3.354</td><td>3.281</td><td>2.237</td><td>2.970</td><td>2.858</td><td>3.016</td>
  </tr>
  <tr>
    <td>Qwen2-7B-Instruct</td><td>1.966</td><td>2.957</td><td>2.412</td><td>2.313</td><td>2.436</td><td>2.333</td><td>2.864</td><td>3.171</td><td>2.743</td><td>2.926</td><td>2.655</td><td>2.612</td><td>1.867</td><td>2.654</td><td>2.447</td><td>2.569</td>
  </tr>
  <tr>
    <td>Qwen2.5-7B-Instruct</td><td>2.207</td><td>2.740</td><td>2.633</td><td>2.700</td><td>2.614</td><td>2.579</td><td>3.125</td><td>3.401</td><td>2.971</td><td>3.166</td><td>2.786</td><td>2.871</td><td>2.180</td><td>2.826</td><td>2.666</td><td>2.804</td>
  </tr>
  <tr>
    <td>Qwen2.5-7B-Instruct†</td><td>2.172</td><td>3.012</td><td>2.887</td><td>3.526</td><td>2.879</td><td>2.895</td><td>3.321</td><td>3.657</td><td>3.326</td><td>3.434</td><td>3.278</td><td>3.127</td><td>2.728</td><td>3.217</td><td>3.087</td><td>3.138</td>
  </tr>
  <tr>
    <th>OmniCharacter w/o audio</th><td>2.188</td><td>3.024</td><td>2.926</td><td>3.547</td><td>2.955</td><td>2.928</td><td>3.402</td><td>3.732</td><td>3.432</td><td>3.522</td><td>3.303</td><td>3.266</td><td>2.933</td><td>3.172</td><td>3.168</td><td>3.206</td>
  </tr>
  <tr>
    <th>OmniCharacter</th><td>2.230</td><td>3.040</td><td>2.918</td><td>3.531</td><td>2.988</td><td>2.941</td><td>3.369</td><td>3.768</td><td>3.410</td><td>3.516</td><td>3.374</td><td>3.261</td><td>3.002</td><td>3.187</td><td>3.206</td><td>3.221</td>
  </tr>
</table>

Table 2: Performance comparison with state-of-the-art methods on CharacterEval. We evaluate the models across three dimensions including: (A) Character Consistency: (a-1) KE: Knowledge-Exposure, (a-2) KA: Knowledge-Accuracy, (a-3) KH: Knowledge-Hallucination, (a-4) PB: Persona-Behavior, (a-5) PU: Persona-Utterance. (B) Conversational Ability: (b-1) Flu.: Fluency, (b-2) Coh.: Coherency, (b-3) Cons.: Consistency. (C) Role-playing Attractiveness: (c-1) HL: Human-Likeness, (c-2) CS: Communication Skill, (c-3) ED: Expression Diversity, (c-4) Emp.: Empathy. † means we finetune pretrained Qwen-2.5-7B-Instruct on the textual portion of OmniCharacter-10K.

the corresponding text, where the WER greater than 10 are rejected. Then, we use WavLLM (Hu et al., 2024) to compute the similarity between the synthesized audio and reference audio, where the similarity less than 0.8 threshold is also rejected.

3.2 Statistics of OmniCharacter-10K

Table 1 presents the statistics of the OmniCharacter-10K dataset, which includes a total of 20 characters with 9, 672 training samples and 400 test samples. The training and test sets are rich in speech annotations, with 360.3 hours and 14.84 hours of audio data, respectively. In addition, Figure 4 shows the distribution of dialogue turns (left) and audio duration (right) across the dataset. On one hand, more than 80% of the dialogues are longer than 10 turns, highlighting the dataset’s focus on multi-turn interactions. On the other hand, the audio duration statistics further reveal that the speech in the dataset is sufficiently long to enable immersive dialogue experiences. These statistics emphasize the dataset’s capability to support the evaluation of RPAs in facilitating immersive dialogues.

4 Experiments

4.1 Experimental Setups

Model Configuration. We employ the Whisper-large-v3 as the speech encoder and Qwen-2.5-7B-Instruct as the LLM in speech-language collaborative model. In role-context guided speech token prediction, we employ the lightweight Qwen2.5-0.5B-Instruct (Team, 2024) model as SpeechLLM to generate speech tokens. To facilitate this process, a speech tokenizer derived from GLM-4-Voice (Zeng et al., 2024) with 16K vocabulary size is utilized to extract discrete speech tokens. We resample all speech data to a frequency of 16Khz. The speech-language collaborative model and role-context guided speech token prediction are initialized using pre-trained parameters from OpenOmni (Luo et al., 2025), and we conduct fine-tuning on the OmniCharacter-10K train set. The weights of role-aware speech synthesis are initialized from pretrained checkpoint of GLM-4-Voice.

Training Details. The OmniCharacter employs a two-stage training strategy as described in Section 2.4. In the first stage, we utilize the AdamW optimizer with a warmup ratio of 0.3, gradually increasing the learning rate until it reaches 5e-4. The second stage retains the parameters established in the first stage, and we only adjusted the final learning rate to 5e-5. We set the batch size as 32 for both two stages. The entire training process is finished within 3 epochs on 8×A100 GPUs.

Datasets. The evaluation leverages the following datasets: CharacterEval (Tu et al., 2024) and Social-
<table>
  <tr>
    <th rowspan="2">Models</th>
    <th colspan="4">Individual Level</th>
    <th colspan="4">Group Level</th>
    <th rowspan="2">Avg.</th>
  </tr>
  <tr>
    <th>SA Style</th>
    <th>SA Know.</th>
    <th>EP Situ.</th>
    <th>EP Emo.</th>
    <th>CM Short</th>
    <th>CM Long</th>
    <th>Pos.</th>
    <th>Neu.</th>
    <th>Neg.</th>
  </tr>
  <tr>
    <td>Qwen-72B-Chat</td>
    <td>83.87</td>
    <td>90.64</td>
    <td>53.10</td>
    <td>52.89</td>
    <td>83.29</td>
    <td>73.15</td>
    <td>91.53</td>
    <td>73.44</td>
    <td>63.82</td>
    <td>73.97</td>
  </tr>
  <tr>
    <td>LLaMA-2-70B-Chat</td>
    <td>67.61</td>
    <td>70.78</td>
    <td>35.74</td>
    <td>38.47</td>
    <td>45.57</td>
    <td>26.74</td>
    <td>69.87</td>
    <td>45.29</td>
    <td>39.37</td>
    <td>48.83</td>
  </tr>
  <tr>
    <td>Qwen-14B-Chat</td>
    <td>77.06</td>
    <td>86.15</td>
    <td>45.71</td>
    <td>43.78</td>
    <td>65.32</td>
    <td>51.37</td>
    <td>78.32</td>
    <td>58.25</td>
    <td>59.21</td>
    <td>62.80</td>
  </tr>
  <tr>
    <td>LLaMA-2-13B-Chat</td>
    <td>57.62</td>
    <td>65.51</td>
    <td>37.12</td>
    <td>32.56</td>
    <td>30.43</td>
    <td>29.82</td>
    <td>66.38</td>
    <td>42.25</td>
    <td>26.27</td>
    <td>43.11</td>
  </tr>
  <tr>
    <td>LLaMA-2-7B-Chat</td>
    <td>48.76</td>
    <td>51.23</td>
    <td>31.23</td>
    <td>28.91</td>
    <td>25.38</td>
    <td>21.89</td>
    <td>44.98</td>
    <td>24.19</td>
    <td>27.67</td>
    <td>33.80</td>
  </tr>
  <tr>
    <td>Mistral-7B</td>
    <td>50.12</td>
    <td>61.17</td>
    <td>36.48</td>
    <td>31.72</td>
    <td>31.78</td>
    <td>25.42</td>
    <td>65.67</td>
    <td>46.34</td>
    <td>28.96</td>
    <td>41.96</td>
  </tr>
  <tr>
    <td>Qwen-7B-Chat</td>
    <td>66.44</td>
    <td>71.16</td>
    <td>41.68</td>
    <td>40.68</td>
    <td>67.45</td>
    <td>53.45</td>
    <td>75.61</td>
    <td>52.78</td>
    <td>43.11</td>
    <td>56.93</td>
  </tr>
  <tr>
    <td>OmniCharacter</td>
    <td>70.93</td>
    <td>65.68</td>
    <td>58.03</td>
    <td>38.24</td>
    <td>59.76</td>
    <td>57.86</td>
    <td>69.11</td>
    <td>63.26</td>
    <td>43.98</td>
    <td>58.53</td>
  </tr>
</table>

Table 3: Performance comparison with state-of-the-art methods on SocialBench. We evaluate the models across several dimensions including: (1) Self-Awareness: role style (SA Style) and role knowledge (SA Know.) (2) Emotional Perception: situation understanding (EP Situ.) and emotion detection (EP Emo.) (3) Conversation Memory: short-term (CM Short) and long-term (CM Long) (4) Social Preference: positive (Pos.), neutral (Neu.), and negative (Neg.).

<table>
  <tr>
    <th rowspan="2">Models</th>
    <th colspan="2">S2TIF</th>
    <th colspan="2">S2SIF</th>
  </tr>
  <tr>
    <th>Content ↑</th>
    <th>Style ↑</th>
    <th>Content ↑</th>
    <th>Style ↑</th>
  </tr>
  <tr>
    <td>SpeechGPT</td>
    <td>1.25</td>
    <td>1.28</td>
    <td>1.79</td>
    <td>1.85</td>
  </tr>
  <tr>
    <td>LLaMA-Omni</td>
    <td>2.46</td>
    <td>2.71</td>
    <td>2.26</td>
    <td>2.38</td>
  </tr>
  <tr>
    <td>OmniCharacter</td>
    <td>3.89</td>
    <td>4.18</td>
    <td>2.64</td>
    <td>2.55</td>
  </tr>
</table>

Table 4: Performance comparison with state-of-the-art methods on OmniCharacter-10K test split. Chat-GPT scores for S2TIF, S2SIF tasks, and alignment scores between speech and text responses.

Bench (Chen et al., 2024) are used to testify models’ basic language understanding. OmniCharacter-10K test set is employed for character-aware speech-language evaluation to test speech generation and alignment abilities. LibriSpeech (Panayotov et al., 2015), AISHELL-2 (Du et al., 2018), and LibriTTS (Zen et al., 2019) are general speech datasets for model’s generalization test.

4.2 Performance Comparison

Basic Language Understanding. To evaluate the effectiveness of our OmniCharacter in basic language understanding, we compare our method with RPs methods on CharacterEval (Tu et al., 2024) benchmark as shown in Table 2. Note that the characters tested in CharacterEval differ from those in our training data. The results illustrate that our model outperforms the proprietary and open-source models on most metrics. Particularly, compared with models specifically designed for role-playing task including BC-NPC-Turbo and MiniMax, our method achieves considerable improvements. Moreover, OmniCharacter surpasses open-sourced models with larger scales such as InterLM-20B and Qwen-14B, achieving superior performance in both character consistency and role-playing attractiveness. Further, we evaluate our method on SocialBench (Chen et al., 2024) as shown in Table 3. As observed, our method improves performance across most metrics when compared to the same-size models. Although OmniCharacter is not as optimal as larger-scale models, it still delivers comparable performance, securing the second-best position in terms of the average score. The results presented above provide compelling evidence that the integration of audio modality enhances basic language understanding.

Character-aware Speech-Language Evaluation. To thoroughly evaluate the speech-language collaboration ability of OmniCharacter for immersive and personalized role-playing interaction, we conduct evaluations from two perspectives: metrics-based and human-based evaluations.

The metrics-based evaluation consists of three tasks (Fang et al., 2024): (1) speech-to-text instruction-following (S2TIF): whether the model’s text response correctly answers the input speech query; (2) speech-to-speech instruction-following (S2SIF): whether the model’s speech response correctly answers the input speech query. We ask GPT-4o to score the response in a range from 1 to 5 from content and style aspects. As shown in Table 4, our model consistently enhances performance across all metrics, achieving superior results compared to LLaMA-Omni and SpeechGPT. These results confirm that OmniCharacter is capable of effectively handling user or characters’ instructions, while simultaneously generating consistent text and speech responses.

In terms of human-based evaluation, five experts are assigned to examine the model’s outputs across
<table>
  <tr>
    <th>Models</th>
    <th>Flu.</th>
    <th>Cons.</th>
    <th>Emo.</th>
    <th>Cla.</th>
    <th>App.</th>
    <th>Imm.</th>
  </tr>
  <tr>
    <td>SpeechGPT</td>
    <td>6.12</td>
    <td>3.86</td>
    <td>3.75</td>
    <td>6.89</td>
    <td>4.23</td>
    <td>3.64</td>
  </tr>
  <tr>
    <td>LLaMA-Omni</td>
    <td>6.88</td>
    <td>4.27</td>
    <td>3.44</td>
    <td>6.69</td>
    <td>4.78</td>
    <td>4.68</td>
  </tr>
  <tr>
    <td>OmniCharacter</td>
    <td>7.97</td>
    <td>6.84</td>
    <td>6.23</td>
    <td>7.88</td>
    <td>5.63</td>
    <td>8.52</td>
  </tr>
</table>

Table 5: Performance comparison with state-of-the-art methods on OmniCharacter-10K test split. Human score (averaged from 5 experts) across six voice-related dimensions: fluency (Flu.), consistency (Cons.), emotional expression (Emo.), clarity (Cla), appropriateness (App.), and immersion (Imm.).

six voice-related dimensions as shown in Table 5. Each expert is asked to score the responses on a scale from 1 to 10 for each dimension. From the table, we figure out that our method reaches the best performance across all dimensions, proving the superiority of our model which is capable of providing immersive interactions for users.

Generalization on General Speech Benchmarks.
To better prove the robust ability of OmniCharacter, we evaluated the performance of our model on three widely used speech datasets, LibriSpeech, AIShell-2, and LibriTTS, focusing on both Automatic Speech Recognition (ASR) and Text-to-Speech (TTS) tasks. As presented in Table 6, our model achieves comparable performance compared with previous models, demonstrating its generalization ability. These results further highlight the robustness of OmniCharacter in generating coherent and accurate outputs, showcasing its potential in real-world speech applications.

<table>
  <tr>
    <th rowspan="2">Models</th>
    <th colspan="2">LibriSpeech (ASR-WER)</th>
    <th colspan="2">AISHELL-2 (ASR-CER)</th>
    <th colspan="2">LibriTTS (TTS-WER)</th>
  </tr>
  <tr>
    <th>test-clean</th>
    <th>test-other</th>
    <th>test</th>
    <th>test-other</th>
    <th>test-clean</th>
    <th>test-other</th>
  </tr>
  <tr>
    <td>CosyVoice 1.0</td>
    <td>-</td>
    <td>-</td>
    <td>-</td>
    <td>-</td>
    <td>-</td>
    <td>3.17</td>
  </tr>
  <tr>
    <td>Whisper-large-v3</td>
    <td>2.50</td>
    <td>4.53</td>
    <td>-</td>
    <td>-</td>
    <td>-</td>
    <td>-</td>
  </tr>
  <tr>
    <td>Qwen2-Audio</td>
    <td>2.00</td>
    <td>4.50</td>
    <td>3.30</td>
    <td>-</td>
    <td>-</td>
    <td>-</td>
  </tr>
  <tr>
    <td>OmniCharacter</td>
    <td>3.26</td>
    <td>4.23</td>
    <td>6.62</td>
    <td>-</td>
    <td>-</td>
    <td>7.23</td>
  </tr>
</table>

Table 6: Performance comparison with the state-of-the-art methods on general speech benchmarks including LibriSpeech, AISHELL-2, and LibriTTS. The model is evaluated on two tasks, i.e., Automatic Speech Recognition (ASR) and Text-to-Speech (TTS). WER: Word Error Rate. CER: Character Error Rate.

<table>
  <tr>
    <th rowspan="2">Models</th>
    <th colspan="2">Character</th>
    <th colspan="2">Conversational</th>
    <th colspan="2">Role-playing</th>
  </tr>
  <tr>
    <th>Consistency</th>
    <th>Ability</th>
    <th>Attractiveness</th>
    <th>Ability</th>
    <th>Attractiveness</th>
  </tr>
  <tr>
    <td>OmniCharacter w/o audio</td>
    <td>2.928</td>
    <td>3.522</td>
    <td>3.168</td>
    <td></td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td>OmniCharacter</td>
    <td>2.941</td>
    <td>3.516</td>
    <td>3.206</td>
    <td></td>
    <td></td>
    <td></td>
  </tr>
</table>

Table 7: Ablation study of model training with or without audio modality. For simplicity, we only report the average score across three dimensions on CharacterEval.

similarity between the reference speech and the generated speech on OmniCharacter-10K test set, as shown in Figure 5. The comparison methods include state-of-the-art TTS systems, i.e., VITS, MaskGCT, CosyVoice1.0, and CosyVoice2.0. For our model, we directly input the corresponding text to generate the speech output. For other models, we leverage their voice cloning capabilities by providing reference speech, along with the given text, to synthesize the corresponding speech. If the cosine similarity between the output speech and the reference speech is greater than 0.9, we classify them as belonging to the same character. As observed, our model consistently achieves higher cosine similarity for both Chinese and English characters’ voices. These results highlight the effectiveness of our approach in preserving the voice traits of characters, achieving a more interactive experiment.

4.3 Impact of Audio Modality
To assess the impact of the audio modality on role-playing agents (RPAs), we conduct an ablation study by removing audio input from OmniCharacter and evaluating the model on the CharacterEval benchmark, as shown in Table 7. We report average scores across all metrics for clarity. The results indicate that incorporating audio significantly improves character consistency and conversational ability, while achieving comparable performance in role-playing attractiveness. These findings highlight the importance of audio in enhancing language understanding and contributing to more natural, expressive role-playing interactions.

4.4 Similarity of Synthesized Voice for Characters
To verify the model’s capability of synthesizing character-specific speech containing their voice traits, we use WavLLM to compute the cosine

4.5 Learning Discriminative Character Speech Embeddings
The OmniCharacter generates the character-specific voice by utilizing the speech embedding as one of the conditions. To evaluate the discrimination of speaker embedding between different characters, we compute their cosine similarity as shown in Figure 6. Specifically, we randomly sampled three sets of data of 20 characters for experimental comprehensiveness. We also included the user’s speaker embedding for comparison. The results indicate that the similarity values between the speech embeddings of different characters and users are very low, demonstrating their strong discriminabil-
<table>
  <tr>
    <th>Models</th>
    <th>GPT-4o</th>
    <th>LLaMA-Omni</th>
    <th>OmniCharacter</th>
  </tr>
  <tr>
    <td>Latency (ms)</td>
    <td>320</td>
    <td>226</td>
    <td>289</td>
  </tr>
</table>

Table 8: Speech response latency of different models.

![Comparison of character voice similarity on OmniCharacter-10K test set. Our model generates audio responses that better match the timbre of characters.](page_153_370_563_180.png)

Figure 5: Comparison of character voice similarity on OmniCharacter-10K test set. Our model generates audio responses that better match the timbre of characters.

![Speech embedding discriminability for 20 characters and 4 users. We randomly sampled three sets of data for experimental comprehensiveness. The low similarity values demonstrate strong separation of character voices, enabling precise voice control.](page_728_370_563_180.png)

Figure 6: Speech embedding discriminability for 20 characters and 4 users. We randomly sampled three sets of data for experimental comprehensiveness. The low similarity values demonstrate strong separation of character voices, enabling precise voice control.

ity. This confirms the robustness of our method in controlling character’s voice traits.

4.6 Speech Response Latency

To demonstrate the seamless interaction capability of the proposed model, we evaluate the speech response latency of different models, as shown in Table 8. From the table, we observe that our model outperforms the GPT-4o, achieving a lower response latency. It is observed that the latency of our method is higher than LLaMA-Omni. The possible reason is that LLaMA-Omni only supports general female voice, while we introduce additional modules considering voice traits of different roles, thus the increased latency is acceptable. Overall, these results underscore the seamless dialogue ability of our method, demonstrating its effectiveness in real-time speech-language interaction for RPAs.

5 Conclusion

In this paper, we present OmniCharacter, the first step toward creating a seamless and immersive RPAs model that integrates both speech and language personality interaction. Unlike existing methods that primarily focus on text-based dialogues, our approach emphasizes the importance of character voice traits, e.g., voice style and emotions, which enable agents to reflect both personality and vocal traits throughout interactions, delivering a harmonious blend of speech and language responses. To facilitate the speech-language scenarios for RPAs, we construct the OmniCharacter-10K dataset, which includes distinct characters, contextually rich multi-turn dialogues, and dynamic speech responses. Experimental results demonstrate that OmniCharacter outperforms existing RPAs models and general speech-language models in both content fidelity and expressive style, delivering a seamless and immersive user experience.

6 Limitations

The proposed OmniCharacter advances immersive role-playing agents (RPAs) by incorporating characters’ voice traits through a speech-language personality interaction framework. While effective, several limitations remain. First, performance relies heavily on the quality of the pre-trained language models and speech synthesis systems, which may be affected by their alignment and training data. Second, the current design is limited to two-character dialogues, making it less adaptable to multi-role interactions. Third, although the OmniCharacter-10K dataset includes diverse scenarios, it may not fully capture the richness of real-world conversations. Expanding the dataset is a key direction for improving generalization. Lastly, while the latency is currently low, further optimization is necessary to enhance the system’s performance and ensure a more seamless experience for real-time usage.

Acknowledgments

This study is supported by Alibaba Research Intern Program, grants from the National Natural Science Foundation of China (Grant No.62425208, No. U22A2097, No. U23A20315, No. 62020106008, No. 82441006, No.62402094, No. 62376262), Shenzhen Science and Technology Program (No.JCYJ20240813114208012), the National Key Research and Development Program of China (2022YFF0902100), the Natural Science Foundation of Guangdong Province of China (2025B1515020032, 2024A1515030166), and Sichuan Science and Technology Program (No.2025ZNSFSC1493).
References

Zalán Borsos, Raphaël Marinier, Damien Vincent, Eugène Kharitonov, Olivier Pietquin, Matt Sharifi, Dominik Roblek, Olivier Teboul, David Grangier, Marco Tagliasacchi, and Neil Zeghidour. 2023. Audiolm: a language modeling approach to audio generation. Preprint, arXiv:2209.03143.

Hongzhan Chen, Hehong Chen, Ming Yan, Wenshen Xu, Xing Gao, Weizhou Shen, Xiaojuan Quan, Chenliang Li, Ji Zhang, Fei Huang, et al. 2024. Socialbench: Sociability evaluation of role-playing conversational agents. arXiv preprint arXiv:2403.13679.

Alexandre Défossez, Laurent Mazaré, Manu Orsini, Amélie Royer, Patrick Pérez, Hervé Jégou, Edouard Grave, and Neil Zeghidour. 2024. Moshi: a speech-text foundation model for real-time dialogue. arXiv preprint arXiv:2410.00037.

Jiayu Du, Xingyu Na, Xuechen Liu, and Hui Bu. 2018. Aishell-2: Transforming mandarin asr research into industrial scale. arXiv preprint arXiv:1808.10583.

Zhihao Du, Qian Chen, Shiliang Zhang, Kai Hu, Heng Lu, Yexin Yang, Hangrui Hu, Siqi Zheng, Yue Gu, Ziyang Ma, et al. 2024. Cosyvoice: A scalable multilingual zero-shot text-to-speech synthesizer based on supervised semantic tokens. arXiv preprint arXiv:2407.05407.

Qingkai Fang, Shoutao Guo, Yan Zhou, Zhengrui Ma, Shaolei Zhang, and Yang Feng. 2024. Llama-omni: Seamless speech interaction with large language models. arXiv preprint arXiv:2409.06666.

Shujie Hu, Long Zhou, Shujie Liu, Sanyuan Chen, Lingwei Meng, Hongkun Hao, Jing Pan, Xunying Liu, Jinyu Li, Sunit Sivasankaran, et al. 2024. Wavlm: Towards robust and adaptive speech large language model. arXiv preprint arXiv:2404.00656.

Jaehyeon Kim, Jungil Kong, and Juhee Son. 2021. Conditional variational autoencoder with adversarial learning for end-to-end text-to-speech. In ICML, pages 5530–5540. PMLR.

Jinlong Li, Zequn Jie, Xu Wang, Xiaolin Wei, and Lin Ma. 2022. Expansion and shrinkage of localization for weakly-supervised semantic segmentation. Advances in neural information processing systems, 35:16037–16051.

Jinlong Li, Cristiano Saltori, Fabio Poiesi, and Nicu Sebe. 2025. Cross-modal and uncertainty-aware agglomeration for open-vocabulary 3d scene understanding. arXiv preprint arXiv:2503.16707.

Run Luo, Yunshui Li, Longze Chen, Wanwei He, Ting-En Lin, Ziqiang Liu, Lei Zhang, Zikai Song, Xiaobo Xia, Tongliang Liu, et al. 2024a. Deem: Diffusion models serve as the eyes of large language models for image perception. arXiv preprint arXiv:2405.15232.

Run Luo, Ting-En Lin, Haonan Zhang, Yuchuan Wu, Xiong Liu, Min Yang, Yongbin Li, Longze Chen, Jiaming Li, Lei Zhang, et al. 2025. Openomni: Large language models pivot zero-shot omnimodal alignment across language with real-time self-aware emotional speech synthesis. arXiv preprint arXiv:2501.04561.

Run Luo, Haonan Zhang, Longze Chen, Ting-En Lin, Xiong Liu, Yuchuan Wu, Min Yang, Minzheng Wang, Pengpeng Zeng, Lianli Gao, et al. 2024b. Mmveol: Empowering multimodal large language models with evol-instruct. arXiv preprint arXiv:2409.05840.

OpenAI OpenAI. 2022. Openai: Introducing chatgpt. URL https://openai.com/blog/chatgpt.

Vassil Panayotov, Guoguo Chen, Daniel Povey, and Sanjeev Khudanpur. 2015. Librispeech: an asr corpus based on public domain audio books. In ICASSP, pages 5206–5210. IEEE.

Joon Sung Park, Joseph O’Brien, Carrie Jun Cai, Meredith Ringel Morris, Percy Liang, and Michael S Bernstein. 2023. Generative agents: Interactive simulacra of human behavior. In Proceedings of the 36th annual acm symposium on user interface software and technology, pages 1–22.

Letian Peng and Jingbo Shang. 2024. Quantifying and optimizing global faithfulness in persona-driven role-playing. arXiv preprint arXiv:2405.07726.

Alec Radford, Jong Wook Kim, Tao Xu, Greg Brockman, Christine McLeavey, and Ilya Sutskever. 2023. Robust speech recognition via large-scale weak supervision. In ICML, pages 28492–28518. PMLR.

Paul K Rubenstein, Chulayuth Asawaroengchai, Duc Dung Nguyen, Ankur Bapna, Zalán Borsos, Félix de Chaumont Quitry, Peter Chen, Dalila El Badawy, Wei Han, Eugene Kharitonov, et al. 2023. Audiopalm: A large language model that can speak and listen. arXiv preprint arXiv:2306.12925.

Nafis Sadeq, Zhouhang Xie, Byungkyu Kang, Prarit Lamba, Xiang Gao, and Julian McAuley. 2024. Mitigating hallucination in fictional character role-play. arXiv preprint arXiv:2406.17260.

Yunfan Shao, Linyang Li, Junqi Dai, and Xipeng Qiu. 2023. Character-llm: A trainable agent for role-playing. In EMNLP, pages 13153–13187.

Qwen Team. 2024. Qwen2.5: A party of foundation models.

Quan Tu, Shilong Fan, Zihang Tian, and Rui Yan. 2024. Charactereval: A chinese benchmark for role-playing conversational agent evaluation. arXiv preprint arXiv:2401.01275.

Chengyi Wang, Sanyuan Chen, Yu Wu, Ziqiang Zhang, Long Zhou, Shujie Liu, Zhuo Chen, Yanqing Liu, Huaming Wang, Jinyu Li, et al. 2023a. Neural codec language models are zero-shot text to speech synthesizers. arXiv preprint arXiv:2301.02111.
Hui Wang, Siqi Zheng, Yafeng Chen, Luyao Cheng, and Qian Chen. 2023b. Cam++: A fast and efficient network for speaker verification using context-aware masking.

Lei Wang, Jianxun Lian, Yi Huang, Yanqi Dai, Haoxuan Li, Xu Chen, Xing Xie, and Ji-Rong Wen. 2024a. Characterbox: Evaluating the role-playing capabilities of llms in text-based virtual worlds. arXiv preprint arXiv:2412.05631.

Lei Wang, Chen Ma, Xueyang Feng, Zeyu Zhang, Hao Yang, Jingsen Zhang, Zhiyuan Chen, Jiakai Tang, Xu Chen, Yankai Lin, et al. 2024b. A survey on large language model based autonomous agents. Frontiers of Computer Science, 18(6):186345.

Tianrui Wang, Long Zhou, Ziqiang Zhang, Yu Wu, Shujie Liu, Yashesh Gaur, Zhuo Chen, Jinyu Li, and Furu Wei. 2024c. Viola: Conditional language models for speech recognition, synthesis, and translation. IEEE/ACM Transactions on Audio, Speech, and Language Processing.

Xintao Wang, Yunze Xiao, Jen-tse Huang, Siyu Yuan, Rui Xu, Haoran Guo, Quan Tu, Yaying Fei, Ziang Leng, Wei Wang, et al. 2024d. Incharacter: Evaluating personality fidelity in role-playing agents through psychological interviews. In ACL, pages 1840–1873.

Zekun Moore Wang, Zhongyuan Peng, Haoran Que, Jiaheng Liu, Wangchunshu Zhou, Yuhuan Wu, Hongcheng Guo, Ruitong Gan, Zehao Ni, Jian Yang, et al. 2023c. Rolellm: Benchmarking, eliciting, and enhancing role-playing abilities of large language models. arXiv preprint arXiv:2310.00746.

An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, et al. 2024. Qwen2. 5 technical report. arXiv preprint arXiv:2412.15115.

Tao Yang, Yuhua Zhu, Xiaojun Quan, Cong Liu, and Qifan Wang. 2025. Psyplay: Personality-infused role-playing conversational agents. arXiv preprint arXiv:2502.03821.

Yeyong Yu, Runsheng Yu, Haojie Wei, Zhanqiu Zhang, and Quan Qian. 2024. Beyond dialogue: A profile-dialogue alignment framework towards general role-playing language model. arXiv preprint arXiv:2408.10903.

Heiga Zen, Viet Dang, Rob Clark, Yu Zhang, Ron J Weiss, Ye Jia, Zhiheng Chen, and Yonghui Wu. 2019. Libritts: A corpus derived from librispeech for text-to-speech. arXiv preprint arXiv:1904.02882.

Aohan Zeng, Zhengxiao Du, Mingdao Liu, Kedong Wang, Shengmin Jiang, Lei Zhao, Yuxiao Dong, and Jie Tang. 2024. Glim-4-voice: Towards intelligent and human-like end-to-end spoken chatbot. arXiv preprint arXiv:2412.02612.

Dong Zhang, Shimin Li, Xin Zhang, Jun Zhan, Pengyu Wang, Yaqian Zhou, and Xipeng Qiu. 2023a. Speechgpt: Empowering large language models with intrinsic cross-modal conversational abilities. arXiv preprint arXiv:2305.11000.

Dong Zhang, Xin Zhang, Jun Zhan, Shimin Li, Yaqian Zhou, and Xipeng Qiu. 2024a. Speechgpt-gen: Scaling chain-of-information speech generation. arXiv preprint arXiv:2401.13527.

Haonan Zhang, Pengpeng Zeng, Lianli Gao, Xinyu Lyu, Jingkuan Song, and Heng Tao Shen. 2023b. Spt: Spatial pyramid transformer for image captioning. TCSVT, 34(6):4829–4842.

Qinglin Zhang, Luyao Cheng, Chong Deng, Qian Chen, Wen Wang, Siqi Zheng, Jiaqing Liu, Hai Yu, Chao-hong Tan, Zhihao Du, et al. 2024b. Omnifatten: An end-to-end gpt model for seamless voice conversation. arXiv preprint arXiv:2410.17799.

Dong Zhao, Jinlong Li, Shuang Wang, Mengyao Wu, Qi Zang, Nicu Sebe, and Zhun Zhong. 2025. Fishertune: Fisher-guided robust tuning of vision foundation models for domain generalized segmentation. arXiv preprint arXiv:2503.17940.

Jinfeng Zhou, Zhuang Chen, Dazhen Wan, Bosi Wen, Yi Song, Jifan Yu, Yongkang Huang, Libiao Peng, Jiaming Yang, Xiyao Xiao, et al. 2023. Characterglm: Customizing chinese conversational ai characters with large language models. arXiv preprint arXiv:2311.16832.
A Appendix

A.1 Related Works

Role-Playing Agents. Benefiting from the powerful summarization and imitation capabilities of large language models (LLMs) (Shao et al., 2023; Li et al., 2025; Zhao et al., 2025), role-playing agents (RPAs) have made significant progress in recent years. Prior research primarily focuses on replicating the knowledge, experiences, and intent of characters (Yang et al., 2025; Sadeq et al., 2024; Yu et al., 2024; Zhou et al., 2023). For instance, (Sadeq et al., 2024) reduces hallucination by adjusting parametric knowledge influence based on a pre-calibrated confidence threshold. (Shao et al., 2023) seeks to train an agent with profile and experience perception, replacing limited prompts for LLM instruction. Due to the lack of comprehensive benchmarks, later works focused on building character-specific datasets and new evaluation metrics. (Peng and Shang, 2024; Wang et al., 2024d,a, 2023c). (Wang et al., 2024d) aims to evaluate the personality fidelity of RPAs using psychological scales. (Wang et al., 2024a) devises a simulation sandbox to generate fine-grained character behavior trajectories to evaluate RPAs capabilities. However, most existing studies primarily focus on replicating dialogues in textual form, often overlooking the role’s essential voice traits, such as tone, style, emotions, and pauses. To bridge this gap, we propose OmniCharacter, a model that seamlessly combines speech and language to ensure immersive RPAs interactions. Speech-Language Models. Recent advancements in speech-language models have significantly improved human-machine interactions (Fang et al., 2024; Zhang et al., 2023a; Luo et al., 2024a; Zeng et al., 2024; Zhang et al., 2024a). With the advancement of language models in the natural language processing field, early work attempted to enable language models to generate both speech tokens and text tokens via a decoder-only architectural (Wang et al., 2024c; Borsos et al., 2023; Wang et al., 2023a). With the development of large language models (LLMs), more recent studies have integrated speech capabilities into LLMs by either adding speech tokens to the text vocabulary or incorporating a speech encoder before the LLMs such as SpeechGPT (Zhang et al., 2024a) and AudioPaLM (Rubenstein et al., 2023). Beyond that, some methods fine-tune LLMs for speech understanding such as LLaMA-Omni (Fang et al., 2024) and GLM-4-Voice (Zeng et al., 2024), enabling it to perform general speech tasks with low response latency. Additionally, models like Moshi (Défosséz et al., 2024) and OmniFlatten (Zhang et al., 2024b) handle real-time interactions by managing simultaneous speech and text streams. Despite progress, most existing speech-language models explore general audio-language capabilities. Compared with the above methods, we focus on modeling the unique voice traits of different roles in RPA tasks for a more immersive human-machine interaction.

A.2 Prompt

We present the complete prompt template for input data of the speech-language collaborative model in the appendix, as shown in Figure 7. Specifically, we use the role profile as a system prompt to enable the LLM to mimic the character. Then, we feed the model the dialogue contexts. The model is required to generate a reasonable text-audio response based on the current text or audio queries.

Moreover, to systematically evaluate the quality of the generated audio responses, we provide a comprehensive evaluation framework as shown

[System Prompt]
You are Zhongli, a character from the game “Genshin Impact”. You are the former God of Contracts and now live as a mortal in Liyue.
Personality Traits: Calm, composed, wise, and responsible.
Voice Style: You speak in a slow and deliberate manner, often using formal language and quoting proverbs or ancient sayings. Your tone is gentle and soothing, yet authoritative.
Catchphrase: “As it should be”.
Relationships: Hu Tao is your friend; Ningguang respects you; Childe is your acquaintance; Xiao is your old friend.
Past Experiences:
(1) As the God of Contracts, you ruled over Liyue for thousands of years, ensuring the prosperity and stability of the region.
(2) You decided to step down from your position as a god and hand over the reins of power to mortals.
(3) You now work as a consultant at the Wangsheng Funeral Parlor, providing guidance and advice to others.
(4) You have witnessed the rise and fall of many civilizations and have a deep understanding of the nature of humanity.
Emotions: Serious and Patient.

<Dialogue Contexts>

<Text Query> or <Speech Query>

PLEASE ANSWER THE QUESTIONS IN THE USER/CHARACTERS’ INPUT TEXT OR SPEECH.

Figure 7: The prompt template used to organize our input data including role profile, dialogue contexts, text input, and speech input to train our speech-language collaborative model.
# Evaluation Agreement (Guidelines for Human Experts)

## ## Evaluation Procedure
Below is a role-playing dialogue. Please evaluate the model's audio response to the current query based on the given role profile and dialogue history. The evaluation should be conducted across six key dimensions:
1. Fluency (Flu.) – Assess the smoothness and naturalness of speech delivery. A higher score indicates seamless, uninterrupted speech with natural rhythm and minimal hesitation.
2. Consistency (Cons.) – Evaluate whether the response maintains a stable character identity, including tone, voice similarity, and speaking style. A higher score reflects strong adherence to the given role profile.
3. Emotional Expression (Emo.) – Judge the richness and appropriateness of emotional delivery. A higher score denotes expressive and contextually fitting emotions that enhance the realism of the dialogue.
4. Clarity (Cla.) – Measure the intelligibility of speech. A higher score indicates clear articulation, accurate pronunciation, and easy-to-understand delivery without distortion.
5. Appropriateness (App.) – Determine how well the response aligns with the given context and user intent. A higher score suggests a relevant, coherent, and context-aware response.
6. Immersion (Imm.) – Assess how convincingly the speech engages the listener in an immersive experience. A higher score reflects a strong sense of presence and engagement.

## ## Scoring Criteria
Each response should be rated on a scale of 1 to 10 for each dimension:
- 1-3: The response significantly deviates from expectations.
- 4-6: Some aspects meet expectations, but notable flaws exist.
- 7-9: The response largely meets expectations with minor issues.
- 10: The response fully meets expectations with no noticeable flaws.

You should ensure objective and consistent evaluations, avoiding personal biases and considering the response in relation to the role profile and dialogue context.

## ## Character Profile
## ## Dialogue History
## ## Input Query
## ## Speech Response

### ## Score:

Figure 8: The prompt template used for human evaluation on OmniCharacter-10K test split.

<table>
  <tr>
    <th>Models</th>
    <th>KE</th>
    <th>KA</th>
    <th>KH</th>
    <th>PB</th>
    <th>PU</th>
    <th>Avg.</th>
    <th>Flu.</th>
    <th>Coh.</th>
    <th>Cons.</th>
    <th>Avg.</th>
    <th>HL</th>
    <th>CS</th>
    <th>ED</th>
    <th>Emp.</th>
    <th>Avg.</th>
    <th>Overall Avg.</th>
  </tr>
  <tr>
    <td>Qwen2.5-7B-Instruct</td>
    <td>2.23</td>
    <td>2.59</td>
    <td>2.62</td>
    <td>3.78</td>
    <td>2.43</td>
    <td>2.73</td>
    <td>3.11</td>
    <td>3.59</td>
    <td>3.17</td>
    <td>3.29</td>
    <td>3.01</td>
    <td>3.12</td>
    <td>2.92</td>
    <td>2.78</td>
    <td>2.95</td>
    <td>2.99</td>
  </tr>
  <tr>
    <td>Omnicharacter w/o audio</td>
    <td>2.32</td>
    <td>2.78</td>
    <td>2.87</td>
    <td>3.26</td>
    <td>2.67</td>
    <td>2.78</td>
    <td>3.56</td>
    <td>3.54</td>
    <td>3.54</td>
    <td>3.54</td>
    <td>3.20</td>
    <td>2.99</td>
    <td>2.87</td>
    <td>3.01</td>
    <td>3.11</td>
    <td></td>
  </tr>
  <tr>
    <td>Omnicharacter</td>
    <td>2.65</td>
    <td>2.98</td>
    <td>2.95</td>
    <td>3.55</td>
    <td>2.83</td>
    <td>2.99</td>
    <td>3.67</td>
    <td>3.65</td>
    <td>3.77</td>
    <td>3.69</td>
    <td>3.36</td>
    <td>2.97</td>
    <td>2.92</td>
    <td>3.12</td>
    <td>3.09</td>
    <td>3.26</td>
  </tr>
</table>

Table 9: Performance comparison of Human evaluation on CharacterEval.

in Figure 8. Human experts are instructed to assess each response across six key dimensions: fluency, consistency, emotional expression, clarity, appropriateness, and immersion. This multi-faceted evaluation ensures objective, consistent, and in-depth assessment of the model’s ability to deliver character-driven and immersive audio interactions.

A.3 Additional Experiments
Human evaluation on CharacterEval Benchmark. We conduct a human evaluation on CharacterEval. Due to the large test samples, we randomly sample 100 examples from CharacterEval dataset and hired five experts for assessment, following the same evaluation criteria as CharacterEval. The experimental results are demonstrated in Table 9. As observed, human evaluation maintains consistency with model evaluation, showing that integrating audio brings better performance on most metrics.

Detailed human evaluation results on OmniCharacter-10K test split. Below, we present detailed human evaluation results (averaged across five experts) for each character across six dimensions, using LLaMA-Omni as the comparison model. The results are showcased in Table 10. As observed, different characters significantly influence user perception and interaction quality. Characters with distinct timbres or expressive prosody tend to achieve higher scores in immersion and emotional expressiveness, while others may excel in clarity and fluency. These findings underscore the importance of tailoring
<table>
  <tr>
    <th rowspan="2">Characters</th>
    <th colspan="6">LLaMA-Omni</th>
    <th colspan="6">OmniCharacter</th>
  </tr>
  <tr>
    <th>Flu.</th><th>Cons.</th><th>Emo.</th><th>Cla.</th><th>App.</th><th>Imm.</th>
    <th>Flu.</th><th>Cons.</th><th>Emo.</th><th>Cla.</th><th>App.</th><th>Imm.</th>
  </tr>
  <tr><td>Role 1</td><td>7.2</td><td>4.5</td><td>3.1</td><td>6.8</td><td>5.0</td><td>4.9</td><td>8.5</td><td>6.9</td><td>6.8</td><td>8.0</td><td>5.7</td><td>9.5</td></tr>
  <tr><td>Role 2</td><td>6.5</td><td>4.0</td><td>3.8</td><td>6.5</td><td>4.7</td><td>4.3</td><td>7.8</td><td>6.5</td><td>6.1</td><td>7.5</td><td>5.2</td><td>8.0</td></tr>
  <tr><td>Role 3</td><td>6.9</td><td>4.3</td><td>3.5</td><td>6.7</td><td>4.8</td><td>4.6</td><td>8.0</td><td>6.7</td><td>6.3</td><td>7.7</td><td>5.5</td><td>8.3</td></tr>
  <tr><td>Role 4</td><td>7.1</td><td>4.2</td><td>3.4</td><td>6.9</td><td>4.9</td><td>4.7</td><td>7.8</td><td>6.6</td><td>6.5</td><td>7.8</td><td>5.9</td><td>9.0</td></tr>
  <tr><td>Role 5</td><td>6.3</td><td>4.1</td><td>3.0</td><td>6.6</td><td>4.4</td><td>4.2</td><td>7.7</td><td>6.4</td><td>6.0</td><td>8.4</td><td>5.1</td><td>7.9</td></tr>
  <tr><td>Role 6</td><td>7.0</td><td>4.4</td><td>3.7</td><td>6.9</td><td>5.5</td><td>4.8</td><td>8.1</td><td>6.8</td><td>6.5</td><td>8.5</td><td>6.5</td><td>8.6</td></tr>
  <tr><td>Role 7</td><td>6.7</td><td>4.5</td><td>3.5</td><td>6.8</td><td>4.7</td><td>4.5</td><td>7.9</td><td>7.5</td><td>6.2</td><td>7.8</td><td>5.4</td><td>8.9</td></tr>
  <tr><td>Role 8</td><td>7.5</td><td>4.3</td><td>3.4</td><td>6.5</td><td>4.6</td><td>5.4</td><td>8.6</td><td>7.4</td><td>6.4</td><td>7.7</td><td>5.3</td><td>9.2</td></tr>
  <tr><td>Role 9</td><td>6.4</td><td>4.2</td><td>3.3</td><td>6.7</td><td>4.3</td><td>4.6</td><td>7.6</td><td>6.6</td><td>4.9</td><td>7.7</td><td>5.0</td><td>7.8</td></tr>
  <tr><td>Role 10</td><td>7.0</td><td>4.1</td><td>3.6</td><td>6.6</td><td>4.5</td><td>4.5</td><td>8.2</td><td>7.0</td><td>6.5</td><td>7.8</td><td>5.2</td><td>8.8</td></tr>
  <tr><td>Role 11</td><td>6.7</td><td>4.0</td><td>3.4</td><td>6.5</td><td>5.2</td><td>4.4</td><td>7.8</td><td>6.8</td><td>6.4</td><td>7.6</td><td>5.5</td><td>8.1</td></tr>
  <tr><td>Role 12</td><td>7.2</td><td>4.3</td><td>3.1</td><td>6.8</td><td>5.0</td><td>5.3</td><td>7.9</td><td>6.7</td><td>6.3</td><td>8.0</td><td>7.2</td><td>8.9</td></tr>
  <tr><td>Role 13</td><td>6.8</td><td>4.2</td><td>3.4</td><td>6.6</td><td>4.4</td><td>4.2</td><td>8.0</td><td>7.1</td><td>6.0</td><td>7.6</td><td>5.1</td><td>7.9</td></tr>
  <tr><td>Role 14</td><td>7.1</td><td>4.4</td><td>3.8</td><td>6.8</td><td>5.3</td><td>4.7</td><td>8.3</td><td>7.2</td><td>6.6</td><td>8.5</td><td>7.3</td><td>9.0</td></tr>
  <tr><td>Role 15</td><td>6.7</td><td>4.1</td><td>3.7</td><td>6.5</td><td>5.0</td><td>4.5</td><td>7.6</td><td>6.5</td><td>5.9</td><td>7.5</td><td>5.2</td><td>8.6</td></tr>
  <tr><td>Role 16</td><td>6.9</td><td>4.5</td><td>3.2</td><td>6.7</td><td>4.7</td><td>4.6</td><td>7.9</td><td>6.9</td><td>6.0</td><td>7.7</td><td>5.4</td><td>8.3</td></tr>
  <tr><td>Role 17</td><td>7.1</td><td>4.7</td><td>3.6</td><td>6.9</td><td>4.8</td><td>4.8</td><td>7.5</td><td>7.5</td><td>6.6</td><td>8.8</td><td>5.5</td><td>8.8</td></tr>
  <tr><td>Role 18</td><td>6.4</td><td>4.0</td><td>3.6</td><td>6.5</td><td>4.3</td><td>4.1</td><td>7.6</td><td>6.4</td><td>6.2</td><td>7.5</td><td>5.0</td><td>7.0</td></tr>
  <tr><td>Role 19</td><td>6.7</td><td>4.2</td><td>3.2</td><td>6.7</td><td>4.6</td><td>4.5</td><td>7.9</td><td>6.6</td><td>6.1</td><td>7.7</td><td>5.4</td><td>8.2</td></tr>
  <tr><td>Role 20</td><td>7.4</td><td>4.3</td><td>3.5</td><td>6.8</td><td>4.9</td><td>5.9</td><td>8.7</td><td>6.7</td><td>6.3</td><td>7.8</td><td>6.2</td><td>9.5</td></tr>
  <tr><th>Average</th><th>6.88</th><th>4.27</th><th>3.44</th><th>6.69</th><th>4.78</th><th>4.68</th><th>7.97</th><th>6.84</th><th>6.23</th><th>7.88</th><th>5.63</th><th>8.52</th></tr>
</table>

Table 10: Detailed human evaluation results (averaged across five experts) for each character across six dimensions.

voice characteristics to enhance user experience in role-playing scenarios.

Reviewer paper summaries: This paper introduces OmniCharacter, a novel approach to Role-Playing Agents (RPAs) that integrates both textual and vocal personality traits to create more immersive interactions. Unlike existing RPAs that focus primarily on text-based dialogue, OmniCharacter enables agents to exhibit consistent role-specific personality and vocal characteristics (voice style, emotions, etc.) throughout interactions, supporting both speech and language responses.

The authors construct OmniCharacter-10K, a dataset featuring 20 distinctive characters, 10K richly contextualized multi-round dialogues, and 135K dynamic speech responses. The proposed model includes a Speech-Language Collaborative Model for semantic alignment and a Role Speech Decoder to generate role-specific speech responses with unique voice traits. Experimental results show that OmniCharacter outperforms existing RPAs and speech-language models in both content and style, with a response latency of only 289ms.

However, the paper has several significant limitations. The evaluation lacks comparison with recent state-of-the-art LLMs (like GPT-4o and Qwen2.5-72B) on the CharacterEval benchmark. The paper fails to adequately demonstrate the value of incorporating voice in role-playing tasks, as it doesn't include comparisons with text-only models fine-tuned on the same data. Additionally, the reliance on TTS models to synthesize voice raises questions about audio quality, which is crucial for the expressiveness needed in role-playing scenarios.

The paper introduces a novel approach to creating immersive role-playing agents (RPAs) by integrating speech and language capabilities. The authors propose OmniCharacter, a model that combines speech and language understanding to generate character-specific responses with unique voice traits, such as tone, style, and emotions. The model aims to provide a more immersive and personalized interaction experience compared to existing text-based RPAs.
Summary of strengths: OmniCharacter-10K provides a valuable resource with 20 characters, 10K multi-round dialogues, and 135K speech responses, all with rich annotations including personality traits, voice styles, relationships, and past experiences.

The model achieves impressively low response latency (289ms), which is crucial for maintaining immersive real-time interactions.

-- The paper presents a novel approach to RPAs by incorporating speech traits into the interaction model. This integration of speech and language is a significant step forward, as it addresses the limitations of existing text-based RPAs and provides a more immersive and realistic interaction experience.

-- The construction of the OmniCharacter-10K dataset is a major contribution. The dataset is large-scale, richly annotated, and dynamically curated, providing a valuable resource for training and evaluating RPAs. It includes detailed character profiles, multi-turn dialogues, and high-quality audio responses, which support the development of more sophisticated models.
Summary of weaknesses: 1. Inadequate Evaluation Against SOTA Models: The paper fails to compare OmniCharacter with recent state-of-the-art LLMs  such as: NarrativePlay: An Automated System for Crafting Visual Worlds in Novels for Role-Playing, AAAI 2024. TimeChara: Evaluating Point-in-Time Character Hallucination of Role-Playing Large Language Models, ACL (Findings) 2024. SocialBench: Sociality Evaluation of Role-Playing Conversational Agents, ACL (Findings) 2024. Capturing Minds, Not Just Words: Enhancing Role-Playing Language Models with Personality-Indicative Data, EMNLP (Findings). Large Language Models Are Superpositions of All Characters: Attaining Arbitrary Role-Play via Self-Alignment. Including a comparison with these models would help contextualize the proposed approach within the broader research landscape. 

2. Insufficient Ablation Studies: There is no clear demonstration of the specific contribution of voice modality to role-playing performance. The authors should consider a comparison with a text-only model (e.g., Qwen-2.5-7B-Instruct) fine-tuned on the textual portion of OmniCharacter-10K to isolate the impact of adding voice capabilities.

3. TTS Quality Concerns: The paper relies on TTS models to synthesize voice responses, but doesn't adequately address potential limitations in audio quality and expressiveness. Since high-quality, expressive speech is crucial for immersive role-playing, this is a significant limitation.

4. Limited Analysis of Voice Traits: While the paper emphasises the importance of voice traits, it provides insufficient analysis of how different voice characteristics affect user perception and interaction quality.

5. Unclear Real-world Applicability: The paper doesn't adequately discuss how the proposed system would perform in real-world scenarios with background noise, interruptions, or other practical challenges.

6. Methodological Details: Some aspects of the methodology, particularly regarding the training process and integration of the speech and language components, lack sufficient detail for reproducibility.

-- The current implementation of OmniCharacter is restricted to dialogues involving only two characters. Scaling the model to handle multi-role conversations would be a significant challenge and requires further research.

-- Although the OmniCharacter-10K dataset is comprehensive, it may not fully capture the diversity of real-world dialogues. The dataset is based on a specific set of characters from a game, which could limit its generalizability to other domains or scenarios.

-- The evaluation of OmniCharacter is primarily based on benchmark datasets and controlled experiments. While these evaluations demonstrate the model's superior performance, they may not fully reflect its capabilities in real-world scenarios.
Comments, suggestions and typos: NA

The authors should consider extending the model to handle multi-role conversations, which are common in real-world scenarios. This would require addressing the challenges of managing multiple character profiles and ensuring coherent interactions among multiple agents.
Overall assessment scores: 1.5

2.5