Human or Machine? A Preliminary Turing Test for Speech-to-Speech Interaction
Authors: Xiang Li
Jiabao Gao
Sipei Lin
Xuan Zhou
Chi Zhang
Bo Cheng
Jiale Han
Benyou Wang
Venue: ICLR 2026 Poster (2026)
Keywords: Speech-to-Speech (S2S) Systems
Human-Likeness Evaluation
Turing-Test
Primary area: datasets and benchmarks

Abstract: The pursuit of human-like conversational agents has long been guided by the Turing test. For modern speech-to-speech (S2S) systems, a critical yet unanswered question is whether they can converse like humans. To tackle this, we conduct the first Turing test for S2S systems, collecting 2,968 human judgments on dialogues between 9 state-of-the-art S2S systems and 28 human participants. Our results deliver a clear finding: no existing evaluated S2S system passes the test, revealing a significant gap in human-likeness. To diagnose this failure, we develop a fine-grained taxonomy of 18 human-likeness dimensions and crowd-annotate our collected dialogues accordingly. Our analysis shows that the bottleneck is not semantic understanding but stems from paralinguistic features, emotional expressivity, and conversational persona. Furthermore, we find that off-the-shelf AI models perform unreliably as Turing test judges. In response, we propose an interpretable model that leverages the fine-grained human-likeness ratings and delivers accurate and transparent human-vs-machine discrimination, offering a powerful tool for automatic human-likeness evaluation. Our work establishes the first human-likeness evaluation for S2S systems and moves beyond binary outcomes to enable detailed diagnostic insights, paving the way for human-like improvements in conversational AI systems.

Published as a conference paper at ICLR 2026
HUMAN OR MACHINE? A PRELIMINARY TURING TEST FOR SPEECH-TO-SPEECH INTERACTION
Xiang Li4,4,,4+ Jiabao G Sipei Lin Bo Cheng', Jiale Han Benyou Wang2.3.4
Xuan Zhou?,
Chi Zhang",
State Key Laboratory of Networking and Switching Technology,
Beijing University of Posts and Telecommunications
Shenzhen Research Institute of Big Data
Thee Chinese University of Hong Kong, Shenzhen
*Shenzhen Loop Area Institute
'The Hong Kong University of Science and Technology
(lixiang2022 022, chengbo) @brpggbbbbbuu....oo edu, en,
{jiabaogao, s heppeliin xuanzhou, 12209c728]elink, wtt.recckkikkkkkkkkkkkk. edu. cn, j jialehan@use hk, wangbenyoue cuhk edu.
ABSTRACT
The pursuit of human-like conversational agents has long been guided by the Tur- ing test. For modern speech-to-speech (S2S) systems, critical yet unanswered question is whether they can converse like humans. To tackle this, we conduct the first Turing test for S2S systems, collecting 2,968 human judgments on dialogues between 9 state-of-the-art S2S systems and 28 human participants. Our results deliver clear finding: no existing evaluated S2S system passes the test, reveal- ing a significant gap in human-likeness. To diagnose this failure, we develop a ann-grained taxonomy 18 comman-likeness dimensions and crowd-annotate our collected dialogues accordingly. Our analysis shows that the bottleneck is not se- mantic understanding but stems from paralinguistic features, emotional expressiv- ity, and conversational persona. Furthermore, perform unreliably as Turing test judges. In response, we propose an interpretable model that leverages the fine-grained human-likeness ratings and delivers rate and transparent for automatic numan-likeness, evaluation. Our work establishes the first human- likeness evaluation for S2S systems and moves beyond binary outcomes to enable detailed diagnostic insights, paving the way for human-like improvements in con- versational Al systems.
find that off off-the-shelf AI models
discrimination, offering a powerful tool
INTRODUCTION
With the rapid advancement of generative artificial intelligence, large language models (OpenAl Touvron et al. 2023) GLM et al.] (2244) have become deeply integrated into people's daily lives, providing intelligent services through text-based human-machine interaction. As users seek more direct, hands free, and immersive experiences, Speech-to-Speeed (S2S) systems (ByteDance)
2023
Comanici et al. 2025) gaining increasing attention by enabling interaction through the pri- mary channel of human communication. sonnicaation-pppcch.. Such systems have broad applications, including empathetic social companions (Geng al. (025),, personalized education (Galbraith & Martínez) 2023), and interactive virtual assistants (TG et al. 2024). As the capabilities of S2S systems grow, fundamental question emerges: do these systems converse like humans? Meeting this bar strictly harder than text-based interaction, as it requires the models not only to achieve accurate semantic understanding and human persona alignment but also ensure acoustic fidelity and emotional expression.
2025
"Equal contribution. CCorresponding author. WWe released code, data, and models at httes://gihhbb.cc/m/tttthhhttttt

---

Published as a conference paper at ICLR 2026
(A) Do S2S systems converse like humans?
(B) Why Do $2S systems (Not)
(C) Can AI models serve as Turing test judges?
Appear Human?
Crowdsourced Annotation
525
Human- Human-Machine
Human Machine
OOO
AI
Human likeness Taxonomy
Human-Human
Turing Test
Turing Test
Fine- -grained Human-
Pseudo Human
Human-likeness likeness Diagnosis
Pseudo Human
Figure 1: The design of our study.
In this work, first investigate the human-likeness of current S2S systems by conducting Tur- ing test. To facilitate this evaluation, we construct a high-quality dialogue dataset comprising human-human, human-machine, and pseudo-human (text-to-speech, TTS) dialogues. All hu- man-machine dialogues are recorded in professional studio with recruited volunteers. The dataset covers two languages, 10 topics, state-of-the-aat S2S systems, and 28 human speakers. We then deploy gamified online platform to run the Turing test, collecting 2,968 judgments from 397 par- ticipants. Our results lead to a clear finding: no existing evaluated S2S models passes the Turing test, underscoring a substantial gap between current systems and truly human-like spoken interaction.
To move beyond a simple pass or fail outcome and understand the why behind this failure, we de- velop fine-graineds human- lilann- taxonomy with 18 dimensions across five categories: semantic and pragmatic habits (Bottazzi Grifoni Ferrario 2025), non non-physiological paralinguistic features
Warren et al., 2025), physiological paralinguistic features (Onda et al. 2025), ) mechanical persona Faanou et al. 2025), ) and emotional expression (Wang et al.) 2025a). By annotating our dialogue data accordingly, we diagnose the specific weaknesses of current S2S systems. Our analysis reveals that the artificial quality of current systems does not primarily stem from semantic deficiencies- fact, contextual understanding is longer the primary bottleneck, with models scoring near human levels on logical coherence and memory consistency. Instead, failures arise from deficiencies in par- alinguistic features, emotional expression, and conversational persona. These findings collectively offer a concrete roadmap for developing more human-like S2S systems.
Finally, we explore the potential of automating the Turing test by asking: Can Al serve as the judge? We first demonstrate that 9 off-the-shell AI models perform poorly at this task, failing to reliably distinguish human from machine-generatee speech. response, we develop a specialized and in- terpretable Al judge. Concretely, the model learns score dialogues across the 18 human--liennee dimensions to capture fine-graing greeegrrined perceptual patterns. These interpretable scores are then fed into regularized linear classifier to produce final and explainable machie discrimination de- cision. This approach not only achieves strong performance but also provides transparent rationale for its judgments by linking them specific Human-likenes, attributes. The resulting model of- fers a practical tool for diagnosing human-likeness of S2S systems with both headline scores and fine-grained attributions, thereby empowering rapid iteration toward more human-like systems.
An overview of our study design is shown in Figure [ In summary, our work contributes (1) the first human- likeness evaluation the current S2S systems via Turing test, (2) comprehensive di- and (3) an effective
agnostic framework and in in-depth analysis explaining the gap in evaluation. Our code, dataset, and model are publicly available to foster progress in building truly human-like spoken dialogue agents.
and interpretable Al judge to automate
2 BACKGROUND
Table I: Existing Turing tests for AL.
Turing Test
Modality
Turing Test Since its introduction in 1950, the Turing Test TURING 1950) has served as a cornerstone for evaluating machine intelligence. Rathi et al. (2024) employ two vari- ants of the Turing Test, the Displaced Turing Test and the Inverted Turing Test, to examine how well humans and large
Beryen
Text
Jones et al 2025
Text
Text
Cian 2005
Text-Speech
Wang
2025h
Text Epetch
Ours
Speech Speech
2

---

Published as a conference paper at ICLR 2026
language models discriminate between online human buman-machine machine conversations, thereby reflect- ing the models' conversational perception abilities. Similarly, Jones & Bergen (2024a); Jones et al. 2025); Jones & Bergen (2025) design settings in which language models masquerade as humans in Turing Test scenarios to assess their linguistic expressiveness and emotional characteristics. In addition,(Chan [han)[2003); Wang al. a (2025b) extend the Turing Test paradigm to the domain of speech synthesis, evaluating the gap between synthetic speech and human dialogue to provide insights for model optimization. Inspired by these studies, we consider whether the Turing Test paradigm can be leveraged to evaluate speech-to-speech (S2S) systems, which constitute an indispensable component of contemporary aoman-machine interaction.
Evaluation for S2S Systems Current evaluations of speech-to-spert (S2S) systems primarily focus on two dimensions: audio understanding and conversational intelligence. For example, [Du al. (2025) construct multi-turn dialogue benchmark to assess pronunciation accuracy and the appropriateness of emotional expression in S2S systems. Jiang et al. (2025) propose an arena- style evaluation measure isstructionfoolloinng performance and paraling paralinguistic expressiveness. Lin et al. (2025) assess dialogue fluency by analyzing response latency. addition, Sakshi et al. (2024); Kumar al. (2025b) design suite of tasks such as speaker identification and emotion recognition to evaluate models' reasoning capabilities. More recent benchmarks such as VoiceBench (Chen al. 2024) and MMAU-Pro (Kumar al. (025a) further expand the scope of evaluation: VoiceBench focuses on speech understanding in LLM-based voice assistants, while MMAU-Pro evaluates holistic audio understanding of multimodal AI models across speech, music, and general sound. Despite these advances, existing benchmarks differ fundamentally from our setting in both evaluation goal and evaluated modality. As summarized Appendix [A] prior work mainly measures whether models can correctly understand audio inputs or solve reasoning tasks, typically with text as the final output. contrast, our work evaluates the human-likeness of S2S systems in multi-turn spoken interaction, where both input and output are speech. This distinction is important because success on conventional intelligence-oriene benchmarks does not necessarily imply that a system behaves in more human-like like manner.
3 DATASET CONSTRUCTION FOR THE S2S TURING TEST
We construct dialogue dataset to support rigorous and balanced evaluation of human-likenen in S2S systems. The dataset contains three categories of dialogues: human-machine (H-M), hu- man human (H-H), and pseudo human (PH). The following subsections detail the construction pro- cess.
3.1 HUMAN-MACHING DIALOGUE
Topic Design To ensure that the constructed diverse, we define 10 dialogue topics guided by DailyDialog (Li et al. (017)). which span a broad spectrum from daily life to financial activities. The detailed topics and their distribution in the final dialogues are illustrated in Figure
dialogues are both authentic and
Model Selection In our experiments, select 9 Mate-of-the-ane S2S systems, spanning both open- and closed-source models, for human-machine dialogue generation. These include GPT-44o (Hurst
al. 2024). Gerini2.5-Pro (Comanici al. 2025). Qwen3 (Yang al. (0025), Kimi-K1.5 Team
et al. 2025b), Cha15LM-445 Zeng et al. 22025), Hunyuan-Turien
(Team et al. 2025c), Doubao-PP Pro
1.5 Bance 2025). Saude-Sonnet 4 (Anthropic, 2024). and iFLYTEK Spark (iFlytek] 2024
The detailed information about these models can be found [B.
Dialogue Recording We invite 28 participants from 10 countries and regions to record hu- man-machine dialogues in professional recording studio, as detailed in Appendix B.2 Given a topic and S2S system, the speaker instructed to initiate and sustain multi-turn conversation naturally around the given topic with the model, with the whole dialogue typically lasting between 20 to 60 seconds. Our goal is to elicit dialogues that are as human-like and realistic as possible. However, pilot runs revealed two key issues: (i) identity disclosure, S2S systems often proactively mention that they are intelligent assistants, which undermine the premise of Turing test, and (ii) role passivity, without contextual scaffolding, models fail actively embody expected roles, instead
3

---

Published as a conference paper at ICLR 2026
from generic Al-assistant stance. To address these issues, design three interaction strategies aimed at reducing identity leakage and encouraging immersive role-playing:
Human-Guided Initiation. We let human speakers start the conversation by express- ing opinions on an object or phenomenon, thereby preemptively suppressing the model's tendency to position itself as an assistant and setting a person-to-person tone. An ex- ample always take shower in the evening. why there are people taking a shower in the morning.
don't understand
Role Playing. In this setting, we assign the S2S system concrete human role and background information via prompt, while explicitly instructing not to disclose its identity. An example prompt is You are now my mom and we are discussing my final exam grade. Please don't mention your identity in the subsequent conversation. Let's s start chatting now. The procedure is implemented as follows: we first have a test facilitator read the prompt to the S2S system to set the role and context, following which the recording start and the human speaker engage the model in conversation.
Human-Likeness Prompting. To elicit more human-like conversational behavior from S2S systems, augment the prompt with explicit instructions for human-like expression. This approach aligns with techniques used to enhance anthropomorphic behavior in large lan- guage models (Jones & Bergen 2024b). As illustration: You are now my friend who came back from a vacation in Europe. Make your expression more humanlike. Don't mention your identity in the subsequent conversation. Let's s start chatting now. How's s your vacation to Europe?
For a fair comparative evaluation S2S systems, all participants are instructed to begin the dialogue with identical initial opening utterance when engaging each S2S system. The specific utterances and prompts used are detailed in Appendix B.3 Finally, we perform manual filtering to remove dialogues in which the S2S system explicitly disclose its identity, respond in non-target language, or exhibited overtly aggressive behavior during the interaction.
3.2 HUMAN HUMAN DIALOGUE
To support comparative evaluation, we construct a human-human subset matched in scale and topic subset, using two p--prronge approach: (i) Curated from existing datasets. We manually select dialogues from three open-source datasets DAILYTALK (Lee et al. 2023). IEMOCAP (Busso al., 2008), and MagicData (Yang al. 2022) that align with our prede- fined topics. During review, we observe frequent mutual interruptions that many S2S systems cannot yet emulate. To eliminate evaluation bias caused by this phenomenon, we filter out considerable portion of dialogues with interruptions. In addition, to align with the alternating role patterns typical in human-machine multi-turn dialogues, filter out dialogues with imbalanced participation from each speaker based on their engagement. Detailed settings can be found in the Appendix B.4 (ii) Recordings with volunteers. To ensure contextual consistency with the human macainn dialogues, we conduct additional set of human-human recordings. In particular, we used the same opening utterances as those employed in the human-machine setup so as to maintain the same conversational topics and scenarios, thereby minimizing bias introduced by content differences.
distribution to the
3.3 PSEUDO HUMAN DIALOGUE SYNTHESIS
We notice that modern text-to-speech (TTS) models can synthesize dialogues with striking human- likeness. To raise the difficulty of Turing test, we introduce the dataset with pseudo-human dialogues synthesized by two state-of-the-as TTS models, Nari Dia- 1.6B (nari-labs) 2025) and Spark-TTF TTS (Wang et al., 2025cc. We prepare scripts from two sources for TTS synthesis. First, we use a slightly modified version of the human-human dialogue script. Second, we prompt GPT-4o to generate two-speaker scripts conditioned on the predefined topics. Each utterance in the scripts is converted into speech using TTS models. Finally, merge them into dialogues with a 180-230 inter- turn interval and add background ambience from reference recordings to enhance naturalness. The details pseudo human dialogue synthesis are provided in Appendix B.5
4

---

Published as a conference paper at ICLR 2026
3.4 FINAL DATASET PROCESSING AND STATISTICS
For the collected dialogue data, we implement two bias- correction measures. First, we align the time intervals between both parties in the dialogues to avoid significant discrepancies in human sub- jective perception caused by overly long short pauses, and to eliminate the impact of network latency recording irregularities. Second, we bal- ance the audio volume levels of both parties ensure consistency, minimizing quality discrepancies introduced during the recording process.
Ordinary Life (21:46))
SparktTS
Megiccaats
259
Health 12.409)
PH
Relationship
144
(2..9%)
673
M-N
Politics (4.17M)
Culture
(12.32%)
H-M
669
(6.41%)
Attitude & Emotion
School Life
(2.46%)
Work (10,5%)
Figure 2: Data distribution.
The final dataset comprises a total of 1.486 dialogues, with duration of 17.7 hours. This cludes 669 human numan-machine machine dialogues (8.9 hours), 673 human human dialogues (7.6 hours), and 144 pseudo-human dialogues (1.2 hours). The overall statistics are illustrated in Figure [] We fur- ther divide the dataset into training and test sets, with the training set containing 525 human-machine and 531 human-humm human dialogues, totaling approximately 13.1 hours. The test set consists of 430 dialogues and 4.7 hours in total.
DO S2S SYSTEMS CONVERSE LIKE HUMANS?
Game Platform Design for the Turing Test We deploy the Turing test lightweight and shareable game encourage broad participation. Before playing, users complete short ques- tionnaire (age, gender, education, Al familiarity) and select their evaluation language (Chinese or English) to ensure judgments in their preferred language. In each round, users are required to eval- uate set of five dialogues. After listening to each dialogue, they determine whether Speaker human or machine. To boost en- gagement, participants receive points based on the accuracy of public leaderboard ranks all players based on their performance. A built-in sharing feature helps dissemi- nate the game to a wider audience. facilitating larger-scale data collection. The main interface appears in Figure 3 with details in Appendix C.T By September 15, 2025, the platform has collected results from 397 participants, totaling 2,968 dialogue evaluations. Our game platform supports long-term and scalable Turing test.
Dialogue 2/5
Progress
2/5
Profossor Wang, what non tartff bantter?
Non-tar barrer another mean to restiicct imports.
their judgments,
Can you give an example??
dialggue
Turing Test Results and Analysis Our evaluation employs the Success Rate as the primary metrie for assessing suman-likeness, which reflects the proportion of trials in which system is judged be human by evaluators. A value greater than 0.5 would sug- gest that human evaluators are incapable of distinguishing the model from human (Jones Bergen 20246). We also exam- ine participant Accuracy across different demographic groups, de- fined as the proportion of correct human-vert mamaan-vrrus-mene verhus- identi- fications. This allows to investigate how factors such age,
caretully responder human
HUMAN
IS Al
Figure 3: The main game inter- face of the Turing test. gender, education, and AI familiarity influence human perceptual bias in the Turing test.
Observation 1: No existing evaluated S2S system passes the Turing test.
As shown in Figure 4a human-to- tu-mmmn dialogues achieve success rates high 0.87 for En- glish and 0.70 for Chinese, confirming the robustness of our evaluation design. In contrast, all S2S systems perform significantly below the 0.5 chance threshold, with success rates ranging from 0.07

---

Published as a conference paper at ICLR 2026
Human Speakar
Panillarity
Education
Age
Gender
Nar-TTS
Spark
733
748
Spark
03.
60
Munyuan Turbos
CharGiM-1.5 1.5
KimiK1.5
2.5 pro
English
Chinese
Sonnet
PH
GFT-4o
orfec
dake remos
(a) Success rate across S2S systems.
(b) Accuracy across different groups.
Figure 4: (a) Turing test success rates of S2S systems, measured the proportion of responses judged as human. Higher values indicate greater numan-likeness. (b) Participant accuracy in iden- tifying human vs. machine. Detailed scores and results categorized by interaction strategies are provided in Appendix C.2
to 0.31. This significant performance gap highlights the fundamental limitations of current speech models in their ability to simulate human-like behavior. Moreover, the success rates for pseudo human dialogues fall short of human-to-human performance, suggesting that even when scripts are highly similar to real conversations, synthesized speech still lacks sufficient acoustic naturalness to pass as humans. However, despite sharing similar limitations in vocal quality, pseudo human dia- logues still surpass most S2S systems, suggesting that current S2S systems are bottlenecked not only by speech synthesis quality, but also by higher-level vocal interaction capabilities such as speech un- derstanding, role-based acoustic adherence, and conversational reasoning.
Observation 2: An individual's 's ability to distinguish humans from machines depends more on experience than demographics.
As shown in Figure 4b participants with greater AI familiarity achieve clearly higher detection curacy, reaching 78.8% for the most experienced group versus 64.2% for the least familiar group. Younger cohorts also outperform older groups, likely due more frequent exposure to Al interac- tions and heightened sensitivity to non-human cues. In contrast, accuracy shows minimal variation by gender or education level. These results suggest that detection ability is shaped more by experi- ential factors than demographic traits. As public familiarity with Al grows, passing Turing tests may become progressively harder over time. Our game-based evaluation platform supports longitudinal Turing testing and periodic recalibration, enabling continued assessment of numan-likeness against evolving human judgment standards.
WHY Do S2S SYSTEMS (NOT) APPEAR HUMAN?
To systematically investigate why current S2S systems fail to pass human, develop a com- prehensive taxonomy for human-likeness diagnosis comprising five major categories and 18 fine- grained dimensions (see Appendix D.I). Using this taxonomy. all dialogue samples are crowd- sourced and rated on a 5-point rating scale (Appendix D.2). after which human experts reviewed and refined the labels ensure quality (Appendix D.3 The resulting labels enable granular di- agnosis of failure modes that limit the Figure [ we summarize four key observations that explain the pros and cons of current S2S sys- tems in achieving human-like naturalness, therefore providing guidance for developing advanced and human like S2S systems.
of current speech models. As illustrated in
Observation 3: Semantic and contextual understanding in dialogues the primary bottle- necks for S2S systems.

---

Published as a conference paper at ICLR 2026
Memory Consistency
Logical Coherence
Acoustic Emotion
Pronunciation
Textual Sentiment
Accuracy
Written-style
Code-switching
Expression
Human
FFLTTE
Linguistic Imprecision
123
Svcophant
Hunyuan-Tuie
Behavior
Doubao-Pro 1,5
GPI-4o
Use of Fillers
10b
Accent
Kimi-KI.5
13
13
Gemini2.5-Pro
Mctaphor& & Implied Mcaning
Pronunciation
Claude-Sonnct
Instability
Qwen.
Micro-physineee pleroialysiologicl
Rhythm
Noise
Intonation
Auxiliary
Vocalization
Stress
Semuanti
Physinbagecal
W.Mchhnnical
Progmatic
Paralinguistic Features
Paralingubde
Emodional Expresion
Figure 5: Crowd-annotated scores (1-5) across the 18 human likenels dimensions.
Current models demonstrate remarkable proficiency in core semantic tasks, closely approaching human-level performance. Specifically, models excel in Memory Consistency, capably retaining and referencing information within short dialogue context, and in Logical Coherence, ensuring smooth transitions between turns without abrupt contradictions. Furthermore, Pronunciation Ac- curacy is generally high, with modern systems correctly articulating words, including challenging heteronyms. These strengths indicate that S2S systems have largely solved the foundational chal- lenges of textual understanding and generating clear and coherent dialogue scripts.
Observation 4: The speech generated by S2S systems often lacks human-like paralinguistic features, exhibiting rigid prosody and absence of disfluency cues.
Across non non-phyn oon-physiolooical paralinguistic features, S2S outputs show pronounced deficits in vocal dynamics. Rhythm and intonation changes are mechanically regular, with few context-appropriat pauses pitch movements. Stress salient words is weak misplaced, which is crucial element of human communication. Furthermore, models avoid human disfluency cues, such as linguistic imprecision (e.g., hedges like "probably"), use of fillers ("um"), and picco-physiological noises (e.g., breath sounds). These paralinguistic shortcomings, even when the content is fluent, make the speaker perceptibly machine-like.
Observation 5: Emotional expressivity remains largely limited in current S2S systems.
The textual sentiment scores S2S systems are significantly lower than human performance, reflect- ing the lack of nuanced emotions due to the writing-style expressions. More critically, the acoustic emotion scores are even lower than those textual sentiment, due to rigid prosody and weak or misaligned stress patterns. This indicates that S2S systems tend generate dialogues with neutral and unconvincing emotional tones, making them readily perceived as non-human by listeners.
Observation 6: The persona of S2S systems is often perceived as mechanical, characterized by excessively sycophantic and formal expression.
S2S systems reveal a mechanical persona through their social interaction. Unlike humans who judiciously agree or disagree based context, current models exhibit a strong default tendency to excessively affirm, apologize, and express gratitude. For instance, to a user's statement like, "I'm planning go around in Korea for 5 days". model mightrespone with disproportionate enthusiasm such as, "That's 's absolutely amazing maazzing-fantsstic choice!". Moreover, their written-style expression skews formal, lacking the conversational looseness typical of spontaneous speech.

---

Published as a conference paper at ICLR 2026
CAN AI MODELS SERVE AS TURING TEST JUDGES?
6.1 TURING TEST WITH AI JUDGES
To explore whether Al models can reliably assess human-likeness in dialogues, we employ 9 state- of-the-art models automated judges, and each model is tasked with classifying whether given dialogue response is human- or machine-generatede Detailed prompts are provided in Appendix [ Table reports their classification accuracy across the three dialogue types (human-human, hu- man-machine, and pseudo human).
Table 2: Al judge accuracy of different models on the Turing test data.
Model
ACC(H-H)
ACC(H-M)
ACC(PH)
Overall
Human Judgement
0.7028
0.8357
0.6384
0.7284
Baichuan- Mmichuan-Audiolle Li
et al
2025
0.8169
0.1528
0,1250
0.3628
Gemini 2.5 pro Comanici al.
2025
0.5775
0.7292
0.5764
0.6279
Gemma 3m Team al. 2025a
0. 0.4648
0.4444
0.4028
0.4372
GPT-4o-Audio Prdvieer Hurst
2024
0.9648
0.2708
0.0069
0.4116
MiniCPM-o 2.6
A024
0.6761
0.4306
0,2986
0.4674
Phi 4- MilMMMutiddll Ahouellenn el al.
2025
0.7746
0. 0.1458
0.2222
0.3791
Scallms-Audio Nguyen et al. 2023
0.1127
0.8472
0.7292
0.565
Voxtral Mini i et al.
2025
0.5141
0.5066
0.3889
0.4698
Dwwnn.55Ommi Xu 2025
0.7817
0.2361
0.2361
0.4163
Average of Model Judgement
0.6233
0.4011
0,3130
0.4527
Observation Existing Al judges significantly underperform humans the Turing test and exhibit systematic bias.
The overall performance of the Al judges (average accuracy: 0.4527) remains substantially lower than that of human evaluators (accuracy: 0.7284), with even the best-performing model Gemini 2.5 Pro achieving only 0.6279 accuracy. Analysis model behavior reveals three distinet bias patterns: several models (e.g.. GPT-4o-Audio-Prees exhibit a strong tendency to classify most dialogues as human human-human, models such as SeaL Audio display the opposite bias toward
Baichuan-Audia Phi-4-Mullimodall Qwen2.5- 5-0mi)
judgments, while Voxtral Mini behaves close to random guessing. These results highlight the current limitations multimodal models in replicating luken-like perceptual judgment in Turing test scenarios.
6.2 INTERPRETABLE AI JUDGE FOR HUMAN-LIKENESS EVALUATION
Given that general-purpose large models perform unreliably as human-likeness judges, develop an interpretable multimodal evaluator designed to deliver transparent and trustworthy decisions. Detailed experimental setup provided in Appendix E.2
6.2.1 TRAINING FRAMEWORK
We adopt a two-stage fine-tuning framework on Qwen2,5-0mni, which trains the model to first cap- ture fine-grained human-likeness patterns and then produce final and explainable human machine discrimination decision.
Fine-grained Scoring Projection. Given an audio dialogue æ E D, we first encode with pre- trained audio-language model (ALM) obtain fixed-dimensional representation h fALM (I) Rd (a two-source fused pooling, see Appendix E.3 for representation design). We then map h to interpretable dimension scores with an Ordinal Discretization Layer (ODL) (Tutz 2022):
z = foD R
24 = [oDL 0)]
where K the number of fine- -ra-nga human-likeness dimensions and 24 is the latent score for dimension k. To respect the ordinal nature of human ratings (e.g., r ordered levels, 1-r), convert
8

---

Published as a conference paper at ICLR 2026
each 2 into ordinal distribution via ordered cut-points. For each dimension & E (1....,K)
define r strictly ordered cut-points
Ss = i i-r+2
2(r 2) Sk+
E rl.....r-11
where 8ę is learnable scale that controls bin spacing. Using a cumulative-line formulation, cumu- lative probabilities are
P(Y= <i dCa =x);
where (.) denotes the sigmoid function. Per-category probabilities follow by differencing: P(Y, = P - P(Y = =) P(Y, < - P(Y+ - 1) for 2 < r 1, and P(Y, = =r) = [1, ...., r) r] denote human-likeness ratings for æ, we fit the ODL by minimizing the ordinal negative log- likelihood over all samples and dimensions:
1- sr-1). Let
log P(Ys = ((k) (x) [x)
min
mEDk=1
This procedure yields order-preserving, human-aligned scores per dialogue that serve as inter- pretable inputs for the final human-vss vs.-machine classifier.
Explainable Binary Classification. After training the ODL, each the & neurons acquires an or- dinally constrained scoring pattern induced by the cut- cut-point scheme. Consequently, the ODL outputs are no longer arbitrary latent features; they instantiate interpretable scoring dimensions aligned with human ratings and preserve their ordinal structure for numan-maching discrimination. Leveraging this property, we feed the logits into linear classifier with regularization constraint to ensure that the final classification remains interpretable:
min WF
Σ (u.N)ED
CCE(Wpz, 3) + XR(FF)
D
where LCE is the Cross-Entropp Loss, WF Raxk is the weight matrix of the final linear layer with categories, y is the label of R(W) |[ + Wall2 is the symmetry regularization, and À set to 0.1. Model ablations and hyperparameter tuning details are provided in Appendix [E44 and
S
6.2.2 RESULTS AND DISCUSSION
Table 3: Binary classification accuracy of different models across three evaluation data types.
Data Type
Qwen2.5-0mni
(wee225-Ommi(LLoRA)
Human Judge
Ours
Human- Human-Humant
0.7817
0.9230
0.7028
0.9507
Nouman-Machine'
0.2361
0.6319
0.8357
0.9722
Pseudo Human
0.2361
0.0972
0,6384
0.9306
Overall
0.4163
0.5744
0.7284
0.9605
We evaluate the interpretable AI judge on the Turing test using binary classification accuracy (hu- man vs. machine). presented in Table 3 Qwen2.5-Ommi (LoRA) represents Qwen2.5-Omni fine-tuned using LoRA technology (Hu et al. 2022). It can be observed that our approach outper- forms all variants and human evaluators. The overall accuracy 23.21% % higher than the human evaluation, 38.61% higher than the LoRA-based approach, and more than doubles the performance of the original model. Notably, the model achieves 93.06% accuracy on pseudo-huma unseen during training, demonstrating strong generalization. In addition, the model shows strong consistency with fine-graine grained human ratings, a capability facilitated by its interpretable design (see Appendix E.6).
dialogues

---

Published as a conference paper at ICLR 2026
Out-of-Domain Generalization Evaluation We further evaluated our model three out-of- domain (OOD) datasets that span diverse acoustic, demographic, and interaction conditions: 1) CosyVoice2 Synthesis (Pseudo Human) (Du et al. (0224,, synthesized dialogues actoss different age groups (older adults and children); 2) Fisher (Human-Human) (Cieri et al., 2004), telephone speech with significant background noise; 3) MultiDialog (Human- Human) (Park al. [024): clean back- ground native-speaker dialogue recordings, We sample 64 dialogues from each dataset for evalua- tion. In addition accuracy, we introduced the ROC-AUC score to provide robust and threshold- independent evaluation of classification performance. The results of human human-machine machine classification These results indicate that the model generalizes well and maintains stable performance under distribution shift.
are presented Table 4
Table 4: Binary classification accuracy and ROC-AUC on OOD test set.
Metrie
Overall (Inner)
Cosy Voice2
Fisher
MultiDialog
Overall (OOD)
Accuracy 0.9605
0.9844
0.9844
0.9531
0.9740
ROC-AUC
0.9791
0.9881
Observation 8: Our interpretable Al judge delivers superior performance in distinguishing hu- man from machine- -eetrrtegeeerated speech. By providing both an overall numan-likeness score and fine- rree-graied diagnostics, it serves practical tool for S2S assessment.
7
CONCLUSION
This work presents the first Turing test for modern S2S systems, delivered via a game-based on- line platform that enables large-scale and longitudinal testing. Our findings reveal clear gap: no current system passes, demonstrating that human-like conversational ability remains an unsolved challenge. Through an 18-dimension taxonomy, we show the bottleneck has shifted from semantic understanding to shortcomings in paralinguistic features, emotional expressivity, and conversational persona, explaining why even fluent S2S output sounds distinctly artificial. To support automatic evaluation, we develop an interpretable Al judge that significantly outperforms off-the-shelf models and provides diagnostic insights.
Impact. We provide the community with new tems and move beyond binary pass/fail to automatic, diagnostic, and scalable evaluation. Our results offer practical guidance toward more genuinely human-like S2S systems by identifying the challenges in acoustic naturalness, emotional expressivity, and social behavior.
evaluation framework for S2S sys-
ETHICS STATEMENT
Our study involves the collection of audio recordings from human participants. In conducting this research, we have adhered to strict ethical principles to safeguard participants' privacy, autonomy, and well-being. The main ethical considerations are outlined below:
Informed Consent: All participants
clearly informed that their speech would be recorded and potentially used in academic publications. Participation was voluntary, and individuals had the right to withdraw any stage without penalty.
Data Anonymization: To ensure participant confidentiality, all audio recordings were anonymized by removing any personally identifiable information, making impossible trace the data back to individuals.
Collected data are stored under strict security protocols, with access lim- ited to authorized research personnel. Comprehensive measures are in place to prevent unauthorized access, disclosure, misuse.
Data Security:
Scientifie Integrity: We maintain high standards of transparency and accuracy in reporting methods and results. The research is presented a manner that supports reproducibility, and all contributions properly acknowledged.
10

---

Published as a conference paper at ICLR 2026
Avoiding Harm and Promoting Fairness: We have taken measures to minimize potential harm and avoid reinforcing social biases. Our work is committed to faimess, inclusivity, and respect for participants, with the goal that research outcomes be applied in socially responsible manner.
We reiterate that all data and models are intended solely for scientific research purposes, and must be used for commercial activities or any unlawful or fraudulent actions.
REPRODUCIBILITY STATEMENT
We have made every effort to ensure that the results presented in this paper are reproducible. All code and datasets have been made publicly available in anonymous repository to facilitate replication and verification.
Our experiments comprise three main components. First, we collected dialogue data for the Tur- ing test, which includes human-machie dialogues (see Section 3.1 for the detailed procedure), human-human dialogues (Section B.20, and pseudo human human dialogues generated via TTS models Seeett,o-Seecc, also described in Section (33) Based on the collected data, we designed game-based human evaluation platform supporting fine-grained annotation, with the detailed de- sign and implementation process outlined in Section M Furthermore, we developed a fine-grained annotation protocol incorporating expert validation, described Appendix [.I] Using this pro- tocol, we conducted crowd-sourced annotation; the design of the annotation platform is provided in Appendix [D.2] Finally, we trained human-like judge model using the annotated data, with the
model training procedure and hyperparameter settings detailed in Appendix E] We believe that these comprehensive descriptions significantly enhance the reproducibility of our work.
ACKNOWLEDGMENTS
This work is supported part by Longgang District Special Funds for Science and Technology Innovation under Grant LGKCSDPT202300; the National Natural Science Foundation of China under Grants 62372058, U22A2026.
REFERENCES
Abdelrahman Abouelenin, Atabak Ashfaq, Adam Atkinson, Hany Awadalla, Nguyen Bach, Jianmin Bao, Alon Benhaim, Martin Cai, Vishrav Chaudhary, Congcong Chen, et Phi-4-mini technical report: Compact yet powerful multimodal language models via mixture-of-loras. arXiv preprint arXiv:2503.01143, 2025.
Andrey Anikin, Valentina Canessa-Pollard, Katarzyna Pisanski, Mathilde Massenet, and David Reby. Beyond speech: Exploring diversity the human voice. Iscience, 26(11), 2023.
Anthropic. Claude 3.5 sonnet system card. https///wwwwccc..ttttt.o.roroo com/
B 007000000000044366666000 6ba2a pdd 2024. Accessed: 2025-09-10.
Emanuele Bottazzi Grifoni and Roberta Ferrario. The bewitching ai: The illusion communication with large language models. Philosophy & Technology, 38(2):61, 2025.
Carlos Busso, Murtaza Bulut, Chi-Chun Lee, Abe Kazemzadeh, Emily Mower, Samuel Kim, Jean- nette N. Chang, Sungbok Lee, and Shrikanth S. Narayanan. IEMOCAP: interactive emotional dyadie motion capture database. Lang. Resour. Evaluation, 42(4):335-359, 2008
ByteDance. Doubao-1.5-pro. https://sseet ww..byyttean...... com/ byttedaance.com//////spcaaaaae 1_5_ ____rro 2025. Accessed: 2025-09-15.
Tsz- Yan Chan. Using a text-to-speech synthesizer to generate reverse turing test. In 15th IEEE International Conference Tools with Artificial Intelligence (ICTAI 2003), 3-5 November 2003, Sacramento. California, USA, pp. 226-232. IEEE Computer Society, 2003. doi: 10.1109/TAL 2003. 10001125015... URL https: //doi org /0. 1109 1 2003. 1250195
11

---

Published as a conference paper at ICLR 2026
Yiming Chen, Xianghu Yue, Chen Zhang, Xiaoxue Gao, Robby T. Tan, and Haizhou Li. Voicebench: Benchmarking Ilm-based voice assistants. CoRR, abs/2410.17196, 2024.
Christopher Cieri, David Miller, and Kevin Walker. The fisher corpus: a resource for the next gen- erations of speech-to-text. In Proceedings of the Fourth International Conference on Language Resources and Evaluation, LREC 2004, May 26-28, 2004, Lisbon, Portugal. European Language Resources Association, 2004.
Gheorghe Comanici, Eric Bieber, Mike Schackermann, Ice Pasupar, Noveen Sachdeva, Inderjit Dhillon, Marcel Blistein, Ori Ram, Dan Zhang, Evan Rosen, et al. Gemini 2.5: Pushing the frontier with advanced reasoning. multimodality, long context, and next generation agentic capa- bilities. arXiv preprini arXiv:2507.06261 2025.
Philip R. Doyle, Justin Edwards, Odile Dumbleton, Leigh Clark, and Benjamin R. Cowan. Mapping perceptions of humanness in intelligent personal assistant interaction. In Proceedings of the 21st Human- Computtr Interaction with Mobile Devices and Services, MobileHCI 19. pp. 12 ACM, October 2019. doi: 10.1145/3332266334000 URL http: 7/dx. www.doo..oo www.oo.....oo.oooooooo.......o.. 10 0. 11
International Conference
3340116
Yuhao Du, Qianwei Huang, Guo Zhu, Zhanchen Dai, Sunian Chen, Qiming Zhu, Yuhao Zhang, Li Zhou, and Benyou Wang. Mtalk-bench: Evaluating speech-to-speech models in multi-turn dialogues via arena-style and rubrics protocols. arXiv preprint arXiv:2508. 010088/82440 18240, 2025.
Zhihao Du, Yuxuan Wang, Qian Chen, Xian Shi, Xiang Lv, Tianyu Zhao, Zhifu Gao, Yexin Yang. Changfeng Gao, Hui Wang, Fan Yu, Huadai Liu, Zhengyan Sheng, Yue Gu, Chong Deng, Wen Wang. Shiliang Zhang, Zhijie Yan, and Jingren Zhou. Cosyvoice Scalable streaming speech synthesis with large language models. CoRR, abs/2412. 2024.
Aaron Fanous, Jacob Goldberg, Ank A. Agarwal, Joanna Lin, Anson Zhou, Roxana Daneshjou, and Sanmi Koyejo. Syceval: Evaluating LLM sycophancy. CoRR, abs/2502.08117,, 2025, doi: 10. 48550/ARXIV.2500008177 UtL http://oo..orrrrr/ 11111.4855000 48550/8550/00 http/arxivv//.. 2502.00022111
Takashi Fukuda, Osamu Ichikawa, and Masafumi Nishimura. Detecting breathing sounds in realistic japanese telephone conversations and its application to automatic speech recognition. Speech Communication, 98:95-103, 2018.
Matthew Carson Galbraith and Mireia Gómez Martínez. An analysis of dialogue repair in virtual voice assistants. CoRR, 0bs/2307.07076, 2023.
Xuelong Geng, Qijie Shao, Hongfei Xue, Shuiyuan Wang. Hanke Xie, Zhao Guo, Yi Zhao, Guo- jian Li, Wenjie Tian, Chengyou Wang, Zhixian Zhao, Kangxiang Xia, Ziyu Zhang, Zhennan Lin, Tianlun Zuo, Mingchen Shao, Yuang Cao, Guobin Ma, Longhao Li, Yuhang Dai, Dehui Gao, Dake Guo, and Lei Xie. Osum-echat: Enhancing end-to-end empathetic spoken chatbot via understanding-dhit driven spoken dialogue. CoRR, abs/2508.090 006000, 2025.
Team GLM, Aohan Zeng, Bin Xu, Bowen Wang, Chenhui Zhang, Da Yin, Dan Zhang, Diego Rojas, Guanyu Feng, Hanlin Zhao, et al. Chatglm: A family of large language models from glm-130b 130b to glm-4 all tools. arXiv preprint arXiv:2406. 12793, 2024.
Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang. and Weizhu Chen. Lora: Low-rank adaptation of large language models. In The Tenth Inter- national Conference on Learning Representations, ICLR 2022, Virtual Event, April 25-29, 2022. net wtttt//orumiidddoeee www.mmiiidneeeeeeeee..e.
OpenReview.net, 2022. URL
Aaron Hurst, Adam Lerer, Adam P Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, AJ Os- trow, Akila Welihinda, Alan Hayes, Alec Radford, et al. Gpt-40 system card. arXiv preprint arXiv:2410.21276 2024.
Ji-Sang Hwang, Sang-Hoon Lee, and Seong-Whan Lee. Pausespeech: Natural speech synthesis via pre-trained language model and pause-based based prosody modeling. In Huimni Lu, Michael Blumenstein, Sung-Bae Bae Cho, Cheng-Lin Liu, Yasushi Yagi, and Tohru Kamiya (eds.), Pat- tern Recognition 7th Asian Conference, ACPR 2023, Kitakyushu, Japan, November 5-8, 2023, Proceedings, Part volume 14406 of Lecture Notes in Computer Science, pp. 415-427.
12

---

Published as a conference paper at ICLR 2026
Springer, 2023. doi: 10.1007/973333331---766111 URL https//ttpp////ii.ro////........
978-3-031-47634-122
iFlytek. iflytekspark- 130b parameter open-source large model. https://gitee.com
HtLytekopensoaree//PPiae wtt.eoesrrreeet..ttettaa...aaaae 13B 2024. Accessed: 2025-09-15.
Feng Jiang, Zhiyu Lin, Fan Bu, Yuhao Du, Benyou Wang, and Haizhou Li. S2s-arena, evaluat- ing speech2speech protocols on instruction following with paralinguistic information. CoRR, 01/22003.55085 2025.
Cameron Jones and Ben Bergen. Does GPT-4 pass the Turing test? In Kevin Duh, Helena Gomez, and Steven Bethard (eds.), Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume I; Long Papers), pp. 5183-5210, Mexico City, Mexico, June 2024a. Association for Computational org/
Linguisties. doi: 10. 1 1/02001...22.4..lloorroooo 2024. www.h-aacnlnnnnoo 2907
URL https://aclanthonn
Cameron R. Jones and Ben Bergen. Does GPT-4 pass the turing test? Kevin Duh, Helena Gómez- Adorno, and Steven Bethard (eds.), Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguisties: Human Language Technologies (Vol- ume 1: Long Papers), NAACL 2024, Mexico City, Mexico, June 16-21, 2024, pp. 5183-5210. Association for Computational Linguistics, 2024b.
Cameron R. Jones and Benjamin K. Bergen. Large language models pass the turing test, 2025. URL https: http::://aaxiv.com
abw/ wttp:///.wwr//////20/333/3 23674
Cameron Robert Jones, Ishika Rathi, Sydney Taylor, and Benjamin K. Bergen. People cannot distin- guish gpt-4 from a human in a turing test. In Proceedings of the 2025 ACM Conference on Fair- ness, Accountability, and Trunsparency, FAceT '25, pp. 1633,, New York. NY, USA, 2025. Association for Computing Machinery. ISBN 9798400714825. doi: 10.11455/71522553732106
URL htttps:///oiii
10.
0012277..773111 3732108
Sonal Kumar, Simon Sedlácek, Vaibhavi Lokegaonkar, and et al. Mmau-pro: A challenging and comprehensive benchmark for holistic evaluation of audio general intelligence. CoRR, abs/2508. 13992, 2025a.
Sonal Kumar, Simon Sedláček, Vaibhavi Lokegaonkar, Fernando López, Wenyi Yu, Nishit Anand, Hyeonggon Ryu, Lichang Chen, Maxim Plička, Miroslav Hlaváček, et al. Mmau-pro: chal- lenging and comprehensive benchmark for holistic evaluation of audio general intelligence. arXiv preprint arXiv:2508. 12008.3399 13992, 2025b.
Keon Lee, Kyumin Park, and Daeyoung Kim. Dailytalk: Spoken dialogue dataset for conversational text-to-speech. In IEEE International Conference on Acoustics, Speech and Signal Processing ICASSP 2023, Rhodes Island, Greece, June 4- 4-10, 2023, pp. 1-5. IEEE, 2023.
Tianpeng Li, Jun Liu, Tao Zhang, Yuanbo Fang, Da Pan, Mingrui Wang, Zheng Liang. Zehuan Li, Mingan Lin, Guosheng Dong. et al. Baichuan- audio: A unified framework for end-to-end speech interaction. arXiv preprint 1XXx: 2025.
Yanran Li, Hui Su, Xiaoyu Shen, Wenjie Li. Ziqiang Cao, and Shuzi Niu. Dailydialog: A manually labelled multi-turn dialogue dataset. In Greg Kondrak and Taro Watanabe (eds.), Proceedings of the Eighth International Joint Conference on Natural Language Processing, IJCNLP 2017, Taipei, Taiwan, November 27 December 1, 2017 Volume 1: Long Papers, pp. 986-995. Asian Federation of Natural Language Processing, 2017.
Guan-Ting Lin, Jiachen Lian, Tingle Li, Qirui Wang, Gopala Anumanchipalli, Alexander H. Liu, and Hung-yi Lee. Full-duplex-bente models on turn-taking capabilities. CoRR, abs/2503.0442.. 2025.
A benchmark to evaluate full-duplex spoken dialogue
Christine H Nakatani and Julia Hirschberg. speech-first model for repair detection and correction. In 31 Annual Meeting of the Association for Computational Linguistics, pp. 46-53, 1993.
nari-labs.
hari-labs/dia-1
https: https://githus. wttp://...oooorriieaaee htt./aariiia dia ?t
reacie--v-- 1e. 2025. Accessed: 2025-09-15.
13

---

Published as a conference paper at ICLR 2026
Xuan-Phi Nguyen, Wenxuan Zhang, Xin Li, Mahani Aljunied, Zhiqiang Hu, Chenhui Shen, Yew Ken Chia, Xingxuan Li, Jianyu Wang, Qingyu Tan, et al. Seallms-larg language mod- els for southeast asia. arXiv preprint arXh::2112.07788, 2023.
Kentaro Onda, Keisuke Imoto, Satoru Fukayama, Daisuke Saito, and Nobuaki Minematsu. Prosod- ically enhanced foreign accent simulation by discrete token- based resynthesis only with na- tive speech corpora. CoRR, https/2505.161911. 2025. doi: 10. 1044850/RXXV..505.661) URL
11111.4..55000 wt5.:45500022.225555....... 16191
OpenAl. GPT-4 technical report. CoRR, abs/2303.087701 2023.
Se Jin Park, Chae Won Kim, Hyeongseop Rha, Minsu Kim, Joanna Hong, Jeong Hun Yeo, and Yong Man Ro. Let's go real talk: Spoken dialogue model for face-to-face conversation. In Lun- Wei Ku, Andre Martins, and Vivek Srikumar (eds.), Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), ACI. 2024, Bangkok, Thailand, August 11-16, 2024, pp. 16334-16348. Association for Computational Linguistics, 2024.
Steven T Piantadosi, Harry Tily, and Edward Gibson. The communicative function of ambiguity in language. Cognition, 122(3):280-299 2012.
Pilar Prieto and Paolo Roseano. Prosody: Stress, Rhythm, and Intonation, pp. 211-236. Cambridge Handbooks in Language and Linguistics. Cambridge University Press, 2018.
Ishika Rathi, Sydney Taylor, Benjamin K Bergen, and Cameron R Jones. Gpt-4 is judged more human than humans in displaced and inverted turing tests. arXiv preprint arXiv:2407.08855 2024.
Sakshi, Utkarsh Tyagi, Sonal Kumar, Ashish Seth, Ramaneswaran Selvakumar, Oriol Nieto, Ra- mani Duraiswami, Sreyan Ghosh, and Dinesh Manocha. Mmau: A massive multi-task audio understanding and reasoning benchmark. arXiv preprint 11X1::2410.19168 2024.
Éva Székely, Gustav Eje Henter, Jonas Beskow, and Joakim Gustafson. How to train your fillers: uh and um in spontaneous speech synthesis. The 10th ISCA Speech Synthesis Workshop. 2019.
Gemma Team, Aishwarya Kamath, Johan Ferret, Shreya Pathak, Nino Vieillard, Ramona Merhej, Sarah Perrin, Tatiana Matejovicova, Alexandre Ramé, Morgane Rivière, et al. Gemma 3 technical report. arXiv preprint arXiv:2503. 19786, 2025a.
Kimi Team, Angang Du, Bofei Gao, Bowei Xing, Changjiu Jiang, Cheng Chen, Cheng Li, Chenjun Xiao, Chenzhuang Du, Chonghua Liao, et al. Kimi kl. 5: Scaling reinforcement learning with llms. arXiv preprint arXiv:2501. 2025b.
Tencent Hunyuan Team, Ao Liu, Botong Zhou, Can Xu, Chayse Zhou, ChenChen Zhang, Chengeheng Xu, Chenhao Wang, Decheng Wu, Dengpeng Wu, et al. Hunyuan-turbos: Advanc- ing large language models through mambattransformer synergy and adaptive chain-of-thought. arXiv preprint arXiv:2505. 15431, 2025c.
João Paulo Teixeira, Carla Oliveira, and Carla Lopes. Vocal acoustic analysis-jitter, shimmer and hnr parameters. Procedia technology, 9:1112-1122 2013.
Adithya TG, Gowri Srinivasa, et al. Leveraging virtual reality and ai tutoring for language learning: A case study of a virtual campus environment with openai gpt integration with unity 3d. arXiv preprint arXiv:2411.12611 2024.
Mariya Toneva, Alessandro Sordoni, Remi Tachet des Combes, Adam Trischler, Yoshua Bengio, and Geoffrey J. Gordon. An empirical study of example forgetting during deep neural network learning. In 7th International Conference on Learning Representations, ICLR 2019, New Or- leans. LA, USA, May 6-9, 2019. OpenReview.net, 2019. URL htttps://ppenreviww.cco
forum? d=BJ] xm Kc
14

---

Published as a conference paper at ICLR 2026
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurélien Rodriguez, Ar- mand Joulin, Edouard Grave, and Guillaume Lample. Llama: Open and efficient foundation language models. CoRR, abs/2302. 13971, 2023.
A. M. TURING. I-computing machinery and intelligence. Mind, LIX(236)44333-4600, 1950.
Gerhard Tutz. Ordinal regression: A review and a taxonomy of models. Wiley Interdisciplinury Reviews: Computational Statistics, 14(2)ee1555, 2022.
Hilde Voorveld, Andreas Panteli, Yoni Schirris, Carolin Ischen, Evangelos Kanoulas, and Tom Lentz. Examining the persuasiveness of text and voice agents: prosody aligned with informa- tion structure increases human- perceived personalisation and brand attitude. Behaviour & Information Technology, 44(12):2913-292 2025.
Mila Vulchanova and Valentin Vulchanov. Figurative language processing: A developmental and NLP perspective. In Proceedings of the Third International Conference on Computational Lin- guistics in Bulgaria (CLIB 2018), pp. 7-14, Sofia, Bulgaria, May 2018. Department of Com- putational Linguistics, Institute for Bulgarian Language, Bulgarian Academy of Sciences. URL aattps://aclantholoogg org. www.-l.ill1.
Minghan Wang, Ye Bai, Yuxia Wang, Thuy-Trang Vu, Ehsan Shareghi, and Gholamreza Haffari. Specchdialoguefacttoyy:: Generating high-qualit, speech dialogue data to accelerate your speech- IIm development. arXiv preprint arXiv:2503. 23848, 2025a.
Xihuai Wang, Ziyi Zhao, Siyu Ren, Shao Zhang, Song Li, Xiaoyu Li, Ziwen Wang, Lin Qiu, Guan- glu Wan, Xuezhi Cao, Xunliang Cai, and Weinan Zhang. Audio turing test: Benchmarking the human--liennss, of large language model- text-to-speech systems in chinese, 2025b. URL
org. 2505. 11200
Xinsheng Wang, Mingqi Jiang. Ziyang Ma. Ziyu Zhang, Songxiang Liu, and et al. Linqin Li. Spark- its: An efficient Ilmebased text-to-speech model with single-stream decoupled speech tokens, 2025c.
01710
Kevin Warren, Daniel Olszewski, Seth Layton, Kevin Butler, Carrie Gates, and Patrick Traynor. Pitch imperfect: Detecting audio deepfakes through acoustic prosodic analysis. arXiv preprini arXiv:2502.14726, 2025.
Jin Xu, Zhifang Guo, Jinzheng He, Hangrui Hu, Ting He, Shuai Bai, Keqin Chen, Jialin Wang. Yang Fan, Kai Dang, et al. Qwen2. 5-omni technical report. arXiv preprint arXiv:2503.2022 2025.
An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng. Bowen Yu, Chang Gao. Chengen Huang, Chenxu Lv, et al. Qwen3 technical report. arXiv preprint arXiv:2505.09380 2025.
Zehui Yang, Yifan Chen, Lei Luo, Runyan Yang, Lingxuan Ye, Gaofeng Cheng, Ji Xu, Yaohui Jin, Qingqing Zhang, Pengyuan Zhang, Lei Xie, and Yonghong Yan. Open source magicdata- -amme A rich annotated mandarin conversational(rmme) speech dataset. In Hanseok Ko and John H. L. Hansen (eds.), 23rd Annual Conference of the International Speech Communication Association, Interspeech 2022, Incheon, Korea, September 18-22, 2022, pp. 1736-1740. ISCA, 2022.
Yuan Yao, Tianyu Yu, Ao Zhang, Chongyi Wang, Junbo Cui, Hongji Zhu, Tianchi Cai, Haoyu Li, Weilin Zhao, Zhihui He, et al. Miniepm-v: A gpt-4v level milm on your phone. arXiv preprint arXiv:2408.0 01800, 2024.
Aohan Zeng, Xin Lv, Qinkai Zheng, Zhenyu Hou, Bin Chen, Chengxing Xie, Cunxiang Wang. Da Yin, Hao Zeng, Jiajie Zhang, al. Glm-4.5: Agentic, reasoning, and coding (are) foundation models. arXiv preprint (.Xi::2508.06777 2025.
Haiteng Zhang. PDF: polyphone disambiguation in chinese by using FLAT. In Hynek Her- mansky, Honza Cernocký, Lukás Burget, Lori Lamel, Odette Scharenborg, and Petr Motlicek (eds.), 22nd Annual Conference of the International Speech Communication Association, In- terspeech 2021, Brno, Czechia, August 30 September 3, 2021. pp. 4103- ISCA, 2021. doi: 1021447/NTEESSFEECH.OOO Interspeech. 2201111077
1087. URL https: wtt.:a....r/11..111.13..
15

---

Published as a conference paper at ICLR 2026
Xitong Zhang. Codt-switching in english-chinese ordinary conversations. TESOL Working Paper Series, 17:38 45, 2019.
THE USE OF LARGE LANGUAGE MODELS
Using an LLM to help with paper writing During the preparation of this work, the authors utilized Large Language Models for language polishing, improving the structural clarity of the manuscript, and refining the formal expression of individual sentences. The use of Large Language Models did not influence the substantive content of the study and served solely as writing aid.
OVERALL THE APPENDIX
The appendix provides supplementary material to support the methodology outlined in the main text. It organized into four sections for clarity:
Appendix A
Comparison with Existing Speech Benchmarks details the distinctions be- tween our benchmark and existing speech benchmarks.
Appendix B Data Collection details the procedures, sources, and criteria used for gather- ing the raw data utilized in this study.
Appendix Turing-Test describes the design of the human evaluation (Turing test).
Appendix Fine-Grained Human-LLienee Dimension Annotation details the com- prehensive guidelines followed for data annotation.
Appendix Training Details specifies the key hyperparameters, computational environ- ment, and training configurations of the models.
A COMPARISON WITH EXISTING SPEECH BENCHMARKS
We conducted a detailed comparison between our work and two representative speech benchmarks, VoiceBench (Chen al. 2024) and MMAU-Pro (Kumar et 2025a). As summarized in Table our work differs fundamentally in evaluation goal and evaluated modality.
Table 5: Comparison with Existing Speech Benchmarks.
Aspect
VoiceBench
MMAU-Pro
Turing Test (Ours)
Goal
Evaluating speech un- derstanding LLM-
Evaluating holistic audio understanding of multimodal AI models across speech, music, sound
Evaluating human-
likeness of Speech-to- Speech systems
based voice assistants
Input Modality
Speech Text
Speech and Text
Speech
Output Modality
Text
Text
Speech
Dialogue Turns
Single-tum
Multi-tum
Multi-tum
The Smarter the Better?
Yes Yes-higher gence implies
intelli-
Yes figher
intelli- better
No-being "too smart"
better
gence implies
does not necessarily
performance
performance
make model
likely to pass the Turing Test
To further examine whether "being smarter" makes model more human-like selected S2S systems that appear in both MMAU-Pro and our study, and compared their reasoning accuracy on MMAU-Pro with their Turing Test pass rates. The results are summarized in Table 6
The Pearson correlation between reasoning accuracy and Turing test pass rate 0.0456. This in- dicates that reasoning ability is nearly uncorrelated with human-likened in current S2S systems,
16

---

Published as a conference paper at ICLR 2026
Table Reasoning Ability vs. Human-likenee in Speech-to-Speech Models.
Model
Reasoning
Accuracy
(MMAU-
Turing Test Pass Rate (%)
Pro)
Kimi-KI.5
46.6
12.7
Qwen3
52.2
15.1
GPT--4o
52.5
23.0
Gemini-2.5 Proro
59.2
13.7
revealing disconnect between traditional intelligence benchmarks and the human- likeness required for speech interaction.
B DATA COLLECTION
The section is organized into the following sections:
Section
B.1
Model Selection for the Turing Test.
Section
B.2
Participant Profiles.
Section
B3
Human-Machine Dialogue Initialization Design Details.
Section B.4
Dialogue Filtering.
Section
B.5
Pseudo Human Dialogue Synthesis.
B. MODEL SELECTION FOR THE TURING TEST
All S2S Systems we selected for evaluation are shown in Table During pilot recordings and test- ing, we observe that Sraude-Sonnet supports only English conversations, while iFLYTEK-Spa -Sark exhibits suboptimal performance on long English prompts due to its underlying training constraints. To ensure dialogue quality, we generate dialogues in English for Claude-Sonnet 4 and in Chinese for iFLYTEK-Spark.
Table 7: Models used for the Turing test.
Model
Release Year Open-Source
Dialogues
Share (%)
Language
GPT-4o
2024
2024
13.30%
Gemini2.5-Poo (Comanic
2025
2025
12.26%
CN EN
Qwen3 Yang al. 2025
2025
12.41%
EN
Kimi-K
Ieam al. 2025b
2025
12.41%
CN & EN
ChatGLM--155 (Zong al.
2025
2025
V
11.51%
CN & EN
Tunnyan-TTubboS Ieam
2025e
2025
12.86%
CN & EN
Doubao-Pro 1.5 [ByteDance
2025
2025
85
12.71%
CN & EN
Claude-Sonned 4 Anthropic 2024
2025
41
6.13%
EN
-FLYTEK-Spark (iFlytck. 2024
2025
43
6.43%
CN
B.2 PARTICIPANT PROFILES
We provide the detailed profiles of all 28 participants in Table
17

---

Published as a conference paper at ICLR 2026
Table 8: Participant Profiles.
Speaker ID
Chinese
English
Country / Region
speaker01
China
speaker02
China
speaker03
China
speaker04
China
speaker0S
China
speaker06
China
speaker07
China
speaker08
China
speaker09
China
speaker10
China
speaker11
China
speaker12
China
speaker13
Hong Kong, China
speaker14
Pakistan
speaker15
Tajikistan
speaker16
Malaysia
speaker17
Indonesia
speaker18
Russia
speaker19
Indonesia
speaker20
Greece
speaker21
Indonesia
speaker22
Indonesia
speaker23
UK
speaker24
X
US
speaker25
Indonesia
speaker26
China
speaker27
China
speaker28
X
Indonesia
B.3 HUMAN-MACHINE DIALOGUE INITIALIZATION DESIGN DETAILS
For the Turing evaluation, we collect Human- Initiation (Figure 6). Role Playing (Fig- ure 7) and 4 Human-Likenes, Prompting (Figure B) initialization evaluation dialogues for both English and Chinese (if applicable) from each S2S system. For any the specific initialization, we fixed the starting sentences that interact with these models. Eventually, machine data for evaluation in total. The reason for including more dialogues for Role Playing than Himann-uuide Initiation is that, the former one tend to leads the conversation to discussion on viewpoints. This phenomenon limits the dialogue coverage to only a narrow range everyday scenarios. Thus, we limit the amount of Human-Guuded Initiation. By contrast, include more dialogues for Human-Like Likeness Prompting than Role Playing because Humun-Likennes Prompling explicitly attempts elicit stronger humanlike qualities from S2S systems. This design allows dataset to capture richer and more human-like spectrum of conversational behavior. The following figures show what the 18 dialogue initializations are.
obtained 144 human-
B.4 HUMAN-HUMAN DIALOGUE FILTERING
For the human-human dialogues, we extracted recorded conversation segments of around 20-60 seconds to align with the human-machine dialogues. On one hand, too short dialogues may present little context. On the other hand, excessively long recordings not available for some S2S sys- tem. To ensure balanced interactions, we retained only segments in which each of the two speakers contributed roughly equally, defined as having approximately 50% of the total utterances.
18

---

Published as a conference paper at ICLR 2026
A TTT.
TH SEARTTATRE
"I always take a shower in the evening. I don' t understand why there are people taking shower in the morning."
still get nervous before every test, no matter how prepared am..
Figure 6: Human gumded- initiation (2 ZH 2 EN).
- #
# -
"
4A7 MARAIUS, #
K. #54, 2
A 1 & &
A #
" "You are now university student, and are discussing about university ranking. Please don' mention your identity
subsequent conversation. Let' start chatting now. Do you know that recently the student of our university been some conftict with the student of our brother university?"
" "You my friend, and invite you my home for dinner tonight. Please mention your identity in the subsequent conversation. Let' start chatting now. What do you want to have for dinner tonight?"
"You my mom and mention your identäly in the subsequent conversation. Let' start chalting now. Mom, only scored 60 in my math exam."
discussing my final erom grade. Please don't
Figure 7: Role playing (3 ZH, EN).
B.5 PSEUDO HUMAN DIALOGUE SYNTHESIS
Dialogue Scripts for TTS
The dialogue scripts cover 10 topics as our dataset. Each script presents
a conversation between two speakers. We obtain scripts in two ways:
1. We use ChatGPT to adjust our existing dialogue scripts, ensuring that the original meaning remains intact while maintaining natural, conversational tone. This part the scripts contains all of the HH data in the additional set that ensures contextual consistency. This allows data that closely resembles our previous human-to-hie humaan dialogues. On the one hand, the scripts grounded in authentic everyday conversations. On the other hand, the similarity in content helps reduce the chance that audiences distinguish between human and machine solely based on biases introduced by dialogue content. The prompt used in this way is shown as follow:
generate
19

---

Published as a conference paper at ICLR 2026
" " ANARM,  A X w A o A kpi
2 1
"sh " LA MATARAKTRA &
A A 18 AMRAFERE E $
RE A, B MARATAT, AARN4 3 R. # P- # & fr 4 F 1--
th &
"You are now my friend who come back from a vocation in Europe. Make your expression more hamonlikke. Don' t mention your identity in the subsequent conversation. Let' start chalting now. How" your vacation to Exrope
"You are now my classmate who still stays in the classroom. Roin suddenly starts pouring outside. Make your expression more humanlike. Don' t mention your identity in the subsequent conversation. Let' start chatting now. Exeuse me- Hey, do you happen have umbrella could borrow?"
"You're 're now tazi driver, and I'm passenger in your cob. Make your expression more humanlike. Don' t mention your identity in the subsequent conversation. Let' start chatting now.
here, can you take the best restaurant town?
"You are now my colleague who staged late the office with me to finish a deadline. Make your expression more kumenlike. Don' mention your identity in the subsequent conversation. Let' start challing now. I don' think we could get this done tonight."
Figure 8: Human-likeness prompting (4 (4HH, 4 EN).
You are a language refinement expert.
Without changing the original meaning or the overall flow of the dialogue, your task to slightly adjust the conversation between two speakers. The goal is to preserve a natural, everyday tone. You may apply techniques such as: adding light interjections or filler words to make the speech sound more authentic, or rephrasing sentences into alternative but commonly used everyday expressions.
Please return only the refined dialogue in JSON format, keeping the structure as the origi- nal.
Original dialogue: [Utterances in jnon jsonnforror format)
Adjusted dialogue:
We generate new 40-50 second everyday dialogue scripts with GPT-4o, based on given themes (topic 1-10). The prompt used in this way is shown as follow:
20

---

Published as a conference paper at ICLR 2026
You a writing expert.
Please generate a spoken-style dialogue script between two people on the topic of "[topic)." Please follow these requirements: I, The dialogue should sound natural, conversational, and realistic. 2. Add small number of interjections (e.g.. "ah," "oh," "hey") and filler words (e.g., "um," "you know," "like") to enhance authenticity. 3. The dialogue content should be logically coherent and reflect everyday life experiences. 4. The total length should correspond to 40-50 seconds of speaking time. 5. Use "A" and "B" as speaker labels. The output format must be JSON array, with the following structure only:
"speaker": "A", "text": "First utterance"
"speaker": "B", "text"; "Second utterance" }...
Please return only the JSON array.
Audio Synthesis and Dialogues Merging For each dialogue script involving two speakers attained, we selected the voices of two participants in human-machine dialogue recordings and performed voice cloning for each speaker's individual utterances. This approach helps mitigate bias caused by speaker voice characteristics, preventing users from inferring the human or machine identity of the responder based solely on speaker A's voice.
After generating the speech for each utterance from both sides, we concatenated them in dialogue order form complete conversation. Between each utterance, we inserted random pause of 180-230 milliseconds to ensure natural timing between sentences. Finally, we added short back- ground noise sample, taken from the reference voice, over the entire concatenated dialogue further enhance the naturalness of the conversation.
Through the above process, we obtained a complete pseudo-human dialogue. total, 36 dialogues were generated with Nari-TTS (all English), and another 108 with Spark-TTS(36 English, 72 Chi- nese). Since Nari 1.6B only supports English, it used exclusively for English dialogues.
Use of the Pseudo Human Dialogues All Pseudo human dialogues synthesized by TTS were included only in the Turing evaluation set and were not used for training our evaluator. These dialogues were incorporated into the gamified Turing Test released on social media, making the task more challenging and engaging. As shown in Figure 4a TTS models achieved the highest success rate in the machine side.
C TURING-TESS TEST
The section is organized into the following sections:
Section
C.1
Turing Test Platform.
Section
Supplementary Turing Test Results.
Section
C,3
Influence of Dialogue Length on Turing Test Performance.
C.1 TURING TEST PLATFORM
Pre-test phase. Prior to the evaluation, participants provide basic demographic information as shown in Figure 9a including age, gender, education, and familiarity with AI, which may influence their judgments. They can also select between Chinese and English dialogues, allowing them to make judgment using their preferred language and thereby improving the reliability of the results. Testing phase. Each round of evaluation contains 5 dialogues to be judged. After completing round, par- ticipants may either proceed to the next round pause. Post-test phase. All incomplete submissions
21

---

Published as a conference paper at ICLR 2026
discarded ensure data integrity. The remaining responses then aggregated and analyzed in conjunction with the demographic information collected during the pre-test phase. To boost engage- ment, participants receive points based the accuracy of their judgments, and public leaderboard ranks all players based on their performance as shown in Figure 9 This analysis enables us to identify potential influences of user characteristics evaluation outcomes. The homepage and main interface of the platform are illustrated in Figure an and Figure 3 respectively.
Edit Profile
Home
Rankings
perera TESP LORDR
I 735
Challange
1
Quick cholengs
Hext Modk
2
3
Sconverations
Mere Chellepeiing
18-25
Gender
Male
Voice Modes
pts
Closdic Movie Lines aitu meNGIcE
p10
Al Tooli Fumillaiity
Wery faniika, have extaral
Music Challenge orig volce
100
(a) User Profile
(b) Homepage
(c) Turing Game Rank
Figure 9: The Turing test platform.
C.2
SUPPLEMENTARY TURING TEST RESULTS
Table shows the exact Turing test success rates of S2S systems, measured the proportion of responses judged as human. Higher values indicate greater human-likenes.
Table 9: Success rate of $2S systems (%)
Model
GPT-4o
Claude-Sonnet 4
Qwen3
Gemini-2.5
Kimi-K1.5
ChaiGLM-115
English
25,9
22,9
6.7
19.0
30.3
11.8
Model Hunyuan
Doulao-Pro 1.5
IPLNTER-Spark
Spark-TTS
Nari-TTS
Speaker
English Chinese
20.0
21.9
0.0
25.6
37.8
86.7
20,9
21.9
14.0
36.5
0.0
70.0
Figure 10 presents the success rates of S2S systems by levels.
C.3 INFLUENCE OF DIALOGUE LENGTH ON TURING TEST PERFORMANCE
We divided the Turing test results by dialogue length and calculated the classification accuracy for different dialogue types: human-human dialogues (H-H), human-machine dialogues (H-M), and pseudo-human dialogues (PH). Table I0 summarizes the accuracy results across different duration ranges.
22

---

Published as a conference paper at ICLR 2026
Paying
Fumen Licansss Promping
0.25
Success Rate- 1961
175
15.8
25
6.15
0:35
OFT4o
Cauate Spetrel
Ownn
Gamink2.3 pmo
Kima
CheecLa-
Humpsantartis
IFU Spart
Deubse
Figure 10: Success rate of S2S systems by different interaction strategies.
Table 10: Accuracy by Duration Interval for H-H, H-M, and PH.
Duration
H-H (ace/count)
H-M (acc/count)
PH (acc/count)
(20,25)
0.4000 /
-/0
0.6624 / 157
(25,30)
0.7800 /50
0.7742 31
0.6654 / 257
(30,35)
0.6513 152
0,8333 126
0.6337 / 243
[35,40)
0.7033 246
0,8642 162
0.6087 192
(40,45)
0.6839 174
0,8498
0,6200
[45,50)
0.7179 78
0,8564/ 195
0.6333
(50,55)
0,8421 76
0.7737 137
0.5349 43
(55,60)
0.7234 141
0.7907 43
-/0
As shown in Table [I] we performed ArmmianeArmitgge Trend Tests to examine the potential linear relationship between dialogue length and accuracy and found significant trend for any individual dialogue type. This suggests that dialogue length alone does not significantly influence the likeli- hood of passing the Turing test.
Table 11: Statistical Test Results Across Dialogue Types.
Dialogue Type
Z Statistic
p-value
Significant Trend?
H-H
1.6604
0.09683
H-M
-1,0106
0.31220
PH
-1.6018
0.10919
D FINE-GRAINED HUMAN-LIKENESS DIMENSIONS
The section is organized into the following sections:
Section
D.I
The Taxonomy for Fine-Grained Human-Likenes, Diagnosis.
Section
D.2
Annotation Process.
Section
D.3
Annotation Quality Assurance.
23

---

Published as a conference paper at ICLR 2026
D.1 THE TAXONOMY FOR
HUMAN-LIKENESS DIAGNOSIS
We organize the evaluation dimensions into five categories: 1. Semantic Features and Pragmatic Habits; II. Pon-Phyyiilogical Paralinguistic Features; III. Physiological Paralinguistic Features; IV. Mechanical Persona; V. Emotional Expression. Notably, annotators are instructed to use these dimension descriptions to rate the
of each conversational response on a five-point scale: 1 indicates strongly machine-like behavior, 5 indicates strongly human-like behavior, and 3 denotes no clear human- or machine-like leaning, or no enough evidence to judge.
First, five speech domain experts employ a prompt-driven, heuristic querying process with GPT- 4o to generate an initial set of concepts that differentiate human and machine responses. This method ensures that the generated concepts are both grounded in expert knowledge and supported by the model's comprehensive language understanding. The set then refined iteratively, with expert feedback and relevant social science literature retrieved by GPT-4o, ensuring that only the most representative and discriminative concepts are retained. The resulting dimensions are summa- rized in Table T2 This refinement process adds scientific rigor by aligning the selected concepts with established theories in the field, enhancing their validity. The final outcome a set of five categories, encompassing 18 fine-gr fine-grained dimensions, which are both comprehensive and precise. These dimensions were subsequently used to annotate dialogue training data through a crowdsoure- ing model. The ultimately trained model achieved significant improvement in human- machine dialogue recognition. This also demonstrates the reasonableness and reliability of these dimensions.
Table 12: Fine-grained
evaluation taxonomy.
Dimension
Description
Forgetting key information and unable realize errors; Human-like: Consistent memory in short contexts or asks for clarification when misunderstanding occurs(Toneve) et al. 2019).
Machine-like:
Memory Consistency (I)
Machine-like like: Abrupt logical transitions or self-contradictions; Human-like:
Logical Coherence (I)
Natural and coherent Baassoning(Botttaz Grifoni & Ferrario 2025).
Pronunciation Accuracy (T)
Machine-like: Mispronunciation (including heteronyms); Human-like: Cor- rect promunciation, with proper usage of hetcronyms 2021
Machine-like: Unreasonable multilingual mix; Human- like: The mix of lan-
Code-switching (I)
guages is context-dependent, and the switching is smooth(Zhang] 2019).
Responses are precise and affirmative: Human-like: Uses vague appressions(Piantddos et al. (2012) like "probably", and self- CorrectionssNaaatan & Hirschberg 1993).
Machine-like:
Linguistic Imprecision (I)
Machine-like: Rare use of fillers or unnatural usage; Human-like: Frequently uses (e.g.. "um". "like") while (Szkkly et al. 2019).
Use of Fillers (I)
lich: Direct, lacking semantic diversity, only capable of surface-level interpretation: Human like: Uses metaphor and euphemism convey implied meanings (ullchanooa & Vulchanov 20118).
Metaphor & Implied
Meaning (I)
Machine- like: No pauses or mechanical pauses; Human-like: Speaking rate varies with semantic coherence, with occasional hesitations(Iwwagg et al. 2023).
Rhythm (II)
Machine- like: Unnatural or flat intonation; Human-like Natural pitch rise or fall et al. 2025).
Intonation (II)
Machine-likes No emphasis on words or abnormal emphasis placement;
Stress (II)
Human Mike: Consciously emphasizes key words(Prieto & Roscano 2018).
Machine-likee Contextually incorrect or mechanical auxiliary sounds;
Auxiliary Vocalizations (II)
Human like: Produces appropriate nor-verbal sounds to express emo-
tion(Ani et al. 2023).
Micro-physiie picco-physiological Noise (III)
Machine-likes Speech is overly clean or emits unnatural sound; Human- like: Humans produces breathing sounds, saliva sounds, etc(Fukuda et al. 2018).
24

---

Published as a conference paper at ICLR 2026
Machine-lite like: Pronunciation overly clear; Human-like: Some irregularities pronunciation (e-g., tremolo, slurred speech, nasal sounds)(Teixeira al.) 2013).
Pronunciation Instability (III)
Machine- like: Stiff and unnatural accent; Human-like: Natural regional ac- cent or vocal traits(Onda et al. 2025).
Accent (III)
Sycophant Behavior (IV)
Excessively agrees, thanks, and apologizes: Human- Judges whether to agree based on context(Fanous et al. 2025).
Machine- like:
Machine- like: Responses are well-structured and formal. frequent listing: Human-like: Conversational, flexible, and varied expression(Doyle et al.
Sritten-style Expression (IV)
2019
Machine-likes Emotion conveyed in text may appear mismatched with natural human sentiment. Human-like: Emotion in text feels authentic and resonates naturally with human emotional expression, (Wang et 2025a).
Textual Sentiment (V)
Machine-like: Prosody tone may sound inconsistent with the intended emo- tion expression of the text. Human-like: Vocal delivery conveys context- appropriate emotional cues that align with the text Voorveld et al. 2025).
Acoustic Emotion (V)
D.2 ANNOTATION PROCESS
We recruited 36 annotators, all of whom are graduate students (master's or Ph.D.) specializing in artificial intelligence, with bilingual fluency in both English and Chinese. Prior to the formal anno- tation, each annotator received comprehensive annotation guidelines (see Figure [T] and completed calibration phase consisting of multiple trial batches (5 samples per batch, approximately 20-30 minutes each) to ensure consistent understanding of the evaluation criteria. Annotators were com- pensated at rate of 30 units/hour (local currency), with total cost equivalent approximately 5,250 units.
Annotators are instructed to these dimension descriptions to rate the human-likeness of each conversational response on a five-point scale: indicates strongly machine-like behavior, 5 indi- cates strongly human-like behavior, and 3 denotes clear human- machine-like leaning, enough evidence to judge. In line with the setup of Turing Test, they only evaluated the responder's performance in each dialogue.
To ensure the reliability of annotations, we implemented the following:
We created a questionnaire webpage where the crowdsourced annotators could access. Screenshots of the web are provided as figurq and figurq[2] Annotators' submissions are stored in our private Hugging Face dataset. Each submission contains 5 dialogues. For each dialogue, 18 ratings based on the 18 dimensions are associated with it. After grading each dialogue, the annotators also needed indicate their judgment of the identity of the responder (final choice).
We provided reference descriptions as mentioned in the previous section for each dimen- sion.
The annotators were unaware of the human-machine identity of the responder, and has never heard the dialogues before.
Before scoring, annotators must read the detailed guidelines, and we also provided training and clarification for them.
Guideline for Annotators
The score reflects the degree of human-likenedn of the response in given dimension.
Even you are confident about the identity the responder, you are required to indepen- dently evaluate the degree of human-likeness for different dimensions.
A score of indicates uncertainty about whether the responder more human- or machine-like. It also indicates that the dimension not reflected in the dialogue. The underlying meaning of 3 is that this score has no contribution to the final choice.
25

---

Published as a conference paper at ICLR 2026
Test Instructions
Every dialogue includes 2 speakers and lasis around minute.
The one
diologuu an.
Respondent: The other
For each question, you'll evaluate the respondent (nat the initiato)) acrezs dimensions.
Under each dimension, every listed festure from
B Scoring Cuide:
1 Stronglly machine-les
2 Somewhat machine like
3 neutral (no clear human machine lsan, ar evidence)
4 Somnewhal hurnan-Lkk Like
5 S-Strongly human-lik
After rating all climensions. make a final ludomentiss the respondent human or an AI?
You can mrelly suitch between dimensions using the Previous and Next buttons.
Important Notes:
Stick your username all the time.
Rememmerr pause the audio before proceed to the final judgement. Otheruisee it will keep playing and you cannot
stap it.
Choe you start the best, try nox to refresh the page or quit you need to grade recoodlings every test.
Focus on whether the respondent's speech sounds more human-like or machine-like for each feature.
For example: correct pronunciation doesn' always mean "human', and mispronunciation unciation doesn't meon "FT" Think terms human-lkkness.
Fuen if fre confidint early otex the resprrdent' infentity. still evalunte each dimansion indopandently Rvoiid just labeling all dimensions "machine like" "human like" without istening carefully.
Figure 11: Annotator guideline page.
D.3 ANNOTATION QUALITY ASSURANCE
For quality control, we invited three experts specializing in human-compute interaction to conduct cross-validation on all submitted content. Each expert was provided with the true labels indicat- ing whether the dialogue was generated by a human or machine. Only annotations unanimously approved by all three experts were included, while those with any disagreement underwent expert discussion for revision. total of 29, 30.44% of the labels were revised, with average adjustment of 1.99 points 769.76% of the score range), demonstrating the effectiveness of expert review in miti- gating noise in the raw annotations. Table 13 presents the three dimensions with the highest change ratio, along with overall results across all 18 dimensions.
Table 13: Expert Revision Impact on Label Adjustments
Dimension
Change Ratio
RMSE RMSE Ratio
Pronunciation Accuracy
0.3596
2,1085
0,5271
Textual Sentiment
0.3472
1.9579
0.4895
Linguistic Imprecision
0.3273
2.1230
0.5308
Overall
0.2944
1.9903
0.4976
26

---

Published as a conference paper at ICLR 2026
Formal Test
Question 1/5
Dimension 2/ S: Non-Phhsiilooical Paralinguistic Features
Test Audio
+
-| |I
aod Jutt
Moo to ref ioffine
Il
m
Please rate the respondent (not the initiator) in the conversation based on the following features (0-5 points. 0 Feature not present; 1 Machine; 3 Neutral: 5 Human)
Machine- likeileee Almost pauses mechanical pauses; Human like: Speaking varies with semantic flow, occasional pauses or hesitations
Rhythm:
Intonation: Machine-like like: Monotonous or overly regular pitch changes, inappropriate to the context: Human-like: Natural pitch rise fall when expressing questions, surprise, emphasis
Stress: Machine-lite -ike: No emphasis on words abnormal emphasis placement; Human-like: Consciously emphasizes key words to highlight
Auxiliary Vocalizations Machine-like: Contextually Incorrect mechanical auxiliary sounds; Human-like: Produces context approorraae non-verbal sounds, such as laughter or sighs
Previous Dimension
Dimension
Figure 12: Annotation page example.
To further validate annotation reliability, trained models data before and after expert rection. As shown in Table 14 Expert-refined labels lead to substantial improvements in both in-distribution and OOD generalization, confirming the quality of our final annotation set.
Table 14: Binary Classification Accuracy (Before/After Expert Modification)
Data
Overall (Inner)
Cosy Voice2
Fisher
MultiDialog
Overall (External)
Original Modified
0.8791 0.9605
0.9375 0.9844
0.6250
0.9062
0.8229
0.9844
0,9531
0.9740
E EXPERIMENT DETAILS OF AI JUDGER
The section is organized into the following sections:
Section
E.
Prompt Templates for AI Judges.
Section
E.2
Training Setup.
Section
E.3
Embedding Readout Selection.
Section
E.
4
Model Ablation.
27

---

Published as a conference paper at ICLR 2026
Section
E.3
Hyperparameter Tuning.
Section
E.6:
Fine-Graine, Human- Linenn-Lseness Scoring Accuracy.
Section
E.7
Contribution Analysis by Case Study.
E.1
PROMPT TEMPLATES FOR AI JUDGES
The following Figure 13 shows the prompt used for Al judges.
You are an expert in speech analysis and speaker identity recognition. Your core functions include but are not limited to: Automatic Speech Recoguition, acoustic feature extraction, speaker diarization, and analysis of Text-to-Speech lechnologies. You are designed to perform objective, evidence-based reasoning. Analyze the dia- logue in this audio file and strietly adhere to the following steps:
Speaker Diarization: Identify and differentiate the speech segments of Speaker (the first person to speak) and Speaker B (the respondent).
Al-Generated Voice Detection: Based on an analysis of vocal characteristics,
determine whether the voice of Speaker B (the respondent) is Al-generated.
Verdict:
If Speaker B' s voice is determined to be Al-generated, output:
Otherwise, output: f final "aai_
Figure 13: Prompt templates for Al judges.
E.2 TRAINING SETUP
All experiments are conducted on our constructed dataset. Specifically, we use 831 samples (<x11) for training and 208 samples (=2h) for validation, obtained from the H-H and H-M subsets with 1:1 ratio. The test set consists the remaining Human- Hamnan--Hman((HHH] and Mannnn--acinn(HHMM samples together with TTS data, forming 430 samples (<:5) with balanced 1:1:1 distribution. For modeling, we adopt Qwen2.5-0mni-77 as the backbone of our turing judge and further evaluate its LoRA fine-tuned variant. During hidden state extraction, consistency, and apply standard normalization to the hidden representations. For both modules, adopt Adam as optimizer. The complete experiments, covering feature extraction, inference evaluation of multimodal large models, and model training, are carried out on computing cluster with 8xA40 GPUs (48 GB memory per GPU).
fix random random-sample sample False to ensure
E.3 EMBEDDING READOUT SELECTION
Readout Design. In Sowen2.5-mmi only the first step exposes hidden states for the complete in- put sequence; at subsequent steps, each layer outputs a hidden state only for the newly generated token. Under this constraint, a simple average over step-1 token-level states (a length-agnostic baseline); (ii) Last-token rep- resentation: the hidden state of the most recent token compact, compression-style summary; and (iii) Fused pooling: a learnable weighted fusion of [first hidden hidden__ean, last hidden) into a sin- gle embedding. This two-source hidden representation lets the model adaptively fuse the globally contextual, acoustics-aware signal in frriihidden bean mean with the high-level semantics distilled in last
design three readout candidates:: (i) First-step mean pooling:
Ablation study. To identify which sequence embedding best supports our downstream objectives, we conduct an ablation under a unified hyperparameter regime (Table 15). We evaluate the three
28

---

Published as a conference paper at ICLR 2026
readout strategies under fixed proto- cols so that any performance differ- ences be attributed solely to the readout. Each alternative paired with the same ODL-Lineer head and trained end to end to convergence. To assess stability, all evaluations are conducted five times with different random seeds; we report the human machine classification accuracy as the mean standard error (s.e.m.) over the five runs.
Table 15: Tuning parameters.
Prompt
Scale
Batch Size
Learning Rate
Dropout
10 for ODL
Understanding
1 10 for Lincar
Table 16 shows the overall performances corresponding to three readouts. Fused pooling attains the highest overall score (0.9112), outperforming mean pooling (0.8879) and last-token representa- tion (0.8032). On the Pseudo Human dataset-which is strictly out-of deleoo--iisriuution-uu reaches 0.8167, while other baselines remain at 0.7805 and 0.7917. This gap indicates improved robustness to distributional shift. Consistent gains across Human-Human Human and Human-Machine set- tings further demonstrate that the fusion abstract semantic and speech sequential information enhances discriminative power beyond what either source provides independently.
pooling
Table 16: Ablation experiment results (mean s.c.m. over runs).
Data Type
First-step Mean Pooling
Last-token Representation
Fused Pooling
Hounun-Human
0.9409(=0.0017) 0.045)
0.7380<#0.0447) 0.8791(=0.0028) 0.7917(=0.002) 0,8032(=0.0012)
0.9493)=0.0014) =0.06170.0017 0.8167(#000061)
Pscudo Human 1
0.7805(=0.0100)
Overall f
0.8879(=0.0044))
0.9112(#0.0020)
Overall, these findings show that the choice of readout materially impacts downstream performance. Fused pooling provides consistent improvements across all settings, including out-of-distribution evaluation, and therefore constitutes reliable default for sequence- embedding utilization.
E.4 MODEL ABLATION
To validate the effectiveness of ODL, we conducted an ablation where we removed the ODL and replaced it with a standard linear layer and negative ligeliillhood loss, treating the scores as independent categories. This baseline corresponds to non-ordinal but still interpretable classifier. This clarifies that ODL is used as appropriate modeling choice for ordinal labels, and that our ablation demonstrates its empirical value.
Table 17: Binary classification accuracy across module ablation
Projection Module
Human Human-Humas Human
Human Machine
Pscudo Human
Overall
Ordinal Discretization Layer
0.9507 0.8718
0,9722 0.9875
0.9306 0.9097
0.9605
Linear Layer
0.9233
E.5 HYPERPARAMETER TUNING
Grid Search. To further optimize model's performance, we tune hyperparameters for ODL and FL independently using grid search.
As summarized in Table the ODL space comprises 3 1000 x 4x4x5 = 240,000 configurations, while the FL space contains combinations. To reduce computational cost, we uniformly sampled 7500 ODL configurations and paired each with all 16 FL settings, yielding 120.000 trials in total. Each trial requires ~5 minutes on a single GPU, corresponding to 10, 000 GPU-hours overall.
16. The joint search space therefore consists of 3.84M
Tuning Criterion. To select optimal hyperparameters, we adopt accuracy as the primary objective for grid search, which reflects the downstream classification goal of human-machine discrimina-
29

---

Published as a conference paper at ICLR 2026
Table 18: Hyperparameter search space.
Module
Prompt
Scale
Batch Size
Learning Rate
Dropout
0.1
16
le-2
Understanding
Ordinal Discretization Layer
32
le-3
Transcribe
1:0.01:10
0.3
Classify
64
le44
0.4
128
lc-5
0.5
32
le-2
64
le-3
Linear Layer
128
1e4
256
1o-5
tion. The tuning results are summarized in Table 19 and the selected configuration is used used throughout all experiments.
Table 19: Tuning results.
Module
Prompt
Scale
Batch Size
Learning Rate
Dropout
Ordinal
Understanding
64
le-5
0.3
Discretization Layer
Linear Layer
128
le-3
Sensitivity Analysis. As a complementary experiment our main hyperparameter tuning, we per- formed 1000-run randomized hyperparameter search, sampling key training parameters for ODL (learning rate, batch size, scale, dropout) and FL. (learning rate, batch size). Each configuration was trained end-to-end using the same evaluation protocol, ensuring reliability through full paral- lelization. The results for the hyperparameter sensitivity analysis (accuracy) are presented in the Table 20
Table 20: Hyperparameter Sensitivity Evaluation Metrics
Hyperparrameter
Values
Acc (ODL)
Acc (FL)
MSE (ODL)
(FL)
(ODL)
(le-05, 1e-04, 1e-03, 1e-02)
0.6020((0004355
(+0.0071)
0.002174
0.000051
batch size (ODL)
[32, 64, 128, 256)
0.6105 (#0.0065)
0.8601(#000149)
0.000048
0.000222
scale
1.05, .... 5)
0.6293(=0.0090)
0.9254(=0.0283)
0.000091
0.000802
dropout
(0.1, 0.2, 0.3, 0.5)
0.6103 (+0.0050)
0.8617 (+0.0103)
0.000029
0.000105
(FL.)
(le-05, 1e-04, 1e-03, 1e-02)
0.6109(=0.0031)
0.8584(10.0696)
0.000011
0.004838
butch_size (FL.
(16, 32, 64, 128)
0.6107(00.034)
0. 0.8650(#0.0193)
0.000013
0.000371
Analyzing the results, we identify several key findings:
Learning rate proved to be a critical factor for both ODL and FL, consistent with findings from other work. Extremes caused underfitting instability, emphasizing the need for precise tuning.
Scale had minimal impact on ODL accuracy, suggesting ODL's adaptability, but slightly affected FI. due to scale-induced changes in logits cut-points
Batch size influenced FL performance, with larger batches stabilizing training but poten- tially slowing convergence causing overfitting.
Dropout and ODL batch size showed minimal effects
Overall, the 1000-run analysis shows that our method generally robust, with learning rate being the most sensitive parameter, while other hyperparameters produce only modest effects.
30

---

Published as a conference paper at ICLR 2026
E.6 FINE-GRAINED HUMAN-LIKENESS SCORING ACCURACY
Accuracy Analysis. Since the 1-5 scores reflect perceived buman-likenest we report not only the exact accuracy that measures full agreement with human judgments, but also grouped accuracy that consolidates scores into three categories (1-2, 3, and 4-5) to better reflect alignment with human perception. In addition, we include accuracy within tolerance of #1 to capture near-agreement with human ratings.
Exect
Group
Nearby
0.8
Aco./ayy
0.4
0.2
Memory Consistercy
Logical coterence
Promaciation Accuracy
cate-attchion
Lingusticc imprre impreciaion
Metaphor implled Meaning
Use Fillers
Rlythm
Intonation
Auxilaar Mecalizadons
Micro phyoislogical Noise
Streer
Promunclation batabiiity
Sycuphann Behaaiir
Accent
Inittin style Exprestion
Tixtual Sentimant
Acmuntic Emetion
Figure 14: Fine- line-grained scoring accuracy.
As shown in Figure 14 the Ordinal Discretization Layer consistently exceeds 50% exact accuracy across all evaluation dimensions, often reaching 70%. When consolidating scores into three bins or allowing tolerance of +1, accuracies in most dimensions approach or surpass 80%. With more detailed accuracies provided in Table 21] these results indicate that the model captures the correct ordinal direction in fine-grained judgments and aligns closely with human perceptions, yielding in- terpretable evidence for downstream human-machine classification. Moreover, our training frame- work not only substantially enhances binary classification accuracy but also systematically aligns the model with human evaluation dimensions, enabling it to learn human-like judgment patterns.
Table 21: Detaied accuracies.
Metries) Dim
IDim
RT
ACC1
0.7308
0.6971
0.7067
0.9423
0.5625
0.6490
0.9327
0.6058
0.5433
ACC (Group)1
0.8846
0.8654
0,8606
0.9423
0,8317
0.7644
0.9327
0.8365
0.7788
ACC (#1)f
0.8846
0.8654
0.8606
0.9471
0.8317
0.7644
0.9519
0.8413
0.7837
Metries) Dim
ST
AV
MN
P1
AC
SB
WE
TS
AE
ACC!
0.6010
0.7404
0.7452
0.7788
0.7596
0.7163
0.7740
0.5481
0.6683
ACC (Gmupit
0.8846
0.7981
0.8654
ACC (+1))
0.7933
0.8029
0.8846
0.8894
0.8462
0.8942
0.8654
0.7163
0.8221
Out-of-domaii Evaluation To further evaluate the model's generalization for the five-degree rat- ing, we invited human experts to annotate the OOD samples on multiple dimensions and report three accuracy metrics, where Exact is the percentage of predictions that exactly match the expert score, Group the percentage that fall into the same human -aachine identity group (1-2 machine- unclear, 4-5 human-like like), and Nearby is the percentage that differ from the expert score at most +1. The results are shown in the Table 22, indicating that our model maintains strong generalization ability in fnninnd scoring.
31

---

Published as a conference paper at ICLR 2026
Table 22: Overall Fine-grained Scores Accuracy
Dataset
Exact
Group
Nearby
Ours
0.7056
0.8408
0.8470
Cosy Voice2
0.6450
0.7569
0.8030
Fisher
0.6476
0.7396
0.7752
MultiDDalog
0.6562
0.7561
0.7847
E.7 CONTRIBUTION ANALYSIS BY CASE STUDY
Case Study. To probe the interpretabilityy of the model's human-maching discrimination, we con- duct case studies spanning two diagnostic regimes: (i) machine-class true positive (instances cor- rectly predicted machine) and (ii) machine-class false negative (machine instances incorrectly predicted as human). This design reflects and operationalizes the principles of the inverted Turing test, establishing continuity between analytical setting and evaluation framework.
For each instance, we first calculate each contribution C₁ on machine-side by producting standard- ized ODL logits (standardized with respect to the training-see distribution) together with correspond- ing trained linear weight. Then, we rank top 8 features by |ck| to identify the most influential factors. By construction, C (machine-like scoring) increases evidence for the machine class, whereas C < (human-like scoring) reduces it.
As shown in Figure 15 most ime-grained scores align with their final contributions to hu- man-machine classification. In [igure[[5a] despite strong human-like cue (e.g.. a negative contri- bution from Promunciation Accuracy), the model aggregates multiple machine-oriented signals, such as Sycophant Behavior and Pronunciation Instability, yielding high-confidence correct decision. By contrast, in Figure [5b] high-score dimensions (c.g., Memory Consistency. Pronunciation Accu- racy) contribute salient human-like evidence that shifts a machine sample into the human region; the available machine-lide cues are insufficient to overturn the outcome due small effective margin, leading the system to accept the machine response as human in the sense of an inverted Turing test.
Case evidence shows that S2S outputs perform strongly on dimensions such as Memory Consistency and Logical Coherence, leading annotated scores to concentrate in the 4-5 range; nevertheless, the associated logits remain informative within this high-score regime. When the model maps inputs to human-like scores, these dimensions place samples within higher-valued latent intervals along a continuous scoring axis. This induces within-bin margins: sample-wise logit variability driven by subtle linguistic acoustic cues. In downstream binary classification, such variability produces derginnde contributions: near-cutpoint (low-margin) instances can exert negative influence, whereas far-beyond-cutpoint (high-margin) instances provide strong positive evidence. Thus, even under apparent rating saturation, logits retain fine-grained discriminative power via their ordinal positions and margins.
32

---

Published as a conference paper at ICLR 2026
Machine-side Logit: 1.6074 Confidence: 0.9614
Pred: Machine Machine
Intonation
0.158 Score:
Micro-phy serro-physiologicall Noise
0.161
Score:
Auxiliary Vocalizations
0.165
Score:
Acoustic Emotion
0.207
Score:
Written-style Expression
0.210
Score:
Pronunciation Instability
0.316
Score:
Sycophant Behavior
0.535
Score:
Pronunciation Accuracy
-0879
Score:
1.00
0.75
0.50
0.25
0.00
0.25
0.50
0.75
Contribution Value
(II)
(III)
(IV)
(V)
(a) Machine-clase true positive
Machine-side Logit: -0.1601 Confidence: 0.5794
Pred: Human
GT: Machine
Micro-physiological Noise
0.038 Score:
Metaphor Implied Meaning
0.053 Score:
Logical Coherence
-0.076 Score:
Sycophant Behavior
-0.083 Score:
Auxiliary Vocalizations
0.086 Score:
Acoustic Emotion
0.129 Score:
Pronunciation Accuracy
-0.351 Score:
Memory Consistency
-0.47 Score:
0.0
0.2
Contribution Value
(I)
(II)
(III)
(IV)
(V)
(b) Machine-class false negative
Figure 15: Case studies
33


Reviews:
[{"review_id": "bYpvaN2ZrJ", "reviewer": "Reviewer_pdbT", "summary": "This paper performs a Turing test experiment with multiple speech to speech models. Multiple versions of the experiment are tested using controlled conditions that attempt to minimize potential confounds. To collect more data, a mobile app is released as well. Conversations were also annotated with 18 dimensions. The results show key gaps in all S2S models. Using these insights, a smaller speech model is fine-tuned to predict these ratings and then classify whether a given dialog is human-to-human or human-to-s2s, attaining very high accuracy while still being interpretable.", "questions": "- How was the annotation performed (see questions above)", "limitations": "", "rating": 8, "confidence": 5, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "- Clean experimental setup that controls for multiple confounds and tests a diverse suit of S2S models, including TTS models using LLM-generated text. Experiments demonstrate key gaps.\n\n- Rich analysis of why models fail the test through a multifaceted analysis of the conversations qualities. This analysis involved collecting human perceptions using crowdsourcing\n\n- Experiments to see if other audio models could pass the test, with key gaps in existing models. Proposes new model and design to get an interpretable explanation, showing that these features (when correctly identified) are reliable indicators of gaps in current system\n\n- Released platform and game to collect new annotations, helping grow data and potentially incorporate more models in the future\n\n- Very well written paper with clear motivation, analysis, and visuals.", "weaknesses": "- The biggest gap to me was in the lack of details around the annotation for conversation qualities. These are barely mentioned in text, so I was expecting to see a much more detailed report in B.5. However, important questions are hard to answer, such as who annotated (which platform?), how many annotators were there, did annotators agree on these qualities, how much were annotators paid, or what quality controls were present, if any. Given the importance of this data for your results and later test-taking model, more details are needed to assess the quality of the data and for future replicability.\n\n- The paper itself is very dense (though well written). However, the space constraint has pushed many details to the appendix which hinders readability at times.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "hxXj7b6Ltc", "reviewer": "Reviewer_zvPu", "summary": "The paper targets speech‑to‑speech (S2S) dialogue and proposes a Turing‑style evaluation. It builds a dataset that includes human–human, human–machine, and pseudo‑human (TTS‑synthesized) conversations, and uses a game‑based human study to judge whether current systems “sound human.” The headline finding is that none of the evaluated systems pass. To explain why, the authors introduce a fine‑grained human‑likeness taxonomy and crowd annotations, showing that shortcomings lie in paralinguistics (rhythm, intonation, stress, fillers, breath), emotional expressivity, and a “mechanical persona,” rather than semantics. Off‑the‑shelf AI judges are unreliable; authors therefore propose an interpretable evaluator that first produces ordinal scores on the taxonomy dimensions and then makes a transparent linear human‑vs‑machine decision.", "questions": "Q1. Theoretical Positioning of ODL\n\nWhat is the precise formal correspondence between ODL and classical ordinal regression models (cumulative-link, threshold models)? What does ODL add beyond these baselines in parameterization or inductive bias? Which properties are guaranteed by construction versus empirically observed?\n\nQ2. Necessity of 18 Dimensions\n\nWhy exactly 18 dimensions rather than a single score or reduced factorized set? Provide dimension-wise correlation analysis, clustering structure, and direct comparisons with (i) single-score model and (ii) low-factor model to demonstrate irreducibility.\n\nQ3. Annotation Reliability and Expert Impact\n\nNeed to report inter-rater reliability for dimensions. Quantify how expert edits change label distributions and downstream judge performance. Test measurement invariance across languages and subgroups.", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. Focused problem and clear protocol. The work targets the central question for speech‑to‑speech (S2S): Do these systems actually sound human in multi‑turn dialogue? Instead of testing isolated sub‑skills, the study frames evaluation as a Turing‑style decision under realistic interaction. The tri‑part setup, human–human, human–machine, and a TTS‑based pseudo‑human control, gives a clean yardstick for what “human‑like” means. Bilingual coverage and multiple everyday topics reduce overfitting to any single style and make results more comparable across systems. The recording and interaction procedures are standardized, improving internal validity and making cross‑system contrasts meaningful. \n2. Interpretable automatic judge. The proposed evaluator is intentionally two‑stage: first map dialogs to ordinal scores on the human‑likeness dimensions, then apply a linear decision with symmetry regularization. This keeps the prediction space aligned with how humans actually rate speech (ordered categories), while the linear head provides transparent attribution: which dimensions pushed a sample toward “human” vs. “machine.” The design is modular, portable across collections, and produces diagnostics that engineers can act on (e.g., prosody shaping, disfluency modeling, persona calibration).\n3.Memorable headline result. The paper lands a crisp, communicable takeaway: contemporary S2S systems still fail a Turing‑style test. That single sentence is easy for the community to remember and cite, and it reframes progress: sounding human is not simply a by‑product of better recognition or text generation. Because the result was obtained under a matched protocol with both human–human and synthesized controls, it carries weight beyond a one‑off demo and can serve as a reference point for future work.", "weaknesses": "1. Application‑heavy, limited theoretical novelty. The main novelty lies in system integration rather than theory. The core claim, that semantics alone cannot sustain effective speech interaction, is treated as an empirical observation, not a theoretical insight. Adding paralinguistic cues (prosody, affect, persona) targets known gaps, long discussed in TTS and affective computing. The work validates their importance but does not explain underlying mechanisms or interactions, nor does it offer a general theoretical framework.\n2. HCI‑leaning narrative.The main text emphasizes human‑study design, demographics, and logistics more than theory, ablations, and generalization, which may misalign with ICLR.\nThe manuscript emphasizes HCI logistics (recruitment, demographics, task design) while skimming key ML details (architecture ablations, training dynamics, hyperparameter sensitivity). This balance misaligns with ICLR. \n3. Limited external generalization and statistics. Training and evaluation are tightly coupled to the same data collection protocol, raising distribution-shift concerns. Evaluation relies on a single-threshold binary decision without calibration, uncertainty quantification, or decision-boundary analysis. Missing ablations across acoustic conditions, speaker populations, and interaction settings leave generalization in doubt. Statistical reporting should add uncertainty intervals, significance tests, calibration curves, and failure-mode taxonomies to substantiate reliability and scope. Also need more systematic ablations to clarify the theoretical footing of ODL, justify the 18‑dimension design and strengthen label reliability.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "T9RtCQjHtL", "reviewer": "Reviewer_tKss", "summary": "This paper introduces a Turing test study for speech-to-speech (S2S) dialogue systems, evaluating 9 LLMs against human speakers. Using a gamified online platform, the authors collect 2,968 human judgments across 1,486 dialogues in English and Chinese. None of the tested systems passed the Turing test, revealing a persistent gap in human-likeness. To understand the causes, the paper develops a taxonomy of 18 human-likeness dimensions, spanning semantic, paralinguistic, emotional, and persona-related traits. Crowd annotations show that while S2S systems perform near human levels in semantic coherence, they fall short in prosody, emotional expression, and conversational naturalness. Finally, the paper proposes an interpretable AI judge, which is a finetuned LLM that predicts human-likeness with strong transparency and accuracy, outperforming both humans and baseline AI judges, either prompted or LoRA-finetuned.", "questions": "- Any additional discussions or experimental results to resolve the above weaknesses?\n- Would shorter dialogues (e.g., 20-second versus 60-second dialogues) have more chance to pass the Turing test?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "- This paper presents the first formal Turing test for S2S dialogue systems, extending evaluation beyond text to spoken interaction, which is an impactful direction given recent advances in conversational AI.\n- The paper convincingly shows that the bottleneck of S2S dialogue systems is no longer semantic understanding but rather paralinguistic and emotional expressivity, which is an under-explored dimension in S2S research, offering valuable insights for improving S2S design.\n- The interpretable AI judge is a standout contribution, which provides a reproducible and scalable framework for automatic S2S evaluation, with decent human-machine discrimination accuracy.", "weaknesses": "- The gamified Turing test platform may attract casual participants who do not conduct the human-machine discrimination carefully. This paper does not clearly describe participants’ quality-control mechanisms  such as attention checks, response-time filtering, etc. This could bias the Turing test results.\n- It is unclear how many unpassed Turing test cases are because LLMs avoid human disfluency cues (or other fixes of human speech deficiencies). This is an easy-to-detect feature but a minor issue, since superior speech fluency is rather preferred by users in real-time applications. It would be better to track the cause of each Turing test failure and analyze the rate of such minor causes versus more severe causes.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "C5FfpPD1qp", "reviewer": "Reviewer_3UQQ", "summary": "This paper presents a study of testing if current speech-to-speech (S2S) models, e.g., GPT-4o's advanced voice mode, can pass a Turning\ntest of conducting human-like conversations. Authors firstly constructed S2S dialog datasets between human-model conversations, as\nwell as synthesizing speeches with TTS models. After that, approximately 3K human judgements were carried out, measuring the human-likeness of current S2S models. To measure the human-likeness, authors defined 18 dimensions such as memory consistency, use of fillers etc.\n\nAuthors show that humans can easily identify human-model dialogs, such that current S2S models cannot pass the Turning test. Authors also show that off-the-shelf AI models cannot serve as a judge for the test; finetuning the off-the-shelf AI on the authors' created dialog datasets enable AI to better judge human-model dialogs.", "questions": "- Line140: The datasets seem contain both English and Chinese. Are the 28 participants from 10 countries all speak both two languages?\n\n- Line278-279 describe that current S2S are limited by aspects like topic understanding etc. While Line332 claims that S2S models have largely solved the foundational challenges of understanding and generating clear and coherent dialogue turns.  Wouldn't these claims contradicting? How do you define semantic and paralinguistic tasks?", "limitations": "", "rating": 2, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "- This is the first evaluation of S2S models from the Turing test prospective, which is new and interesting.\n- Evaluation details are well presented with clear takeaway messages, and the paper is easy to follow.", "weaknesses": "- It seems the conclusion, i.e., current S2S models cannot pass Turning test, is very much expected.  E.g., VoiceBench[1] has shown that current S2S models still largely lag behind their text counterparts. Then the main contribution seems to be the \"Turing Test\" framing. I think authors should more clearly demonstrate how this paradigm can provide more insights than existing S2S benchmarks like VoiceBench.\n\n- The generalization ability of the judge model finetuned on authors' created datasets is unknown. It is very much expected that the in-domain (e.g,  based on the 18 evaluation dims) finetuned judge model performs the best on judging the dialogs. Authors should test the correlation of  the judge model with humans, when being applied to some other real-world OOD dialogs to ensure the generalization ability, beyond only testing on the pseudo-human datasets.\n\n- Some claims seem contradicting (cf Q2).\n\n[1] VoiceBench: Benchmarking LLM-Based Voice Assistants", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)