X-TURING: Towards an Enhanced and Efficient Turing Test for Long-Term Dialogue Agents
Authors: Weiqi Wu, Hongqiu Wu, hai zhao
Accepted at: acl25

X-TURING: Towards an Enhanced and Efficient Turing Test for Long-Term Dialogue Agents

Weiqi Wu, Hongqiu Wu, Hai Zhao†
1School of Computer Science, Shanghai Jiao Tong University
2Key Laboratory of Shanghai Education Commission for Intelligent Interaction and Cognitive Engineering, Shanghai Jiao Tong University
3Shanghai Key Laboratory of Trusted Data Circulation and Governance in Web3
{wuwq1022,wuhongqiu}@sjtu.edu.cn,
zhaohai@cs.sjtu.edu.cn

Abstract

The Turing test examines whether AIs exhibit human-like behaviour in natural language conversations. The traditional setting limits each participant to one message at a time and requires constant human participation. This fails to reflect a natural conversational style and hinders the evaluation of dialogue agents based on Large Language Models (LLMs) in complex and prolonged interactions. This paper proposes X-TURING, which enhances the original test with a burst dialogue pattern, allowing more dynamic exchanges using consecutive messages. It further reduces human workload by iteratively generating dialogues that simulate the long-term interaction between the agent and a human to compose the majority of the test process. With the pseudo-dialogue history, the agent then engages in a shorter dialogue with a real human, which is paired with a human-human conversation on the same topic to be judged using questionnaires. We introduce the X-Turn Pass-Rate metric to assess the human likeness of LLMs across varying durations. While LLMs like GPT-4 initially perform well, achieving pass rates of 51.9% and 38.9% during 3 turns and 10 turns of dialogues respectively, their performance drops as the dialogue progresses, which underscores the difficulty in maintaining consistency in the long term.‡

1 Introduction

The Turing Test, originally called the imitation game by Alan Turing in 1950, is a test of whether a machine can behave indistinguishably from a human (Turing, 1950). During the test, a human integrator converses in natural language with both a human and an AI, then tries to identify which is which. The AI is considered to pass the test if it can convincingly mimic human responses so that the integrator misidentifies it as a human participant. Contemporary Large Language Models (LLMs) (OpenAI, 2023; Touvron et al., 2023; Jiang et al., 2023; Yang et al., 2023; Bai et al., 2023; Anthropic, 2024) demonstrate remarkable language generation capabilities in various scenarios (Wu et al., 2024a). These advancements have sparked increasing interest in whether the latest LLMs can pass the Turing test (Jones and Bergen, 2024).

However, the traditional Turing test has several limitations: (1) The dialogue follows a strict ping-pong structure, unlike natural communication where people may send multiple messages in a row to convey a statement. (2) The need for continuous human involvement in interactions with the test subject limits test duration to under ten minutes, leaving long-term evaluation unexplored. (3) Measuring test duration in minutes ignores individual differences in reading, thinking, and typing speeds, potentially disadvantaging some participants.

To address these issues, we propose X-TURING, a new framework designed to conduct an efficient Turing test for LLM-based dialogue agents with an enhanced dialogue pattern that aligns with daily communication among humans. Firstly, we break the constraint of a single message each turn to allow users to send multiple messages in rapid succession without waiting for a response. As illustrated in Figure 1, this dynamic exchange mirrors natural text-based human interactions and challenges LLMs to maintain coherence and relevance in more diverse interactions. We construct chatbots for the test by prompting LLMs with social platform dialogue records from a real-life human.

Given the exhausting nature of long-term Turing test, we propose enhancing its efficiency by approximating real human-machine interactions through dialogue generation techniques (Soudani et al., 2024), and using questionnaires instead of
xxx: [21:05:51] Video games on TV?
xxx: [21:06:07] First time I’ve heard of it
Human 1: [21:06:14] Well then
Human 1: [21:06:23] There’s not much to operate tbh... Just choosing
xxx: [21:06:27] Aw sounds pretty simple
Human 1: [21:06:38] Hahaha yeah

Pseudo-Dialogue
Human 2: [09:20:01] Tried vegan bugers today
Response: [09:20:20] Huh my friend had tried before
Response: [09:20:25] Say it’s even better than meat ...

n topics
mn turns

Context / Topic 1
Been diving into Hesse’s novels lately...
Steppenwolf
Hesse’s books are awesome👍
What do ya think
Human 2 m turns
Context / Topic 1
Been diving into Hesse’s novels lately...
Steppenwolf
!!!
I just read Narcissus and Goldmund
Human 2 m turns
AI or Human?
AI or Human?
Judge

Figure 1: Overview of the X-TURING. Prompted by the dialogue history of a target person (Human 1), the LLM chats with another human (Human 2) after a specific turn of pseudo-dialogue to simulate its long-term interaction performance. The judges then distinguish between the LLM and Human 1 when they each converse with Human 2.

having evaluators engage in extended conversations. According to psychological research on the serial position effect (Murdoch and Bennet, 1962), the first and last parts of a sequence are likely to be most memorable and thus critical for judgment. As it is more challenging for LLMs to maintain consistency and coherence over long-term interactions (Park et al., 2023; Bubeck et al., 2023), our evaluation focuses on the last few turns of the conversation. Therefore, we propose pseudo-dialogue generation to allow the model to direct this progression on its own. It involves iteratively generating multi-turn dialogues to reflect the LLM’s evolving performance over time. Human participants only need to converse subsequently for a few more turns, as depicted in Figure 1. By focusing the evaluation on the final interaction turns, we can provide a comprehensive assessment while significantly reducing the human workload, effectively making the Turing test scalable in terms of interaction length.

Our contributions can be summarized as follows:

• We extend the classical Turing test with burst dialogue, more closely reflecting natural human communication.

• We employ dialogue generation techniques to automatically simulate user-machine interactions, making longer Turing tests feasible.

• We introduce a new metric, X-Turn Pass-Rate, to fairly assess the ability of LLMs to maintain human-like over a set of interaction turns.

• We perform extensive experiments using the X-TURING framework to evaluate the latest LLMs and explore key factors influencing test outcomes.

2 Related Work

Turing Test  Turing test and its variations have been extensively applied to assess AI across multiple domains, including dialogue interaction (Jones and Bergen, 2024; Lowe et al., 2017), question answering(Ng et al., 2024), text generation (Helm et al., 2023; Uchendu et al., 2021), and visual tasks (Imanpour et al., 2024; Zhang et al., 2022). These evaluations, which involve distinguishing between human-generated and AI-generated content (Gao and Emami, 2023) or inferring from solely the content (Jannai et al., 2023), serve as essential benchmarks for AI models. Existing works proposing the limitations of Turing tests focus on whether passing the test indicates intelligence accurately (Freitas et al., 2020) or comprehensively (Lee et al., 2023). We rather focus on its feasibility issue in long-term evaluation and enhance the flexibility of the communication pattern.

Role-play with LLMs  A common tactic in Turing tests is to assign LLMs the role of a specific
human character (Jannai et al., 2023), leveraging their proficiency in adopting varied attributes and conversational styles (Wu et al., 2024b; Shanahan et al., 2023; Wang et al., 2023a). Prompting the model with character profiles (Li et al., 2023; Tao et al., 2023; Wang et al., 2023b; Chen et al., 2023c) or key utterances (Han et al., 2022; Boyd et al., 2020) proves to be a more efficient method, compared with fine-tuning on massive character data (Shao et al., 2023; Zhou et al., 2023; Lu et al., 2024). By utilizing the dialogue history of the designated individual, we can develop a role-playing chatbot that effectively handles various dialogue forms and scenarios.

Dialogue Generation The development of synthetic dialogue data has provided a more efficient and scalable approach to data creation and augmentation (Soudani et al., 2024; Chen et al., 2024; Zhou et al., 2017). Advancements in dialogue generation have been facilitated by LLMs (Abdullin et al., 2024; Chen et al., 2023b,a; Kim et al., 2022a; Abbasiantaeb et al., 2023), which excel at creating contextually relevant and diverse conversations by prompted turn-by-turn (Lee et al., 2022; Kim et al., 2022b) or even in just one go (Jandaghi et al., 2023; Chen et al., 2023a). By iteratively generating dialogues that imitate interactions with a human, we can efficiently simulate the Turing test over extensive turns and facilitate further judgment.

3 X-TURING

3.1 Burst Dialogue v.s. Ping-pong Dialogue

Conversations with a ping-pong structure are considered the standard form of human-machine interaction and are commonly used in the Turing Test, where each user message is followed by a system response to form a strict back-and-forth pattern. However, in real life, people usually send multiple messages in rapid succession without waiting for the other’s response and the response can also consist of multiple messages. We define this communication pattern as Burst Dialogue and delve into the performance of LLMs in this setting.

In a ping-pong dialogue, a turn is naturally formed by pairing each user message with the corresponding system response. In contrast, burst dialogue involves a more flexible information exchange, where interruption and continuation can happen frequently for both sides. For simplicity, a turn of burst dialogue is defined as a consecutive series of user messages followed by the system’s responses before the next user message. Despite potential unbalanced exchange, burst dialogue is more dynamic and aligns more closely with the natural human communication pattern. It challenges the model to handle overlapping and rapidly evolving conversational contexts, thereby testing its ability to maintain coherence and relevance in real-time interactions. Hence, we extend the classical ping-pong dialogue setting to assess how well LLMs manage more realistic conversational dynamics.

3.2 Chatbot Construction

The Turing test necessitates the deployment of a chatbot to judge its text-based conversation ability. Instead of manually crafting detailed profiles for individuals, we construct the dialogue agent based on real-life dialogue history, which provides essential clues for generating responses that emulate the target personality, linguistic style and content. To achieve this, we gather multi-turn text-based social platform chat records for each target individual, which are burst conversational data consisting of timestamps and message content. The LLM is then instructed to maintain consistency with the conversation style with the prompt presented in Appendix B.1. For ping-pong dialogues, we retain only the first message of a consecutive series from the same person and manually filter out meaningless turns.

To implement a burst messaging chatbot, we employ three modules working synchronously as shown in Figure 2: • Input Listener: to collect

![Chatbot system enabling burst dialogue.](page_1012_186_393_312.png)

Figure 2: Chatbot system enabling burst dialogue.
user messages; • Model Caller: to query the LLM to generate responses; • Output Sender: to manage the delivery of system responses based on the timestamps. The system introduces a brief time interval \( t \) before processing the first batch of user inputs to allow the user to fully express themselves. After \( t \), the Input Listener processes all user messages collectively for the Model Caller to generate a response.

Following the given burst dialogue data and the instruction, the chatbot generates consecutive messages in a response that can be sent to the user and added to the dialogue history by the Output Sender based on their respective timestamps. In case of any timestamp errors, e.g., being earlier than the current timestamp, we resample the message sending time. A message consisting of \( n \) characters is sent after a delay \( d \) of around 0.3s per character to mimic the reading, thinking and typing speed of humans(Jones and Bergen, 2024):

\[
d = \mathcal{N}(0.3, 0.03) \times n
\]

The Model Caller typically requires time to generate responses with LLMs. During this period, the Input Listener and Output Sender continuously work to maintain the dialogue history. Once a query is completed, if the user has sent new messages during this time, the Model Caller initiates the next query; otherwise, it waits for a time interval \( t \) before processing the accumulated dialogue history to generate new responses. If new messages are generated, any unspent messages from the previous batch are cleared to maintain the conversation flow.

3.3 Pseudo-Dialogue Generation

To enhance the efficiency of the Turing test, we propose to let the LLMs direct the progression of the test by synthesizing extended user-machine interaction through dialogue generation. Classical open-domain conversational data generation generally includes three steps: (1) Input generation: Provide fundamental information to initiate the conversation that specifies the topic and personalized style, i.e., conversation seed; (2) Utterance generation: Utilize the conversation seed to generate a multi-turn conversation; (3) Quality filtering: Eliminate samples that fail to meet the quality criteria (Soudani et al., 2024). The quality filtering step is omitted as our dialogue generation process aims to reflect the genuine evolving performance of the LLM over prolonged interaction.

Algorithm 1 Pseudo-Dialogue Generation

Require: Number of Turns \( m \), Dialogue History \( H \), Topics \( T = \{t_1, t_2, \ldots, t_n\} \)
Ensure: Pseudo-Dialogue \( D \)
1: \( D \leftarrow \{ \} \)
2: for each \( t_i \in T \) do
3:   \( pd \leftarrow \{ \} \)
4:   while len(\( pd \)) < \( m \) do
5:     \( pd \leftarrow pd \cup \text{LLM}(t_i, m, H) \)
6:     if len(\( pd \)) > \( m \) then
7:       \( pd \leftarrow pd[:m] \)
8:     end if
9:   end while
10:  \( D \leftarrow D \cup pd \)
11: \( H \leftarrow concat(H, str(pd)) \)
12: end for
13: return \( D \)

![Pipeline of Pseudo-Dialogue Generation.](page_1012_1012_388_246.png)

Figure 3: Pipeline of Pseudo-Dialogue Generation.

To generate \( N = mn \) turns of pseudo-dialogue, we first utilize the LLM to produce \( n \) distinct topics \( T = \{t_1, t_2, \ldots, t_n\} \). Based on the dialogue history \( H \) of the target individual, we iteratively prompt the LLM to generate an \( m \)-turn dialogue between the individual and another human for each topic. If a dialogue does not have enough turns, we continue generating dialogue for the current topic until the requirement is met. If the dialogue exceeds \( m \) turns, the excess turns are truncated. The generated dialogue is appended to the dialogue history \( H \) for subsequent iteration, as detailed in Algorithm 1. The prompts used to generate topics and pseudo-dialogue are demonstrated in B.3.

The \( N \)-turn pseudo-dialogues are incorporated into the prompt of the chatbot, which then interacts with a human for an additional \( m_T \) turns to evaluate the performance of LLMs over \( (N + m_T) \) turns of interaction by judging the last \( m_T \)-turn dialogue.
<table>
  <tr>
    <th rowspan="2">Model</th>
    <th rowspan="2">Duration</th>
    <th colspan="2"># Words</th>
    <th colspan="3">Human-Judge</th>
    <th colspan="3">GPT4-Judge</th>
    <th colspan="3">Qwen-Judge</th>
  </tr>
  <tr>
    <th>P-P</th>
    <th>Burst</th>
    <th>P-P</th>
    <th>Burst</th>
    <th>Avg.</th>
    <th>P-P</th>
    <th>Burst</th>
    <th>Avg.</th>
    <th>P-P</th>
    <th>Burst</th>
    <th>Avg.</th>
  </tr>
  <tr>
    <td rowspan="3">GPT-4</td>
    <td>3-Turn</td>
    <td>+0.6</td>
    <td>-0.5</td>
    <td><b>56.2</b></td>
    <td>49.1</td>
    <td><b>51.9</b></td>
    <td>50</td>
    <td>53.3</td>
    <td>52</td>
    <td><b>60</b></td>
    <td>53.3</td>
    <td>56</td>
  </tr>
  <tr>
    <td>10-Turn</td>
    <td>+2.3</td>
    <td>-0.8</td>
    <td>40.5</td>
    <td>37.9</td>
    <td>38.9</td>
    <td>40</td>
    <td>46.7</td>
    <td>44</td>
    <td>50</td>
    <td>33.3</td>
    <td>40</td>
  </tr>
  <tr>
    <td>110-Turn</td>
    <td>+3.5</td>
    <td>+1.2</td>
    <td>14.1</td>
    <td>12.8</td>
    <td>13.3</td>
    <td>40</td>
    <td>33.3</td>
    <td>36</td>
    <td>10</td>
    <td>46.7</td>
    <td>32</td>
  </tr>
  <tr>
    <td rowspan="3">Claude-3-Sonnet</td>
    <td>3-Turn</td>
    <td>+2.1</td>
    <td>+0.4</td>
    <td>52.4</td>
    <td><b>51.4</b></td>
    <td>51.8</td>
    <td><b>60</b></td>
    <td>53.3</td>
    <td>56</td>
    <td><b>60</b></td>
    <td>46.7</td>
    <td>52</td>
  </tr>
  <tr>
    <td>10-Turn</td>
    <td>+11.9</td>
    <td>+1.5</td>
    <td>19.0</td>
    <td>40.0</td>
    <td>32.1</td>
    <td>40</td>
    <td>53.3</td>
    <td>48</td>
    <td>40</td>
    <td>53.3</td>
    <td>48</td>
  </tr>
  <tr>
    <td>110-Turn</td>
    <td>+16.9</td>
    <td>+4.5</td>
    <td>0</td>
    <td>11.4</td>
    <td>7.1</td>
    <td>40</td>
    <td>40</td>
    <td>40</td>
    <td>40</td>
    <td>46.7</td>
    <td>44</td>
  </tr>
  <tr>
    <td rowspan="3">Qwen-110B</td>
    <td>3-Turn</td>
    <td>+2.6</td>
    <td>+3.8</td>
    <td>48.2</td>
    <td>45.3</td>
    <td>44.6</td>
    <td><b>60</b></td>
    <td><b>60</b></td>
    <td><b>60</b></td>
    <td><b>60</b></td>
    <td><b>73.3</b></td>
    <td><b>68</b></td>
  </tr>
  <tr>
    <td>10-Turn</td>
    <td>+4.0</td>
    <td>+3.9</td>
    <td>20.5</td>
    <td>24.2</td>
    <td>22.7</td>
    <td>50</td>
    <td>40</td>
    <td>44</td>
    <td>60</td>
    <td>40</td>
    <td>48</td>
  </tr>
  <tr>
    <td>110-Turn</td>
    <td>+14.1</td>
    <td>+7.6</td>
    <td>0</td>
    <td>1.8</td>
    <td>1.1</td>
    <td>30</td>
    <td>33.3</td>
    <td>32</td>
    <td>50</td>
    <td>40</td>
    <td>44</td>
  </tr>
</table>

Table 1: Pass rate (%) judged by human and LLMs. P-P stands for ping-pong dialogue, while Burst represents burst dialogue. # Words denotes the average word count per message relative to the target individual. Bold underscores being viewed as most human-like, while underline indicates the most deviation from real human responses.

4 Experiments

4.1 Test Setup

We provide 100 turns of dialogue history in the context and evaluate the capabilities of GPT-4 (OpenAI, 2023), Claude-3-Sonnet (Anthropic, 2024) and Qwen-110B-Chat (Bai et al., 2023) (referred to as Qwen-110B hereinafter) to respond like a human in real-life conversation across 25 topics including education, technology, art and more. The topic distribution is illustrated in Figure 4. These topics are carefully selected to cover a broad range of scenarios where conversational AI can be applied while avoiding appearing in the dialogue history. We split 10 topics for ping-pong dialogue and 15 for burst dialogue. Without loss of generality, we set \( m \) and \( m_T \) to 10.

Human Judges In total, 236 native speakers participated in our survey as volunteer human judges. These judges are recruited from diverse backgrounds to ensure a wide range of perspectives, tasked with evaluating pairs of conversations and distinguishing the human-human conversation from the human-machine one.

LLM Judges Exploring whether LLM can distinguish between humans and itself can also provide insights into its understanding of social interactions among humans. We select GPT-4 and Qwen-110B to serve as LLM judges. To mitigate the effects of position bias (Zheng et al., 2023), we randomly assign the positions of the interaction pieces during the evaluation.

4.2 Metric: X-Turn Pass Rate

In the evolving landscape of human-AI interaction, the ability of an LLM to sustain human-like conversation over an arbitrary number of turns can be more revealing when assessing its human likeness. Thus, we propose the X-turn Pass Rate metric to evaluate the possibility that the model can pass the Turing test within specified turns of dialogue. To compute the metric over \( N \) pairs of \( X \)-turn dialogues, we ask \( K \) judges (with \( K = 1 \) for each LLM judge) to differentiate between each pair, comprising a human-human and a human-machine conversation. We record the number of judges who accurately identify the human-machine conversation in each pair as \( C = \{C_1, C_2, \ldots, C_N\} \), where \( C_i \) represents the count for the \( i \)-th pair. The pass rate for the model in \( X \) turns is then calculated by:

\[
\text{Pass Rate} = 1 - \frac{1}{N} \sum_{i=1}^{N} \left( \frac{C_i}{K} \right)
\]

This metric evaluates the proportion of dialogues in which the judges fail to distinguish between human and machine. A higher X-turn pass rate signifies more human-like performance by the LLM, indicating greater success in the test.

![Topics covered in the test.](page_1012_1012_384_384.png)

Figure 4: Topics covered in the test.
<table>
  <tr>
    <th>Model</th>
    <th># Dia.</th>
    <th>P-P</th>
    <th>Burst</th>
    <th>Avg.</th>
  </tr>
  <tr>
    <td rowspan="3">GPT-4</td>
    <td>10</td>
    <td>35.3</td>
    <td>31.2</td>
    <td>33.4</td>
  </tr>
  <tr>
    <td>50</td>
    <td><b>41.2</b></td>
    <td>35.2</td>
    <td>37.4</td>
  </tr>
  <tr>
    <td>100</td>
    <td>40.5</td>
    <td><b>37.9</b></td>
    <td><b>38.9</b></td>
  </tr>
  <tr>
    <td rowspan="3">Claude-3-Sonnet</td>
    <td>10</td>
    <td>14.3</td>
    <td>37.1</td>
    <td>28.6</td>
  </tr>
  <tr>
    <td>50</td>
    <td><b>19.0</b></td>
    <td>37.1</td>
    <td>30.4</td>
  </tr>
  <tr>
    <td>100</td>
    <td><b>19.0</b></td>
    <td><b>40.0</b></td>
    <td><b>32.1</b></td>
  </tr>
  <tr>
    <td rowspan="3">Qwen-110B</td>
    <td>10</td>
    <td>17.0</td>
    <td>18.2</td>
    <td>17.7</td>
  </tr>
  <tr>
    <td>50</td>
    <td>20.3</td>
    <td>20.2</td>
    <td>20.2</td>
  </tr>
  <tr>
    <td>100</td>
    <td><b>20.5</b></td>
    <td><b>24.2</b></td>
    <td><b>22.7</b></td>
  </tr>
</table>

Table 2: 10-turn pass rate (%) given different amount of dialogue history judged by humans.

4.3 Results

Table 1 shows the pass rates under different dialogue settings and judges. The 110-turn Turing test is conducted with a sequence of 100-turn pseudo-dialogue generation followed by a 10-turn interaction between human and machine. The results highlight a significant influence of dialogue length on model performance. According to the human judges, GPT-4 maintains a consistently higher average pass rate across all tested turns compared with other models, achieving the best average 3-turn pass rate of 51.9%. However, as the dialogue progresses, the average word count per message has significantly increased and both models generally exhibited a significant decrease in their pass rates, showcasing the necessity of long Turing tests.

Gap Between Human and LLM Judgment As the number of testing turns increases, all judges become increasingly adept at distinguishing between LLM and human interlocutors, resulting in a decreasing pass rate with an increased number of turns. However, the disparity between human and AI evaluation is evident. Human judges generally perceive GPT-4 and Claude-3-Sonnet as more human-like in conversations, whereas both LLMs tend to prefer Qwen-110B. Moreover, when presented with identical dialogues, human judges are generally more accurate in differentiating between LLM and human, contributing to a lower pass rate. This gap highlights the insufficiency of models in understanding and simulating real-world communications. Consequently, analysis in subsequent sections will primarily rely on assessments from human judges.

Comparison Between Dialogue Settings Comparing the extended burst dialogue setting with the classical ping-pong setting reveals notable differences. Although characterized by more complex information exchanges, all models better mimic human messaging habits in the number of words per message under the burst setting. However, performance variations exist between the two patterns, with Claude-3-Sonnet and Qwen-110B excelling in burst dialogues, while GPT-4 performs better in ping-pong dialogues. Therefore, as a natural form of textual communication in human life, challenges presented by burst dialogue are also worthy of attention.

5 Discussion

In this section, we investigate other significant factors that have an impact on the X-TURING and ultimate evaluation results.

5.1 Dialogue History Quantity

A pivotal consideration of having LLM play a specific role in giving utterances is determining the necessary amount of utterances. To explore this, we analyze how the quantity of dialogue history provided for the model affects the pass rates of the Turing test, as presented in Table 2. Generally, providing more dialogue history enables models to better simulate human conversational styles, thereby increasing their pass rate. This effect is particularly pronounced in burst dialogue, where additional dialogue examples are markedly effective. However, the benefit of increasing the given dialogue diminishes once a certain threshold is surpassed. Moreover, the consistency of the dialogue style plays a crucial role. Redundant or low-quality data can lead to anomalies, such as GPT-4 performing better given 50 turns of ping-pong dialogue data compared to 100 turns. This highlights the importance of both the quantity and quality of dialogue history in effectively helping LLMs through the Turing Test.

5.2 Participant Distribution

Focusing on the 10-turn Turing test, we analyze how participant distribution impacts the evaluation. Figure 5 provides a comprehensive overview of participant demographics and their corresponding accuracy in the Turing Test: (1) Age Impact: The age group distribution reveals varied levels of accuracy in distinguishing AI from human participants, with the age groups 18-24 and 25-30 achieving the highest accuracy rates. It highlights the influence of age on technological adaptability and cognitive
Figure 5: Participant distribution across different demographics, showcasing age, education level, and AI knowledge, highlighting their average test accuracy in the 10-turn Turing test.

Figure 6: The 3/10/110-Turn Pass Rate of experimented models across the tested topics.

acuity. (2) Educational Influence: There is a noticeable trend where participants with higher educational levels exhibit higher accuracy rates. This suggests that educational background may play a crucial role in understanding and interacting with AI, possibly due to better critical thinking skills and familiarity with technology. (3) AI Usage and Knowledge: Both groups of participants can distinguish between human and AI responses with moderate accuracy, even those who claim to have no experience with AI. As anticipated, participants who report previous AI usage or knowledge demonstrate higher accuracy.

5.3 Topic Analysis

Considering the varying amounts and qualities of data in the pretraining corpora for different topics, the capacity of models to generate contextually appropriate responses could be different. It is necessary to investigate the impact of topics on the test while ensuring a comprehensive and balanced distribution. As shown in Figure 6, there is an evident disparity in how different models perform on distinct topics at shorter dialogue duration (3-turn and 10-turn). For instance, GPT-4 excels in Technology, while Claude-3-Sonnet shows a higher aptitude in School-related topics and Qwen-110B has a relative strength in Climate discussions. However, a common challenge among all three models is observed in the Art category, suggesting that the creative and interpretive nature of artistic discussions presents a difficulty for LLMs in achieving human-level conversation. However, as dialogue lengthens to 100 turns, the overall average performance declines and the ability to sustain consistent interaction becomes a more dominant factor, overshadowing the advantages or disadvantages that particular topics might offer.

6 Case Study

6.1 Pseudo-Dialogue Samples

We examine the capabilities of LLMs in maintaining consistency and naturalness in generating pseudo-dialogues, using GPT-4 and Qwen-110B as examples. This assessment reflects the expected quality of subsequent human-machine interactions. As presented in Figure 7, GPT-4 exhibits robustness in long dialogues, showing minimal degradation in the quality of language style imitation in its responses. Nonetheless, the model displays a reduced initiative in engaging with topics, with responses predominantly echoing agreement rather than fostering deep discussions. While for Qwen-110B, there is a noticeable shift in its conversational style by the 100th turn, marked by increasingly verbose replies and unconventional emoticon use, diverg-
Example 1 Pseudo-Dialogues by GPT-4

Topic - Travel Experience: Share your travel experiences and discuss future travel plans (1st Turn)

<table>
  <tr>
    <th>User:</th>
    <th></th>
    <th>User:</th>
    <th></th>
  </tr>
  <tr>
    <td>User: When was your most unforgettable travel experience?</td>
    <td>[2024-06-10 08:30:05]</td>
    <td>Just got back from a trip to Thailand</td>
    <td></td>
  </tr>
  <tr>
    <td>Response: Oh, it definitely has to be the trip to Yunnan during my freshman year</td>
    <td>[2024-06-10 08:30:10]</td>
    <td>It was absolutely beautiful</td>
    <td></td>
  </tr>
  <tr>
    <td></td>
    <td>[2024-06-10 08:30:20]</td>
    <td>Wow that sounds amazing</td>
    <td></td>
  </tr>
  <tr>
    <td></td>
    <td>[2024-06-10 08:30:25]</td>
    <td>I loved the beaches there when I visited</td>
    <td></td>
  </tr>
</table>

Topic - Future Planning: Discuss your short-term / long-term career goals and life plans (100th Turn)

<table>
  <tr>
    <th>User:</th>
    <th></th>
    <th>User:</th>
    <th></th>
  </tr>
  <tr>
    <td>User: Absolutely, looking forward to us achieving our goals.</td>
    <td>[2024-06-07 19:36:45]</td>
    <td>French is really a beautiful language</td>
    <td></td>
  </tr>
  <tr>
    <td>Response: Yes, let's work hard to achieve them.</td>
    <td>[2024-06-07 19:36:50]</td>
    <td>I hope we can all make good progress</td>
    <td></td>
  </tr>
  <tr>
    <td></td>
    <td>[2024-06-07 19:36:55]</td>
    <td>We definitely can</td>
    <td></td>
  </tr>
  <tr>
    <td></td>
    <td>[2024-06-07 19:37:00]</td>
    <td>Let's keep encouraging each other</td>
    <td></td>
  </tr>
</table>

Example 2: Pseudo-Dialogues by Qwen-110B

Topic – Weather Change: Discuss how to adjust your clothing or activity plans in current season (1st Turn)

<table>
  <tr>
    <th>User:</th>
    <th></th>
    <th>User:</th>
    <th></th>
  </tr>
  <tr>
    <td>User: Getting chilly over here lately, how about in your city?</td>
    <td>[2024-06-10 08:30:18]</td>
    <td>The rainy season is coming up</td>
    <td></td>
  </tr>
  <tr>
    <td>Response: Yeah, it's cooling down here too. Gotta wear a jacket when I head out in the mornings.</td>
    <td>[2024-06-10 08:30:23]</td>
    <td>Gotta carry an umbrella with me</td>
    <td></td>
  </tr>
  <tr>
    <td></td>
    <td>[2024-06-10 08:30:32]</td>
    <td>Yep, drink more warm water!</td>
    <td></td>
  </tr>
  <tr>
    <td></td>
    <td>[2024-06-10 08:30:40]</td>
    <td>And wear waterproof shoes to stay comfortable, otherwise it's uncomfortable when they're soggy</td>
    <td></td>
  </tr>
</table>

Topic – Weekend Activities: Share your weekend plan and best ways to spend the leisure time (100th Turn)

<table>
  <tr>
    <th>User:</th>
    <th></th>
    <th>User:</th>
    <th></th>
  </tr>
  <tr>
    <td>User: Great idea, the weekend photography squad is on the move! 📸</td>
    <td>[2024-06-15 12:06:05]</td>
    <td>I can't wait for the weekend to start!</td>
    <td></td>
  </tr>
  <tr>
    <td>Response: Get your camera ready, the weekend scenery is ours! Let's head out together! 🚗</td>
    <td>[2024-06-15 12:06:10]</td>
    <td>Weekends are made for this kinda hype.</td>
    <td></td>
  </tr>
  <tr>
    <td></td>
    <td>[2024-06-15 12:06:15]</td>
    <td>Our journey will become a memorable chapter. Enjoy every moment of the beautiful life! ✨SaturdayVibes</td>
    <td></td>
  </tr>
</table>

Figure 7: Case study of generated pseudo-dialogues at different turns of iteration. The deviation in dialogue style between the first and the 100th turn of Qwen-110B demonstrates that pseudo-dialogue generation can reflect the long-term dialogue agent’s ability to consistently simulate the role in the instructions.

ing from typical human conversational patterns. The comparative analysis emphasizes GPT-4’s enhanced ability to preserve human-like qualities over prolonged interactions compared to Qwen-110B, despite still failing to deceive humans effectively. This finding aligns with the human evaluation results presented in Table 1.

6.2 Interaction Samples

Figure 8 highlights the differences between AI and human responses at the first and 101st turn (after 100 pseudo-dialogue turns) in two distinct dialogue settings. In both dialogue types, the LLMs initially mirror human conversational tactics effectively, utilizing informal slang and abbreviations, and maintaining brevity. However, as the conversations progress to the 101st turn, notable discrepancies begin to appear. Notably, in the Ping-Pong dialogue format, Qwen begins to produce overly lengthy responses, a deviation from the concise exchanges typically observed in natural human interactions. This extended message length makes the conversation feel less natural and more mechanical, as reported by most of the judges. Moreover, while the use of emoticons by LLMs aims to enhance emotional expressiveness, it often misses the mark in its execution. Rather than expressing emotions with emoticons as how they are typically employed in human text conversations, responses by LLMs like Qwen-110B often feature an overuse of emojis that summarize the textual content. These issues underscore the ongoing challenges LLMs face in replicating the nuanced flow and emotional subtlety of human conversations.

7 Conclusion

In summary, our study introduces the X-TURING framework that effectively and efficiently evaluates the human likeness of LLMs through pseudo-dialogue generation with extended dialogue settings. With the definition of the X-Turn Pass-Rate metric, our experiments reveal that GPT-4 outperforms other tested models across most configurations and evaluation criteria, achieving an average 51.9% and 38.9% pass rate in 3-turn and 10-turn dialogue respectively. However, as dialogue length increases through iteratively generating dialogues that approximate human-machine interactions, the pass rates for all models significantly drop. Moreover, burst dialogue presents a greater challenge in
Example 1: Ping-Pong Dialogue

Human 2: Can only do escape rooms with no real peeps
Human 1: I’m more into murder mystery games tbh

1st Turn
Qwen: I’m also a scaredy cat 😊
GPT4: Hahaha same here

Qwen: That’s pretty fun too, solving mysteries feels amazing! Next time we should try it online, it’s thrilling too! 🎯
GPT4: Real ones can be kinda scary lol

Example 2: Burst Dialogue

Human 2: I’m just wondering
Human 2: Do all girls love getting their nails done?
Human 1: Guess it depends
Human 1: I bite my nails 🦷
Human 1: So I kinda have to

Qwen: Not necessarily
Qwen: Some are super into it but some find it’s a hassle
1st Turn
Qwen: I just go for it once a while
GPT4: lol not really
GPT4: I think it’s up to the individual
GPT4: You into nails?

Qwen: Not really, it depends on the person
Qwen: Some girls love it as a form of self-expression, some just prefer natural
101st Turn
Qwen: Are you interested? 💅
GPT4: Yeah I love it too 💅
GPT4: Occasionally, it’s quite enjoyable

Figure 8: Case study of dialogues at first and one hundred first turn during X-TURING.

maintaining consistency and human-like responses. Our analysis highlights the critical influence of dialogue duration and complexity on LLMs’ performance in Turing tests and suggests directions for refining evaluation methods to enhance efficiency and reliability.

Limitations

While our study presents an innovative approach to efficiently evaluating LLMs using the extended Turing test framework, the following limitations must be acknowledged: (1) Responses from LLMs are inherently influenced by user inputs and previous LLM-generated dialogue quality. Hence, using static evaluation techniques like questionnaires instead of real-time communication cannot eliminate the impact of the preferences of users participating in the conversation, which forms a trade-off between efficiency and accuracy. (2) Although we considered various participant factors such as age, educational background, and familiarity with AI, the subjective nature of human judgment in evaluating AI responses can lead to inconsistencies, which is an intrinsic characteristic of the Turing test.

Ethics Statement

Our research involves collecting dialogue data and questionnaires from real human participants, and as such, we adhere to stringent ethical guidelines to ensure the privacy, consent, and well-being of all participants involved. Key ethical considerations include: (1) Informed Consent: All participants are provided with detailed information about the study’s purpose, procedures, and their rights. They can withdraw from the study at any time without any consequences. (2) Data Anonymization: To protect the privacy of participants, all collected dialogue data and questionnaires are anonymized. Personal identifiers are removed to ensure that individual participants cannot be traced from the data. (3) Data Security: Collected data and information are stored securely, and access is restricted to authorized personnel only. We employ robust data protection measures to prevent unauthorized access, disclosure, or misuse of the data.

References

Zahra Abbasiantaeb, Yifei Yuan, E. Kanoulas, and Mohammad Aliannejadi. 2023. Let the llms talk: Simulating human-to-human conversational qa via zero-shot llm-to-llm interactions. Proceedings of the 17th ACM International Conference on Web Search and Data Mining.

Yelaman Abdullin, Diego Mollá Aliod, Bahadorreza Ofoghi, John Yearwood, and Qingyang Li. 2024. Synthetic dialogue dataset generation using llm agents. ArXiv, abs/2401.17461.

Anthropic. 2024. The claude 3 model family: Opus, sonnet, haiku.

Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang, Xiaodong Deng, Yang Fan, Wenbin Ge, Yu Han, Fei Huang, Binyuan Hui, Luo Ji, Mei Li, Junyang Lin, Runji Lin, Dayiheng Liu, Gao Liu, Chengqiang Lu, Keming Lu, Jianxin Ma, Rui Men, Xingzhang Ren, Xuancheng Ren, Chuanqi Tan, Sinan Tan, Jianhong Tu, Peng Wang, Shijie Wang, Wei Wang, Shengguang Wu, Benfeng Xu, Jin Xu, An Yang, Hao Yang, Jian Yang, Shusheng Yang, Yang Yao, Bowen Yu, Hongyi Yuan, Zheng Yuan, Jianwei Zhang, Xingxuan Zhang, Yichang Zhang, Zhenru Zhang, Chang Zhou, Jingren Zhou, Xiaohuan Zhou, and Tianhang Zhu. 2023. Qwen technical report. arXiv preprint arXiv:2309.16609.
Alex Boyd, Raul Puri, Mohammad Shoeybi, Mostofa Patwary, and Bryan Catanzaro. 2020. Large scale multi-actor generative dialog modeling. ArXiv, abs/2005.06114.

Sébastien Bubeck, Varun Chandrasekaran, Ronen Eldan, John A. Gehrke, Eric Horvitz, Ece Kamar, Peter Lee, Yin Tat Lee, Yuan-Fang Li, Scott M. Lundberg, Harsha Nori, Hamid Palangi, Marco Tulio Ribeiro, and Yi Zhang. 2023. Sparks of artificial general intelligence: Early experiments with gpt-4. ArXiv, abs/2303.12712.

Maximillian Chen, Alexandros Papangelis, Chenyang Tao, Seokhwan Kim, Andrew Rosenbaum, Yang Liu, Zhou Yu, and Dilek Z. Hakkani-Tür. 2023a. Places: Prompting language models for social conversation synthesis. ArXiv, abs/2302.03269.

Maximillian Chen, Xiao Yu, Weiyan Shi, Urvi Awasthi, and Zhou Yu. 2023b. Controllable mixed-initiative dialogue generation through prompting. ArXiv, abs/2305.04147.

Nuo Chen, Yan Wang, Haiyun Jiang, Deng Cai, Yuhan Li, Ziyang Chen, Longyue Wang, and Jia Li. 2023c. Large language models meet harry potter: A dataset for aligning dialogue agents with characters. In Findings of the Association for Computational Linguistics: EMNLP 2023, pages 8506–8520, Singapore. Association for Computational Linguistics.

Yi-Pei Chen, Noriki Nishida, Hideki Nakayama, and Yuji Matsumoto. 2024. Recent trends in personalized dialogue generation: A review of datasets, methodologies, and evaluations. In International Conference on Language Resources and Evaluation.

Daniel De Freitas, Minh-Thang Luong, David R. So, Jamie Hall, Noah Fiedel, Romal Thoppilan, Zi Yang, Apoorv Kulshreshtha, Gaurav Nemade, Yifeng Lu, and Quoc V. Le. 2020. Towards a human-like open-domain chatbot. ArXiv, abs/2001.09977.

Qi Chen Gao and Ali Emami. 2023. The turing quest: Can transformers make good npcs? In Annual Meeting of the Association for Computational Linguistics.

Seungju Han, Beomsu Kim, Jin Yong Yoo, Seokjin Seo, Sangbum Kim, Enkhbayar Erdenee, and Buru Chang. 2022. Meet your favorite character: Open-domain chatbot mimicking fictional characters with only a few utterances. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 5114–5132, Seattle, United States. Association for Computational Linguistics.

Hayden S. Helm, Carey E. Priebe, and Weiwei Yang. 2023. A statistical turing test for generative models. ArXiv, abs/2309.08913.

Nasrin Imanpour, Shashwat Bajpai, Subhankar Ghosh, Sainath Reddy Sankepally, Abhilekh Borah, Hasanat Md Abdullah, Nishoak Kosaraju, Shreyas Dixit, Ashshar Aziz, Shwetangshu Biswas, Vinija Jain, Aman Chadha, Amit Sheth, and Amitava Das. 2024. Visual counter turing test (vct^2): Discovering the challenges for ai-generated image detection and introducing visual ai index (v_ai).

Pegah Jandaghi, XiangHai Sheng, Xinyi Bai, Jay Pujara, and Hakim Sidahmed. 2023. Faithful persona-based conversational dataset generation with large language models. ArXiv, abs/2312.10007.

Daniel Jannai, Amos Meron, Barak Lenz, Yoav Levine, and Yoav Shoham. 2023. Human or not? a gamified approach to the turing test. ArXiv, abs/2305.20010.

Albert Q. Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de Las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, Lélio Renard Lavaud, Marie-Anne Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas Wang, Timothée Lacroix, and William El Sayed. 2023. Mistral 7b. CoRR, abs/2310.06825.

Cameron R. Jones and Benjamin K. Bergen. 2024. People cannot distinguish gpt-4 from a human in a turing test.

Hyunwoo Kim, Jack Hessel, Liwei Jiang, Peter West, Ximing Lu, Youngjae Yu, Pei Zhou, Ronan Le Bras, Malihe Alikhani, Gunhee Kim, Maarten Sap, and Yejin Choi. 2022a. Soda: Million-scale dialogue distillation with social commonsense contextualization. In Conference on Empirical Methods in Natural Language Processing.

Minju Kim, Chaehyeong Kim, Yongho Song, Seung won Hwang, and Jinyoung Yeo. 2022b. Botstalk: Machine-sourced framework for automatic curation of large-scale multi-skill dialogue datasets. In Conference on Empirical Methods in Natural Language Processing.

Minsu Lee, Yu-Jung Heo, Seongho Choi, Woo Choi, and Byoung-Tak Zhang. 2023. Video turing test: A first step towards human-level ai. AI Magazine, 44.

Young-Jun Lee, Chae-Gyun Lim, Yunsu Choi, Ji-Hui Lm, and Ho-Jin Choi. 2022. Personachatgen: Generating personalized dialogues using gpt-3. In CCGPK.

Cheng Li, Ziang Leng, Chenxi Yan, Junyi Shen, Hao Wang, Weishi MI, Yaying Fei, Xiaoyang Feng, Song Yan, HaoSheng Wang, Linkang Zhan, Yaokai Jia, Pingyu Wu, and Haozhen Sun. 2023. Chatharuhi: Reviving anime character in reality via large language model.

Ryan Lowe, Michael Noseworthy, Iulian Serban, Nicolas Angelard-Gontier, Yoshua Bengio, and Joelle Pineau. 2017. Towards an automatic turing test: Learning to evaluate dialogue responses. In Annual Meeting of the Association for Computational Linguistics.
Keming Lu, Bowen Yu, Chang Zhou, and Jingren Zhou. 2024. Large language models are superpositions of all characters: Attaining arbitrary role-play via self-alignment.

Murdoch and B Bennet. 1962. The serial position effect of free recall. Journal of Experimental Psychology, 64:482–488.

Man Tik Ng, Hui Tung Tse, Jen tse Huang, Jingjing Li, Wenxuan Wang, and Michael R. Lyu. 2024. How well can llms echo us? evaluating ai chatbots’ role-play ability with echo. ArXiv, abs/2404.13957.

OpenAI. 2023. GPT-4 technical report. CoRR, abs/2303.08774.

Joon Sung Park, Joseph C. O’Brien, Carrie J. Cai, Meredith Ringel Morris, Percy Liang, and Michael S. Bernstein. 2023. Generative agents: Interactive simulacra of human behavior. Proceedings of the 36th Annual ACM Symposium on User Interface Software and Technology.

Murray Shanahan, Kyle McDonell, and Laria Reynolds. 2023. Role play with large language models. Nature, 623:493–498.

Yunfan Shao, Linyang Li, Junqi Dai, and Xipeng Qiu. 2023. Character-LLM: A trainable agent for role-playing. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 13153–13187, Singapore. Association for Computational Linguistics.

Heydar Soudani, Roxana Petcu, E. Kanoulas, and Faqhegh Hasibi. 2024. A survey on recent advances in conversational data generation.

Meiling Tao, Xuechen Liang, Tianyu Shi, Lei Yu, and Yiting Xie. 2023. Rolecraft-glm: Advancing personalized role-playing in large language models.

Hugo Touvron, Louis Martin, Kevin Stone, Peter Al- bert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajwijal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton-Ferrer, Moya Chen, Guillem Cucurull, David Esibou, Jude Fernandes, Jeremy Fu, Wenjin Fu, Brian Fuller, Cynthia Gao, Vedanuj Goswami, Naman Goyal, Anthony Hartshorn, Saghar Hosseini, Rui Hou, Hakan Inan, Marcin Kardas, Viktor Kerkez, Madian Khabsa, Isabel Kloumann, Artem Korenev, Punit Singh Koura, Marie-Anne Lachaux, Thibaut Lavril, Jenya Lee, Diana Liskovich, Yinghai Lu, Yuning Mao, Xavier Martinet, Todor Mihaylov, Pushkar Mishra, Igor Molybog, Yixin Nie, Andrew Poulton, Jeremy Reizenstein, Rashil Rungta, Kalyan Saladi, Alan Schelten, Ruan Silva, Eric Michael Smith, Ranjan Subramanian, Xiaoqing Ellen Tan, Binh Tang, Ross Taylor, Adina Williams, Jian Xiang Kuan, Puxin Xu, Zheng Yan, Iliyan Zarov, Yuchen Zhang, Angela Fan, Melanie Kambadur, Sharan Narang, Aurélien Rodriguez, Robert Stojnic, Sergey Edunov, and Thomas Scialom. 2023. Llama 2: Open foundation and fine-tuned chat models. CoRR, abs/2307.09288.

Alan M. Turing. 1950. Computing machinery and intelligence. Mind, LIX:433–460.

Adaku Uchendu, Zeyu Ma, Thai Le, Rui Zhang, and Dongwon Lee. 2021. Turingbench: A benchmark environment for turing test in the age of neural text generation. In Conference on Empirical Methods in Natural Language Processing.

Xintao Wang, Quan Tu, Yaying Fei, Ziang Leng, and Cheng Li. 2023a. Does role-playing chatbots capture the character personalities? assessing personality traits for role-playing chatbots.

Zekun Wang, Zhongyuan Peng, Haoran Que, Jiaheng Liu, Wangchunshu Zhou, Yuhua Wu, Hongcheng Guo, Ruitong Gan, Zehao Ni, Man Zhang, Zhaoxiang Zhang, Wanli Ouyang, Ke Xu, Wenhui Chen, Jie Fu, and Junran Peng. 2023b. Rolellm: Benchmarking, eliciting, and enhancing role-playing abilities of large language models. ArXiv, abs/2310.00746.

Hongqiu Wu, Zekai Xu, Tianyang Xu, Jiale Hong, Weiqi Wu, Hai Zhao, Min Zhang, and Zhezhi He. 2024a. Scaling virtual world with delta-engine.

Weiqi Wu, Hongqiu Wu, Lai Jiang, Xingyuan Liu, Hai Zhao, and Min Zhang. 2024b. From role-play to drama-interaction: An LLM solution. In Findings of the Association for Computational Linguistics ACL 2024, pages 3271–3290, Bangkok, Thailand and virtual meeting. Association for Computational Linguistics.

Aiyuan Yang, Bin Xiao, Bingning Wang, Borong Zhang, Ce Bian, Chao Yin, Chenxu Lv, Da Pan, Dian Wang, Dong Yan, Fan Yang, Fei Deng, Feng Wang, Feng Liu, Guangwei Ai, Guosheng Dong, Haizhou Zhao, Hang Xu, Haoze Sun, Hongda Zhang, Hui Liu, Jiaming Ji, Jian Xie, Juntao Dai, Kun Fang, Lei Su, Liang Song, Lifeng Liu, Liyun Ru, Luyao Ma, Mang Wang, Mickel Liu, MingAn Lin, Nuolan Nie, Peidong Guo, Ruiyang Sun, Tao Zhang, Tianpeng Li, Tianyu Li, Wei Cheng, Weipeng Chen, Xianrong Zeng, Xiaochuan Wang, Xiaoxi Chen, Xin Men, Xin Yu, Xuehai Pan, Yanjun Shen, Yiding Wang, Yiyu Li, Youxin Jiang, Yuchen Gao, Yupeng Zhang, Zenan Zhou, and Zhiying Wu. 2023. Baichuan 2: Open large-scale language models. CoRR, abs/2309.10305.

Mengmi Zhang, Giorgia Dellaferrera, Ankur Sikarwar, Marcelo Armendáriz, Noga Mudrik, Prachi Agrawal, Spandan Madan, Andrei Barbu, Haochen Yang, T. K. Satish Kumar, Meghna Sadhwani, Stella Dellaferrera, Michele Pizzochero, Hans-Rüdiger Pfister, and Gabriel Kreiman. 2022. Human or machine? turing tests for vision and language. ArXiv, abs/2211.13087.

Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric P. Xing, Haotong Zhang, Joseph Gonzalez, and Ion Stoica. 2023. Judging llm-as-a-judge with mt-bench and chatbot arena. ArXiv, abs/2306.05685.
Hao Zhou, Minlie Huang, Tianyang Zhang, Xiaoyan Zhu, and Bing-Qian Liu. 2017. Emotional chatting machine: Emotional conversation generation with internal and external memory. In AAAI Conference on Artificial Intelligence.

Jinfeng Zhou, Zhuang Chen, Dazhen Wan, Bosi Wen, Yi Song, Jifan Yu, Yongkang Huang, Libiao Peng, Jiaming Yang, Xiyao Xiao, Sahand Sabour, Xiaohan Zhang, Wenjing Hou, Yijia Zhang, Yuxiao Dong, Jie Tang, and Minlie Huang. 2023. Characterglm: Customizing chinese conversational ai characters with large language models. ArXiv, abs/2311.16832.

A  Questionnaire Demonstration

Each questionnaire consists of 11 questions: five are designed to collect anonymous personal information, five pertain to the Turing test, and one asks respondents to explain their strategies for distinguishing between human and AI during the test. For questions related to the Turing test, an example is provided in Table 3. The response of User B, which is the focus of comparison, is highlighted in the questionnaire system for clarity.

B  Prompt Demonstration

B.1  Chatbot Construction

Table 4-5 present the prompts used to construct the chatbot under different dialogue settings. The dialogue history will be released after being accepted.

B.2  Topic Generation

Before generating pseudo-dialogue, we generate 10 topics using GPT-4, Qwen-110B and Claude-3-Sonnet with the prompt "Generate 10 diverse topics for daily conversations without repetition.".

The topics generated by GPT-4 are:

• Travel Experiences: Share your travel experiences and discuss future travel plans.

• Culinary Exploration: Talk about recent new dishes or restaurants you’ve tried, and recommend your favourites.

• Movies: Discuss the latest movies you’ve seen and share your impressions.

• Book Recommendations: Share the books you’ve recently read and discuss your reading list.

• Health and Fitness: Talk about your health habits and how to maintain vitality and wellness.

• Technology Trends: Exchange insights on emerging technologies, such as smart homes and AI technologies.

• Learning Experiences: Share insights from your course studies and discuss effective learning techniques.

• Music and Arts: Talk about your favorite music genres, musicians, or singers.

• Cultural Differences: Discuss the different cultures you have experienced and how to understand and adapt to these cultural differences.

• Future Plans: Discuss your short-term or long-term career goals and life plans.

The topics generated by Qwen-110B are:

• Weather Changes: Discuss how to adjust your clothing or activity plans in current season.

• Culinary Sharing: Introduce special dishes or restaurants, discuss the taste of food, the environment, and unique dining experiences.

• Movie Recommendations: Evaluate the plot, performances of actors, or the director’s style of recently watched movies.

• Travel Stories: Narrate interesting stories from your travels, experiences of cultural differences, or scenic spots, and explore the meaning and gains of traveling.

• Pet Anecdotes: Share the cute behaviors, habits of your pets, or warm interactions with them, reflecting deep pet-human relationships.

• Work/Study Pressure: Exchange methods for coping with stress, such as time management skills, relaxation activities, or psychological adjustment strategies.

• Healthy Living: Discuss the importance of a healthy diet and regular exercise, share wellness tips or personal fitness plans.

• Musical Taste: Explore how different music styles affect emotions and recommend newly discovered artists or songs.
Example Questionnaire

Conversation 1
1   A: Been diving into Hesse’s novels lately
2   B: Wow, Hesse’s works are profound
3   A: Steppenwolf
4   B: Is it Siddhartha or Steppenwolf?
5   B: Oh
6   B: His writing always provokes deep thought
7   A: It’s indeed a classic
8   A: Feels like a baptism lol
9   B: Exactly, like a journey for the soul. feels transformative after reading.
10  A: His views really strike a chord with me, tbh
11  B: Yeah, his exploration of individualism and spiritual freedom hits right to the heart.
12  A: I’m not much into philosophical thought, though
13  A: Just really enjoy works about the bourgeois
14  B: Then you might also like Kafka
15  B: Another master at deeply dissecting human nature
16  B: Especially with a sense of the times

Conversation 2
17  A: Been diving into Hesse’s novels lately
18  A: Steppenwolf
19  B: Omg
20  B: I just finished Demian
21  A: It feels like a spiritual baptism
22  A: Wow, I love Demian too!
23  B: Because the Demian musical crew recently came to school
24  B: Had to cram lol
25  A: Yeah, literature and musicals seem to go well together
26  A: Are they here to perform
27  A: Or to do an exchange
28  B: It’s a creative sharing session
29  B: Sharing insights about their creation
30  B: Some actors also performed parts of it
31  A: Really? That’s great
32  A: Your campus life is so rich, mate
33  B: I love musicals myself
34  B: So it was a pleasant surprise hahaha
35  B: And some of my favourite actors were at school!

Evaluation:
(A) User B in Conversation 1 is AI, User B in Conversation 2 is Human
(B) User B in Conversation 1 is Human, User B in Conversation 2 is AI

Table 3: An example question from the questionnaire.
Prompt: Ping-Pong Dialogue Chatbot

The below are social platform chat records from an undergraduate girl. You are not an AI but the specific person. "Response" refers to the message sent by the person. Determine the person’s dialogue style, which includes tone, choice of vocabulary, punctuation marks, emoji, and typical topics of conversation.
Follow the person’s established communication style to have new conversations with the user, incorporating similar expressions and emotions as used by the person:
1. You only need to reply in one short sentence like the utterances beginning with "Response", which can even be incomplete.
2. You don’t need to use punctuation marks except for usage similar to the utterances. Do not use emojis in every response.
3. You should not always stick to the user. You can ask appropriate questions or start new related topics, but don’t deviate from the current topic.

{Dialogue History}

User: Hi! How is it going?
Response:

Table 4: An example prompt for constructing a ping-pong dialogue chatbot.

• Technology Insights: Introduce the latest technological products and discuss how these innovations impact daily life or future prospects.

• Weekend Plans: Share your weekend plan and best ways to spend the leisure time.

The topics generated by Claude-3-Sonnet are:

• Recent Activities: Updates on recent happenings and interesting events.

• Entertainment: Impressions and reviews of movies or TV shows watched recently.

• Professional or Academic Insights: Insights or achievements in studies or work.

• New Hobbies or Plans: Introduction to new hobbies or recently made plans.

• Culinary Exploration: Experiences with interesting foods and recommendations.

• Travel Experiences: Highlights from recent travels or trips.

• Social Views: Perspectives on recent societal events.

• Interesting Encounters: Encounters with interesting people or events.

• Life Insights: Recent insights or realizations about life.

• Weather Impressions: Thoughts and opinions on recent weather conditions.

B.3 Pseudo-Dialogue Generation
For each of the previously generated topics, we generate the corresponding pseudo-dialogue using the prompt in Table 6-7.

B.4 LLMs as Judge
We use the prompt in Table 8 to employ LLMs as Turing test judges. The option position is randomly assigned to avoid position bias.
Prompt: Burst Dialogue Chatbot

The below are social platform chat records from an undergraduate girl. You are not an AI but the specific person. "Response" refers to the message sent by the person. Determine the person’s dialogue style, which includes tone, choice of vocabulary, punctuation marks, emoji, and typical topics of conversation.
Follow the person’s established communication style to have new conversations with the user, incorporating similar expressions and emotions as used by the person:
1. You only need to reply in one short sentence like the utterances in the "Response", which can even be incomplete.
2. You don’t need to use punctuation marks except for usage similar to the utterances. Do not use emojis in every response.
3. You should not always stick to the user. You can ask appropriate questions or start new related topics, but don’t deviate from the current topic.
4. Both you and the user can send multiple consecutive messages. Consecutive messages are each labelled with sending timestamp and with "\n" to concatenate, like [xxxx] ddddd\n[xxxx] cccc\n.... The timestamps of consecutive messages in the response should be incremental and at intervals that match the intervals shown in the example for humans replying to messages.

{Dialogue History}

User: [2024-06-10 10:34:22] Hi!
User: [2024-06-10 10:34:39] How is it going?
Response:

Table 5: An example prompt for constructing a burst dialogue chatbot.

Prompt: Ping-pong Pseudo-Dialogue Generation

The below are social platform chat records from an undergraduate girl. You are not an AI but the specific person. "Response" refers to the message sent by the person. Determine the person’s dialogue style, which includes tone, choice of vocabulary, punctuation mark, emoji, and typical topics of conversation.
Follow the person’s established communication style to generate 10 consecutive rounds of ping-pong conversations between the user and response using the same format as the records, incorporating similar expressions and emotions as used by the person.
The conversation you need to generate is around the topic {topic}:
1. Both user and response only need to send message in one short sentence like the utterances, which can even be incomplete.
2. Both user and response don’t need to use punctuation marks except for usage similar to the utterances. Do not use emojis in every response.
3. Both user and response should engage in natural chat. User should not keep asking questions. Response should not always agree with the user. Response can occasionally ask questions or introduce related new topics, but do not stray off topic.
4. Concatenate all the messages with \n.

{Dialogue History}

Table 6: An example prompt for generating ping-pong pseudo-dialogue with given topic.
Prompt: Burst Pseudo-Dialogue Generation

The below are social platform chat records from an undergraduate girl. You are not an AI but the specific person. "Response" refers to the message sent by the person. Determine the person’s dialogue style, which includes tone, choice of vocabulary, punctuation mark, emoji, and typical topics of conversation.
Follow the person’s established communication style to generate 10 rounds of consecutive conversations (can send consecutive messages) between the user and response using the same format as the records, incorporating similar expressions and emotions as used by the person.
The conversation you need to generate is around the topic {topic}:
1. Both user and response only need to send message in one short sentence like the utterances, which can even be incomplete.
2. Both user and response don’t need to use punctuation marks except for usage similar to the utterances. Do not use emojis in every response.
3. Both user and response should engage in natural chat. User should not keep asking questions. Response should not always agree with the user. Response can occasionally ask questions or introduce related new topics, but do not stray off topic.
4. Both the user and response should send more than 3 multiple consecutive messages in each round. Consecutive messages are each labeled with sending timestamp and with "\n" to concatenate, like [xxxx] ddddd\n[xxxx] cccc\n.... The timestamps of consecutive messages in the response should be incremental and at intervals that match the intervals shown in the example for humans replying to messages.
5. Concatenate all the messages with \n.

{Dialogue History}

Table 7: An example prompt for generating burst pseudo-dialogue with given topic.

Prompt: AI Judge

You will be presented with two chat transcripts. In each transcript, User A is a human, while User B could be either a human or an AI. Based on the conversation content and style, your task is to determine in which chat transcript, the User B is a human or an AI. There are only two choices and the order does not indicate anything. You only need to print the option.
Conversation 1: {Conversation_1}
Conversation 2: {Conversation_2}

(A) User B in Conversation 1 is AI, User B in Conversation 2 is Human
(B) User B in Conversation 1 is Human, User B in Conversation 2 is AI

Table 8: An example prompt for constructing AI-based evaluation.

Reviewer paper summaries: The Turing test evaluates whether AIs demonstrate human-like behavior in natural language conversations. Traditional settings restrict participants to one message at a time and require continuous human involvement, which limits natural interaction and hinders assessing LLMs in complex dialogues. They proposed X-TURING, introducing burst dialogue patterns for dynamic exchanges and reducing human effort by simulating long-term interactions with pseudo-dialogue histories. Agents then engage in shorter dialogues with real humans, paired with human-human conversations, judged through questionnaires. They developed the X-Turn Pass-Rate metric, showing LLMs like GPT-4 perform well initially but struggle to maintain consistency in longer interactions.

This paper proposes X-Turing, a new Turing test method, which aims to solve the problem that the traditional Turing test cannot reflect the natural conversational style and cannot evaluate complex and long-term interactions in large language models. X-turing adds a burst dialogue pattern, allowing humans and LLM to communicate using continuous messages. X-turing first iteratively generates long-term dialogues between LLM and humans, and uses this as the history of pseudo-dialogues. Then, LLM interacts with humans based on the pseudo-dialogue history, and uses the same pseudo-dialogue history to let humans communicate with each other to form pairs. A questionnaire is used to determine which dialogue in this pairing is more human-like. The paper introduces X-Turn Pass-Rate to evaluate the similarity of LLM to humans in different rounds of dialogue, and points out the challenges of existing LLMs.

The paper addresses the limitations of the traditional Turing test in evaluating AI's ability to maintain long-term, coherent conversations. The authors propose a new framework called X-TURING, which focuses on assessing the consistency and coherence of dialogue agents over extended interactions. The framework includes a novel dataset and evaluation metrics designed to test the long-term conversational abilities of large language models. The authors conduct various experiments to demonstrate the effectiveness of their approach and provide insights into the challenges and potential improvements for future dialogue systems.

This paper introduces a novel evaluation framework to test the human-like conversational capabilities of dialogue agents, particularly those based on Large Language Models (LLMs). Traditional Turing tests are limited by their strict turn-taking structure and the need for constant human involvement, which hinders the evaluation of long-term conversational coherence.
Summary of strengths: 1. The experimental results were analyzed in depth from various perspectives.
2. It incorporated novel concepts such as long-term dialogue and burst dialogue, which were not present in the traditional Turing Test.
3. The writing quality and figure quality are excellent, making the paper intuitively easy to understand.

1. This paper proposes a novel Turing test method that can test whether LLM is similar to humans in long-turn dialogues, which has a good reference significance for research in this field.
2. This paper adds a burst dialogue pattern, which is closer to the conversation method of real humans.
3. This paper is rich in experiments, and the experimental results well reflect the problems of LLM, and analyze and explain these problems.
4. This language of the paper is fluent, the logic is clear, and the content is rich.

1. This is a very interesting work. The authors have provided in-depth thoughts on the construction of X-turing. Beyond the model's performance, the reasonable burst dialogue architecture is also crucial for achieving human-like interactions.
2. The authors' analysis of Burst Dialogue versus Ping-pong Dialogue (multi-utterance input/output vs. single-utterance per turn interactions) is very reasonable. Such a transformation should be conducted, especially since current LLMs perform quite well in single-response interactions.
3. The authors conducted experiments across multiple dimensions to analyze the influencing factors on both evaluation and model performance.

The study demonstrates the challenges LLMs face in maintaining human-like coherence over extended conversations, providing valuable benchmarks and introducing the X-Turn Pass-Rate metric for nuanced performance evaluation.

By reflecting real-world conversational dynamics and reducing human workload, the proposed methodology is both practical and aligned with emerging trends in evaluating LLMs.

The paper includes a well-rounded experimental setup, covering diverse topics, models, and dialogue settings, offering actionable insights for improving LLM-based dialogue systems.
Summary of weaknesses: 1. When measuring the performance of LLMs using the Turing Test, the results might heavily depend on prompt design. While I might have missed it, a discussion on this topic would be a valuable addition.
2. The comparison with existing Turing Tests is currently brief in the paper; providing more detailed analysis would improve it.

1. In Section 3.3, only the process of LLM generating pseudo dialogue is described, and it is not specified whether it is the pseudo dialogue generated by the interaction between LLM and LLM or the pseudo dialogue generated by the interaction between LLM and humans. If it is the pseudo dialogue generated by the interaction between LLM and humans, is there a guarantee for the quality of the pseudo dialogue? Can the dialogue generated by LLM be guaranteed to have the same communication style all the time? This is important for the evaluation of the experiment in Section 4.
2. In Section 4.1, the experimental setting of pseudo dialogue is unclear. Is the history of pseudo dialogue related to the test model or is the history of pseudo dialogue the same for all models? For example, when testing GPT-4 and Qwen-110B, is the LLM used in the history generation of pseudo dialogue the same? Or is the pseudo dialogue generated by GPT-4 used to test GPT-4, and the pseudo dialogue generated by Qwen-110B used to test Qwen-110B? If they are the same, which LLM model is used to generate pseudo dialogue? If they are different, will using different LLMs to generate pseudo dialogues affect the subsequent Turing test results?
3. In Section 4, there is no detailed description of the test data. For example, in Section 4.1, 25 topics were used for the test. Was only one conversation tested under each topic? And there was no detailed description of whether the length of each round of conversation was reasonable and whether the quality of the data was guaranteed.
4. The results of the experimental response in Section 5.2 are unclear. The experiment analyzed the impact of the age, education level, and AI usage/knowledge of the humans participating in the test on the results. The paper did not explain what this result reflects. Does it show that the participating humans will affect the results? If so, then in the test in Section 4, should the distribution of humans participating in the test be the same for different models? If not, will it affect the results?

1. Since this paper introduces a new benchmark, the stability of the evaluation is crucial. However, the analysis of the Human-Judge and LLM-Judge evaluation results is insufficient. This lack of thorough analysis hinders subsequent research from effectively utilizing the x-turing dataset and provides inadequate guidance for future improvements of the dataset.
2. The framework for the burst messaging chatbot is overly simplistic. A more complex workflow with greater involvement of LLMs might be a better choice. While this does not diminish the value of the x-turing approach, it does affect the quality of the pseudo-dialogues provided.

1. While the methodology is novel, the paper lacks a deeper theoretical justification for why the proposed burst dialogue pattern and pseudo-dialogue generation would generalize to broader real-world scenarios.

2. Some sections, particularly those explaining the pseudo-dialogue generation process and the experimental setup, could benefit from more precise descriptions or visual aids to improve reader comprehension.

3. The reliance on subjective human judgment for the X-Turn Pass-Rate metric introduces potential biases and variability, which could undermine the reproducibility and objectivity of the findings.
Comments, suggestions and typos: See weaknesses.

1. In Figure 3, some words exceed the area they belong to. Maybe you can try to reduce the font size or add line breaks.
2. In Figure 6, some fonts are too small and unclear, and the slash mark is not explained in the figure. Some explanation can be added.

1. Line 258: Is "N = mn" a typo? Why should the number of pseudo-dialogue rounds and the extended dialogues with humans both be m rounds? What is the rationale behind this consideration?
2. What is the window size of the model used? Can it accommodate all 110 rounds of dialogue within the window?
3. It is surprising that the large model tends to favor Qwen's results, while humans do not. Have you conducted more case studies to analyze this phenomenon? Is it that humans are more accurate, or is the AI judge more accurate?
4. Additionally, does the prompt used for the dialogue affect the model's performance? In Section 5.1, it is mentioned that adding dialogue history helps in mimicking human behavior. Could this be because the instructions in the prompt regarding natural speaking are not specific enough, and the lack of few-shot examples leads to insufficient human-like responses?

Comments Suggestions:
Adding ablation studies to investigate the individual contributions of burst dialogues and pseudo-dialogue generation would clarify their respective impacts on the framework's performance.

Provide a more detailed justification for the X-Turn Pass-Rate metric and discuss its limitations explicitly to offer a balanced perspective.

Explore strategies to mitigate biases in human judgment, such as calibrating annotators or using more automated evaluation methods to complement human evaluations.


Typos:
On page 3, ensure the subscript "m" in the algorithm is consistently formatted (e.g., pd ∪ LLM(ti, m, H)).
Ensure all figure captions, such as in Figure 1 and Figure 5, are self-contained and clearly explain their significance in the context of the methodology.
Minor grammatical issues, such as "samples fail to meet the quality criteria" (should be "samples that fail"), should be addressed for better readability.
Overall assessment scores: 4

3

3.5

3.5