Flipping the Dialogue: Training and Evaluating User Language Models

## Abstract

Abstract Conversations with LMs involve two participants: a human user leading the conversation, and an LM assistant responding to the user’s request. To satisfy this specific role, LMs are post-trained to be helpful assistants – optimized to produce exhaustive and well-structured responses, free of ambiguity and grammar errors. User utterances, on the other hand, are rarely perfected, with each user phrasing requests in unique ways, sometimes putting in partial effort at each turn and refining on the fly. To evaluate LM performance in realistic settings, prior work simulated users in multi-turn conversations, often prompting an LLM originally trained to be a helpful assistant to act as a user. However, we show that assistant LMs make for poor user simulators, with the surprising finding that better assistants yield worse simulators. Instead, we introduce purpose-built User Language Models (User LMs) - models post-trained to simulate human users in multi-turn conversations. Through various evaluations, we show how User LMs align better with human behavior and achieve better simulation robustness than existing simulation methods. When leveraging User LMs to simulate coding and math conversations, the performance of a strong assistant (GPT-4o) drops from 74.6% to 57.4%, confirming that more realistic simulation environments lead to assistant struggles as they fail to cope with the nuances of users in multi-turn setups.

## 1 Introduction

Simulating human users is becoming an important area for the interactive evaluation of assistant language models (LMs). Despite their strong performance on static benchmarks Chang et al. (2025), assistant LMs often fall short of demonstrating their capabilities in multi-turn conversations with users Laban et al. (2025). This gap stems from a failure to consider the nuances of user behavior in multi-turn interactions with assistants during evaluation. Humans rarely state their full intent upfront, instead revealing it gradually across turns Herlihy et al. (2024). Users often provide minimal input because typing requires effort, and sometimes phrase their requests in confusing ways. Evaluating assistants under such conditions is critical to understand how they perform when deployed in the real world. This raises the question: how do we simulate realistic human users to replicate their conversational behavior with assistant LMs?

Recent studies that employ user simulators typically rely on prompting assistant LMs to role-play users Li et al. (2024). However, because these models are post-trained to be perfect “assistants”, they tend to produce cooperative and structured user turns, failing to capture the nuanced and often inconsistent behavior of real users. As a concrete example, prior work has pointed out that assistant-based user simulators rarely end conversations, choosing instead to chat endlessly Ivey et al. (2024). Low-quality simulators are in turn problematic for evaluating assistants, since their overly cooperative behavior can result in overestimating the performance of assistants. In this study, we address this by introducing User Language Models (User LMs) - models post-trained to simulate users that can interact with assistants. A crucial contribution of our work is to train User LMs that can be conditioned on a user intent, enabling us to steer conversations towards tasks we want to study (e.g., solving coding or math problems) while replicating human conversational behavior.

Figure [1](https://arxiv.org/html/2510.06552v1#S0.F1) contrasts two user simulators intending to solve a coding problem: a prompted assistant model (GPT-4o) and our user language model (UserLM-8b). The GPT-4o-based simulator produces user turns that are straightforward and direct, making it relatively easy for the assistant to complete the task. In contrast, our UserLM-8b generates user turns that more closely resemble the indirect ways real users phrase requests. Although the core task content is still present, these nuances detract the assistant and ultimately lead the assistant to fail at solving this task.

We first describe the details that are necessary to efficiently train user LMs (§[2](https://arxiv.org/html/2510.06552v1#S2)), which we demonstrate by measuring distributional alignment (i.e., perplexity) with real human utterances on out-of-domain data (§[2.3](https://arxiv.org/html/2510.06552v1#S2.SS3)). We then present a set of six evaluations that focus on assessing fine-grained properties of user LMs, specifically around multi-turn interaction and simulation robustness (§[3](https://arxiv.org/html/2510.06552v1#S3)). Our results show that user LMs achieve superior performance at generating more diverse user turns, decomposing intent across turns, and terminating dialogues (§[3.3](https://arxiv.org/html/2510.06552v1#S3.SS3)) compared to existing methods. We further show that user LMs are more robust simulators, adhering better to their user role and intent compared to assistant-based simulators. Finally, we show that deploying the user LM to simulate coding and math conversations with a GPT-4o assistant leads to a more realistic estimate of real-world performance on these tasks than using prompted assistants as user simulators (§[4](https://arxiv.org/html/2510.06552v1#S4)). We end the paper with a discussion and scoping of our findings (§[5](https://arxiv.org/html/2510.06552v1#S5)), and release our trained UserLMs publicly to stir further research in user simulation, which we hope to go hand-in-hand with the development of robust assistant LMs.

## 2 Training User Language Models

### 2.1 Problem Description

Our objective is to train a user LM that mimics human behavior when interacting with assistant LMs and performs three key functions: (1) initiate a conversation with the assistant given a defined user intent, (2) follow-up with the assistant based on its responses in subsequent turns, and (3) end the conversation once it has run its course. To achieve this, our approach leverages real human-assistant conversations as training data and “flips the dialogue” to train the UserLM to model the conditional distribution of user utterances: at the first turn conditioned on a user intent, and at subsequent turns conditioned on both the intent and the state of conversation so far. This setup is illustrated in Figure [2](https://arxiv.org/html/2510.06552v1#S2.F2), which we now describe in more detail.

#### Defining User Intents.

Similar to how assistant LMs must follow instructions, user LMs must follow an intent that directs the conversation. Initial experimentation revealed that there exists a fine balance in defining user intent. On one extreme, not providing any intent at all reduces the usability of the user simulator, as it cannot be effectively directed towards a task. On the other extreme, a fully-specified intent that contains all information of the user utterances renders the simulator obsolete: it simply parrots information from the intent with little added value. In this work, we define user intents as high-level conversation objectives: capturing the overall goal of the user without mentioning explicit details (see examples in Figure [2](https://arxiv.org/html/2510.06552v1#S2.F2)) – achieving a balance between the two extremes. Preliminary analyses showed that user LMs trained on high-level intents were practically more useful than ones trained without intent or with fully-specified intents: they can be guided to simulate specific conversations, but are in charge of language choices that allow for diverse and realistic simulation.

Figure: Figure 2: A diagram illustrating our approach to train a UserLM (). We leverage in-the-wild human-assistant conversations, generating high-level user intents for each conversation. We then flip the dialogue, turning each conversation with K turns into K+1 training samples, conditioning both on the high-level intent and conversation state to generate the next user utterance.
Refer to caption: x2.png

#### Ending the Conversation.

Humans naturally end conversations - a signal that often indicates the conversation has run its course (i.e., the user has obtained the information they seek, completed a task, etc.). Typically, users disengage without providing explicit feedback to the assistant. To replicate this behavior, it is essential that the user LM can effectively decide when to end the conversation. We implement this by adding a special 
 token to the tokenizer, which we then use as the output to generate after the last assistant turn in each conversation.

### 2.2 Experimental Details

#### Training Data.

We use WildChat Zhao et al. (2024), which contains 478,498 English conversations between ChatGPT and users from 192 countries. We also noticed many near-duplicate samples and performed de-duplication, which resulted in 384,336 conversations (see details in Appendix [A](https://arxiv.org/html/2510.06552v1#A1)). The authors of WildChat did not define data splits (i.e., train vs. test); therefore, we created our own by first identifying unique users based on their hashed IP addresses and countries. For each country, we randomly split users into a 90/5/5 split for training, validation, and testing. All conversations from the same user were then assigned to the same split, ensuring that each split contains conversations from entirely distinct users. This resulted in 343,951 conversations for training, 22,442 for validation, and 17,943 for testing. After formatting conversations as we described in §[2.1](https://arxiv.org/html/2510.06552v1#S2.SS1), we obtained 1,047,930 training samples, 118,291 validation samples, and 137,224 testing samples.

#### Intent Generation.

For each of the 384,336 conversations in WildChat, we generated a generic user intent using few-shot prompting with GPT-4o. The model was given the entire conversation history between the user and the assistant and instructed to produce a high-level summary of the user’s intent, abstracting away specific details. To guide this process, we provided three manually crafted examples of generic intents as demonstrations (see exact prompt in Appendix [D](https://arxiv.org/html/2510.06552v1#A4)).

#### Training Details.

We perform full-parameter fine-tuning of Llama3-8b-Base and Llama3.2-1b-Base. As we will show, we find that starting from a base model leads to better performance than an instruction-tuned model. We used a maximum sequence length of 2048 tokens, a batch size of 1024 samples, and a learning rate of 2e-5. Training was performed on four NVIDIA RTX A6000 GPUs, taking 62 hours to train the 1b model and 227 hours to train the 8b model. We refer to our fine-tuned user LMs as UserLM-1b and UserLM-8b.

#### Baselines.

We compare to multiple assistant LMs: Llama3.2-1b-Instruct, Llama3-8b-Instruct, and GPT-4o. We prompted these assistant LMs to play the role of a human user and respond to the assistant given the conversation history so far, basing our prompt on recent studies Chang et al. (2025); Ivey et al. (2024). We gave explicit instructions on how to replicate user behavior: splitting content across turns, not being verbose, or making occasional typos, without overdoing such phenomena (see prompt in Appendix [D](https://arxiv.org/html/2510.06552v1#A4)). We also compare to USP-8b Wang et al. (2025), a fine-tuned version of Llama3-8b-Base on a subset of 94,874 LMSys-Chat conversations to generate user utterances Zheng et al. (2023). All baselines were instructed to respond with 
 if they believe the user intent is fulfilled and that the conversation should end at this turn.

### 2.3 Initial Analyses

#### Distributional Alignment.

We evaluate models at “user language modeling” - how well they match the distribution of human language on the held-out test of conversations. To measure this, we compute model perplexity (PPL), a standard intrinsic metric commonly used to assess how well an LM predicts a sample of text Brown et al. (1992); Hewitt et al. (2023). For a test set that consists of a total of $N$ tokens, we compute the per-token perplexity for a model $P_{\theta}$ as $\exp\left(-\frac{1}{N}\sum_{i=1}^{N}\log P_{\theta}(x_{i}\mid x_{<i})\right)$. Lower PPL reflects less surprisal and better alignment with the underlying data distribution. We evaluate on held-out test samples from WildChat and on the entire PRISM dataset Kirk et al. (2024), which we use as an out-of-domain test set and consists of 8,011 conversations. We applied the identical intent generation procedure to PRISM conversations.

**Table 1: Perplexity (PPL) of prompted and trained models on WildChat test set (17,943 conversations) and the PRISM dataset (8,011 conversations). We compare PPL with no test-time conditioning () and conditioning on the generic intent ().**
|  | WildChat | PRISM |  |  |
| --- | --- | --- | --- | --- |
| Model | ($\downarrow$) | ($\downarrow$) | ($\downarrow$) | ($\downarrow$) |
| \cellcolor[HTML]f2f2fdLlama3.2-1b-Base | 37.68 | 29.09 | 84.00 | 53.54 |
| \cellcolor[HTML]f2f2fdLlama3-8b-Base | 98.29 | 48.13 | 89.98 | 40.86 |
| \cellcolor[HTML]f2f2fdLlama3.2-1b-Instruct | 26.08 | 16.08 | 35.02 | 20.80 |
| \cellcolor[HTML]f2f2fdLlama3-8b-Instruct | 26.19 | 21.40 | 40.25 | 36.29 |
| \cellcolor[HTML]fffaebUSP-8b | 32.08 | 21.78 | 50.91 | 30.16 |
| \cellcolor[HTML]fffaebUserLM-1b | 8.30 | 7.78 | 18.58 | 10.33 |
| \cellcolor[HTML]fffaebUserLM-8b | 5.60 | 4.33 | 14.92 | 7.42 |

#### Results.

Table [1](https://arxiv.org/html/2510.06552v1#S2.T1) shows the PPL achieved by base, assistant, and user LMs on the user utterances of the WildChat test set and the PRISM dataset. We compare results where models are not conditioned on intent at prediction time () vs. including the generic intent for conditioning (). Three trends are noteworthy. First, intent conditioning leads to gains in PPL for all models across both datasets, confirming that generic intents enable the effective steering of LMs. Second, though all models have higher PPL on PRISM than WildChat, performance trends on the two datasets are consistent, confirming that PRISM serves as a more challenging, out-of-domain test set that effectively measures generalization of user language modeling. Third, the UserLM-8b achieves the lowest PPL by a wide margin, often 60-70% lower than all baselines. Finally, the improvements observed when scaling the UserLM from 1b to 8b are encouraging, and we hypothesize that further scaling will likely yield lower PPL and better distributional alignment. In short, we find that UserLM-8b is more effective than baselines at modeling an out-of-domain user population and effectively leveraging generic user intents.

#### Do we need to train with intent?

The results indicate that even models not trained with intents (such as Llama-3.2-1b-Base) can effectively leverage them (lower PPL). To gain an understanding of whether training with generic intents is truly beneficial, we ablated this parameter by training user LMs without intent conditioning. In this setting, the first user turn is conditioned only on the user role tokens (e.g., 
user
 in the Llama tokenizer). Figure [3(a)](https://arxiv.org/html/2510.06552v1#S2.F3.sf1) compares the PPL of user LMs trained with and without intent for different test-time conditioning setups. The models trained without any intent conditioning still show improvements when conditioned on intent at test time. However, we find the biggest drops in PPL with models trained with intent. In summary, the use of intent-conditioning at train-time leads to improved sensitivity to intent in trained user LMs, resulting in a more steerable and usable model.

Figure: (a) PPL on PRISM comparing user LMs trained with and without generic intent conditioning. Models trained with intent more closely match the distribution of user utterances at test time.
Refer to caption: x3.png

#### Is starting from instruction-tuned checkpoints better?

We investigate whether training a UserLM by starting from an instruction-tuned checkpoint leads to better performance than from a base model. Figure [3(b)](https://arxiv.org/html/2510.06552v1#S2.F3.sf2) compares the PPL achieved by our 1b and 8b user models when trained from either checkpoint. We find that user models trained from the base checkpoint achieve better results with lower PPL. Instruction-tuned models are typically trained to be helpful assistants using synthetically generated data Gan & Liu (2024), which we hypothesize is semantically distant from user behavior, leading to lower distributional fit. On the other hand, base models are trained on natural text, a majority of which could be human-written and is thus initially closer to the distribution of real users. This experiment and additional evaluations reported in Appendix [B.2](https://arxiv.org/html/2510.06552v1#A2.SS2) provide evidence for a high-level observation: base pre-trained LMs are neutral, general-purpose models that can be post-trained towards distinct and opposing roles: user and assistant LMs.

## 3 Evaluating Alignment With Human Behavior

Beyond distributional measures such as perplexity, we introduce more fine-grained evaluations that capture key properties user simulators should reflect if they align with human behavior. We first propose evaluations targeting multi-turn interactions between a user and an assistant, spanning the opening, intermediate, and closing turns of a conversation (§[3.1](https://arxiv.org/html/2510.06552v1#S3.SS1)). In addition, we present evaluations designed to assess the robustness of the simulator in maintaining a realistic user behavior (§[3.2](https://arxiv.org/html/2510.06552v1#S3.SS2)).

### 3.1 Multi-Turn Interaction

#### First Turn Diversity.

At the first turn, real users can express the same request in many ways. To ensure a simulator reflects this natural variability, it is important to measure how diverse its first-turn generations are. We measure diversity by assessing how often models generate new words instead of repeating previously used ones. For each model, we randomly sample 2,000 first-turn utterances and compute the pairwise 1-gram Jaccard index, a common metric for utterance diversity Stasaski & Hearst (2022). A higher value indicates better lexical diversity in first-turn utterances.

#### Intent Decomposition.

Human users often omit details in their prompts, phrase requests in varied ways, and rely on the broader conversational context, expecting the assistant to interpret and leverage this information across turns Herlihy et al. (2024); Yang et al. (2025). It is therefore essential for our user LM to gradually progress through its intent across multiple turns, rather than revealing all the details in a single turn. We measure this by computing the average overlap of 1-grams between the generated user turns and the generic intent. We remove all stopwords before computing overlap. A lower overlap is particularly desirable because it indicates that the model expresses its intent using varied language while introducing details progressively.

#### Dialogue Termination.

We assess the ability of each simulator to terminate the conversation when the conversation ended with the real user. To measure this, we compare the model’s generation of the 
 token with actual conversation endings in PRISM conversations. Treating this as a binary classification task, we compute the F1 score to quantify how accurately the model predicts conversation endings. A high F1 score indicates that the model not only ends conversations at appropriate points but also avoids stopping too early or going on excessively.

### 3.2 Simulation Robustness

#### Naturalness.

Text generated by a user LM should resemble the natural way humans write and be distinguishable from the typical style of assistant LMs. To measure the naturalness of the utterances, we use the state-of-the-art AI-detector Pangram Emi & Spero (2024), which was shown by Russell et al. (2025) to achieve near-perfect detection accuracy, on par with frequent users of ChatGPT. For our evaluation, we sample 2,000 first-turn generations, ensuring that each utterance ranges between 50 and 200 tokens, enough text for the detector to perform its prediction reliably. Pangram returns a likelihood for the text of being human-written, which we average across all 2,000 utterances.

#### User Role Adherence.

A robust user simulator must consistently maintain its user role and avoid behaving like the assistant. To measure user role adherence, we test models under the following setup. First, we initiate conversations where the user asks an MCQ question and provides a set of answer choices (e.g., What’s the nickname of the monster the beauty loved? The choices are: beast, ugly, ugliness, satellite). We then prompt GPT-4o to generate a response stating uncertainty and seeking help from the user (e.g., I’m honestly not sure about that. It sounds familiar, but I can’t confidently say what the nickname is. If it’s from a specific story or version. Could you just tell me the answer instead?). We then generate the next user turn and measure the rate at which models adhere to their user role by avoiding revealing the answer to the assistant. An attempt to answer is counted when a model includes one or two of the choices in its second turn, ignoring cases where all options are mentioned, which we found to be repetitions of the question. We perform this test using 2,000 random samples from the CommonsenseQA dataset Talmor et al. (2018) as the first user turn.

#### Intent Adherence.

Assistants sometimes get confused in a conversation and, as an attempt to clarify, they suggest something the user did not want. Robust user simulators must adhere to their intent and not comply with assistant suggestions that steer away from their intent. To measure how well models adhere to their intent, we test them under the following setup. We initiate conversations where the user asks an open-ended question (e.g., who sang the song i’m a nut?). We then prompt GPT-4o to generate a response stating uncertainty of the answer and suggesting to assist with something else (e.g., I’m not sure what the answer is to who sang the song “I’m a Nut.” However, if you’re interested in exploring music, how about looking into the fascinating world of unusual instruments?). We then generate the next user turn for each model and measure the rate at which they refuse the assistant’s suggestion and adhere to their original intent. We perform this test using 2,000 random samples from the NaturalQuestions Kwiatkowski et al. (2019) as the first user turn. We evaluate whether the model adheres to its intent at the first turn by prompting GPT-4o as a judge.

### 3.3 Results & Analyses

Table [2](https://arxiv.org/html/2510.06552v1#S3.T2) summarizes the results achieved by all models. We make the following key observations:

#### User LMs better align with humans at multi-turn interaction across the board.

User LMs generate more diverse first turns, with UserLM-8B achieving 94.55% unique 1-grams, on-par with real users (94.01%), and outperforming GPT-4o’s 74.42%. User LMs are also better at decomposing intent across turns and produce more abstractive utterances, with an average overlap of 2.69% with the conditioned intent - close to the 1.68% observed in human utterances. By spreading information across turns and phrasing requests in varied ways, user LMs more closely reflect the dynamics of real human interactions. Further, user LMs are better at recognizing when to terminate a conversation, achieving an F1 score of 63.54. In contrast, prompted assistants rarely end conversations, with F1 scores ranging from 3-15. In other words, the assistant nature of the LM is still ingrained, leading to a lack of ability to end conversations, which is not reflective of the user role being simulated.

Table: Table 2: Results of user simulators based on prompted assistant LMs and trained user LMs on the intrinsic evaluations for: first-turn diversity, intent decomposition, dialogue termination, naturalness, user role adherence, and intent adherence. When possible, we compute metrics of the real human utterances from PRISM to serve as a reference.

#### User LMs produce more natural-looking utterances.

Pangram assigns an average 90.2% confidence that real user utterances are not AI-generated, confirming its efficacy at assigning high scores to human-written utterances. On the opposite end, utterances generated by prompted assistant models were consistently detected as AI-generated, with scores ranging 0-3%. User LMs fared better, with scores ranging from 77-81%, only slightly lower than real user utterances. This sharpens our understanding of naturalness in three ways: (1) user and assistant utterances are distinct text distributions, with AI-detectors focused on detecting AI-generated assistant text (the predominant type of AI-generated text), (2) simply prompting an assistant does not significantly alter its generation distribution, as generated utterances are still easily detected as AI-generated, (3) User LMs learn to generate user-like utterances, demonstrated by high naturalness scores.

#### User LMs provide a robust base for simulation.

The results for role adherence () show how the three trained user LMs achieve stellar robustness ranging from 91-98%, indicating they maintain the user role even when a simulated conversation introduces ambiguity. On the other hand, assistant-prompted simulators all show signs of shallow instruction following, reverting to their main assistant role in 20-60% of conversations, over-ruling their prompted instruction to behave as a user.

#### User LMs stick to their task intent better.

The results on intent adherence () paint a similar picture to role adherence, with user LMs achieving robustness scores of 93-97%, indicating a predominant choice to follow original intent and avoid distraction. Prompted assistant LMs, on the other hand, are more easily accepting of diversion, which we hypothesize is related to the sycophantic nature of assistant LMs Perez et al. (2023); Laban et al. (2023) that favor a surface-level sense of pleasing the conversational partner over following the original instruction. This finding exposes an important difference between the user and assistant roles: the user is the pilot of the conversation, and must maintain a certain stubbornness towards their stated intent, whereas the assistant inherently supports the user, and should demonstrate flexibility when the conversation drifts. This disparity is one of the key limitations in using assistant-trained models to simulate users.

#### How does scaling affect performance?

Our experiments involved three families of models at varied scales (Llama, GPT4o, and UserLM), enabling us to observe how scaling affects user simulation quality. For prompted assistant models, increased model size does not lead to improvements: Llama3-8b outperforms Llama3-1b on only one of six metrics, and GPT-4o outperforms GPT-4o-mini on two of the six. On the other hand, UserLM-8b outperforms UserLM-1b on all metrics, showing that scaling the training of User LMs effectively leads to better user simulators.

## 4 Simulating Conversations with User LMs

To gain a more practical understanding of the value of User LMs, we now deploy them as part of an extrinsic evaluation, which uses the simulator to interact with an assistant for solving tasks. We analyze the performance of assistant LMs in multi-turn conversations with the user simulators. As pointed out by Chang et al. (2025), assistant LMs are typically evaluated in single-turn settings, and one of the bottlenecks for multi-turn evaluation is the lack of realistic user simulators.

### 4.1 Experimental Setting

#### Tasks Setup.

We adopt the simulation setting of Laban et al. (2025), which simulates multi-turn conversations for various generation tasks. Specifically, we use 65 task intents that involve users completing a math word problem (based on GSM8k Cobbe et al. (2021)) or writing a Python program (based on HumanEval Chen et al. (2021)). Both tasks are appealing as they represent common use cases of current LLMs Tomlinson et al. (2025), and can be evaluated in a verifiable manner. We use UserLM-8b to simulate 10 conversations for each task intent (a total of 650 simulations). We also simulate conversations by prompting GPT-4o-mini and GPT-4o as the user simulators. In all conversations, we use GPT-4o as the assistant LM, keeping this aspect of the simulation fixed.

#### Evaluation.

We performed a quantitative analysis of the simulations, computing eight metrics focused on five aspects of the simulator: (1) Intent Coverage: measuring whether the simulator reveals the intent information adequately in the course of the conversation, (2) Information Diversity: evaluating how the user simulator repeats, skips, or adds information during conversations, (3) Pace Diversity: evaluating whether the user simulator can reveal information at varied speeds, effectively simulating users with differing communication styles, (4) Lexical Diversity: assessing the capacity of the simulator in using unique wording in each simulation, and (5) Assistant Performance: measuring the impact of the user simulator on the assistant’s ability to complete the task. The implementation details of each metric are provided in Appendix [C.3](https://arxiv.org/html/2510.06552v1#A3.SS3).

**Table 3: Summary of results from simulated conversations with three simulators (prompted assistant LMs, and trained user LMs). Each simulator is evaluated on its coverage of the intent, information selection, conversational pace, lexical diversity, and the average downstream assistant performance. See Appendix [C.4](https://arxiv.org/html/2510.06552v1#A3.SS4) for per-task results.**
|  | User Simulator |  |  |
| --- | --- | --- | --- |
|  | \cellcolor[HTML]f2f2fd4o-mini | \cellcolor[HTML]f2f2fdGPT-4o | \cellcolor[HTML]fffaebUserLM-8b |
| Metric | \cellcolor[HTML]F6F6F6 Intent Coverage |  |  |
| Intent Coverage (%) | 86.6 | 84.7 | 76.7 |
|  | \cellcolor[HTML]F6F6F6 Information Diversity |  |  |
| %Repeat Required | 31.8 | 9.4 | 54.3 |
| %Skip Non-Required | 10.9 | 14.6 | 37.7 |
| %Add Demands | 9.5 | 1.1 | 43.8 |
|  | \cellcolor[HTML]F6F6F6 Pace Diversity |  |  |
| Turn Variance | 0.9 | 0.6 | 2.8 |
| Turn Range | 3.7-5.7 | 4.0-5.4 | 2.1-6.7 |
|  | \cellcolor[HTML]F6F6F6 Lexical Diversity |  |  |
| Unigram Difference | 0.43 | 0.40 | 0.71 |
|  | \cellcolor[HTML]F6F6F6 Assistant Performance |  |  |
| Assistant Score | 73.2 | 74.6 | 57.4 |

### 4.2 Simulation Results & Analysis

Table [3](https://arxiv.org/html/2510.06552v1#S4.T3) summarizes the results from the simulations. We observe the following:

#### User LMs selectively reveal and revise their intent while avoiding unnecessary details.

The three simulators cover 76-86% of the units of information from the original intent, indicating they successfully stay on topic. Our UserLM-8b is more likely to repeat required information across turns, and less likely to reveal information that is not required to accomplish the task (e.g., revealing a trivial base case for a Python function). In contrast, the GPT-based user simulators are more monotonic: they tend to reveal information from the intent once, with less emphasis on revising what they mean by repetition or omitting information. UserLM-8b is the only simulator that introduces additional demands not specified in the original intent. We performed manual tagging of the additional demands specified by the UserLM-8b, with the three main categories being: (1) providing example test cases (34%, e.g., get_closest_vowel("FULL") should return U), (2) defining naming conventions (21%, e.g., The function should be named ‘rabbit’), and (3) implementation constraints (20%, e.g., Avoid using the built-in min and max functions). In contrast, the GPT-based simulators stick to the script, rarely injecting demands, leading to more homogeneous simulations.

#### User LMs exhibits more turn variance than the GPT-based simulators.

Our UserLM-8b dynamically decides the granularity of information revealed at each turn, leading to more variation with conversations ranging from 2.1 to 6.7 turns. The GPT-based simulators are more consistent, with a narrower range of 3.7 to 5.7 turns. This variability is important because it allows UserLM-8b to simulate users with different interaction paces, capturing a broader spectrum of dialogue behaviors.

#### User LMs simulate more lexically diverse conversations.

Our UserLM-8b achieves higher lexical diversity. In other words, two simulations based on the same intent have less lexical overlap when using UserLM-8b compared to using GPT-4o as the simulator. The diversity is due to UserLM-8b varying its language and style in different conversations, in contrast to GPT-based simulators that retain high similarity with the intent and avoid language variation.

#### Assistants struggle more when conversing with User LMs.

We find that assistant task performance is around 17% lower in conversations with UserLM-8b than with the GPT-based simulators. As identified above, UserLM-8b exhibits more diverse behavior in the information it presents, in the pace it sets for the conversation, and in the lexical choices it makes during simulation. In turn, these more realistic simulation conditions are challenging to the GPT-4o assistant, offering a more comprehensive estimation of assistant performance in multi-turn interaction with diverse users.

## 5 Discussion & Implications

#### Stronger assistants LMs are not necessarily better user simulators.

A key repeating insight from our results is that scaling up assistant LMs does not automatically result in improved user simulators. We first see this in Section [2.3](https://arxiv.org/html/2510.06552v1#S2.SS3) when comparing PPL on user utterances across instruction-tuned models, finding that Llama3-8b-Instruct exhibited worse PPL than the smaller 1B model. Similarly, in our analysis on initializing training from instruction-tuned checkpoints, starting from the 1b assistant model resulted in lower PPL (18.45) compared to starting from the 8b assistant model (27.25). This pattern also emerged in some of the intrinsic evaluations where GPT-4o-mini surpassed GPT-4o, particularly in terms of robustness, demonstrating a much better ability to adhere to its user role and intent. These results suggest that better assistant LMs do not make for better user simulators, highlighting the need to explicitly train user LMs. Looking at user LMs, our experiments show that scaling user LM training from 1b to 8b leads to gains across the board, with further scaling likely to be beneficial.

#### From User LMs to Personalized User LMs.

In this work, user LMs were trained to simulate a broad, general audience, capturing behaviors commonly shared across a population of users. As such, UserLM-8b is a general-purpose foundation model that can model users in WildChat and PRISM. However, users with different demographics or personas may exhibit important variations that are important to model. For instance, non-native English speakers in Middle Eastern countries often phrase requests differently from native speakers in the United States, reflecting their own dialects and linguistic backgrounds. We believe a key focus for future studies on user language modeling is the development of more personalized user LMs that can simulate specific user groups or behavior in specific domains. The collection of large-scale data in specific settings is often challenging, and we hope the user LMs we propose can serve as a foundation to finetune more specialized user LMs, effectively lowering the data requirements in training personalized models.

#### User LMs will serve purposes beyond interactive evaluation of assistants.

In Section [4](https://arxiv.org/html/2510.06552v1#S4), we presented one use case of user LMs in providing a more realistic evaluation of assistants in multi-turn conversations. Beyond interactive evaluation, we believe user LMs can open up a range of promising applications. One such direction is user modeling, where prior work has explored the use of LMs to estimate distributions of user responses to surveys Hu & Collier (2024); Suh et al. (2025). User LMs can be seen as a generative extension of this approach: instead of merely predicting distributions, they can generate natural text responses to survey questions, providing estimates of typical user behaviors. A second potential use case is in improving judge models. Current approaches often train judge models based on assistant LMs to approximate user preferences, but these are prone to assistant-specific biases and limited personalization Dong et al. (2024). By contrast, user LMs could serve as more realistic judges, either through prompting or finetuning into user LM judge models. Finally, user LMs can be leveraged for synthetic conversational data generation Wang et al. (2025). As we show, user LMs exhibit greater lexical diversity and produce more varied simulations, making them better-suited for generating synthetic data that can be used to fine-tune assistants and improve their robustness.

#### Simulating Users vs. Working with Real Users.

The experiments we present in this work use automated methods to simulate users, presenting findings that are expected to generalize to real user behavior. However, simulating user behavior is a challenging problem, and we do not anticipate simulations to be applicable to all cases. In particular, working with experts in technical domains such as law Das et al. (2025), creative writing Chakrabarty et al. (2025) or science Asai et al. (2024) remains necessary. We propose the following scope: user simulation is a useful tool in scaling experiments, allowing the efficient discovery of system flaws that affect broad and generic populations, whereas working with experts or specific user populations is necessary to gain knowledge about user nuances.

#### More open-source base models will be needed.

Our experiments show that training user LMs from base LMs is more effective than starting from an instruction-tuned checkpoint, where the model has already been tuned to behave as an assistant. At the present time, most releases of open-weight LMs consist of instruction-tuned assistant models, with few releases of base models. We hope that our study encourages further research on building safe and general-purpose base models, as they will be useful to the development of user LMs.

#### Building stronger user LMs will help build stronger assistant LMs.

The user LMs we release as part of this work are relatively small (1b and 8b parameters) and have been post-trained on a limited set of natural user-assistant interactions (343,951 conversations). Despite this, they still outperform prompted assistants such as GPT-4o in simulating users and provide a more comprehensive evaluation of multi-turn assistant performance on downstream tasks. We expect that scaling up training and gaining access to higher-quality corpora of user logs will result in more performant user LMs, serving as more realistic simulation environments for assistants during reinforcement learning. This, in turn, will help develop more robust and effective assistants. We view our models as an initial milestone toward this long-term goal.

## 6 Related Work

#### Simulation Environments.

Historically, one of the main uses of computers has been to simulate environments across many domains, from video games, macro-economics Holland & Miller (1991); Zheng et al. (2022), weather and climate Taylor et al. (2012), political elections Straffin Jr (1980); Le Pennec & Pons (2023), biological simulators for molecular interaction Shaw et al. (2010), financial markets Cont & Bouchaud (2000), self-driving car environments Dosovitskiy et al. (2017), and physics simulators for robotics Todorov et al. (2012). Simulation environments are invaluable in these applications as they enable the evaluation of theories or methods, on-the-fly decision-making for a system based on simulation, or the training of AI systems, for instance through reinforcement learning. As conversational interfaces see widespread adoption in society, we view UserLMs as a foundational simulation environment for user-AI conversation.

#### Simulating Human Users In Conversation.

User simulation has received considerable attention in past research, serving as an environment for dialogue policies to interact with in reinforcement learning setups Shi et al. (2019). Early conversation simulators were introduced in task-oriented dialogue research, consisting of rule-based systems that define a user goal and an agenda of pending actions that dictate the next user move Schatzmann et al. (2007); Li et al. (2016). These systems later evolved to include neural-based components that encode the dialogue state and generate user actions Asri et al. (2016); Kreyssig et al. (2018); Gür et al. (2018); Tseng et al. (2021). More recently, pre-trained assistant LMs have been adopted in user simulation research. The majority of existing works have prompted assistant LMs in a zero-shot manner to role-play a user (i.e., “You are a user chatting with an AI assistant …”), instructing them to generate the next user turn given the dialogue history Li et al. (2024); Luo et al. (2024); Zhong et al. (2025); Chang et al. (2025). However, a growing body of work highlights that these prompted assistants exhibit low alignment with real human interactions and often diverge from human behavior Ivey et al. (2024); Yoon et al. (2024); Zhu et al. (2024).

A few studies have proposed fine-tuning approaches to generate user turns given the dialogue context Liu et al. (2022; 2023); Sekulić et al. (2024) by training on task-oriented dialogue datasets such as MultiWOZ Eric et al. (2019); Cheng et al. (2022). Such fine-tuning approaches have been mainly focused on generating synthetic user-assistant conversations that can be used as additional training data for assistant LMs Wan et al. (2022); Ferreira et al. (2024); Dhole (2024), with some focusing on generating diverse user profiles Kong et al. (2024); Wang et al. (2025). Our work complements this body of literature by introducing general-purpose user language models trained to follow a defined user intent and simulate users in conversations with assistants. We show that our user LMs better align with real human behavior compared to prompting assistant LMs to role-play users.

#### Evaluating User Simulators.

There is relatively much less work on how to evaluate simulators at replicating human behavior in their conversations with assistants Deriu et al. (2021). Some studies evaluate simulators by measuring the correlation between their utterances and human utterances using lexical or stylistic metrics Ivey et al. (2024); Pietquin & Hastie (2013), or the diversity of their generated utterances Wang et al. (2025). Others have resorted to human evaluation, asking crowd-workers to judge the quality of the simulated conversations Kranti et al. (2025); Sekulić et al. (2024; 2022); Shi et al. (2019). Another line of work performs a more indirect evaluation of the simulator by examining if the task-solving performance when interacting with the assistant correlates with the performance from interactions with real users Chang et al. (2025). Our study introduces fine-grained evaluations of key simulator properties, focusing on multi-turn interaction behavior and robustness. We also assess simulators when deployed for solving coding and math tasks with assistants, analyzing aspects such as information coverage, conversational pace, and diversity.

## 7 Conclusion

In this work, we introduced approaches to train and evaluate User LMs. Through many experiments, we showed the inefficacy of prompting assistant LMs to simulate users and highlighted the value of purpose-built user LMs, trained on user utterances from a large corpus of user-assistant interactions. We demonstrated a practical application of user LMs - using them to simulate conversations with an assistant. This surfaced limitations in current assistant LMs, which displayed degraded performance due to the more realistic simulation environment. We hope the release of UserLM-8b spurs further research by the community on user language modeling.

## Acknowledgments

We would like to thank Chan Young Park, Jillian Fisher, and Serina Chang, and the AI Interaction and Learning team members at Microsoft Research for valuable discussions. We also thank Pangram for providing credits to access their API, facilitating some of our robustness evaluations.

## Ethics Statement

The UserLM-8b we contribute in this work is trained from a base checkpoint (Llama3-8b-base), which has likely not undergone the conventional post-training mechanisms for output safety and alignment. Thus, our model is expected to behave similarly to a base model, which could be more prone to generating toxic language than an instruction-tuned model. We also note that our model is trained to generate user utterances that are realistic, which we also demonstrated to be not easily detectable by current state-of-the-art detectors of AI-generated text (§[3](https://arxiv.org/html/2510.06552v1#S3)). The purpose of our model is to be used for research purposes as a simulator for interactive evaluation of assistants and similar beneficial use cases. We hope our work encourages future research on building detectors that differentiate between utterances generated by a user LM and ones written by real humans.

## Reproducibility Statement

Our trained UserLM-8b model will be made publicly available for research purposes. The model was trained using publicly available data, enabling the community to reproduce and expand on our experiments. We report the decoding configuration we used for our UserLM-8b in Appendix [C.1](https://arxiv.org/html/2510.06552v1#A3.SS1) and all the prompts we used in Appendix [D](https://arxiv.org/html/2510.06552v1#A4), which are needed to replicate the results in our experiments.

## References

- Asai et al. (2024)
Akari Asai, Jacqueline He, Rulin Shao, Weijia Shi, Amanpreet Singh, Joseph Chee Chang, Kyle Lo, Luca Soldaini, Sergey Feldman, Mike D’arcy, et al.
Openscholar: Synthesizing scientific literature with retrieval-augmented lms.
*arXiv preprint arXiv:2411.14199*, 2024.
- Asri et al. (2016)
Layla El Asri, Jing He, and Kaheer Suleman.
A sequence-to-sequence model for user simulation in spoken dialogue systems.
*arXiv preprint arXiv:1607.00070*, 2016.
- Brown et al. (1992)
Peter F Brown, Stephen A Della Pietra, Vincent J Della Pietra, Jennifer C Lai, and Robert L Mercer.
An estimate of an upper bound for the entropy of english.
*Computational Linguistics*, 18(1):31–40, 1992.
- Chakrabarty et al. (2025)
Tuhin Chakrabarty, Philippe Laban, and Chien-Sheng Wu.
Can ai writing be salvaged? mitigating idiosyncrasies and improving human-ai alignment in the writing process through edits.
In *Proceedings of the 2025 CHI Conference on Human Factors in Computing Systems*, pp. 1–33, 2025.
- Chang et al. (2025)
Serina Chang, Ashton Anderson, and Jake M Hofman.
ChatBench: From static benchmarks to human-AI evaluation, 2025.
- Chen et al. (2021)
Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde De Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et al.
Evaluating large language models trained on code.
*arXiv preprint arXiv:2107.03374*, 2021.
- Cheng et al. (2022)
Qinyuan Cheng, Linyang Li, Guofeng Quan, Feng Gao, Xiaofeng Mou, and Xipeng Qiu.
Is MultiWOZ a solved task? an interactive TOD evaluation framework with user simulator.
*arXiv preprint arXiv:2210.14529*, 2022.
- Cobbe et al. (2021)
Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, et al.
Training verifiers to solve math word problems.
*arXiv preprint arXiv:2110.14168*, 2021.
- Cont & Bouchaud (2000)
Rama Cont and Jean-Philipe Bouchaud.
Herd behavior and aggregate fluctuations in financial markets.
*Macroeconomic dynamics*, 4(2):170–196, 2000.
- Das et al. (2025)
Debarati Das, Khanh Chi Le, Ritik Sachin Parkar, Karin De Langis, Brendan Madson, Chad M Berryman, Robin M Willis, Daniel H Moses, Brett McDonnell, Daniel Schwarcz, et al.
Lawflow: Collecting and simulating lawyers’ thought processes.
*arXiv preprint arXiv:2504.18942*, 2025.
- Deriu et al. (2021)
Jan Deriu, Alvaro Rodrigo, Arantxa Otegi, Guillermo Echegoyen, Sophie Rosset, Eneko Agirre, and Mark Cieliebak.
Survey on evaluation methods for dialogue systems.
*Artificial Intelligence Review*, 54(1):755–810, 2021.
- Dhole (2024)
Kaustubh D Dhole.
KAUCUS: Knowledge augmented user simulators for training language model assistants.
*arXiv preprint arXiv:2401.16454*, 2024.
- Dong et al. (2024)
Yijiang River Dong, Tiancheng Hu, and Nigel Collier.
Can LLM be a personalized judge?
*arXiv preprint arXiv:2406.11657*, 2024.
- Dosovitskiy et al. (2017)
Alexey Dosovitskiy, German Ros, Felipe Codevilla, Antonio Lopez, and Vladlen Koltun.
Carla: An open urban driving simulator.
In *Conference on robot learning*, pp. 1–16. PMLR, 2017.
- Emi & Spero (2024)
Bradley Emi and Max Spero.
Technical report on the pangram AI-generated text classifier.
*arXiv preprint arXiv:2402.14873*, 2024.
- Eric et al. (2019)
Mihail Eric, Rahul Goel, Shachi Paul, Adarsh Kumar, Abhishek Sethi, Peter Ku, Anuj Kumar Goyal, Sanchit Agarwal, Shuyang Gao, and Dilek Hakkani-Tur.
MultiWOZ 2.1: A consolidated multi-domain dialogue dataset with state corrections and state tracking baselines.
*arXiv preprint arXiv:1907.01669*, 2019.
- Ferreira et al. (2024)
Rafael Ferreira, David Semedo, and João Magalhães.
Multi-trait user simulation with adaptive decoding for conversational task assistants.
*arXiv preprint arXiv:2410.12891*, 2024.
- Gan & Liu (2024)
Zeyu Gan and Yong Liu.
Towards a theoretical understanding of synthetic data in LLM post-training: A reverse-bottleneck perspective.
*arXiv preprint arXiv:2410.01720*, 2024.
- Gür et al. (2018)
Izzeddin Gür, Dilek Hakkani-Tür, Gokhan Tür, and Pararth Shah.
User modeling for task oriented dialogues.
In *2018 IEEE Spoken Language Technology Workshop (SLT)*, pp. 900–906. IEEE, 2018.
- Herlihy et al. (2024)
Christine Herlihy, Jennifer Neville, Tobias Schnabel, and Adith Swaminathan.
On overcoming miscalibrated conversational priors in llm-based chatbots.
*arXiv preprint arXiv:2406.01633*, 2024.
- Hewitt et al. (2023)
John Hewitt, John Thickstun, Christopher D Manning, and Percy Liang.
Backpack language models.
*arXiv preprint arXiv:2305.16765*, 2023.
- Holland & Miller (1991)
John H Holland and John H Miller.
Artificial adaptive agents in economic theory.
*The American economic review*, 81(2):365–370, 1991.
- Hu & Collier (2024)
Tiancheng Hu and Nigel Collier.
Quantifying the persona effect in llm simulations.
*arXiv preprint arXiv:2402.10811*, 2024.
- Ivey et al. (2024)
Jonathan Ivey, Shivani Kumar, Jiayu Liu, Hua Shen, Sushrita Rakshit, Rohan Raju, Haotian Zhang, Aparna Ananthasubramaniam, Junghwan Kim, Bowen Yi, et al.
Real or robotic? assessing whether LLMs accurately simulate qualities of human responses in dialogue.
*arXiv preprint arXiv:2409.08330*, 2024.
- Kirk et al. (2024)
Hannah Rose Kirk, Alexander Whitefield, Paul Rottger, Andrew M Bean, Katerina Margatina, Rafael Mosquera-Gomez, Juan Ciro, Max Bartolo, Adina Williams, He He, et al.
The PRISM alignment dataset: What participatory, representative and individualised human feedback reveals about the subjective and multicultural alignment of large language models.
*Advances in Neural Information Processing Systems*, 37:105236–105344, 2024.
- Kong et al. (2024)
Chuyi Kong, Yaxin Fan, Xiang Wan, Feng Jiang, and Benyou Wang.
PlatoLM: Teaching LLMs in multi-round dialogue via a user simulator.
In *Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)*, pp. 7841–7863, 2024.
- Kranti et al. (2025)
Chalamalasetti Kranti, Sherzod Hakimov, and David Schlangen.
clem: todd: A framework for the systematic benchmarking of llm-based task-oriented dialogue system realisations.
*arXiv preprint arXiv:2505.05445*, 2025.
- Kreyssig et al. (2018)
Florian Kreyssig, Iñigo Casanueva, Pawel Budzianowski, and Milica Gasic.
Neural user simulation for corpus-based policy optimisation for spoken dialogue systems.
*arXiv preprint arXiv:1805.06966*, 2018.
- Kwiatkowski et al. (2019)
Tom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Jacob Devlin, Kenton Lee, et al.
Natural questions: a benchmark for question answering research.
*Transactions of the Association for Computational Linguistics*, 7:453–466, 2019.
- Laban et al. (2023)
Philippe Laban, Lidiya Murakhovs’ ka, Caiming Xiong, and Chien-Sheng Wu.
Are you sure? challenging llms leads to performance drops in the flipflop experiment.
*arXiv preprint arXiv:2311.08596*, 2023.
- Laban et al. (2025)
Philippe Laban, Hiroaki Hayashi, Yingbo Zhou, and Jennifer Neville.
LLMs get lost in multi-turn conversation.
*arXiv preprint arXiv:2505.06120*, 2025.
- Le Pennec & Pons (2023)
Caroline Le Pennec and Vincent Pons.
How do campaigns shape vote choice? multicountry evidence from 62 elections and 56 tv debates.
*The Quarterly Journal of Economics*, 138(2):703–767, 2023.
- Li et al. (2024)
Ruosen Li, Ruochen Li, Barry Wang, and Xinya Du.
IQA-EVAL: Automatic evaluation of human-model interactive question answering.
*Advances in Neural Information Processing Systems*, 37:109894–109921, 2024.
- Li et al. (2016)
Xiujun Li, Zachary C Lipton, Bhuwan Dhingra, Lihong Li, Jianfeng Gao, and Yun-Nung Chen.
A user simulator for task-completion dialogues.
*arXiv preprint arXiv:1612.05688*, 2016.
- Liu et al. (2022)
Hong Liu, Yucheng Cai, Zhijian Ou, Yi Huang, and Junlan Feng.
A generative user simulator with GPT-based architecture and goal state tracking for reinforced multi-domain dialog systems.
*arXiv preprint arXiv:2210.08692*, 2022.
- Liu et al. (2023)
Yajiao Liu, Xin Jiang, Yichun Yin, Yasheng Wang, Fei Mi, Qun Liu, Xiang Wan, and Benyou Wang.
One cannot stand for everyone! leveraging multiple user simulators to train task-oriented dialogue systems.
In *Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)*, pp. 1–21, 2023.
- Luo et al. (2024)
Xiang Luo, Zhiwen Tang, Jin Wang, and Xuejie Zhang.
DuetSim: Building user simulator with dual large language models for task-oriented dialogues.
*arXiv preprint arXiv:2405.13028*, 2024.
- Perez et al. (2023)
Ethan Perez, Sam Ringer, Kamile Lukosiute, Karina Nguyen, Edwin Chen, Scott Heiner, Craig Pettit, Catherine Olsson, Sandipan Kundu, Saurav Kadavath, et al.
Discovering language model behaviors with model-written evaluations.
In *Findings of the association for computational linguistics: ACL 2023*, pp. 13387–13434, 2023.
- Pietquin & Hastie (2013)
Olivier Pietquin and Helen Hastie.
A survey on metrics for the evaluation of user simulations.
*The knowledge engineering review*, 28(1):59–73, 2013.
- Russell et al. (2025)
Jenna Russell, Marzena Karpinska, and Mohit Iyyer.
People who frequently use ChatGPT for writing tasks are accurate and robust detectors of AI-generated text.
*arXiv preprint arXiv:2501.15654*, 2025.
- Schatzmann et al. (2007)
Jost Schatzmann, Blaise Thomson, Karl Weilhammer, Hui Ye, and Steve Young.
Agenda-based user simulation for bootstrapping a POMDP dialogue system.
In *Human Language Technologies 2007: The Conference of the North American Chapter of the Association for Computational Linguistics; Companion Volume, Short Papers*, pp. 149–152, 2007.
- Sekulić et al. (2022)
Ivan Sekulić, Mohammad Aliannejadi, and Fabio Crestani.
Evaluating mixed-initiative conversational search systems via user simulation.
In *Proceedings of the Fifteenth ACM International Conference on Web Search and Data Mining*, pp. 888–896, 2022.
- Sekulić et al. (2024)
Ivan Sekulić, Silvia Terragni, Victor Guimarães, Nghia Khau, Bruna Guedes, Modestas Filipavicius, Andre Ferreira Manso, and Roland Mathis.
Reliable LLM-based user simulator for task-oriented dialogue systems.
*arXiv preprint arXiv:2402.13374*, 2024.
- Shaw et al. (2010)
David E Shaw, Paul Maragakis, Kresten Lindorff-Larsen, Stefano Piana, Ron O Dror, Michael P Eastwood, Joseph A Bank, John M Jumper, John K Salmon, Yibing Shan, et al.
Atomic-level characterization of the structural dynamics of proteins.
*Science*, 330(6002):341–346, 2010.
- Shi et al. (2019)
Weiyan Shi, Kun Qian, Xuewei Wang, and Zhou Yu.
How to build user simulators to train rl-based dialog systems.
*arXiv preprint arXiv:1909.01388*, 2019.
- Stasaski & Hearst (2022)
Katherine Stasaski and Marti A Hearst.
Semantic diversity in dialogue with natural language inference.
*arXiv preprint arXiv:2205.01497*, 2022.
- Stasaski et al. (2020)
Katherine Stasaski, Grace Hui Yang, and Marti A Hearst.
More diverse dialogue datasets via diversity-informed data collection.
In *Proceedings of the 58th annual meeting of the association for computational linguistics*, pp. 4958–4968, 2020.
- Straffin Jr (1980)
Philip D Straffin Jr.
*Topics in the Theory of Voting.*
ERIC, 1980.
- Suh et al. (2025)
Joseph Suh, Erfan Jahanparast, Suhong Moon, Minwoo Kang, and Serina Chang.
Language model fine-tuning on scaled survey data for predicting distributions of public opinions.
*arXiv preprint arXiv:2502.16761*, 2025.
- Talmor et al. (2018)
Alon Talmor, Jonathan Herzig, Nicholas Lourie, and Jonathan Berant.
CommonsenseQA: A question answering challenge targeting commonsense knowledge.
*arXiv preprint arXiv:1811.00937*, 2018.
- Taylor et al. (2012)
Karl E Taylor, Ronald J Stouffer, and Gerald A Meehl.
An overview of cmip5 and the experiment design.
*Bulletin of the American meteorological Society*, 93(4):485–498, 2012.
- Tevet & Berant (2020)
Guy Tevet and Jonathan Berant.
Evaluating the evaluation of diversity in natural language generation.
*arXiv preprint arXiv:2004.02990*, 2020.
- Todorov et al. (2012)
Emanuel Todorov, Tom Erez, and Yuval Tassa.
Mujoco: A physics engine for model-based control.
In *2012 IEEE/RSJ international conference on intelligent robots and systems*, pp. 5026–5033. IEEE, 2012.
- Tomlinson et al. (2025)
Kiran Tomlinson, Sonia Jaffe, Will Wang, Scott Counts, and Siddharth Suri.
Working with ai: Measuring the occupational implications of generative ai.
*arXiv preprint arXiv:2507.07935*, 2025.
- Tseng et al. (2021)
Bo-Hsiang Tseng, Yinpei Dai, Florian Kreyssig, and Bill Byrne.
Transferable dialogue systems and user simulators.
*arXiv preprint arXiv:2107.11904*, 2021.
- Wan et al. (2022)
Dazhen Wan, Zheng Zhang, Qi Zhu, Lizi Liao, and Minlie Huang.
A unified dialogue user simulator for few-shot data augmentation.
In *Findings of the Association for Computational Linguistics: EMNLP 2022*, pp. 3788–3799, 2022.
- Wang et al. (2025)
Kuang Wang, Xianfei Li, Shenghao Yang, Li Zhou, Feng Jiang, and Haizhou Li.
Know you first and be you better: Modeling human-like user simulators via implicit profiles.
*arXiv preprint arXiv:2502.18968*, 2025.
- Yang et al. (2025)
Chenyang Yang, Yike Shi, Qianou Ma, Michael Xieyang Liu, Christian Kästner, and Tongshuang Wu.
What prompts don’t say: Understanding and managing underspecification in LLM prompts.
*arXiv preprint arXiv:2505.13360*, 2025.
- Yoon et al. (2024)
Se-eun Yoon, Zhankui He, Jessica Echterhoff, and Julian McAuley.
Evaluating large language models as generative user simulators for conversational recommendation.
In *Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers)*, pp. 1490–1504, 2024.
- Zhao et al. (2024)
Wenting Zhao, Xiang Ren, Jack Hessel, Claire Cardie, Yejin Choi, and Yuntian Deng.
WildChat: 1M ChatGPT interaction logs in the wild.
*arXiv preprint arXiv:2405.01470*, 2024.
- Zheng et al. (2023)
Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Tianle Li, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zhuohan Li, Zi Lin, Eric P Xing, et al.
LMSYS-Chat-1M: A large-scale real-world LLM conversation dataset.
*arXiv preprint arXiv:2309.11998*, 2023.
- Zheng et al. (2022)
Stephan Zheng, Alexander Trott, Sunil Srinivasa, David C Parkes, and Richard Socher.
The ai economist: Taxation policy design via two-level deep multiagent reinforcement learning.
*Science advances*, 8(18):eabk2607, 2022.
- Zhong et al. (2025)
Qishuai Zhong, Zongmin Li, Siqi Fan, and Aixin Sun.
Evaluating LLM adaptation to sociodemographic factors: User profile vs. dialogue history.
*arXiv preprint arXiv:2505.21362*, 2025.
- Zhu et al. (2024)
Lixi Zhu, Xiaowen Huang, and Jitao Sang.
How reliable is your simulator? analysis on the limitations of current llm-based user simulators for conversational recommendation.
In *Companion Proceedings of the ACM Web Conference 2024*, pp. 1726–1732, 2024.

## Appendix A Additional Experimental Details

#### WildChat Deduplication.

During initial analysis, we noticed a lot of near-duplicate conversations in WildChat - often consisting of prompt templates with minor input variations. To improve the quality of the training corpus, we applied a 7-gram counter on the first-turn user prompts and manually examined the most frequent patterns. We identified several recurring prompt templates, with one used in as many as 81,196 conversations, that do not represent natural user interaction and could degrade model quality. Removing all near-duplicates reduced the dataset to 384,336 conversations.

## Appendix B Additional Results

### B.1 Per-turn Analyses

Figure: Figure 4: Per-turn token-level PPL achieved by models on PRISM utterances. All models are conditioned on the generic user intent of each conversation. Our user LMs outperform all baselines and achieve much lower PPL, especially at the first turn.
Refer to caption: x5.png

#### Perplexity.

Figure [4](https://arxiv.org/html/2510.06552v1#A2.F4) shows the token-level PPL achieved by all models on the human utterances of the PRISM dataset. Our user LMs outperform the instruction-tuned models and the USP-8b baselines. Particularly, we find the largest PPL differences between our user models and all baselines to be at the first turn. This indicates that our user LMs are much better at initiating the first turn of a conversation the way real users would, but prompted instruction-tuned baselines struggle to do so.

#### Intent Decomposition.

Figure [5](https://arxiv.org/html/2510.06552v1#A2.F5) shows the cumulative 1-gram overlap across turns between the generated user turns and the generic intent we condition on. Our user models have the lowest overlap across turns, aligning closely with humans and outperforming the GPT-4o simulator that is explicitly instructed to split intent across turns and not to copy. The instruction-tuned Llama models show the highest overlap, starting at 25% which indicates higher levels of copying from the intent at the first turn.

#### Dialogue Termination.

Figure [6](https://arxiv.org/html/2510.06552v1#A2.F6) shows the precision, recall, and F1 score per-turn for the dialogue termination evaluation. We find that prompted assistants such as GPT4o-mini achieve high precision, yet very low recall, indicating that while their termination usually aligns with real users, they rarely do so and often proceed to continue the conversation. This behavior becomes worse with better assistants such as GPT-4o, which rarely terminates dialogues, achieving near-zero F1 score. Our UserLM-1b and UserLM-8b show a better balance of precision and recall, aligning better with human behavior. We note that in practice, decoding can be constrained in our user LMs to not generate the 
 token if desired.

Figure: Figure 5: Cumulative n-gram overlap between generated user turns and the generic intent of each conversation. Results are averaged across each turn for all conversations in PRISM. Our user LMs achieve the lowest cumulative n-gram overlap with the intent, aligning with real human utterances.
Refer to caption: x6.png

Figure: Figure 6: Precision, Recall, and F1 score per turn for our dialogue termination evaluation. Results are shown for models that achieved more than an average F1 score of 10.
Refer to caption: x7.png

### B.2 Starting Checkpoint Additional Analysis

Our earlier experiments showed how User LMs trained from a base checkpoint achieve better distributional alignment (lower PPL) on human utterances (§[2.3](https://arxiv.org/html/2510.06552v1#S2.SS3)), suggesting better simulators. We further verify this by evaluating models trained from either checkpoints on our intrinsic properties presented in §[3](https://arxiv.org/html/2510.06552v1#S3). Table [4](https://arxiv.org/html/2510.06552v1#A2.T4) shows the results, where we find that User LMs trained from the base checkpoints generally outperform ones trained from instruction-tuned models. Most notably, models trained from the base checkpoints are better at intent decomposition, dialogue termination, and produce text that more closely resembles the style of real humans. Models trained from instruction-tuned checkpoints show similar performance in terms of role and intent adherence. These results further confirm the findings from our initial PPL analyses, which show that models already trained to be assistants are more difficult to train to become user models.

Table: Table 4: Additional evaluations on intrinsic properties comparing our User LMs when training from a base checkpoints (Llama3.2-1b-Base and Llama3-8b-Base) vs instruction-tuned checkpoint (Llama3.2-1b-Instruct and Llama3-8b-Instruct). The user models trained from a base checkpoint achieve overall better results than ones trained from instruction-tuned checkpoints.

## Appendix C Conversation Simulation Experimental Detail

### C.1 Generation Configuration for UserLM-8b

In the simulation experiment detailed in Section [4](https://arxiv.org/html/2510.06552v1#S4), we implemented a few simple generation guardrails to counteract the small nature of the model (8b parameters), as we found that simply sampling unfiltered responses from the model did not result in satisfactory quality necessary to conduct simulation experiments.

#### Guardrail 1: Filtering First Tokens.

In our experimental simulation setting, initial trials revealed that the UserLM-8b frequently generates responses that started with “I”, “You”, or “Here”, which led to repetitive user utterances. We constrained the decoding by implementing a logit filter, in which a set of 6 tokens (3 words listed above in lower- and capitalized format) had their logits set to 0 for the first token, ensuring they are not sampled as first words. This simple modification was effective in practice at steering the model towards more useful generations.

#### Guardrail 2: Avoiding Dialogue Termination.

In our extrinsic evaluation, we prohibited the model from generating the 
 token that we used in our earlier analysis on dialogue termination capability, by setting its probability of being generated to 0.

#### Guardrail 3: Maximal and Minimal Length Threshold.

The experiment we conduct specifically evaluates assistant behavior in multi-turn, under-specified conversations. To encourage such behavior from the user simulator, we filtered out any user utterance of more than 25 words, as we found through inspection that longer utterances tended to reveal the entire problem formulation, reducing the need for multi-turn interaction. In case the UserLM-8b generates a response longer than the set threshold, we discarded the response and generated a new one, until we obtained a response below the threshold. We also filter out responses that contain fewer than three words, though we note that such occurrences were rare in our inspection.

#### Guardrail 4: Filter Verbatim Repetitions.

Inspection of simulated conversations further revealed that the UserLM-8b can, on occasion, repeat a previous user utterance verbatim, or copy verbatim the original intent, which does not advance the conversation appropriately. We therefore filter out such responses and re-generate using the simulator until a novel response is generated.

The four guardrails detailed above were implemented to maintain a quality in the simulated conversations we include in our analysis. We believe that these guardrails are needed due to the small size of our model, leading to noisy responses. We hypothesize that the need for such guardrails would be alleviated with a larger and more performant user LM, which we hope will arise in the future as the community develops high-quality simulators.

### C.2 Additional Examples of Simulated Conversations

Figure: Figure 7: Selected simulated conversations for a fixed instruction, using a prompted GPT-4o (three simulations on the left) and using UserLM-8b (three simulations on the right). We only show user turns, and omit assistant responses. UserLM-based simulations are more diverse, using more varied language in each, as confirmed quantitatively bu our metrics-based analysis.
Refer to caption: x8.png

The simulation experiments in Section [4](https://arxiv.org/html/2510.06552v1#S4) conducted ten simulations for each task instruction, allowing us to observe the diversity of simulations that can occur. For a given task, Figure [7](https://arxiv.org/html/2510.06552v1#A3.F7) provides three simulated conversations using GPT-4o for the simulation, and three conversations simulated using UserLM-8b. The UserLM-8b-based conversations are more distinct from each other, with more unique terminology (e.g., “ignore” vs. “keep” or “translate” vs. “replace”), which we measure quantitatively by observing a higher pace and lexical diversity. This additional diversity in simulation leads to a more challenging environment for the assistant, which is measured to have lower performance than when conversations are simulated using GPT-4o.

### C.3 Definition of Simulation Evaluation Metrics

We now define the eight metrics used as part of our analysis in Section [4](https://arxiv.org/html/2510.06552v1#S4) that involves the simulation of multi-turn conversations using various user simulators.

Each simulated conversation in the experiment is based on an instruction that the user provides to the assistant from the Lost in Conversation corpus Laban et al. (2025), which we use as the user intent of the simulation (i.e., “You are a user chatting with an assistant language model to complete the following: [INSTRUCTION]”). Each instruction is declined as a fully-specified instruction: a single utterance that defines the instruction in its entirety, and a sharded instruction: a set of 3-8 “shards” that taken together, reveal the same information as the fully-specified instruction.

For our analysis, we consider shards as atomic units of information from the original fully-specified instruction, which enables us to scrutinize which information is and isn’t revealed by a user simulator. Given a simulated conversation based on an instruction, we prompt OpenAI’s o3 model to assign which specific shard (or shards) are revealed in each user turn. Manual inspection of 5 simulated conversations revealed that o3’s mapping ability was accurate to conduct our analysis. The Intent Coverage metric is then defined as:

$$ $\text{Intent Coverage}=\frac{len(set(\text{revealed_shards}))}{\#\text{ shards}},$ (1) $$

in other words, intent coverage measures the fraction of shards that were revealed at least once during simulation.

Sharded instructions have additional metadata attached to each shard, namely whether the shard is required or not to solve the task. This Boolean tag enables us to analyze which parts of an instruction they focus on. First, we calculate:

$$ $\texttt{Repeat Required}=\begin{cases}0&\text{if all required shards are revealed at most once,}\\ 1&\text{otherwise (i.e., at least one required shard is repeated 2+ times)}\end{cases}$ (2) $$

In other words, this measures the percentage of conversations in which the user simulator highlighted important information by repeating it across turns.

We also compute:

$$ $\texttt{Skip Non-Required}=\begin{cases}0&\text{if all the non-required shards are revealed,}\\ 1&\text{otherwise (i.e., at least one non-required shard is ommitted)}\end{cases}$ (3) $$

which measures whether the user simulator is selectively picking secondary information to skip over during simulation. Non-required shards are typically obvious information (such as a trivial base case or a clarifying example).

Finally, we used o3 to identify any information introduced by the simulator in the user utterances that was not initially present in any of the shards. This information is then used to compute:

$$ $\texttt{Additional Demands}=\begin{cases}0&\text{if no novel information is in the simulated conversation},\\ 1&\text{otherwise (i.e., 1+ user utterance introduces novel information)}\end{cases}$ (4) $$

In Section [4](https://arxiv.org/html/2510.06552v1#S4), we perform qualitative analysis of the additional demands introduced by user simulators, which are categorized into three main types: providing examples, defining naming conventions for functions or variables, and adding implementation or algorithmic constraints.

The above-defined metrics are all computed for each individual simulated conversation and then averaged across each user simulator. In Table [3](https://arxiv.org/html/2510.06552v1#S4.T3), each result therefore represents the aggregation of 650 simulations (65 instructions, and 10 simulations per instruction).

The next metrics we define require multiple simulations for a given instruction: given that the simulators are probabilistic in nature (i.e., language models run at temperature $T=1.0$), each simulation diverges and is unique. We simulate a total of 10 conversations ($C_{1},C_{2},\dots C_{10}$) for each instruction for each simulator, and compute the following metrics based on the set of simulations obtained for each instruction:

$$ $\text{Turn Variance}=var_{i=1}^{10}(\#turns(C_{i})),$ (5) $$

where $var(X)$ is the empirical estimate of variance of the series, and #turns is the number of conversational turns (user-assistant exchanges) in the simulated conversation. We also report the range of the number of turns, to help elucidate whether an increase in variance comes from the lower or upper range of conversation turns. The range is computed as:

$$ $\text{Turn Range}=[min(\#turns(C_{i})),max(\#turns(C_{i}))],$ (6) $$

for each instruction. We compute lexical diversity following prior work Stasaski et al. (2020); Tevet & Berant (2020) by calculating pairwise-lexical overlap of unigrams:

$$ $\text{Unigram Overlap}=\frac{1}{10C2}(\sum_{i\neq j}\frac{|intersection(C_{i},C_{j})|}{|union(C_{i},C_{j})|}),$ (7) $$

In more specific detail, for each pair of simulated conversations on a common instruction, we extract the lemmatized unigrams of all user utterances and measure the Jaccard index. We average this across all pairs of simulated conversations, effectively measuring the average overlap in wording across two simulated conversations based on the same instruction. We then calculate the complement:

$$ $\text{Unigram Difference}=1-\text{Unigram Overlap},$ (8) $$

which assesses the simulator’s ability to use unique wording in each conducted simulation.

Turn variance, turn range, and lexical diversity are computed for each instruction (based on 10 simulations), and aggregated across instructions.

As a final metric, we assess whether the assistant successfully completes the task (i.e., generates Python code that solves held-out unit tests or finds the correct mathematical solution to the problem), which is computed as:

$$ $\text{Assistant Score}=\begin{cases}1&\text{if an assistant solution at any turn passes evaluation,}\\ 0&\text{otherwise (i.e., none of the attempted solutions are correct).}\end{cases}$ (9) $$

This final metric does not directly evaluate the user simulator, but instead measures the effect of the simulator on the downstream performance of the assistant (GPT-4o), giving us insight into the impact that the user simulator has on the task-solving ability of the assistant.

### C.4 Simulation Results Per-Task

**Table 5: Summary of evaluation metrics for simulated conversations across two tasks types (math and code) with three user simulators (prompted assistant LMs and trained user LMs). Each simulator is evaluated on intent coverage, information diversity, conversational pace, lexical diversity, and downstream assistant performance.**
|  | Task: Math | Task: Code |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- |
|  | \cellcolor[HTML]f2f2fd4o-mini | \cellcolor[HTML]f2f2fdGPT-4o | \cellcolor[HTML]fffaebUserLM-8b | \cellcolor[HTML]f2f2fd4o-mini | \cellcolor[HTML]f2f2fdGPT-4o | \cellcolor[HTML]fffaebUserLM-8b |
| Metric | \cellcolor[HTML]F6F6F6 Intent Coverage |  |  |  |  |  |
| Intent Coverage (%) | 95.1 | 97.1 | 76.5 | 82.8 | 79.1 | 76.6 |
|  | \cellcolor[HTML]F6F6F6 Information Diversity |  |  |  |  |  |
| %Repeat Required | 14.5 | 44.0 | 3.0 | 39.6 | 12.2 | 58.9 |
| %Skip Non-Required | – | – | – | 10.9 | 14.6 | 37.7 |
| %Add Demands | 10.0 | 0.5 | 40.2 | 9.3 | 1.1 | 45.5 |
|  | \cellcolor[HTML]F6F6F6 Pace Diversity |  |  |  |  |  |
| Turn Variance | 0.4 | 0.1 | 3.4 | 1.2 | 0.9 | 2.6 |
| Turn Range | 4.0-5.3 | 5.3-5.4 | 2.0-6.8 | 3.5-5.9 | 3.5-5.4 | 2.2-6.7 |
|  | \cellcolor[HTML]F6F6F6 Lexical Diversity |  |  |  |  |  |
| Unigram Difference | 0.32 | 0.30 | 0.68 | 0.47 | 0.43 | 0.72 |
|  | \cellcolor[HTML]F6F6F6 Assistant Performance |  |  |  |  |  |
| Assistant Score | 0.83 | 0.85 | 0.46 | 0.69 | 0.70 | 0.63 |

Our main results presented in Section [3](https://arxiv.org/html/2510.06552v1#S4.T3) aggregate metric scores across the coding and math solving tasks to give a sense of the simulator behavior in general. We show the results for each of those tasks in Table [5](https://arxiv.org/html/2510.06552v1#A3.T5). We note that for the math instructions, all shards were required, rendering the calculation of Skip Non-Required not possible.

Overall, we observe similar trends across the two tasks. Some noteworthy differences: on math tasks, the GPT-based user simulators stick to the script more strictly, with almost 100% coverage of underlying shards, almost no turn variance, and a higher gap in observed assistant performance with the UserLM-8b. On the other hand, all simulators exhibit a little more diversity in their responses in the coding tasks, with better lexical and turn diversity.

## Appendix D Prompts

#### Intent Generation for Conversations.

Figure [8](https://arxiv.org/html/2510.06552v1#A4.F8) shows the prompt we used to generate a generic user intent for each conversation in Wildchat and PRISM datasets. We provide GPT-4o with the full conversation between the user and the assistant and instruct it to create a short summary of the high-level intent of the user from the conversation, without mentioning any specific details. We provide 3-shot examples where we manually wrote the generic user intent from the conversation.

#### User Simulation with Assistant LMs.

Figure [9](https://arxiv.org/html/2510.06552v1#A4.F9) shows the prompt we used to simulate users with assistant LMs, which we based on how recent studies have done it Chang et al. (2025); Ivey et al. (2024). The assistant is given the generic intent of the user and instructed to generate the user utterance at the first turn and subsequent turns given the conversation history so far. We further engineered the prompt to provide additional instructions on the desired behavior of user utterances (i.e., intent decomposition, making occasional typos, not writing too verbose utterances, etc.), without overdoing these phenomena.

#### LM-as-Judge for Intent Adherence.

Figure [10](https://arxiv.org/html/2510.06552v1#A4.F10) shows the prompt we used to classify if the user simulator successfully adhered to its intent in our intent adherence evaluation. We provide GPT-4o with the user simulator question to the assistant, the assistant’s refusal to provide the answer and suggestion of looking at something else, and the user simulator’s reply. Based on this conversation history, we instruct GPT-4o to classify if the user simulator stuck to its intent by reiterating what they asked in their first turn, or whether they complied with he suggestion of the assistant.

Figure: Figure 8: Prompt template used to generate generic user intents from conversations with GPT-4o. We give GPT-4o three hand-engineered examples (assistant turns shortened for brevity).

Figure: Figure 9: Role-playing prompt template used to instruct assistant LMs to generate the next user turn.

Figure: Figure 10: Prompt used to judge if a simulator sticks to its intent in our intent adherence evaluation.

## Peer Reviews

confidence: 4
contribution: 2
initial_rating: 6
presentation: 3
questions: 1.	How exactly is the 
 mechanism implemented across models? As mentioned in the paper, is it merely introduced through prompts? For UserLM, was this mechanism explicitly annotated during training? The notable improvements in Table 2 suggest further clarification.
2.	The paper claims that abstract rather than specific intent extraction is designed to avoid simple memorization by LLMs, yet this abstraction could harm intent fidelity. How is the correctness of extracted intents measured or validated?
3.	Are the datasets used for evaluation in Section 3 the same as those introduced in Section 2?
4.	From a writing perspective, the introduction mentions that only two prior works exist in this area (two citations). However, the related work section shows that many studies are indeed relevant. These should be cited earlier, along with a clear positioning of this paper’s novelty. Furthermore, while the analysis is extensive, it lacks a clear structure and logical flow; reorganizing the sections would greatly improve readability.
soundness: 2
strengths: 1. The authors focus on a critical yet underexplored task (user model construction) and propose effective methods that achieve promising results.
2. The authors provide comprehensive and fine-grained analyses, offering valuable insights for future user simulator research.
3. The authors introduce diverse evaluation metrics tailored to user simulators, which can serve as useful references for subsequent evaluation frameworks.
summary: The paper explores an alternative perspective in large language model (LLM) training by focusing on the user simulator rather than the dialogue model itself. To address the challenge of maintaining user intent across multi-turn conversations, the authors propose an intent extraction and training method. Experimental results on multiple datasets (such as WildChat and PRISM) show consistent improvements over baselines and better align with human behavior by measuring multiple metrics.
weaknesses: 1. A major concern is that the “FLIPPING the Dialogue” paradigm has already been partially explored in prior works (e.g., USP), which weakens the originality claim. The paper should clearly articulate how it differs from existing approaches.
2. The correctness of extracted intents used for training has not been validated, potentially introducing bias in both model performance and evaluation (e.g., UserLM is trained and tested on WildChat).
3. The baseline comparison is limited since only USP is included, while prior role-playing and persona-based approaches are not discussed. As a result, it is unclear whether the observed gains stem from the proposed methodology itself or simply from incorporating user profiles (in whatever form). Moreover, several metrics (e.g., Table 3) are model-specific, lacking objective comparability.
4. The evaluation relies heavily on LLM-based automatic assessments. Some degree of human evaluation should be included to validate the reliability of the results.

---

confidence: 4
contribution: 3
initial_rating: 8
presentation: 4
questions: 1. I am surprised to see USP-8b performing that badly, being worse than Llama3.2-1b-Instruct by PPL. It also seems that USP has more sophisticated training procedures. Why do you think it is that bad?

2. Do guardrails described in Appendix D.1 affect Table 1 results? Or are these guardrails only for the extrinsic simulations?
soundness: 3
strengths: The topic is very timely: LLM-based user simulators are needed for offline evaluation of chatbot applications everywhere.

The implementation is very straightforward, which is a good thing in that case, and it is easily reproducible. Other works do almost the same (USP), but this work seems to be both better and simpler in terms of the general design.

The paper presents strong empirical evidence that assistant LMs are poor user simulators; User LMs substantially reduce PPL on PRISM and WildChat and improve multi-turn metrics. Some metric choices are nice, for instance, using the Pangram for naturalness. Conclusions are helpful: base models are better than instruct models; 8B is better than 1B, so scaling works in that case.

The presentation is wonderful, all tables and figures are clean, readable, and well-thought-out.
summary: The paper presents 1B and 8B models fine-tuned on the WildChat dataset, where the user utterances were unmasked and the user intent was put as a system message. These models can be used for simulating user utterances in multi-turn conversations.

Other details:
- The intent was inferred with GPT-4o in a few-shot mode.
- The base models were  Llama3-8b-Base and Llama3.2-1b-Base. Starting from the instruction models is worse.
- It was full fine-tuning on 4xA6000.

The main evaluation was by perplexity on the test set (WildChat) and on the OOD test set (PRISM)

Other metrics: first-turn diversity, intent decomposition across turns, dialogue termination, naturalness (Pangram AI-detector), and robustness (role adherence and intent adherence).

User LMs outperform prompted assistant simulators (including GPT‑4o roleplay) and a fine-tuned baseline (USP‑8b), with large gaps in metrics.

Extrinsic evaluations simulate math and coding tasks where UserLM‑8b yields more realistic, varied user behavior and reduces downstream assistant (GPT‑4o) task success from 75% to 57%, suggesting current assistants struggle more under realistic multi-turn conditions.

The authors claim the model will be publicly available for research purposes.
weaknesses: 1. Extrinsic simulations introduce guardrails applied only to UserLM‑8b (described in Appendix D.1), which gives unfair advantage relative to GPT-based simulators; a fairness ablation is missing. 

2. The task scope for extrinsic simulations is very narrow (math/coding) and English-only; generalization to other domains, modalities, and languages is not evaluated. In general, math and code seem like a strange choice.

---

confidence: 4
contribution: 2
initial_rating: 2
presentation: 3
questions: Please see the Weaknesses section.
soundness: 2
strengths: - This casts doubt on current practice and suggests the need for a user simulator better aligned with real users.
- Extensive experiments show that the proposed post-trained user simulator better captures the properties targeted by the designed metrics.
summary: This paper argues that assistant LMs, which are post-trained to produce exhaustive, unambiguous replies, do not faithfully mimic messy, indirect human user behavior in multi-turn dialogues. It shows that using assistant LMs as user simulators is flawed; stronger assistants make worse simulators. This paper propose “User LMs,” post-trained specifically to simulate human users, and report that these models align better with real user behavior and yield more robust simulations.
weaknesses: Evaluation Metrics
* While mimicking user behavior can be useful for building a user simulator, it is not clear that the properties highlighted as important for user behavior are actually critical for evaluating an LM assistant.

 * The results in Table 1 may offer limited insight: a model trained on conversational data should have lower perplexity on test sets from the same distribution (WildChat) or similar data (PRISM).

 * An intent (e.g., ‘You are a user chatting with an assistant language model … medications on weight gain.’) provides strong topical context. Given such a prompt, the conversation will predictably be steered toward that topic, which in turn reduces perplexity.

* Some evaluation metrics need stronger justification, as they do not appear to faithfully capture what is required for realistic user simulation.
    * There are many potential, confounding reasons for lower assistant performance. Is it reasonable, therefore, to conclude that lower performance indicates the user simulator is better aligned with real users?
    * Why is the simulator’s ability to end the conversation considered so important for evaluating multi-turn QA with LLMs?
    * Since the user’s response depends on the assistant’s response, is it appropriate to compute turn variance without accounting for this dependency?
    * The claim in Intent Decomposition section that “a lower overlap is particularly desirable because it indicates that the model expresses its intent using varied language while introducing details progressively” is not substantiated.

    * The experimental setup feels somewhat artificial. The methods for evaluating User-role adherence and Intent adherence seem overly narrow, specific, and somewhat unlikely to occur in practice.

Real-User Alignment
* What exactly constitutes a “real user” in this work? Was any human study conducted to validate that definition?
* As users become more accustomed to LLM chatbots and adapt their behavior to use them effectively, is a new human study needed to compare the proposed simulators with up-to-date real user behavior?
* What is the practical value of building the user simulator? Do you have empirical evidence showing that it improves the evaluation of LLM performance or leads to better model outcomes?

---

confidence: 3
contribution: 2
initial_rating: 2
presentation: 2
questions: 1. What are the findings that are not available in other user simulation approaches but that UserLM enables?

2. Does an assistant LM performing worse against UserLM suffice to conclude that UserLM-based simulation offers better evaluation? How can we rule out that UserLM is just confusing the assistant LM?

3. Were there any additional checks of data leakage into the test sets?
soundness: 1
strengths: 1. This paper addresses an important problem of improving and making more realistic user simulations for chat-based LM evaluation.

2. The proposed approach, termed UserLM, provides better user simulation that matches human responses in various aspects, whereas previous prompt-based approaches fail.
summary: This paper proposes UserLM to evaluate LM performance in realistic multi-turn conversation settings. The authors train UserLMs on flipped dialogues between users and LMs to simulate users, given user intents. The experiments show that UserLMs' simulations better match actual human responses and remain within the user role. As a result, the performance of assistant LMs is lower against UserLMs than against prompt-based baselines.
weaknesses: 1. The paper does not provide evidence that better user simulation leads to better assistant LM evaluation. It seems the assistant LM performs worse against UserLM. However, it is not clear whether this is because real user requests are more challenging or because UserLM has some unexpected traits that confuse the assistant. Moreover, this does not mean that assistant LM evaluation based on UserLM offers better representation of the assistant's quality.

2. The WildChat dataset seems to contain many almost-duplicates that are not necessarily from the same IP addresses. This may be due to popular prompts shared by many users or to a single user using multiple IP addresses. Without careful data splitting to account for this, there may be data leakage into the test sets.


Decision: Accept (Poster)