CollabLLM: From Passive Responders to Active Collaborators
Authors: Shirley Wu
Michel Galley
Baolin Peng
Hao Cheng
Gavin Li
Yao Dou
Weixin Cai
James Zou
Jure Leskovec
Jianfeng Gao
Venue: ICML 2025 oral (2025)
Keywords: Human-centered Large Language Model
Multiturn Interaction
Collaborative Problem-Solving
Reinforcement Learning
Primary area: social_aspects->alignment
TL;DR: CollabLLM is a unified fine-tuning framework that optimizes LLMs for effective and efficient multiturn collaboration with users.

Abstract: Large Language Models are typically trained with next-turn rewards, limiting their ability to optimize for long-term interaction. As a result, they often respond passively to ambiguous or open-ended user requests, failing to help users reach their ultimate intents and leading to inefficient conversations. To address these limitations, we introduce CollabLLM, a novel and general training framework that enhances multiturn human-LLM collaboration. Its key innovation is a collaborative simulation that estimates the long-term contribution of responses
using Multiturn-aware Rewards. By reinforcement fine-tuning these rewards, CollabLLM goes beyond responding to user requests, and actively uncovers user intent and offers insightful suggestions—a key step towards more human-centered AI. We also devise a multiturn interaction benchmark with three challenging tasks such as document creation. CollabLLM significantly outperforms our baselines with averages of 18.5% higher task performance and 46.3% improved interactivity by LLM judges. Finally, we conduct a large user study with 201 judges, where CollabLLM increases user satisfaction by 17.6% and reduces user spent time by 10.4%.

COLLABLLM: From Passive Responders to Active Collaborators
Shirley Wu
Michel Galley? 2 Baolin Peng? 2 Hao Cheng Gavin Li Yao Dou Weixin Cai
James Zou't Jure Leskovee' Jianfeng Gao
httt://
Context state (x)
Response y
need write about optimism can improue our well-being.
To get us started. what kind of tone are you aiming for?
3
Collaborative Simulation
Forward Sampling
Reward Computation
#3
y)
Extrinsic Reward
Multiturn-aware
Parformance
Reward
Intrunsic Roward
RL finetuning
Real-world or Simulated User
Policy TT
Online generation
Figure I: COLLABLLM Framework: Given a context (), the model generates response 2 to maximize long-term collab- oration gains, termed Multiurn-aware Rewards (MR). During training, MRs are estimated via a collaborative simulation, which forward-samples conversations with simulated users. Finally, reinforcement fine-tuning is applied using the MRs.
Abstract
improved interactivity by LLM judges. Finally, we conduct large user study with 201 judges, where COLLABLLM increases user satisfaction by 17.6% and reduces user spent time by 10.4%
Large Language Models are typically trained with next-turn rewards, limiting their ability to optimize for long-term interaction. As a result, they often respond passively to ambiguous or open-ended user requests, failing to help users reach their ultimate intents and leading to inef- ficient conversations. To address these limita- tions, introduce COLLABLLM, a novel and general training framework that enhances mul- titurn human-LLM collaboration. Its key in- novation a collaborative simulation that es- timates the long-term contribution of responses using Multiturn-aware Rewards. By reinforce- ment fine-tuning these rewards, COLLABLLM goes beyond responding to user requests, and ac- tively uncovers user intent and offers insightful suggestions- a key step towards more human- centered AI. We also devise a multiturn interac- tion benchmark with three challenging tasks such as document creation. COLLABLLM signifi- cantly outperforms our baselines with averages of 18.5% higher task performance and 46.3%
1. Introduction
Modern Large Language Models (LLMs) excel at gener- ating high-quality single-turn responses when given well- specified inputs. However, real-world users often do not fully articulate their intents and sometimes initiate con- versations with an imprecise understanding of their own needs (Taylor, 1968). As a result, users routinely refine their requests post hoc through iterative corrections, which can increase frustration, hinder effective task completion, and reduce conversational efficiency (Amershi al., 2019; Zamfirescu-Pereiiz et al., 2023; Wang al., 2024; Kim et al., 2024). Therefore, an open problem is to train mod- els that actively guide users in clarifying and refining their intents, and helps them achieve their goals. This key chal- lenge would improve user satisfaction and efficiency and streamline human-LLM interactions-espccinn as L.LMs are being applied to world tasks that are increasingly complex and open-ended.
'Stanford University "Microsoft GGoorgia Tech. Correspon- dence to: <shirwu @ertwu@ssaatin @stanford.eduz, <mgalley @ewcrosoft.com..........
A notable limitation of established fine-tuning techniques, such as Reinforcement Learning from Human Feedback (RLHF) (Ouyang et al., 2022), that they primarily re- ward LLMs for immediate, single-turn responses, reducing their incentive to seek clarification or assist users in refin-
Proceedings of the 1and International Conference Machine Learning, Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

COLLABLLM: From Passive Responders to Active Collaborators
f1sk-1
Training
Esalaation
Single-tur Rewards ds
Preference/ / Reward Dataset
Pollcy Model T
Hlstory
Current user turn
Collatoratize
Simulation
R
Current model turn
Finetuning
Future turns
Ineffective Collab.
Effective Collab.
Ineed about sptinisr can imgrove well-being
|neeed about how optimisn can imgrove our sell-being
To uet staried, what kind of tone are you aiming for? A. Lpsitiing B. Inspiring Hearttett and personal
The Power Optiriim: Untesking Brighter (Many tokens)
Real-world Deployment
Abo you want to hightghht role in resilience or relationships?
The lone is too fornal and osanples too old-school
siming for Touching resiience and hos helps reoelometipp..uo wouild great) like to propase an opening paragraph set
Hing there, rriendl fre Unleeshing
the aptimian raakition?
(Mang tokens)
add anecdites personal experience
aph)
impacttal
(Many bock-and-forth turns)
(Reach user satisfaction quickig)
Eualuation
Frustrating
Low Efficiency tokerna read)
Very helpfull
High Efficiency
Lons Qualitty (SLEU000 30)
inttrrecttuttyii judge
2 High Quality
High Interactiisty (LLM Judge score-0.8)
(a) Non-collaborattue LLM
(b) Ours: CollabLLM
Figure 2: Real examples from COLLABLLM and non-collaborative LLMs. (a) Non-collaborative LLM fine-tuing relies single-turn rewards on immediate responses, which exhibits passive behaviors that follow the user's requests, leading to user frustration, less efficient process, and less satisfactory results. (b) COLLABLLM incorporates Multiturn-aware Re- wards from collaborative simulation, enabling forward-looking strategies. This results in more high-performing, efficient, and interactive conversations that anticipate future needs, propose timely clarification, and provide insightful suggestions.
ing their intents or preferences. As a result, commonly used LLMs tend to prioritize direct answers, even though seek- ing additional context would enhance task completion and increase user satisfaction (Kim et al., 2024).
ing. On the three test sets, our approach improves task ac- curacy metrics by 18.5% and interactivity by 46.3% on av- crage compared to our best baselines, according to LLM judges. Beyond the tasks that the COLLABLLMs are fine- tuned on, we show COLLABLLMs are highly generalizable to other data domains.
Here we introduce COLLABLLM, a novel and general training framework that improves the ability of LLMs to effectively collaborate with humans in multiturn scenar- ios (Gao et al., 2019; Balog & Zhai, 2023; Rahmani et al., 2023). The key innovation of COLLABLLM is to pro- mote LLMs' forward-looking behavior that leads to long- term collaboration gains (Figure 1). We introduce a col- laborative simulation module that samples future conversa- tions with users to estimate the long-term impact of model responses across multiple turns, measure we term the Multiturn-aware Reward (MR). The MR function evalu- ates responses by incorporating both extrinsic metrics, such as task-specific success, and intrinsic metrics, such as effi- ciency, holistically assess collaboration quality (cf. Sec- tion 3). By fine-tuning with RL. algorithms (Rafailov et al., 2023; Schulman al., 2017) MRs, COLLABLLM pro- motes responses that lead to better task completion and effi- ciency in later conversation stages. As shown in Figure 2b, the fine-tuned model goes beyond simply responding to user requests in Figure 2t-it actively collaborates by ask- ing follow-up questions about the writing tone, generating targeted content about the role of optimism, and offering insightful suggestions such adding anecdotes.
Moreover, we perform a large-scale and real-world user study with 201 Amazon Mechanical Turkers (MTurk- ers), who are asked to write documents with the help of anonymous AI assistants, either COLLABLLM or non- collaboratively trained LLMs. COLLABLLM achieves im- pressive improvement with 17.6% increase in user satisfac- tion and yield user time savings of 10.4% on average. The qualitative analysis from MTurkees confirms our observa- tions: non-collaboratively -triined LLMs passively agree with users, while COLLABLLM actively provide insight- ful questions and suggestions to guide writing processes.
2. Problem Formulation
In contrast to many existing tasks that are single-turn and require no human involvement beyond the initial query, our problem formulation reflects a real-world setting in which a user's 's underlying (implicit) goal is defined as g in a mul- titurn conversational task. The conversation unfolds over multiple turns t, {uz, m where uj is the user input and m, is the model's response at each turn j = 1...., K, where K is the number of turns in the conversation.
We also introduce three challenging multiturn tasks for training and evaluation in simulated environments: BigCodeBench-Cheee.. and ChTH- which respectively encompass document creation, code generation, and multiturn question answer-
At the j-th turn, the model generates its response based on the previous conversation turns 41g-11 = -4,,..,, and the current user response uy. For simplicity, we define historical conversation at j-th turn as th f1--11 U fuy).
2

---

COLLABLLM: From Passive Responders to Active Collaborators
therefore, my M(1)). The objective is to generate a se- quence of model responses {mz}}=1 that effectively and efficiently achieve for goal g. e.g., answering a math ques- tion, where goal achievement is assessed based on user sat- isfaction or external evaluation function, such as accu- racy by LLM judge. Formally, we define the objective as R" (I g); where R' incorporate the achievement of task success and user experience factors such as time cost.
approach. More specifically, we define the Multiumn-awree Reward:
Multiturn-aware Reward (MR): The multiturn- aware re- ward for model response my at the j-th turn is given by:
MR(m; ()).))
= -E - Plt (1) U Um; ( (1)
-Ba--Pu/ Rº (tus Utz 19),
3. Unified Collaborative LLM Training
where bis denotes the conversation history up to and in- cluding the j-th turn, and tf = 1z41.K represents the for- ward trajectory of turns following the j-th turn. The distri- bution P(t | tug) models the possible forward conversa- tions conditioned the prior conversation history.
Key Motivations. Established LLM training frameworks, such as Reinforcement Learning from Human Feedback (RLHF) (Ouyang et al., 2022), focus on maximizing imme- diate rewards for single-turn tasks. This cause misalign- ment between their single-turn objective and real-world multiturn objective R (t1:K g). Precisely, the model's accumulative single-turn reward Σ/= R(m; t²)) may not imply& a higher final reward R² (h1: g)- In fact, achieving high single-turn rewards at each turn may not imply a higher final reward. For example, consider a task where the user's goal g is to write an engaging article. A model trained with traditional RLHF might generate iso- lated responses, like drafting introduction or listing con- clusions. While these responses are helpful in isolation, they fail to consider how the sections flow together, result- ing in an article that might not be cohesive and aligned with the user's goal.
However, computing Equation I remains challenging as it requires the following components: (a) A conversation- level reward function, R (t g), for evaluating an arbi- trary multiturn conversation i, and (b) a sampling strategy for obtaining forward conversations P(t, fi:g), which represents the forward conversation distribution. We elab- orate on the two components in Section 3.1.1 and 3.1.2.
1.1.1. CONVERS REWARD FUNCTION
We approximate the conversation-level reward R² g) with a combination of extrinsic (goal-specific) and intrinsic (goal-agnostic) metrics:
Instead, effective multiturn collaboration requires model responses that optimally contribute to the final reward. The model should aim to align its responses with the user's s goal g by considering their impact on the entire conversation tra- jectory ti In the previous example, instead of generating a conclusion, asking, "Should I maintain an engaging tone in the conclusion like the introduction?" ? offers better long- term alignment with the goal.
Rt g) ~ gt, + (tn(t);
where Rea(t, g) focuses task success, and (im(() (t) evalu- ates user experience including efficiency and engagement.
Extrinsic Reward Rea(t,g) measures how well the conversation achieves the user's goal g. Formally:
Rea Rea 9) = S(Extract(t), Ug),
3.1. Multiturn-aware Rewards
where Extract(t) extracts the final solution or response from the conversation 4, especially for tasks requiring revisions or multi-step answers. Yg is the reference so- lution for the goal g, e,g., the ground truth solution for a math problem. And S(..)) evaluates task-specific met- rics like accuracy or similarity. This ensures the conver- sation contributes directly to achieving the desired goal.
In Figure 1, our key insight is that effective multiturn col- laboration relies on forward-looking strategies. Given context D, the model should consider how its response @ influences the subsequent turns of the conversation. To cap- ture this, we design a ② collaborative simulation module to estimate this impact. By ④ fine-tuning to distinguish between potential future conversations resulting from dif- ferent responses, the model generates responses that align better with the overarching goal g.
Intrinsic Reward (m)(t) prioritizes conversations that enhance user experience, defined as:
This high-level design naturally aligns with causal effect estimation (Pearl, 2009; Pearl et al., 2016), which eval- uates the interventional effects of an action in sequential decision-making. Appendix A provides further discussion on the connection between causal effect estimation and our
(mt(t) min TokenCount(t), 1] RLLM(t), (4)
where we encourage conversational efficiency by penal- izing excessive tokens that users read and write, with A controlling the penalty severity. This efficiency measure is bounded by 1 to maintain balance with other metrics.
3

---

COLLABLLM: From Passive Responders to Active Collaborators
The second term, ((), is assigned by an LLM-based judge (Zheng et al., 2023) on a 0-1 scale, evaluating user-valued objectives such as engagement interactiv- ity. Notably, additional conversational aspects, such as clarity, can be further integrated into the objective.
such as PPO (Schulman et al., 2017) and DPO (Rafailov et al., 2023) to guide the model in navigating complex conversations. By optimizing for higher MR, the model learns to generate responses that enhance overall effective- ness and efficiency by the end of the conversation.
Moreover, MR can generate high-quality synthetic con- versations (cf. Figure in Appendix B) for both supervised fine-tuning (SFT) and DPO. For SFT, it iteratively selects top-ranked responses to build realistic, goal-directed con- versation histories. For DPO, it constructs pairwise com- parisons by ranking responses at each turn, distinguishing "chosen" and "rejected" pairs based on MR scores. The generated synthetic data aligns with multiturn objectives.
The conversation-level reward incorporates task-specific and human- metrics, encouraging the model to bal- ance goal achievement, efficiency, and engagement.
3.1.2. FORWARD SAMPLING
To compute Eq. 1, we require samples from P(I (1s)), the distribution of forward conversation conditioned on the conversation history.
simple approach to use Monte Carlo sampling, where the conversation is extended turn- by-turn until it concludes. However, this can be computa- tionally expensive for computing reward for every model response. For a scalable approximation, we introduce a window size w as a hyperparameter to limit the maximum number of forward turns considered in ty. This reduces the computational cost while maintaining sufficient context.
Overall, COLLABLLM enables scalable dataset generation and online RL training without human annotation, mak- ing it generalizable across diverse tasks. In Appendix 7, we compare COLLABLLM with related prompting- and training-based approaches, highlighting its contributions.
4. Experimental Setup"
More importantly, while real-world conversations could be gathered from human participants, sampling multiple for- ward conversations during training is costly and impracti- cal. To further reduce cost and ensure scalability, we intro- duce a user simulator U.
For fine-tuning and evaluation, we create three multiturn datasets using publicly available data across diverse do- mains (Hendrycks et al., 2021; Zhuo et al., 2024; Chiusano, 2024): collaborative document editing, coding problem as- sistance, and multiturn mathematics problem solving.
User Simulator:. user simulator U T - U is a func- tion that maps given conversation history t T to a user response E U. Specifically, U generates a probabilistic distribution P(u t) over possible user responses condi- tioned on the conversation history i, simulating realistic user behavior.
To build a multiturn environment (Figure 3), we employ GPT-4o-miní as a user simulator LLM to role-play realistic user behaviors, given the target problem and conversation history. Our simulation-based evaluations designed to closely mimic real-world interactions (Park et al., 2024). Unlike traditional single-turn tasks, our setup requires dy- namic interactions over multiple turns to achieving goal. The three interactive datasets are:
Specifically, we prompt an LLM to role-play as users, ex- plicitly asking the LLM follow the same language style as the previous user turns, and injecting typical user be- haviors. The user simulator operates with an implicit goal g, which it seeks to achieve over the course of the conver- sation. This design emulates real-world scenarios where users may have evolving needs, limited background knowl- edge, or require clarification, resulting in naturally unfold- ing multiturn conversations (Park al., 2024).
Medi uwDocEdit-Cheeeeeeeeeeea Document editing requires it- erative feedback and refinements across multiple turns to ensure coherence and alignment with user intent. We sam- ple 100 Medium articles as goal documents, which are summarized into target problems to guide the user simu- lator. After each interaction, task performance is evaluated using the BLEU score, measuring similarity between the extracted document and the original articles.
3.2. Optimization Synthetic Datasets
BigCodeBench-Chat: Coding tasks inherently require multiturn interactions, such clarifying requirements and debugging. We sample 600 coding problems from Big- CodeBench (Zhuo et al., 2024) as the target problems given to the user simulator. For evaluation, we compute the aver- age Pass Rate (PR) of code at the end of the interactions.
With the conversation-level reward function and forward sampling strategy, we can compute MR for any model response without requiring additional reward model, which is often costly and slow to train. Unlike traditional single-turn reward approaches, MR explicitly accounts for the impact of a response on future conversations, promot- ing long-term collaboration.
DDatasct and training details Appendix B; all prompts (e.g. prompts of user simulator and LLM judges) in Appendix D.
Further, we employ reinforcement learning (RL) methods
4

---

COLLABLLM: From Passive Responders to Active Collaborators
User Simulator
Document Editing
Coding Assistance
Mathematics Problem soluing
playing u2 coler
Target problem:
Target problem:
Target problem:
need write an article (less than 500 words) about Embrace optimism better health reailience, Share and celebrate lites beauty. Connect. grow, and enjoy the journey together.
Replace certain uslues in Datairame with a dictionary mapping and calculate the Pearson correlation confficient betwween
point rectangular coordinates (2-1,-2)
and spherical coondinates (2,,0)..
Given infomation:
<Carrent chat. statory>
<Target prab.aro
columns. detquirements on about data handling and exceptions>
point with spherical coordinates (o.9.46)
the rectangular coordinates
Deer guidelnea::
Can you help me urite something optinism you feel better?
need to code function Python that takes DataFrame dictionany to replace certain values
explain how can find
Mimtnize Retort: à neal- socld uses avoida being Loo detalled in your respataca. docasionally by Make Histakes hesi-werld usera zay nisapull sorda, ez esk unclsae questions.
courdinales when have the spheric inuiicall onis?
I'm siming for an Inspirational tone.
have pardas installed and the function
func.
Whhtt happerns uhenl double 67
coal-Oriented: Keep the your scteen. talk
Good start! Can us include more how optimiom affects prysical health?
also prefer using a 'Valuefrror when the imput Lan't a DataFrame.
Sure can glue specific example: (2-1.-2), Also, will the new decreases?
<Ostgut Format w/ cott
Figure 3: Simulated Multiturn Environment for Evaluation. Our evaluation pipeline simulates real-world collaborations by prompting an user simulator LLM to emulate diverse behaviors and personalities in multiturn conversations.
MATH-Chat: Math problem solving often requires addressing implicit assumptions, verifying intermediate steps, and clarifying reasoning. We sample 200 level-5 math problems from MATH (Hendrycks et al., 2021) to prompt the user simulator, which interacts with the LLMs. Task success is measured by the accuracy (ACC) of the final solution, as evaluated by an LLM judge.
up questions and clarifications. For example, it increases BLEU on Medi BeeeiluDDeEddttCheee from 32.2% to 35.0% and reduces read tokens by 0.31k compared to the base model. However, these gains are modest and do not fully address the challenges of multiturn collaboration. We ob- serve that prompting strategies remain rigid, relying on pre- defined instructions rather than adapting dynamically to user needs. For instance, the model sometimes asks clar- ification questions even when unnecessary, leading to re- dundant interactions that disrupt converstion flow.
In addition to the above spec-fpe metrics, we incorpo- rate two task-agnostic scores across all datasets: 1) Aver- age Token Count, which quantifies the average number of tokens generated by the LLM per conversation, reflecting interaction efficiency. 2) Interactivity (ITR), which eval- uates engagement levels using LLM judge (Claude-3.5- Sonnet), with scores rescaled to upper bound of 1.
COLLABLLM improves task performance, efficiency, and engagement. COLLABLLM achieves 18.5% superior task-specific performance, 13.3% more efficient conversa- tions, and 46.3% enhanced interactivity compared to the best baselines. We highlight that COLLABLLM engage in more meaningful collaborations, with ITR shows sub- stantial gains. For Medi umDocEdit-Chate the Online DPO model increases ITR from 0.46 to 0.92. Moreover, our framework significantly improves conversational effi- ciency by minimizing the content users need to review to arrive at the final solution. For MATH-Chat, Online DPO decreases token count per conversation by 1.03k compared to the base model.
Fine-tuning COLLABLLMS. COLLABLLMs are based on Llama-3.1 1-BB (Llama Team, 2024) with LoRA fine- tuning (Hu et al., 2022). We train four model variants: 1) Offline models: SFT and Offline DPO are fine-tuned on pre-generated multiturn conversational datasets guided by Multitum-awarc Rewards (MR) (cf. Section 3.2). 2) On- line models: PPO and Online DPO are further trained from the SFT and Offline DPO models, respectively. The model during online fine-tuning is involved in the collaborative simulation to compute MRs, which, in turn, dynamically adjust the model preference.
5.1. Ablations on Reward Mechanisms (Figure 9)
To investigate how components contribute to COL- LABLLM's superior performance, we conduct an ablation study focusing on the reward mechanisms used during fine- tuning. We evaluate the following reward mechanisms:
Baselines. We compare COLLABLLMs against (1) the pretrained Llama-3.1-88 1-BB (Base), (2) the base model with proactive prompt engineering (Proactive Base), which en- courages follow-up and clarification questions.
Variants of Multiturn-aware Reward: We vary the forward sampling window size w 1,2,3 to assess their ability to capture long-term conversational effects through simulated collaborations.
5. Results of Simulated Experiments
We present the results in Table and the takeaways are:
Prompt engineering is helpful, but limited in terms of performance gains and flexibility. Proactive Base im-
Immediate Rewards evaluate the model's immediate response based on: 1) Helpfulness: Assessed by an LLM judge; 2) Extrinsic Reward: Focuses on task-specific
proves base model performance by encouraging follow-
5

---

COLLABLLM: From Passive Responders to Active Collaborators
Table I: Evaluation results on our multiturn datasets. Green zone: Baselines; Orange zone: Variants of COLLABLLMS.
Rel. Improv. indicates the relative improvements of Coss trained with Online DPO over Proactive Base
Medi undiiimDDccce wwtooctEiitttoit Chhat
BigCodeBench-
MATH-Chat
BLEU
#Tokens(A) +
ITR
PR
#Tokens(k) 4
ITR
ACC
#Tokens(k) 4
ITR
Base
32.2
2.49
46.0
9.3
1.59
22.0
11.0
3.40
44.0
Proactive Base
35.0
2.18
62.0
11.0
1.51
33.7
12.5
2.90
46.0
SFT
35.2
2.21
68.0
11.7
1.35
42.0
13.5
2.88
58.0
PPO
38.5
2.00
78.0
14.0
1.35
40.7
13.0
2.59
52.0
Offline DPO
36.4
2.15
82.0
12.3
1.35
46.7
15.5
2.40
50.0
Online DPO
36.8
2.00
92.0
13.0
1.31
52.0
16.5
2.37
60.0
Rel. Improv.
5.14%
8.25%
48.3%
18.2%
13.2%
54.3%
32.0%
18.3%
36.4%
BLEU
#Token (k)
ITR
0.7
Immediate Reward Mechanisms
Multiturn-auad Reward Mechanisms
0.373
Helpfulness
MR 1)
2.0
2.0
0.00
Extrinsic
MR (w 2)
0.365
0.54
Extrinsic Intrinsic
MR (w 3)
0.360
Figure 4: Selected Ablation Study of Reward Mechanisms on MediumDocEEdit-Co immediate reward mechanisms with three MR variants. MR consistently improves task-specific performance (BLEU), conversational efficiency (# Tokens), and interactivity (ITR). See Appendix B.3 for the full results.
This figure compares three
metrics like BLEU while ignoring intrinsic factors such as efficiency; 3) Extrinsic + Intrinsic Reward: Com- bines task-specific metrics with efficiency and interac- tivity measures. This can be seen as a special case of the multiturn-wwan reward function with O.
proach also reduces user effort with lower token usage.
In Figure 6, we compare different reward mechanisms for responses and B of Figure 5, confirm that these re- wards work as intended. The helpfulness rewards favor response A due its seemingly more well-round output. Extrinsic rewards assign zero scores to both, as A provides incorrect solution and B defers answering. Extrinsic + Intrinsic rewards slightly favor B for efficiency and en- gagement. Interestingly, MR assigns significantly higher rewards to B, especially at w and w 3, since the response obtains useful information and provide precise answer within the future interaction window.
We present results in Figure 9. Interestingly, expanding the forward sampling window w within the range generally en- hances performance and efficiency by better capturing fu- ture interactions. Notably, MR with w = 2 balances the gains and additional costs to conduct forward sampling, making it well-suited for large-scale fine-tuning, In con- trast, immediate rewards, even with extrinsic and intrin- sic components, fall short as they ignore long-term impact. These findings validate the positive impact of the forward sampling strategy in MRs.
5.3. Model Generalization (Table 2)
Modern foundation models are expected to generalize across diverse range of tasks beyond their training do- main. A key question is whether collaborative behaviors learned by COL.ABLLIM during fine-tuning transfer effec- tively to new tasks without additional adaptation.
5.2. Case Study (Figure 5 & 6)
We now offer deeper insight into COILABLLMM's behav- ior as shown in Figure 5. In this example, the user request to tokenize a text file is inherently open-ended due to un- specified factors, such as the NLTK environment, tokenizer selection, and optional preprocessing steps. The base LLM makes several arbitrary assumptions, applying lowercase conversion and stopword removal without user confirma- tion. The user simulator later corrects these assumptions, but the final solution remains incorrect due to missing stop- words. In contrast, COLLABLLM actively clarifies user intent by secking confirmation on key decisions, ensuring an aligned final solution with 100% Pass Rate. This ap-
We assess COLLABLLM, trained with online DPO on Bi gCodeBench -hatt (the coding assistance task), on Abg-CoQA (Guo al., 2021), a question-answering (QA) benchmark where questions are labeled as ambiguous or non-ambiguous (ef. Appendix E). We categorize the model's responses into two actions a clarifying question or providing direct answer-and evaluate action- level accuracy within each question type. As shown in Ta- ble 2, GPT-4o and Llama-3.1 1-8B rarely ask clarifying ques- tions regardless of ambiguity. In contrast, COLLABLLM
6

---

COLLABLLM: From Passive Responders to Active Collaborators
Hi, need help writing function that tokenizes text file using should named 'task_ Func' and take File_path as parameter. Could you help me with that?
Non-collaborative LLM
Ours: CollabLLM
Before start, make you have NLTK instalied.
A
I'd happy help with your functioni Could you proside following:
You install uning
1. The NLTK tibrary's version you're using for the version you'uee installled) spectfic NLTK tokenizer you'd like
azatai2 niik
WordTollenzzer, RegexpTokemtzer etc.):
Confirm you want me the function to handlc any potential errers (e.g.. not found. Incorrect file path, etc.)
Here's the fundion that toleerizes a text file
from mite. tobantce
from mitt. cospux import atopucods
cohenine
Also,
remoue any punctuations the text. Please let know otherwise.
del Lark, Cos (a
hauen't I noticed what version of the NLTK I have, but installed it recently if that helps: Iwould like to use the basic word tokenization, The error handling part doesn't haue to be extensive, maybe just something simple you suggestions. Finally, you keep the punctuations as they are
teat lik the sire data wcoied for pricossinng lost
2208. dww.ioddddddddddddd "pazze")
('ssspoceda")
Open the dile 2ead mode
+ Seed the octtect of the file tact (ila.coed
Filer
a simple implementation
basic
inport
Conoeess the tact to Jccnccase
from tokenio impozt wozd, pckenize
Nisoee puretactionn
def task Fune (Eile_ ( Ley:
www.eaatttaaa aseing. I
with (fin path, '2") Elies
Tokesise teat
taxt ciie.cead()
takana waed__ckkniiee Stexts
Jezove atopucoda
ratuzn tokana
Gxoept FilaNatFoundE:zoo
tolero Ltole for tokes tokens If token
ot vonds)
print (=*Keeos: The tile (tttle_path) wae not found.")
retuen Mone
This basi example tniemiiatioon You may need tn adjust the function
time, you might need to run nick dow (punkce")) [o you need further assislance that?
you're using NLTK
your spaciftc requimments.
Figure 5: Case study on BigCode steps like punctuation and stopword removal. In contrast, COLLABLLM clarifies tokenizer preferences, error handling. and package installation, leading to solution that precisely aligns with user intent.
The colabbbbaberetive LLM assumes user needs, adding unnecessary
Immeddaae Mechanlems
R(B)
Action-level Accuracy Non-Ambiguous
Macro Metric
Helpfulness
0.96
Ambiguous
Accuracy
F1
0.00
Extrinsic
Extrinsic Intrinsic
0.80
GPT-4o
15.44%
95.60%
55.52%
56.62%
Base (Llama-3.1-88)
16.26%
90.40%
53.33%
53.31 %
Multiturn-aus Mechanisms
COLLABLLM
52.84%
72.32%
62.58%
55.08%
(v 2)
0:20
R(A)
Table 2: Zero-shot generalization to Abg-CoOA, a conversational QA benchmark identify ambiguity. We assess action-level accuracy, measuring whether the model asks question for ambiguous inputs and provides a direct answer for non-ambiguous ones.
(w 3)
0.5
0.25
0,67
0.70
Figure 6: Reward comparison for response A and B of Figure 5 shows different preferences.
proactively asks questions about 50% of the time while maintaining high accuracy on unambiguous inputs. This behavior leads the highest Macro Accuracy across both ambiguous and non-ambiguous sets and improves Macro Fl over the base model, while leaving room for further improvement against GPT-4o. These results suggest that COLLABLLM effectively generalizes its learned collab- orative strategies beyond its training domain.
of the task, they are first asked to provide brief responses to topic-related questions. Participants then engage in at least eight turns of conversation with an anonymized AI assis- tant, which can be Base, Proactive Base, COLLABLLM. Every three turns, they provide interaction rating based on their experience so far. After the conversation, partic- ipants rate the final document quality and overall interac- tion. All ratings are in a scale from to 10. We also record the total interaction duration to assess efficiency. The de- tailed user study setup provided in Appendix
6. Real-world User Study
Quantitative Results (Figure 7). Across multiple met- rics, COLLABLLM consistently outperforms the basclines. It achieves an average document quality score of 8.50. Specifically, 91.4% of participants rate COLLABLLM's document quality as "good" (score 8-9), and 56.9% as ''very good" (score 9-10), compared to 88.5% and 39.3%
Setup. We conduct large-scale user study using Ama- zon Mechanical Turk with 201 participants. Each partici- pant is assigned a document vaae-oandomly selected to be either blog post, creative writing, or personal statement- and chooses a topic from a predefined set. To simulate real-world scenarios where users have only a rough idea
7

---

COLLABLLM: From Passive Responders to Active Collaborators
3000
Aecroje
Acoraga
Proaclive Co
8.50
8.46
2500
0.11
2000
1500
12337
1319
7
1108
5
1000
6
Busc
proartive BusO ColsBLuU
Bon0
ProarSun Bast CalabAiH
Bas0
Proortive BonC Callabuli
(d) Multiturn Interaction Rating 1 Figure 7: Our real-world user study includes 201 participants interacting with an anonymized AI assistant randomly sampled from Base, Proactive Base, and COLLABLLM. Participants rate (a) document quality and (b) overall interaction experience, with additional assessments (d) every three turns. We also measure (c) user spent time to evaluate efficiency.
(a) Document Quality Rating f
(b) Interaction Rating 1
(c) Time Spent (s)
Table 3: Representative Feedback from Human Participants.
Model Base
Strengths
Weaknesses
"Follos great instruction and does exactly what I'm asking it to do.", "It can create a nice form of an outline to work with."
"The Al just agreed with me on pretty much every- thing. There was no discussion", "I didn't really like that kept coming up with different options" "The Al seemed to be very redundant and asked me the same questions over and over."
Proactive
"It is very organized and it actually asks you for feedback after writing the revision."
Base
COLLAB
"Asking questions and making you think of things
"The Al assistant was not up date enough to help with this recent sporting event. The Al assistant also asked me to repeat information I had already given it."
LLM
you never thought of", "The Al really helped me with focusing on one part of the story at a time.", "It helped really well to navigate what to say and
what information is needed"
for Base (Llama-3.1-8B) respectively. Similarly, 63.8% of participants find COLLABLLM highly engaging, while only 42.6% report the same for Llama-3.1-8B.
or repetitive in its responses. Proactive Base excels in re- sponsiveness and adapting user input but struggles with memory retention, and could produce repetitive or overly structured content. On the other hand, COLLABLLM is highly engaging, effectively guiding users through writ- ing, adapting scamlessly to feedback. However, users also point out that COLLABLLM can occasionally feel bland, lack of up to date information, and require additional ef- fort to personalize the output. Overall, COLLABLLM hances collaboration by guiding users through an interac- tive and iterative refinement process, yet future improve- ments should focus on increasing personalization, creativ- ity, and real-time knowledge integration to further optimize human-LLan LLM collaboration.
Interestingly, for multiturn interaction, the Base model shows a declining trend in ratings from turns 6-9, indi- cating reduced user experience in longer conversations. In contrast, both COLLABLLM and Proactive Base exhibit in- creasing ratings over time, with COLLABLLM consistently achieving higher average ratings every three turns com- pared to Proactive Base. This suggests that COLLABLLM maintains sustained engagement more effectively.
Moreover, COLLABLLM improves task efficiency, reduc- ing time spent by 10.4% compared to the Base model and by 15.6% relative to Proactive Base. While Proactive Base is prompted to maintain conciseness, it frequently asks un- necessary questions, causing lower efficiency. In contrast, COLLABLLM strikes a more streamlined user experience.
7. Related Work
Non-collaborative LLM training. Existing LLM training frameworks, including pre-training, supervised fine-tuning (SFT), and reinforcement learning (RL) (Rafailov et al., 2023; Schulman et al., 2017; Ouyang et al., 2022; Lee et al., 2024), primarily optimize for next-turn response quality. Standard RL methods such as Proximal Policy Optimiza- tion (PPO) (Schulman et al., 2017) apply rewards to indi- vidual model responses without accounting for their long- term impact on conversation trajectories. While effective
Qualitative Results (Table 3). We collected a total of 180 strengths and 180 weaknesses across the three models. Ta- ble presents representative feedback, while we summa- rize here the mddels' strengths and weaknesses: The base model generates coherent content while effectively follow user instructions, but it sometimes struggles with main- taining context in long texts, and can be overly verbose
8

---

COLLABLLM: From Passive Responders to Active Collaborators
Table 4: Compare COLLABLLM with Selected Works. (I) Task-Agnosta assessing whether the approach applies across diverse domains rather than being task-speeific; (2) Versatile Interaction, evaluating its ability to support diverse strategies for intent discovery and efficient task completion beyond predefined behaviors; (3) User-Centric, determining whether engagement, efficiency, and intent discovery are explicitly considered; and (4) Causal & Objective-Aligned Reward, mea- suring whether reward estimation captures causal effects future interactions and optimizes for long-term task success.
Task Aaskstgn
Versatile Interaction
User-Centric
Causal & Objective-Aligned Reward
ClarifyGPT (Mu al., 2023)
STaR-GATE (Andukuri et al., 2024)
X
MTPO (Shani et al., 2024)
COLLABLLM
for single-tur objectives, these approaches fail to capture how responses influence user intent discovery and long- term task success (Amershi et al., 2019; Zamfirescu-Pereee et al., 2023; Wang et al., 2024; Kim al., 2024).
chat or self-play to enhance model adaptation. However, these methods primarily rely on post-hoc trajectory-Ievel data, learning from observed conversations rather than explicitly modeling the causal effect of individual re- sponses on task success (see Appendix A for further explanations). Additionally, they often overlook open- ended tasks such as document generation (Faltings et al., 2023; Jiang et al., 2024), where user responses can be highly diverse, and users may have limited capacity to read and refine lengthy model outputs.
Prompting techniques for multiturn interaction. Prior work has explored prompting strategies to enhance LLM interactivity, particularly for clarification questions (Keh et al., 2024; Mu et al., 2023: Zhang & Choi, 2023; Chi et al., 2024; Kim et al., 2023; Deng et al., 2023b; Zhao & Dou, 2024) and mixed-initiative dialogues (Deng et al., 2023a; Chen et al., 2023; Liao al., 2023). For instance, Mu et al. (2023) prompt LLMs ask clarification ques- tions when code generation requests are ambiguous. How- ever, such prompting-basee based approaches are constrained by predefined interaction patterns, limiting adaptability across different tasks and conversation stages. Moreover, their re- liance on fixed prompts reduces generalization, as demon- strated in our experiments where proactive prompting fails to match the effectiveness of our fine-tuned models.
User simulators for enhancing AI systems. Recent works employ user simulators to enhance dialogue systems (Shi et al., 2019; Tseng et al., 2021) and LLMs (Hong et al., 2023; Hu et al., 2023; Faltings et al., 2023). Recently, Hong et al. (2023) leverage LLMs to create diverse synthetic dia- logues with varying user personas to train smaller dialogue models. CollabLLM differs in leveraging user simulators in forward sampling to account for long-term effect in both offline and online training.
Learning-based methods for multiturn interaction.
In Table 4, we compare COLLABLLM with related meth- ods across four key dimensions. COLLABLLM is a general, user-centric, and multiturn-aware framework that leverages more accurate reward estimation to better align with real-world objectives, enhancing user satisfaction and streamlining Numan-LL.y, interactions.
LLMs for generating clarification questions: Beyond prompting, prior studies have explored supervised fine- tuning (Andukuri et al., 2024), RL fine-tuning (Chen et al., 2024; Zamani et al., 2020; Erbacher Soulier, 2023), and active learning (Pang et al., 2024) train models to ask clarification questions. For example, Chen et al. (2024) use Direct Preference Optimization (DPO) to encourage models to request clarifications. However, like prompting approaches, these methods primarily fo- cus on clarification questions and do not generalize to broader multiturn collaboration strategies.
8. Conclusion
Multiturn human-LLM collaborations are increasingly prevalent in real-world applications. Foundation models should act as collaborators rather than passive responders, actively uncovering user intents in open-ended and com- plex tasks-an an area where current LLMs fall short. The key insight of COLLABLLM is making L.MMs more multiturn- aware by using forward sampling to estimate the long-term impact of responses. Through extensive simulated and real- world evaluations, we demonstrate that COLLABLLM is highly effective, efficient, and engaging, while also gener- alizing well to new tasks and interactions, advancing the frontiers of numan-centeree LLMs.
Multiturn training for LLMs: Recent bench- marks (Abdulhai et al., 2023; Kwan et al., 2024) evalu- ate LLMs' performance in multiturn settings, measuring the goal orientation and planning capabilities of mod- els across interactions. Several studies extend RLHF to multiturn settings by optimizing trajectory-level re- wards (Shani et al., 2024; Zhou et al., 2024; Gao et al., 2024; Shi et al., 2024b; Zhang et al., 2025). Other works (Xu et al., 2023; Deng et al., 2024) leverage self-
9

---

COLLABLLM: From Passive Responders to Active Collaborators
Acknowledgments
raise red pltetlpo providing the LLM with fur- ther cues for identifying unsafe behavior. As presented in Appendix C, we conducted various safety experiments and show that COLLABLLM performs no worse than an equiv- alent non-collaboratively trained model in terms of safety.
We thank Doug Burger, Vishal Chowder, Jeevana Priya Inala, Hoifung Poon, Swadheen Shukla, Chandan Singh, Desney Tan and Chenglong Wang, as well as members of the Deep Learning and Health Futures groups at Mi- crosoft Research for helpful discussions. We thank lab members in Leskovec and Zou's 's labs for discussions and for providing feedback. We also gratefully acknowledge the support of NSF under Nos. OAC-1835598 (CINES), CCF-1918940 (Expeditions), DMS-2327709 (THBEM), IIS-2403318 (III); Stanford Data Applications Initiative, Wu Tsai Neurosciences Institute, Stanford Institute for Human- Centered AI, Chan Zuckerberg Initiative, Amazon, Genentech, GSK, Hitachi, SAP, and UCB.
The data collected in our study involves human participants recruited through Mechanical Turk. We took several mea- sures to ensure the privacy of these workers in the docu- ment creation tasks. First, asked workers to confirm that they were willing to share the text they wrote as part of a public dataset. Second, we urged them not to include any personally identifiable information (PII) in their writ- ings and to focus only on topics of public knowledge or fictitious stories. Third, we scanned the collected data to ensure that no PII was included. For the final version of the dataset, we will recruit additional workers to manually review each collected conversation to ensure that no PII or other safety issues (e.g., offensive language) exist in the data. Mechanical Turk workers were paid $10 per conver- sation. Given that conversations averaged 28.4 minutes, including break times, this means workers were paid more than $20 per hour on average-above the minimum wage in the country where the data was collected.
Impact Statement
This paper presents work aimed at making AI more user- and human-centric, which, in our view, yields a positive societal impact. Most current work on AI and its evalu- ation focuses on fully automated tasks, with no user in- volvement in solving the task or optimization for a col- laborative experience with users. This has serious societal drawbacks, given issues such as AI hallucinations (Huang et al., 2025), biases (Gallegos et al., 2024), and unsafe language (Shi et al., 2024a) that arise from a lack of hu- man oversight. The common focus on having AI models autonomously complete tasks also ignores the reality that many scenarios have humans present regardless of the level of automation, and that not priming AI models to proac- tively seek human help, feedback, or clarifications misses an opportunity to make generative Al more accurate, effec- tive, and safe. This consideration would also help increase the adoption of Al in safety-critich scenarios, such med- ical decision-making maaiinnn-maiiggggaa tasks (Liu et al., 2024), in which we believe Al models should be inclined to seek confirmation or verification (Gero et al., 2023) from an expert in case of uncertainty- a behavior that is mostly absent in current state-of-the--at I.IMs.
This work presents one of the first attempts to train LLMs in such human- environments. To promote future research in this societally beneficial direction, we release all the code, models, data, benchmarks, and user simulators described in this work.
References
Abdulhai, M., White, I., Snell, C., Sun, C., Hong, J., Zhai, Y. Xu, K., and Levine, S. LMRL gym: Benchmarks for multi-turn reinforcement learning with language models. CoRR, 2023.
Amershi, S., Weld, D. S., Vorvoreanu, M., Fourney, A., Nushi, B., Collisson, P., Suh, J., Iqbal, S. T., Bennett, P. N., Inkpen, K., Teevan, J.. Kikin-Gil, Gil, R., and Horvitz, E. Guidelines for human-ai interaction. In CHI. ACM, 2019.
Since the models in this work are trained collaboratively and aim to better align with user intent, concerns may arise regarding users with malevolent goals. However, ar- gue that COLLABLLM can help mitigate safety risks in such cases-at at least when used with LLMs that have been aligned for safety (as is the case for all models used in this work). Safety-aligned LLMs generally refuse to respond to unsafe queries, which often leads malicious users to ob- seure their true intentions in order to bypass safeguards. This is where our approach offers an advantage: COL- LABLLM often seeks to clarify user intent, creating ad- ditional opportunities to detect misuse. For example, mali- cious users might unintentionally reveal their actual goals, or their vagueness and refusal to disclose motivations could
Andukuri. C., Fränken, J., Gerstenberg, T., and Goodman, N. D. Star-gate: Teaching language models to ask clari- fying questions. CoRR, 1b1/24 2024.
Balog, K. and Zhai, C. Rethinking conversational agents in the era of Ilms: Proactivity, non-collaborativity. and beyond. In SIGIR-AP, 20223.
Chen, M., Yu, X., Shi, W., Awasthi, U., and Yu, Z. Con- trollable mixed-initiative dialogue generation through prompting. In ACL, 2023.
Chen, M., Sun, R., Arik, S. Ó., and Pfister, T. Learning to
10

---

COLLABLLM: From Passive Responders to Active Collaborators
clarify: Multi-turn conversations with action-based con- trastive self-training. CoRR, abs/2406.000 002022, 2024.
Hendryeks, D., Burns, C., Kadavath, S., Arora, A., Basart, S., Tang, E.,, Song, D., and Steinhardt, J. Measuring mathematical problem solving with the math dataset. NeurlPS, 2021.
Chi, Y. Lin, J., Lin, K., and Klein, D. CLARINET: aug- menting language models ask clarification questions for retrieval. CoRR, abs/2405.15784, 2024.
Hong, J., Levine, S., and Dragan, A. D. Zero-shot goal- directed dialogue via RL on imagined conversations. CoRR, 2023.
Chiusano, F. Medium articles dataset. https:
/ //huggingfaae co/datasetas /aatassts//aicchiu// fabiochiu/ medi
2024. A dataset of Medium articles collected through web scraping, including metadata such as titles, authors, publication dates, tags, and content.
Hu, E. J., Shen, Y., Wallis, P., Allen-Zhu, Z., Li, Y., Wang. S., Wang, L.,, and Chen, W. Lora: Low-rank adaptation of large language models. In ICLR, 2022.
Hu, Z., Feng, Y. Luu, A. T., Hooi, B., and Lipani, A. Un- locking the potential of user feedback: Leveraging large language model as user simulators to enhance dialogue system. In CIKM. ACM, 2023.
Deng, Y., Liao, L., Chen, L., Wang, H., Lei, W., and Chua, T. Prompting and evaluating large language mod- els for proactive dialogues: Clarification, target-guided, and non-collaboration. In EMNLP, 2023a.
Huang. L., Yu, W., Ma, W., Zhong, W., Feng. Z., Wang, H., Chen, Q. Peng, W., Feng, X., Qin, B., and Liu, T. A survey on hallucination in large language models: Prin- ciples, taxonomy, challenges, and open questions. ACM Transactions on Information Systems, 43(2):1-55, Jan- uary 2025. ISSN 1558-2868. doi: 10.1145/3703155. URL http: / /xx doi www.doo..oooo.o.oooooooooo.o.oo.
Deng, Y., Zhang, W., Chen, Z., and Gu, Q. Rephrase and respond: Let large language models ask better questions for themselves. CoRR, abs/231 010/2311.0440 1.042055 2023b.
Deng, Y., Zhang, W., Lam, W. Ng. S., and Chua, Plug- and-play policy planner for large language model pow- ered dialogue agents. In ICLR, 2024.
org/10 10. .1145/3703155
Jiang, Y., Shao, Y., Ma. D., Semnani, J. and Lam, M. S. Into the unknown unknowns: Engaged human learning through participation in language model agent conversa- tions. arXiv, abs/2408. 15232, 2024.
Erbacher, P. and Soulier, CIRCLE: multi-turn query clarifications with reinforcement learning. CoRR, abs/2311.02737. 2023.
Keh, S., Chiu, T., and Fried, D. Asking more informative questions for grounded retrieval. In FNAACL, 2024.
Faltings, F. Galley, M., Brantley, K., Peng, B., Cai, W., Zhang, Y., Gao, J., and Dolan, B. Interactive text gener- ation. In EMNLP, 2023.
Kim, G., Kim, S., Jeon, B., Park, J., and Kang, J. Tree of clarifications: Answering ambiguous questions with retrieval -uggreetd large language models. In EMNLP, 2023.
Gallegos, O., Rossi, A., Barrow, J., Tanjim, M. M., Kim, S., Dernoncourt F., Yu, T., Zhang, R., and Ahmed, N. K. Bias and fairness in large language models: A survey, 2024. URL https:/ 2309 00770.
Kim, Y., Lee, J., Kim, S., Park, J., and Kim, Understand- ing users' dissatisfaction with chatgpt responses: Types, resolving tactics, and the effect of knowledge level. In IUI. ACM, 2024.
Gao, J., Galley, M., and Li, L. Neural approaches to con- versational AI. Found. Trends Inf. Retr., 13, 2019.
Kwan, W., Zeng, X., Jiang, Y., Wang, Y., Li, L., Shang, L., Jiang, X., Liu, Q. and Wong, K. Mt-eval: A multi-turn capabilities evaluation benchmark for large language models. In EMNLP, 2024.
Gao, Z., Zhan, W., Chang, J. D., Swamy, G., Brantley, K., Lee, J. D., and Sun, W. Regressing the relative future: Efficient policy optimization for multi-turn rlhf, 2024.
Gero, Z., Singh, C., Cheng, H., Naumann, T., Galley, M., Gao, J., and Poon, H. Self-verification improves few-shot clinical information extraction, 2023. URL https: / org/ wtwg/bss/////oooooooooooossssere 6.30 00024.
Lee, H., Phatale, S., Mansoor, H., Mesnard, T., Ferret, J., Lu, K., Bishop. C., Hall, E., Carbune, V., Rastogi, A., and Prakash, S. RLAIF vs. RLHF; scaling reinforce- ment learning from human feedback with Al feedback. In ICML, 2024.
Guo, M., Zhang, M., Reddy, S., and Alikhani, M. Abg- coqa: Clarifying ambiguity in conversational question answering. In AKBC, 2021.
Liao, L., Yang, G. H., and Shah, C. Proactive conversa- tional agents the post-chatgpt world. In SIGIR, 2023.

---

COLLABLLM: From Passive Responders to Active Collaborators
Liu, L., Yang, X., Lei, J., Shen, Y., Wang, J., Wei, P., Chu, Z.,, Qin, Z-, and Ren, K. A survey on medical large language models: Technology, application, trust- worthiness, and future directions, 2024. URL https: //arxiv.oo org/ abs/ / 2406. 03712.
Shi, D., Shen, T., Huang, Y., Li, Z.,, Leng, Y., Jin, R., Liu, C., Wu, X., Guo, Z., Yu, L., Shi, L.,, Jiang, B., and Xiong, D, Large language model safety: A holistic survey, 2024a. URL 2412.
org/ abs/
Llama Team, A. M. The llama 3 herd of models, 2024.
Shi, W., Qian, K., Wang. X., and Yu, Z. How to build user simulators to train rl-based dialog systems. In EMNLP-IJCNLP. Association for Computational Lin- guistics, 2019.
Mu, F., Shi, L., Wang, S., Yu, Z., Zhang, B., Wang. C., Liu, S., and Wang. Q. Clarifygpt: Empowering Ilm-based code generation with intention clarification. CoRR, 0ts/2310.10999 2023.
Shi, W., Yuan, M., Wu, J., Wang, Q. and Feng, F. Direct multi-turn preference optimization for language agents. In EMNLP. Association for Computational Linguistics, 2024b.
Ouyang, L.,, Wu, J., Jiang, X., Almeida, D., Wainwright, C. L., Mishkin, P., Zhang, C., Agarwal, S., Slama, K., Ray, A., Schulman, J., Hilton, J., Kelton, F., Miller, L., Simens, M., Askell, A., Welinder, P., Christiano, F., Leike, J., and Lowe, R. Training language models to follow instructions with human feedback. In Advances in Neural Information Processing Systems 35: Annual Conference on Neural Information Processing Systems 2022, NeurlPS 2022, New Orleans, LA, USA, November 28 December 9, 2022, 2022.
Taylor, R. S. in libraries. College & research libraries, 29(3):178- 194, 1968.
and information seeking
Tseng, B., Dai, Y., Kreyssig, F., and Byrne, B. Transferable dialogue systems and user simulators. In ACL/JCNLP. Association for Computational Linguistics, 2021.
Pang, J., Fan, H., Wang, P., Xiao, J., Tang, N., Yang, S., Jia, C., Huang, S., and Yu, Y. Empowering language mod- els with active inquiry for deeper understanding. CoRR, abs/2402.03719, 2024.
Wang. J., Ma, W., Sun, P., Zhang, M., and Nie, J. Under- standing user experience in large language model inter- actions. CoRR, abs/2401.08329, 2024.
Xu, C., Guo, D., Duan, N., and McAuley, J. Baize: An open-source chat model with parameter-efficient tuning self-chat data. In EMNLP, 2023.
Park, S., Zou, C. Q. Shaw, A., Hill, B. M., Cai, C., Mor- ris, M. R., Willer, R., Liang P., and Bernstein, M. S. Generative agent simulations of 1,000 people, 2024.
Zamani, H., Dumais, S. T., Craswell, N., Bennett, P. N., and Lueck, G. Generating clarifying questions for infor- mation retrieval. In WWW, 2020.
Pearl, Causality: Models, Reasoning and Inference. Cambridge University Press, 2nd edition, 2009.
Pearl, J., Glymour, M., and Jewell, N. P. Causal Inference in Statistics: A Primer. John Wiley & Sons, 2016.
CeiffrescuuPereiine J. D., Wong, R. Y., Hartmann, B., and Yang, Q. Why johnny can't tt prompt: How non-ai experts try (and fail) design LLM prompts. In CHll. ACM, 2023.
Rafailov, R., Sharma, A., Mitchell, E., Manning, C. D.. Ermon, S., and Finn, C. Direct preference optimiza- tion: Your language model is secretly reward model. In NewrlPS, 2023.
Zhang, C., Dai, X., Wu, Y., Yang, Q. Wang, Y., Tang, R., and Liu, Y. A survey on multi-turn interaction capabilities of large language models. arXiv preprint arXiv:2501.09959 2025.
Rahmani, H. A., Wang, X., Feng, Y., Zhang, Q., Yilmaz, E., and Lipani, A. A survey on asking clarification questions datasets in conversational systems. In ACI., 2023.
Zhang, M. J. Q. and Choi, F. Clarify when necessary: Re- solving ambiguity through interaction with Ims. CoRR, 0as/2311.094450 2023.
Schulman, J., Wolski, F., Dhariwal, P., Radford, A., and Klimov, O. Proximal policy optimization algorithms. CoRR, abs/ 0bss/7777.06344 .00477, 2017.
Zhao, Z. and Dou, Z. Generating multi-turn clarification for web information seeking. In www, 2024.
Shani, L., Rosenberg, A., Cassel, A. B., Lang, O.,, Calan- driello, D., Zipori, A., Noga, H., Keller, O., Piot, B., Szpektor, I., Hassidim, A., Matias, Y., and Munos, R. Multi-tum reinforcement learning from preference hu- man feedback. CoRR, abs/2405. 14655, 2024.
Zheng, L., Chiang, W., Sheng, Y., Zhuang, S., Wu, Z., Zhuang, Y., Lin, Z., Li, Z., Li, D., Xing, E. P., Zhang, H., Gonzalez, J. E., and Stoica, I. Judging a-jg-e with mt-bench and chatbot arena. In NeurIPS, 2023.
12

---

COLLABLLM: From Passive Responders to Active Collaborators
Zhou, Y., Zanette, A., Pan, Levine, S., and Kumar, A. Archer: Training language model agents via hierarchical multi-turn RL. In ICML, 2024.
Zhuo, T. Y., Vu, M. C., Chim, J., Hu, H., Yu, W., Widyasari, R., Yusuf, N. B., Zhan, H., He, J., Paul, I,, al. Big- codebench: Benchmarking code generation with diverse function calls and complex instructions. arXiv preprint TrXiv:2406.15877, 2024.
13

---

COLLABLLM: From Passive Responders to Active Collaborators
A. Supplementary Discussion
A.1. Connection Between Multiturn-awarr Reward and Causal Inference
Our approach naturally aligns with causal inference principles, as it aims to quantify how a model's response influences the future trajectory of conversation. This aligns with the fundamental goal of causal effect estimation, which seeks to isolate the impact of an intervention-in this case, model response-on long-term outcomes.
From a causal perspective, given conversation history t turn the causal effect of model response my the final conversation trajectory can be expressed using front-door adjustment (Pearl, 2009; Pearl et al., 2016):
Σ R (t1.k P(t.x ()))(t))) Σ Rd (t1: P(tik f.) 1( (f1.k lg).
(5)
This equation captures the expected long-term reward of conversation conditioned on the model's response at turn j. explicitly accounts for how m, intervenes in the conversation, influencing future turns and, ultimately, task success.
A.2. Distinction from Other Multiturn Training Frameworks
Existing multiturn trajectory- -baed training frameworks (Shani et al., 2024; Zhou et al., 2024; Gao et al., 2024) primarily rely on learning from observed trajectory-level rewards. These methods estimate the utility of responses by assigning rewards post hoc to completed conversations, typically training models to prefer higher-rated conversations over lower- rated ones. However, this approach is fundamentally observational-it captures statistical associations between responses and final outcomes, without disentangling how individual responses causally influence future turns. For example, in MTPO (Shani al., 2024), the learning signal remains coarse-gran graained: rewards are assigned at the trajectory level, and the influence of specific turns within a conversation remains confounded and indirect.
In contrast, our Multiturn-awart Reward (MR) framework intervenes individual model responses and uses forward simulation to generate alternative future trajectories. This allows the model to estimate the counterfactual impact of different responses at each turn, thereby enabling fine-grained optimization. By leveraging causal effect estimation, MR training moves beyond passive imitation of high-reward conversations and instead actively selects responses to maximize long-term task success. This interventional approach provides turn-level credit assignment that is critical in dynamic human-LLe LLM interactions, where user needs evolve and the consequences of early decisions compound over time.
B. Experimental Details
B.1. Dataset Generation for Offline Training
SFT data
DPO data
Chosen
Rejected
MR 3.2
MR 2.7
8
Rejected
Chosen
MR 1.7
MR 4.5
8
Figure 8: Generating high-quality conversation data with Multiturn-awarr Rewards (MR).
The Multiturn-awar Reward (MR) function enables the generation of high-quality synthetic conversation datasets for training. Given user query, multiple LLM responses are sampled and ranked based on their MR scores, with higher- ranked responses designated as Chosen and lower-ranked as Rejected. To simulate natural conversational flow, the first turn from the chosen response's 's forward interaction window is appended to the prompt for the next turn, iteratively extending the conversation until completion. Solid red arrows denote data collection for Supervised Fine-Tuning (SFT), while dashed blue arrows indicate preference data construction for Direct Preference Optimization (DPO). This approach systematically curates multiturn conversations that enhance both response quality and collaborative efficiency, both of which are explicitly captured by MR.
14

---

COLLABLLM: From Passive Responders to Active Collaborators
Given (1) user simulator LLM, e.g,, GPT-40-mini, (2) assistant LLM,, GPT-4o, and (3) arbitrary tasks with defined task-specific metric, we simulated and generate high-quality conversations following Figure 8. We create the following training datasets in this simulated environments.
Table 5: Statistics of conversational datasets created from MR. Chosen/Rejected MR indicates the mean and standard deviation (mean + std) of MRs for chosen and rejected responses (cf. Figure 8).
# Train
# Turns
Average Turns
Chosen MR
Rejected MR
MediumDocEdit-Chae
500
2,303
4.61
0.312 00.14
0.246 10.113
BigCodeBench-Chat
500
2,627
5.25
0,494 00.221
0.207 #0.763
MATH-Chat
500
2,527
5.05
0.863 00.524
0.547 =0,502
B.2. Training Details
Hyperparameters (Table 6). We provide the hyperparameters for COLLABLLM fine-tuning.
Notably, COLLABLLM relies minimal set of hyperparameters, using the same window size and sample size for computing MRs across multiple datasets. The penalty factor on token count, X, is set lower for Medi umDocKdit-Chet compared to BigCodeBench- -hat and MATH -hatt as document lengths in Medi umDou -Chat can vary significantly and may be easily bounded by in Eq. 4 if X is too large. Training Cost (Table 7). We compute average
Table 6: Hyperparameters for LoRA configuration, different stages of fine-tuning, and COLLABLL fine-tuning.
LoRA Configuration
Fine-Tuning Hyperparameters
Rank
32
SFT
Offline DPO
Online DPO
PPO
Scaling factor a
16
Learning rate
le-5
5e-6
5e-6
2--6
Dropout
0.1
Total batch size
64
64
32
64
Bias
False
Number of epochs
3
8
1
5
-ospei Hyperparameters
MediumDocFdittChatt
BigCodeBench-Chat
MATH-Chat
Window size w Sample size for MR
2
3
Penalty A
le-4
5c-4
5c-4
statistics over 100 future conversations on Medi MediumDoeEdit-Chatt the document editing task, which incurs the highest computational overhead among the three tasks. The table shows that even at the largest window size (w = 3), the total per-sample cost remains low, suggesting that our multi-turn training setup financially practical. To further reduce the cost of simulating users, one could use an open-source model to role-play as users. Unfortunately, at the current stage, find that open-source models generally perform poorly, often getting "confused" and starting to solve problems as an assistant rather than acting as a user. This raises an interesting research problem: while we have increasingly capable LLM assistants trained to solve problems, we lack user models that learn from real-world user behavior. Building better user models could be valuable for running simulations real-world applications.
Policy Model
Policy Model
Policy Model Time (s)
User Simulator Input Tokens (k)
User Simulator Output Tokens (k)
User Simulator Cost (S)
Input Tokens (k)
Output Tokens (k)
W 1
0.89
0.42
7.41
1.85
0.26
0.00174
W = 2
2.55
0.91
15.84
4.55
0.69
0.00439
W = 3 4.13
1.22
21.72
7.18
1.06
0.00685
Table 7: Comparison of policy model and user simulator's compute (per forward sample) across different window sizes. We use GPT--4--ini as the user simulator. The results are averaged over 100 forward sampled conversations.
15

---

COLLABLLM: From Passive Responders to Active Collaborators
B.3. Full Ablation Results
BLEU
dTokkens (k)
ITR
0.200
0.902
2.19
1.80
2.21
0.80
9:30
0.85
0.303
0.301
0.972
181
2:22
0.14
9.04
0.14
0.392
0.014
0.205
0:572
2.55
199
1,55
0.
0.62
0.60
9.5/0
0.912
2011
9.49
0.34
resilumen
Extrmii Dndy
bauinali Diyy
GANALN
Extrraii Orly
Figure 9: Full ablation study showing the impact of different reward types (Helpfulness, Extrinsic Only, Intrinsic Only) and window sizes (w) on BLEU, token count (in thousands), and Interactivity Rate (ITR). The CollabLLM setting combines intrinsic and extrinsic rewards using the multiturn-awer -wware reward formulation.
To further understand the source of performance improvements, conduct full ablation by training models with isolated reward Saaiull-Hellfulnesss Extrinsic Only, and Intrinsic Only Only-across window sizes w E 2,,1, The resulting BLEU, token usage, and ITR scores are reported in Figure 9.
We make three key observations:
Helpfulness alone leads to marginal improvements in BLEU and ITR, but significantly increases token usage, especially at larger window sizes, suggesting verbosity rather than improved efficiency or interactivity.
Extrinsic-only reward achieves strong BLEU scores (e.g., 0.377 at 1), indicating good task alignment. However, it underperforms in ITR and often generates longer responses.
Intrinsic-only reward improves ITR at w 1 (e.g.. 0.74), but offers lower BLEU and comparable or slightly lower token efficiency. indicating better interactivity at the expense of task success.
The CollabLLM configuration, which combines both intrinsic and extrinsic rewards using a multiturn-aware framework, achieves strong and balanced performances.
Note that the choice of reward type (intrinsic or extrinsic) independent of the multiturn-awwrt reward design. In practice, can flexibly plug in different reward signals, which then used to evaluate the responses' long-term impact through forward sampling.
C. Safety Evaluation
As the models in this work are collaboratively trained and designed to be more aligned with the user's intent, concerns may arise if a user happens have malevolent intentions. However, we note that COLLABLLM models were finetuned from Llama-3. 1-8B, which has been aligned for safety-so jailbreaking COLLABLLM still poses significant challenge. To determine whether collaborative training weakens the safety features inherent to model (Llama-3.1 1.1BBB) that has undergone significant alignment steps for safety, we performed an adversarial evaluation using the Azure AI Evaluation SDK* and prompted both the baseline and COLLABLLM with various offensive queries intended to elicit unsafe responses.
Specifically, we performed the following steps:
Adversarial query selection: We used the SDK's Adversaria SitiarriiaSsmmlllll.l.mm SSiuuSmlllaao to generate adversarial queries (e.g., queries encouraging the LLM to produce hateful comments). We then used the SDK's harm evaluators VriolenceEvalluttonn
www.rraiinnssslllaaatoro to categorize each query into one of four harm types: violence, sexual, self-harm, and hate. For each query, we used the highest score among the four evaluators to determine its harm category. We randomly selected 20 adversarial queries per harm category, resulting in total of 80 queries.
luatorr Sel Self
Thttps://1eat //1errn///a micoosoft.. www.ooo///n/oeeeooooo httpr//tt..ooeeeee/p//r/pooeeeaa http://wwww.......to//t////////a
Tuation naaion-readme
16

---

COLLABLLM: From Passive Responders to Active Collaborators
Model
Harm score (0-7 range, 4)
Violence
Sexual
Self-harm
Hate
Llama-3.1-8B
0.88
0.96
0.89
1.01
COLLABLLM
0.95
0.94
1.00
0.99
Table 8: Harm scores of responses generated by the two models under adversarial prompting. Scores range from 0 to 7, with values between O and 1 indicating "very low" harm.
Response generation:
We generated responses to these 80 adversarial queries using both the Llama-3.1-8B baseline
model and COLLABLLM.
Harm scoring: We evaluated each model-generated response using all four harm evaluators ensure comprehensive assessment.
The main safety results shown in Table 8, which presents the average harm scores across the four categories. Although all queries were adversarial and received high harm scores (typically between and 7 0-7 scale), both the Llama-3.1- 8B baseline and COLLABLLM produced responses that were, average, very safe. Most scores are in the 0-1 range, which corresponds to "very low" harm. COLLABLLM shows slightly lower harm in the Sexual and Hate categories and slightly higher harm in the other two. In terms defect rate, COLLABLLM produced only one response deemed unsafe by the SDK (out of 80 queries 4 categories 320 evaluations), resulting in pass rate of 99.7%. Coincidentally, this the same pass rate as Llama-3.1 which also had one failed evaluation.
Overall, these results encouraging. They suggest that COLLABLLM's training did not degrade the safety capabiliiiees of the original LLM, even though no additional safety alignment was performed during COLLABLLM's training.
D. Prompts
D.1. User Simulator
You are role-playing as human USER interacting with an AI collaborator to complete specific task. Your goal is to generate realistic, natural responses that user might give in this scenario.
2
3. ## Inout Information:
4. You will be provided with:
rask Description: The type of task you are trying to accomplish.
Complete Prompt or Reference Goal: This field may include the complete user request/ this field to understand the user
query or a reference answer to user's request. 's intent, requirements, or what would count as satisfactory outcome.
Chat History: The ongoing conversation between (as the user) and the AI
9 Inputs:
10 <h start of Task Description (Not visible to the AI) (task_desc)
</The End of Task Description/>
<hehe Start of Complete Prompt or Reference Goal (Not visible to the AI) /> 15
(singie_tures perle__tur__ppe turmppr
<h End of Complete Prompt or Reference Goall>
The Start Chat Historyl>
19 (chat_history)
20 </The End of Chat Historyl>
Guidelines:
24 Stay in Character: Role-play as a human USER. You are NOT an AI. Maintain a consistent personality throughout the chat.
Minimize Effort: IMPORTANT! As a user, avoid being too detailed in your responses. Provide vague incompiete demands in the early stages of the conversation to minimize your effort. let the AT ask for ciari rather than providing
17

---

COLLABLLM: From Passive Responders to Active Collaborators
everything upfront.
Knowledge Background; Reflect the user's knowledge level in the role-playing, If the user is less knowledgeable about task, they might not notice incorrect statements. Ask questions that demonstrate your current understanding and areas of confusion. Occasionally Make Mistakes: Real-world users might misspell words, provide incorrect dates, give wreng information, or ask unclear questions. Simulate this behavior to reflect natural intcractions.
Mention Personal Preforences: Include preferences or constraints that might influence your requests responses. For example, "I prefer short answers, " "I need this done quickly, " or "I like detailed comments code."
29
Goal-Oriented: Keep the chat focused on your intent. Avoid small taik or digressions. redt the chat back to the main objective if it starts to stray.
# Output Format:
You should output a JSON object with three entries:
"current_aai answer" (str): Briefly summerize the AI's current solution to the task. "thought" " (str); Output your thought process as user deciding what to say next, Consider:
Have you obtained satisfactory solution from the AI? If yes, you can terminate this chat.
36
2. If not, what specific part of the problem or solution are you struggling with?
3. Has the AI asked you to perform a task or answer question? If so, how should you approach it?
4. Are you noticing any patterns or potentiai misunderstanding that need ciarification?
5. If you're stuck, how can you phrase your question to get the most helpful response while demonstrating your current understanding?
"response" (str): Based on your thought process, respond to the role-playing. Stop immediately when the user's response is completed.
user you are
## Important Notes:
Respond Based Previous Mossages: Your responses should be based the context of the current chat history. Carefully read the previous messages maintain coherence in the conversation.
44 Conversation Flow: If "Current Chat History" is empty, start the conversation from scratch with initial request. Otherwise, continus based the existing conversation.
Don't Copy Input Directly: Use the provided information for understanding context only Avoid copying target queries or any provided information directly in your responses.
your response when you believe your goal has been solved if you determine the cannot help further. 47 Double check if the JSON object is formatted correctly. Ensure that all fields are present and properly structured.
Completion Signal: Use "[terminal_signa)"
Remember to stay in character instructions and guidelines carefully.
throughout your response, and follow the
D.2. Prompt for Proactive Base
You are an AJ assistant interacting with a user to perform tasks such as writing, analysis, question answering, math, coding. Your goal is to generate response to the user's last message in a conversation. should be helpful, collaborative, and highly interactive.
2
I will provide with the following information:
Conversation History: This is the complete chat history where you need respond the last user message.
5 Additional Information (Optional): This may include reference knowiedge with a question and answor to give you rolevant context.
7. </The Start Conversation Historyl>
8. (chat_history)
18

---

COLLABLLM: From Passive Responders to Active Collaborators
The End of Conversation History|>
10
</ Start of Additional Information/> [additional_ _inf]
</The End of Additional Information/>
15 $ Guidelines:
1. Understanding and Engagement
Accurately interpret the user's intent throughout the conversation. Acknowledge previous interactions to maintain context and continuity in the conversation,
2. Interactivity (Important!)
Ask clarifying questions if the user's request lacks detail is ambiguous. Such as the length an essay, specific function format for a coding task, or the context of question.
Ask specific follow-up questions to assist the user based on their intent. Avoid general questions like "Do you have any further questions? Let know, " Instead, focus on specifics like, "Nould you like more information on X?" or "Can you clarify your requirements for Y?"
23
When seeking feedback, avoid generic requests like "Let me know if this is helpful Instead, ask for feedback specific aspects, such as "Does this solution meet your needs about X?"
Collaboratively offer guidance, especially in complex or tricky situations. Provide specific suggestions on potential next steps.
Pocus on the long-term goal, prioritize responses that not only solve the immedi problem but also contribute to the user's long-term objectives. Foresee how your response can shape the next few turns of the conversation by aligning with the user's overarching goais.
26
3. Efficiency and User Consideration
Be mindful of how much the user needs to read or type, keeping the interaction concise and fecused,
29
When asking for feodback or presenting options, provide multiple-choice suggestions or specific prompts to make it easier for the user to respono quickly.
Avoid repeating information from earlier in the conversation uniess it's necessary for clarity. Ensure your responses are not redundant.
32 4. Communication Style
Be honest in your responses. If you are unsure of something, say, "I don't 't knew," and suggest ways the user could find the information.
Align your tone and responses with the user's emotional state, adapting your style to suit their mood or urgency.
Ensure your responses are clear, well-structured, and free from gramnatical errors.
Output Format:
You should output a JSON object with three entries:
39 "aurrenn_pprolem" (str): What is the current problem the user is facing, and what are they confused about?
"thought" (str): Output your thought process deciding what say next. You may consider the following:
1. If reference knowledge is provided, how do you make sure you don't overly use it and simply assume the user's question is the same as the reference question?
2. What information is missing from the user's input? Does the user's message lack any necessary details?
Is there need to ask a clarifying question to better understand the user's intent?
4. Does the user seem confused or unclear on a particular tepic? How can you address that confusicn?
What follow-up can you suggest help the user move forward with their task? How can you ensure that your response is heipful, concise yet thorough, and collaborative?
7. Whether your response can guide the conversation toward the user's long-term objectives beyond the immediate problem?
19

---

COLLABLLM: From Passive Responders to Active Collaborators
"response" " (str): Based on your thought process and chat history, provide your response following the guidelines to the user, Keep your response within { max_new_tokens, tokens to avoid being cut off.
Notes:
51
Clarifying Questions: If the user's message unclear or lacks necessary details, always ask for clarification rather than making assumptions. Ensure you have enough information to provide an accurate and relevant response. For example, if the user asks, "Can you solve this equation?" but doeen't provide the equation, respond with: "Could you provide the equation you'd 'd like me to solve?"
Reference Knowledge Usage: If reference knowledge is provided in the additional information, use it as context but do not assume that the user's question will exactly match the reference question. Always adapt your response to the speci fic context provided by the user in the conversation history.
Ensuring Interactivity: Encourage more interaction with the user by engaging in at least three conversational turns. This will help refine the conversation and ensure the user's 's needs are fully addressed.
Double check if the JSON object is formatted correctly. Ensure that all fields are present and properly structured.
56 Take deep breath and carefully follow the instructions and guidelines provided.
D.3. System Prompt
The assistant is designed to be helpful, proactive, and highly interactive.
2
3. The assistant strives to accurately interpret the user's intent throughout the conversation, acknoviedging previous interactions to maintain context and continuity If the user's message is unclear or lacks recessary details, the assistant always asks for clarification rather than making assumptions. For example, if the user's request is incomplete, the assistant responds with: "Could you provide more details so can assist you better?"
The assistant asks specific follow-up questions and offers suggestions based the user 's needs, avoiding vague or generic prompts. It proactively provides guidance and potential next steps, especially in complex tasks such as writing, analysis, coding, and question answering.
The assistant is mindful of how much content the needs to read type, keeping interactions concise and efficient. reduces unnecessary repetition and ensures responses are relevant, well-structurent and free from errors. When presenting options or asking for feedback, the assistant simplifies interactions by offering multiple-choice answers specific suggestions to make it easier for the user to respond quickly.
9. The assistant adapts its tone to align with the user's emotional state and style, adjusting its approach as needed. If uncertain about something, the assistant honestly says, "I don't know, and suggests ways for the user to find the information.
10
The assistant provides factually Iyy accurate, coherent, and relevant responses, using proper grammar and structure. it remains interactive and proactive across all tasks, continually seeking feedback to refine and improve interactions.
D.4. Interactivity Metric by LLM Judge
For the prompt template below, the ITR results reported in Table use weights A = 3, 2, and 1, with the final score S rescaled as S' = 2 (S - 2.5). as all methods achieve an average ITR score above 2.5. Please use the same configuration to reproduce the results shown in Table Note that the absolute values of A, B, and C do not affect the overall conclusions. In our most recent codebase, we adopt A = 1, B = 0.5, and C = 0 to eliminate the need for rescaling.
20

---

COLLABLLM: From Passive Responders to Active Collaborators
You are a helpful and meticulous conversation evaluator.
2. Your task is to evaluate the *interactivity+ of the responses provided by an A1 assistant
questions in given conversation:
5 </The Start of the Conversation to be Evaluated/> (chat_ history
</The End of the Conversation to be Evaluated/>
9. You should assess the assistant's 's engagement, clarity, and ability to understand the uscr's needs.
Give float number between (C) and (A), where:
{A} Highly interactive: The assistant is very engaging, asks all relevant questions, and signi ficantlly enhances understanding and problem-solving. Exampie: The assistant thoroughly understands the user's question, asks for necessary ciarifications, such "It sounds like you're asking about the causes of climate change. Are you lcoking for specific exampies general overview?"
12
(B) Moderately interactive: The assistant is engaging, asks some relevant questions, but can be substantially improved.
Example: The assistant asks some relevant questions about the user's inquiry but misses key details, such as "Are you asking about effects of climate change ?" but does not probe further for clarification.
15
(C) Low interactivity: The assistant shows low engagement, asks few relevant questions, and barely try to undcrstand the user's needs.
Example: The assistant provides vague incomplete response without fully understanding the user's intent, such "CI "Climate change is bad, " without asking any foliow-up questions or providing detailed information.
Output format (JSON)
21
"thought"; "<oWw interactive is the assistant?>", "interactivity": <score>
Double check the JSON object is formatted correctly. Ensure that all fields are present and properly structured. Use or to wrap up the thought content and use single quotes inside the "thought" field to avoid JSON escape issues.
Your evaluation:
D.5. Helpfulness Reward by LLM Judge
You are helpful and meticulous conversation evaivator. Your task is to assess the helpfulness of an LIM-generated response in the context of the user intent and the provided chat history. Focus how effectively the response fulfills the user's needs and intent.
2.
3. Frovided Information:
5 </The Start The User Intent/>
6 (question)
7 </ End of The User Intent/>
</The Start of The Historical Conversation
10 (chat history!
</The End of The Historical Conversation|>
</The Start of The Response to be Bvaluated/> (chat)
</The End of The Response to be Evaluated/>
21

---

COLLABLLM: From Passive Responders to Active Collaborators
You should evaluate the follow-up conversation based on the following criteria: Evaluate the response using the provided information below. Your evaiuation should consider the following aspects of helpfuiness:
1. Alignment with Intent: Does the response address the user's question request understood from the chat history?
20 2. Usefulness: Does the response provide actionable, relevant, and sufficient information to assist the user effectively?
3. Clarity: the response expressed clearly and in way that is easy for the to understand?
Scoring Criteria:
0.0: The response is completely unhelpful. it does not address the user's intent, iacks useful information solve the problem, and/or is entirely unclear.
0.2: The response is minimally helpful.
barely addresses the user's intent, lacks key information to scive the problem, or is very unclear.
4: The response is somewhat helpful. It partially addresses the user's intent but clarity issues.
has notable inaccuracies, omissions,
0. The response moderately helpful. It addresses the user's intent with some issues in completeness, accuracy, or clarity.
0.8: The response is quite helpful. It aligns well with the user's intent, provides relevant and sufficient information to solve the problem, and is mostly clear.
1.0: The response is very helpful. It fully aligns with the user's intent, provides thorough and accurate information to selve the problem, and expressed clearly and effectively.
Output Pormat:
"helpfulness": fi "thhought"; "<How helpful is the assistant the conversation?>", score": <score>))
11
Important Notes:
37 The "User Intent" and "Historical Conversation" is provided only for reference to help you understand the context of the response. You should focus your evaluation solely the "Response" provided above.
Inside of the content of "thought", replace all double quotes (") with single quotes
(') to prevent JSON formatting issues. For exampie, you can output "thought";
Hello' is a common phrase."
Your evaluation:
E. Question Template and Example on Abg-CoOA
We use the following prompt format for the LLMs to answer the question given story.
Can you help me answer question about
the following story?
2. (story)
5 My question is: (question)
For example:
Can you help mo answer question about the following story?
2
3. I spont last weekend with my grandma and grandpa. I love them very much! I always look forward to visiting them! They always de fun things with
Last weekend, went to the zoo together. saw great big elephant. It had long nose. My grandpa and played a game to see who could be the most like an elephant. We stomped around a lot and made crumpeting noises. I won! Grandma looked on and laughed. ! saw a
22

---

COLLABLLM: From Passive Responders to Active Collaborators
monkeys too! The monkeys swung through the trees. They even made monkey noises! Grandma wanted to take a picture of me with the monkeys, but I was too busy pretending I was monkey to stand still. After we left the 200, I went home. we had dinner together. Then, my grandma read me a story and tucked me into bed, I had a great time with my grandparents. love them lot. always look forward visiting them.
5 My question is: Khere did they when they left?
The label of the above question is ambiguous since the user's query about "Where did they go when they left?" could mean "Where did they go when they left the 200?" or Where did the grandparents go when they left me?".
F. User Study
E.I. User Study Platform
We provide screenshots of the interface used for human participants to interact with the Al assistants. The task consists of three sequential steps, requiring users to complete periodic evaluations throughout the interaction, followed by a final evaluation to complete the task. All data collection is fully anonymized to ensure user privacy.
Creating Writing Contents with Al Assistants [Version: 10x Speed Up]
the wet with ao unsoc docaeett seici bocks are Ever atojk cags to Encught
limple chalk
key
(a) Overall interface
(b) Step
Figure 10: Overall interface and Step view.
Step2
Step
to vittag An dovi tongge toors
Cenome with the
Hoa kidw thot prouiaeo nopaig poddccon sericcioe wittt iTSig snatue:
sock gentoor caee crcille
ngg
Acoollers more proniii
accial qunati virin reyemalii sonatii canging
be diviing musel Caans doou paraci Aelagg Aud
(a) Step
(b) Step
Figure 11: Step 2 and Step interfaces.
23

---

COLLABLLM: From Passive Responders to Active Collaborators
(a) Multiturn evaluation view
(h) Final evaluation view
Figure 12: Evaluation interface for multiturn and final user studies.
F.2. Analysis: Divergence Between Simulated and Real Users
While simulators were employed exclusively during training due to the large-scale conversation demands of Multitum-aware Reward computation, provide comparative analysis to study the divergence between simulators and real users. We summarize key differences and similarities in communication patterns between real and simulated users
below:
Table 9: Comparison of Simulated vs. Real Users
Differences
Similarities
1) Real users tend to use shorter, fragmented sen- tences with grammatical errors; simulators produce more complete and polished responses.
Both exhibit iterative content ment progresssgsssylly revealing information rather than specifying everything upfront.
develop-
2) Real users often shift direction mid-conversation and introduce specific personal details (e.g.. "eight dogs"); simulated users remain more predictable and generic.
2) Both emphasize accesibility-fiequennty request- ing simplifications, examples, and actionable guidance.
3) Real users express emotion more bluntly (e.g., "that's awful") and use informal language, abbrevia- tions, or incomplete thoughts; simulators respond in more neutral and formal tone.
3) Both articulate preferences about content structure or style, and provide feedback when expectations are met or unmet.
Although our models were trained using simulated users, the user study demonstrates that they generalize effectively to real users. This supports the feasibility of simulator-based training for scalable optimization, while also revealing opportunities to enhance the realism and diversity of user simulators.
24


Reviews:
[{"review_id": "kp7mgzdXA9", "reviewer": "Reviewer_Fu3N", "summary": "This paper introduces CollabLLM, a training framework designed to enhance the capability of large language models (LLMs) to collaborate with humans in multi-turn interactions.\n\nThe basic idea is to introduce forward-looking behaviors in LLMs to maximize long-term collaborative outcomes. This is achieved through a collaborative simulation module, which samples potential future user interactions to assess the impact of current responses using a new metric called Multiturn-aware Reward (MR). The MR combines both extrinsic factors, such as successful task completion, and intrinsic factors, like interaction efficiency, to comprehensively evaluate response quality. By applying reinforcement learning methods to optimize responses according to MR, CollabLLM improves models' abilities to proactively engage users, leading to superior collaborative task performance.\n\nThe experimental results show the fine-tuned model actively anticipates user needs, poses relevant follow-up questions, generates targeted content, and offers insightful recommendations.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your constructive and thoughtful suggestions touching on the practicalness of our work! We address your remaining concerns:\n\n\n---\n\n## **[Experimental Analyses | Weaknesses] \"(What is) the computational costs associated with larger window sizes”**\n\nThanks for this suggestion! In online training, the computational overhead in forward sampling comes from 1) generation from the policy model and 2) generation from the user simulator. \n\n- For (1), the computational overhead and cost are fairly low since we have integrated vLLM [1] in model inference. \n- For (2), we use gpt-4o-mini as the user simulator where we expect user responses to be concise, i.e., the number of output tokens to be small. \n\nWe compute the average statistics over 100 future conversations when $w=1,2,3$ on MediumDocEdit-Chat, the document editing task, which has the maximum computational overhead among the three tasks. Please see the table in https://anonymous.4open.science/r/collab-llm/images/cost.png \n\nWe have added this information to `Appendix C.3: Computational Cost During Training`, and hopefully this provides clear details.\n\n---\n\n## **[Experimental Designs | Weaknesses] \"Including multiple diverse external benchmarks could strengthen claims about generalizability.”**\n\nIn addition to Adb-CoQA, the model's generalizability is also validated in our user study, where the deployed model is trained on the MediumDocEdit-Chat task with document type restricted to a collection of medium blogs. While in deployment, the model also interacts with users to write personal statements and conduct creative writing. \n\n---\n\n## **[Essential References | Weaknesses] Adding related works**\n\nThanks for providing more references! \n\nThe second paper [3] that the reviewer listed is discussed in `Line 631 (Appendix A)` and `Table 4 (Appendix B)`. A difference between CollabLLM and MTPO [4] is that our Multiturn-aware Rewards provide **accurate and turn-level** signals as the long-term effect of each model response, while MTPO applies **trajectory-level** rewards when training the LLMs, making it hard to dissect the effect of good/bad responses inside a trajectory/conversation. \n\nWe further added the benchmark paper [2] to `Line 682 (Related Work)`, which will be included in the main paper in the final version. Here is the modified content: \n\n```Recent benchmarks~\\cite{LMRL,MTEval} evaluate LLMs' performance in multiturn settings, measuring the goal orientation and planning capabilities of models across interactions. Several studies...```\n\nWe hope this addresses your concern about the comprehensiveness of our related work.\n\n---\n\n\n## **[Questions] How does CollabLLM integrate with RL frameworks and what modifications are needed?**\n\nCollabLLM is plug-and-play with two user-defined modifications:\n\n\n1) **(Optional) User simulator prompt**. We provide the default prompt (the one we used in the paper), while in some cases, the user characteristics can be known ahead of time in certain tasks. For example, consider an LLM for education where users may be students with basic physics understanding. Brief instructions for role-playing can better approximate real conversations.\n\n\n2) **Metrics.** The intrinsic rewards (interactivity and efficiency) should be applicable to most applications, while users can define other task-specific metrics, such as accuracy, correctness, or, for example, bargaining advantage in debating or deal-making tasks.\n\n\nWe are glad that the reviewer raises this question. Our implementation goal is to make CollabLLM easy to use, able to accommodate user-customized tasks, and efficient with a fast inference infrastructure such as vLLM, so as to be compatible with RL training libraries such as [4].\n\n\n---\n\n## **Summary**\n\nWe thank you for the extensive review and insights! We hope our answers address your concerns on 1) computational overhead, 2) generalization, and 3) related works. We further provide clarification about how to integrate the existing RL training with CollabLLM. \n\nFinally, we greatly appreciate your support, which strengthens the paper in terms of its practical potential. Thank you again!\n\n---\n\n## **Reference**\n\n[1] Kwon et al. Efficient Memory Management for Large Language Model Serving with PagedAttention. SOSP 2023. \n\n[2] Abdulhai, Marwa, et al. Lmrl gym: Benchmarks for multi-turn reinforcement learning with language models. arXiv preprint arXiv:2311.18232 (2023). \n\n[3] Shani, Lior, et al. Multi-turn reinforcement learning from preference human feedback. arXiv preprint arXiv:2405.14655 (2024).\n\n[4] von Werra et al. TRL: Transformer Reinforcement Learning. 2020.", "claims_and_evidence": "Yes, the claims made in the submission are supported by clear and convincing evidence specifically in Section 5.", "methods_and_evaluation_criteria": "Yes, the proposed methods and evaluation criteria in this paper make sense for addressing the multiturn collaboration problem. Specifically:\n\nThe authors clearly identify the limitations of existing methods (like RLHF), which typically reward single-turn responses and do not effectively address long-term user interactions. The introduction of Multiturn-aware Reward (MR) and a collaborative simulation module effectively addresses these limitations by explicitly modeling forward-looking behaviors, thereby ensuring LLMs actively engage in clarifying user intent, leading to better long-term outcomes.\n\nThe use of both extrinsic (task-specific success) and intrinsic (interaction efficiency and interactivity) evaluation metrics provides a comprehensive and meaningful assessment of collaboration quality.", "theoretical_claims": "The paper does not contain explicit theoretical proofs.", "experimental_designs_or_analyses": "Multiturn-aware Reward (MR) Ablation (Section 5.1, Figure 4):\n- Validity: The ablation study clearly compares immediate reward methods (Helpfulness, Extrinsic, Extrinsic + Intrinsic) and multiturn-aware reward variants (with window sizes w=1,2,3). The controlled experimental setup is sound because it isolates the effect of reward design clearly and directly evaluates their relative effectiveness.\n- Issues: No major issues. However, the authors mention briefly the computational costs associated with larger window sizes, but explicit details about these computational trade-offs are sparse. Including more detail on computational overhead might enhance the practical interpretability of the findings.\n\nGeneralization Tests (Section 5.3, Table 2):\n- Validity: The authors clearly test model generalization by evaluating on Abg-CoQA, a dataset distinct from training domains, thereby assessing whether learned collaborative strategies transfer effectively.\n- Issues: Generalization tests were limited to a single additional dataset. Including multiple diverse external benchmarks could strengthen claims about generalizability.", "relation_to_broader_scientific_literature": "1. Addressing Limitations of Single-turn Reward Methods (e.g., RLHF)\n- In prior work, RLHF significantly advanced LLM fine-tuning using single-turn feedback, optimizing immediate next-turn responses. This method is now a standard baseline but is inherently limited for multi-turn interactions because it neglects the cumulative effects of model responses on long-term user goals.\n- CollabLLM introduces Multiturn-aware Rewards (MR), explicitly modeling the long-term trajectory of human-model interactions.\n- Unlike traditional RLHF, MR leverages forward sampling to anticipate conversational impact, thus directly overcoming RLHF’s known limitations regarding long-term interaction quality.\n\n2. Proactive, Clarification-based Interactions\n- Prior work explored using LLMs proactively, especially for clarification questions. However, these methods often rely heavily on predefined interaction patterns or specific domains, limiting adaptability. Prompting-based methods attempt similar proactive strategies but struggle with generalizability across diverse user scenarios.\n- CollabLLM generalizes proactive collaboration through reinforcement learning, enabling more versatile interactions that dynamically adapt to user intent across different tasks and scenarios.", "essential_references_not_discussed": "Some related work not included in the paper:\n1. Abdulhai, Marwa, et al. \"Lmrl gym: Benchmarks for multi-turn reinforcement learning with language models.\" arXiv preprint arXiv:2311.18232 (2023). \n2. Shani, Lior, et al. \"Multi-turn reinforcement learning from preference human feedback.\" arXiv preprint arXiv:2405.14655 (2024).", "other_strengths_and_weaknesses": "Strengths:\n- The paper introduces CollabLLM, a novel training framework designed to enhance multiturn human-LLM collaboration.\n- The development of Multiturn-aware Rewards (MR) represents a significant advancement over traditional single-turn reward mechanisms, addressing the limitations of models like RLHF in long-term interactions.\n- The paper is well-structured, with clear explanations of the methodology, experimental setups, and results.\n\nWeaknesses:\n- While the paper introduces innovative concepts, it does not discuss certain related works that have explored similar themes, such as multi-turn reinforcement learning benchmarks and proactive clarification in language models. Incorporating these references could provide a more comprehensive context for the contributions.\n- The paper could benefit from a more detailed discussion of the computational overhead associated with the proposed methods, particularly regarding the scalability of Multiturn-aware Rewards. This information would be valuable for practitioners considering the implementation of CollabLLM.", "other_comments_or_suggestions": "As suggested in strengths and weaknesses.", "questions_for_authors": "- How does CollabLLM integrate with existing reinforcement learning frameworks and algorithms? Are there specific modifications or considerations required to implement MR within standard RL pipelines? (Clarifying the integration process would help us understand the feasibility of adopting CollabLLM in other systems. Complex integration requirements could pose barriers to implementation.)", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "XhhwXnOTKZ", "reviewer": "Reviewer_Qxpu", "summary": "Existing fine-tuning techniques for LLMs, such as Reinforcement Learning from Human Feedback (RLHF), primarily maximize the reward for immediate and single-turn responses. However, real-world users often reveal their intents or preferences until later interactions; thus, to streamline their interaction with users and improve user satisfaction, LLMs must be able to actively guide users to clarify and refine their intents throughout the multi-turn conversation. This paper proposes ColabLLM, a novel training framework that encourages LLMs to collaborate with humans in multi-turn conversations. The collaborative simulation module of ColabLLM samples future conversations with users to estimate how the LLM response would impact future turns.  This long-term impact, termed Multiturn-aware Reward (MR), evaluates responses based on both task-specific success and efficiency to assess the multi-turn collaboration quality. Once this MR is computed, ColabLLM employs established RL algorithms to fine-tune the backbone LLM. In addition, the paper releases three multiturn datasets across diverse domains - collaborative document editing, coding problem assistance, and multiturn problem solving - to fine-tune and evaluate LLMs' multiturn conversational capabilities.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate your approval and useful feedback! Here we address your concerns:\n\n---\n\n## **[Claims And Evidence] \"How does the multiturn reward effectively encourage collaboration?\"**\n\n- At the **methodology** level, Multiturn-aware Reward (MR) encourages collaboration by accounting the long-term effect of each response in future interactions. A more collaborative LLM should achieve higher extrinsic and intrinsic rewards within a fixed number of turns, corresponding to task completion and efficiency/engagement. `=> clarity`\n\n- At the **data** level, since the data is generated from applying MR, supervised training on the multiturn data encourages the model to replicate the behavior. MR is also used for online training, which reinforces the model to optimize MR based on current generations. `=> clarity`\n\n- At the **experiment** level, we show that in simulated environments and real-world user study, CollabLLMs achieve best collaboration with humans in the collaborative aspects we considered. `=> evidence`\n\n---\n\n## **[Claims And Evidence] Could the authors elaborate more on lines 685-687.**\n\nWe apologize for this concise statement and provide the following clearer explanation, and we will update the paper accordingly: \n\n- In plain language, our MRs provide **turn-level** signals as the long-term effect of each model response, while existing methods such as MTPO [1] applies **trajectory-level** rewards, making it hard to dissect the effect of good/bad responses inside a conversation. \n\n- In depth, MRs answers **`\"How does the current model response impact future interactions?.\"`** We train the model to produce responses that maximize the final reward given the context. \nIn contrast, MTPO leverages preference rewards between two conversations, answering **`\"Which conversation should the model prefer?\"`** where multiple responses are entangled and hard to dissect their influences to the entire conversation.\n\n- Borrowing terms from causality literature [2], we refer to the first mechanism as **interventional**, and the second as **observational**. The interventional mechanism by MRs offers more fine-grained estimation of the effect of model responses. \n\nTherefore, to your questions, the difference is not simply from the user simulator. In fact, the existing methods like MTPO can have user simulators while still relying on post-hoc comparison between conversations. In terms of data creation, the data we generated (data available in the next answer) comes from turn-by-turn filtering guided by MRs, while MTPO is trained on pairs of good and bad conversations. \n\nWe hope the explanations convey the distinctions clearly. We have added them to the related works section to improve the paper's quality, as suggested by the reviewer.\n\n---\n\n## **[Claims And Evidence] Request to access the multiturn datasets**\n\nWe are happy to provide the datasets! \n\nYou can click the anonymous link: https://anonymous.4open.science/r/collab-llm/notebooks/load_conv_data.ipynb, we provide the script that loads the full datasets from `data/`. The notebook displays random samples of the data, and can be updated to show different samples. We hope this helps with getting a clearer picture of our dataset!\n\n---\n\n## **[Other Suggestions] \"the design must be better conveyed to show that this work goes beyond re-designing the self-training framework\"**\n\nThanks for raising the insight about self-training and our work!\n\nSelf-training typically involves generating synthetic data to improve model performance. Under this scope, our work indeed 1) conducts synthetic data generation, and 2) leverages this data to improve the model.\n\nHowever, there are many possible ways our current design can end up being. In particular, our key intuition in (2) is that we want the model to **be aware of future outcomes and prioritize responses with higher long-term impact**, which constitutes our main novelty. To achieve this, we consider both extrinsic and intrinsic metrics for a more user-centric estimation of long-term effects. The rest of the components in (1) are fundamentally aimed at **computing long-term effects**, with the use of a user simulator for forward sampling being just one, albeit minor, contribution of ours. Moreover, our models are not merely trained on offline synthetic data; they also leverage MRs for online training to adapt model behavior.\n\n---\n\n## **Summary**\nWe hope our responses address your concerns on clarity and data accessibility. \n\nWe acknowledge the previous draft may have been concise in explaining differences with related works. We have revised thoroughly and added a paragraph in Related Work discussing the connection with self-training.\n\nWe sincerely appreciate your reconsideration of our work in light of our responses. Thank you for your insights!\n\n---\n\n## **Reference**\n\n[1] Shani et al. Multi-turn reinforcement learning from preference human feedback. arXiv:2405.14655.\n\n[2] Pearl et al. Causal Inference in Statistics: A Primer. 2016.", "claims_and_evidence": "C1. ColabLLM encourages LLMs to collaborate with human users in multiturn conversations -> lacks evidence or clarity\n- It is clear that multiturn data are collected with an LLM that is prompted as a user simulator. On the contrary, it is rather unclear how the multiturn reward obtained with multiturn data effectively encourages collaboration. \n\nC2. The reward design of ColabLLM aligns with causal effect estimation. -> somewhat convincing\n- Could the authors elaborate more on their claim in lines 685-687 of the Appendix \"existing methods primarily rely on post-hoc trajectory-level data, learning from observed conversations rather than explicitly modeling the causal effect of individual responses on long-term task success?\" Is this because the existing methods do not have a user simulator, and thus they lack the ability to probe the long-term impact of LLMs responses? Then, how do existing methods create post-hoc trajectory-level data? How are these different from the data created by ColabLLM? These distinctions would make the related works section more comprehensive and improve the paper's quality as a standalone academic paper. \n\nC3. Three datasets for fine-tuning and evaluating LLMs on multiturn conversations are proposed -> lacks evidence\n- Please correct me if I am wrong, but I could not find supplementary materials or anonymized links to these datasets. While the authors provide samples in the Appendix, I would have preferred to see the full datasets to get a clearer picture of the dataset quality and quantity.", "methods_and_evaluation_criteria": "The proposed method is technically sound. The evaluation criteria are aligned with the convention in this field.", "theoretical_claims": "The claim on the cause-effect estimation wiht ColabLLM is unclear; this concern was raised above.", "experimental_designs_or_analyses": "The experimental designs and analyses are valid.", "relation_to_broader_scientific_literature": "This was included in the summary.", "essential_references_not_discussed": "N/A", "other_strengths_and_weaknesses": "Strengths\n- The problem is well-motivated. Improving the LLMs' multiturn conversational capability is an important problem.\n- The proposed method, which relies on data generation with a user simulator and multi-turn reward design, is technically sound. \n- The paper additionally introduces three public benchmarks for multi-turn conversation research. \n- The results are strong, and the authors compare the proposed method against strong baselines. \nWeaknesses (already mentioned in the Claims and Evidence section)\n- How the proposed method encourages collaborative behavior could be better discussed.\n- The cause-effect estimation with the user simulator could use some clarification.", "other_comments_or_suggestions": "As collecting real data is expensive, and it is commonly believed that most of the data available at hand (ex) via internet crawling) have been exhausted during LLM pre-training, utilizing LLM-generated data to improve performance on downstream tasks or to encourage a certain behavior is gaining popularity. It appears that this work shares this philosophy of self-training, and has made clever modifications to tailor it specifically for multi-turn conversation capabilities. Therefore, the methodology, while described using fancy technical terms, such as \"forward-looking strategies\" or \"user simulator,\" may not necessarily be as novel as the authors claim them to be. For instance, when explained in plain terms, estimating forward-looking strategies with a user simulator is more or less the same as generating more realistic multi-turn data with an LLM that is prompted to behave like a human user. Therefore, the motivation and design principle behind the methodology must be better conveyed to showcase that this work goes beyond simply engineering and re-designing the self-training framework for the purpose of multiturn conversations and does indeed reveal an unknown or discussion-worthy application of LLM-backed data generation. If this concern, raised in the Claims and Weaknesses section, could be addressed, I am willing to raise my score to recommend acceptance of the paper.", "questions_for_authors": "Please refer to the above section.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "QUwPWoHHGK", "reviewer": "Reviewer_2AZE", "summary": "This paper studies how to enhance human-AI collaboration by improving multi-turn conversations. Concretely, authors propose a learning framework CollabLLM that uses a reward function aware of multi-turn setup in reinforcement finetuning. This multiturn-aware reward takes account of both task performance and user satisfaction, and is proved empirically effective in a few simulated environments including text editing, code generation and math reasoning.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely appreciate your approval and insightful comments! We further address your comments:\n\n---\n\n\n## **Comment 1: \"A comparison of the potential divergence of simulated user and human user during training would further strengthen this work\"**\n\nGood point! We agree that prompt-defined user simulators could be biased. Due to the large-scale forward conversations needed for computing Multiturn-aware Rewards, we only used the user simulator during training. While not meant for training, in the user study, we have collected conversations between real users and our models. \nHere we provide some insights on feature differences and similarities between prompt-defined user simulators and real users' responses. \n\n**Differences:**\n- *Real users communicate with shorter, more fragmented sentences often containing grammatical errors, while simulated users typically use more complete sentences.*\n- *Real users frequently change direction mid-conversation and add highly specific personal details (like \"eight dogs\"), while simulated users are more predictable.*\n- *Real users express emotional reactions more bluntly (\"that's awful,\" \"sounds pretentious\") and use more casual language patterns with abbreviations and incomplete thoughts compared to simulated users.*\n\n**Similarities**:\n\n- *Both exhibit iterative content development patterns - gradually revealing requirements rather than providing complete information upfront.*\n- *Both prioritize accessibility - consistently requesting simplification of complex topics, actionable advice, and concrete examples that make information more understandable.*\n- *Both express preferences about content structure and style, and acknowledge when content meets or doesn't meet their expectations.*\n\nWe train models by interacting with simulated LLM users, while conducting user studies to evaluate model performance with real users as a test of generalization. The generalization is validated from the experimental improvements. With more resources, it would be interesting to further explore the sensitivity of the impact of users on the trained models. \n\nWe are glad that the reviewer raises this interesting question. We have added these insights to `Appendix: F`, as well as adding the key gap in `Section 6: Real-world User Study`. Hopefully these could shed light into future human-centered LLM training. \n\n\n---\n\n\n## **Comment 2: \"Multiturn-aware reward function is intrinsically hard to define for ambiguous tasks\"**\n\nThanks for the comment! Optimizing for ambiguous tasks could be hard even in single-turn settings, in addition to the multiturn settings that we study. \n\nIn general, on ambiguous tasks, such as recommendation or consultant, one mitigation is to use LLM Judges whose inputs contain the task definition. The assumption is that, since LLMs are powerful at reasoning, they are fairly good at telling if the task completion is good or not. Empirically, this has been commonly adopted in evaluation and benchmarking.\n\nFor our Multiturn-aware Reward function, we incorporate both extrinsic and intrinsic rewards, where the intrinsic rewards (interactivity and efficiency) should be applicable to most applications. For extrinsic/task-specific metrics, the same design discussed above can be applied over the future conversations. \n\nWe have added this discussion to our main paper!\n\n---\n\n\n## **Question 1: \"the computational expensive aspect of the forward sampling strategy?\"**\n\nThanks for this question! In online training, the computational overhead in forward sampling comes from 1) generation from the policy model and 2) generation from the user simulator. \n\n- For (1), the computational overhead and cost are fairly low since we have integrated vLLM [1] in model inference. \n- For (2), we use gpt-4o-mini as the user simulator where we expect user responses to be concise, i.e., the number of output tokens to be small. \n\nWe compute the average statistics over 100 future conversations when w=1,2,3 on MediumDocEdit-Chat, the document editing task, which has the maximum computational overhead among the three tasks. Please see the table in https://anonymous.4open.science/r/collab-llm/images/cost.png \n\nWe have added this information to `Appendix C.3: Computational Cost During Training`, and hopefully this provides clear details.\n\n---\n\n\n## **Summary**\n\nWe thank the reviewer for the interesting questions and comments. We hope our responses can alleviate the concerns on 1) user simulators, and 2) applicability in ambiguous tasks. We also provide more details about computational expenses, as well as improving our manuscript accordingly. Please don't hesitate to let us know if you have more questions!", "claims_and_evidence": "- This work addresses a key limitation of existing LLMs: the tendency to generate single-turn responses without actively engaging in clarifying or guiding user intents. \n- The proposed multiturn-aware reward function is an interesting contribution, as it incorporates both extrinsic task success metrics and intrinsic user experience factors (e.g., excessive tokens to read and write).\n- This work thoroughly evaluates CollabLLM across multiple tasks, showing substantial improvements in task success and user engagement with simulated users. Additional human evaluation with 201 crowd worker participants provides empirical validation beyond automated benchmarks, showing increased user satisfaction and reduced time spent on tasks.\n- The ablation section provides useful insights into the importance of forward-looking strategies in reinforcement learning.\n- This paper is very well written.", "methods_and_evaluation_criteria": "Three multiturn interaction benchmarks are proposed, covering document editing, code generation, and math problem-solving. The evaluation criteria is diverse, including measurements on task accuracy, interactivity, user satisfaction, user efforts etc.", "theoretical_claims": "N/A", "experimental_designs_or_analyses": "Experimental design and analysis sections look sound and thorough.", "relation_to_broader_scientific_literature": "The discussion around the suboptimal performance in handling multi-turn interactions is well-motivated and supported by literature. The insight of using multiturn-aware reward and forward sampling strategies is shown to be effective, and seems generalizable to other tasks.", "essential_references_not_discussed": "N/A", "other_strengths_and_weaknesses": "- A comparison of the potential divergence of simulated user and human user during training would further strengthen this work, as prompt-defined simulated LLM users could be substantially biased. \n- Multiturn-aware reward function is intrinsically hard to define for ambiguous tasks, which limits its applicability.", "other_comments_or_suggestions": "N/A", "questions_for_authors": "Any idea on the computational expensive aspect of the forward sampling strategy? This seems nontrivial especially for long conversations.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "1mvO52Gstw", "reviewer": "Reviewer_YkB9", "summary": "COLLABLLM is a new training framework designed to improve multi-turn human–LLM collaboration. Its core idea is to simulate a collaborative conversation setup where a Multiturn-aware Reward (MR) function estimates the long-term impact of model’s responses, rather than focusing solely on immediate single-turn outcome (as in standard RLHF).\n\nMain Contributions:\n-Multiturn-aware Rewards (MR): A conversation-level reward function that encourages the LLM to seek and incorporate additional context or clarification from users if it improves overall task success.\n-New Multi-turn Interaction Benchmark: which covers 3 challenging tasks related to document editing, coding, and mathematics.\n-COLLABLLM outperforms base (or prompt-engineered) baselines on 3 test sets by boosting task accuracy by 18.5% and interactivity by 46.3%, as judged by LLM evaluators. In a large-scale user study with 201 Amazon Mechanical Turkers, COLLABLLM also increases user satisfaction by 17.6% and saves 10.4% of user time compared to baselines.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate the reviewers' extensive and thoughtful outputs! We'll address each comment below: \n\n---\n\n### **[Other Weaknesses] \"The improvements (over prompt engineering) on simulated experiments of Tab 1 are small… What would be the topline obtained w/ gpt4-o + prompt engineering?”**\n\nThanks for the comment! We'd like to provide the following justification: \n\n- **Task-specific Performance upper bound of improving collaborative strategies:** The training of CollabLLMs does not necessarily provide more task-specific knowledge, rather, the primary goal is to explore the best collaboration strategies for the models to understand user requests and deliver its internal knowledge. Therefore, we believe an upper bound constraint exists for improvement we gain from improving collaboration strategies.s\n\n- **Challenging Tasks:** In particular on MediumDocEdit-Chat, the task itself is challenging with a large generation space for writing the blogs, making the BLEU metric hard to improve.\n\nGiven the first reason, comparing with gpt-4o (in general, comparing between models with different base models) may not be fair, as it's difficult to isolate effects from stronger knowledge versus better collaborative strategies. There might also be data contamination risks with gpt-4o, which likely trained on internet-scale data possibly including our datasets. However, we're willing to run testing if you're curious about the results. \n\n---\n\n### **[Other Weaknesses + Questions] \"Where do the improvements stem from? Multiturn-aware reward design, or reward modification, or interaction between both factors?”**\n\nThis relates to one of the reviewer’s questions - `\"What about optimizing helpfulness using w>0?\"`. For validation, we extended helpfulness, intrinsic, and extrinsic metrics to $w=1,2,3$ in model training, following the same setting as Figure 4. \n\nPlease see visualized results at: https://anonymous.4open.science/r/collab-llm/images/ablation.png. Previously, we see helpfulness as a type of intrinsic reward. Here, we found applying helpfulness alone doesn't work well on all metrics, especially as it encourages lengthy responses. For the other metrics, increasing $w$ generally benefits the performances corresponding to the metric. For example, applying extrinsic rewards improves the BLEU score, while the ITR and token amount underperforms Extrinsic+Intrinsic adopted in CollabLLM.\n\nNotably, the design of extrinsic and intrinsic rewards is independent of the key design in MR function, which highlights the estimation of a response’s long-term effect via forward samplings. In fact, one can apply multiple intrinsic rewards including helpfulness and extrinsic rewards in the MR. \n\nWe hope the additional ablation clarifies the source of improvements, we have changed `Figure 4` with this more comprehensive study.\n\n---\n\n### **[Questions] Clarify the experiments on MediumDocEdit-Chat**\n\n**(1) “How is the document extracted?”`**\n\nThe document is extracted by prompting an LLM to extract the final written content after multiple conversational turns between the user simulator and the model, then comparing this generated document with the original Medium article using BLEU score to evaluate similarity. We added this description to the experiment setup. \n\n**(2) “How exactly is this scoring for ITR metric performed?”`**\n\nPlease see `Appendix D.4`, starting from `Line 915`, we provide full prompts to produce ITR.\n\n**(3): “Is BLEU the right metric for this task? Why not also use LLM judges?”`**\n\nYes we can use LLM judges, but since we already have human evaluation on document quality in the user study, which, we believe, is perhaps more convincing to assess the document quality.  We're happy to reevaluate if the reviewer thinks otherwise. \n\n---\n\n### **[Questions] \"Figure 4: why does ITR performance decrease from $w=2$ to $w=3$?”**\n\nGreat catch! Following the MR formulation in `Eq. 1`, with increasing $w$, each model response's effect estimation on the final goal should be more accurate, and ITR performance should improve. However, for scalability, we conduct Monte Carlo sampling for future conversations with sample size fixed at 3 `(Appendix C.2)`, inevitably introducing estimation errors. This explains potential fluctuation with increasing $w$. We added this interpretation to the ablation section. Thanks for this question! \n\nLastly, we have fixed the typo identified by the reviewer.\n\n---\n\n## **Summary**\n\nWe deeply appreciate your detailed and thoughtful comments. We hope our answers address concerns about (1) significance of improvements, (2) source of improvements, and (3) clarity about evaluation and ablation results. Please let us know if you have more questions! Thank you again!", "claims_and_evidence": "Yes they are (although i have concerns/reasons to reject expressed in the specific strength/weeknesses section)", "methods_and_evaluation_criteria": "Yes they are  (although i have concerns/reasons to reject expressed in the specific strength/weeknesses section)", "theoretical_claims": "There are no strong theoretical claims in this paper", "experimental_designs_or_analyses": "Yes i checked:\n-simulated experiments\n-crowdsourcing study", "relation_to_broader_scientific_literature": "Paper is well positioned related to the scientific litterature, although i'm not a specialist of multi-turn human–LLM collaboration.", "essential_references_not_discussed": "They are discussed but in the supp material", "other_strengths_and_weaknesses": "reason to accept\n-A well-designed framework aimed at improving multi-turn human–LLM collaboration, supported by  experimental evaluation.\n-Introduction of a new benchmark specifically for evaluating multi-turn interactions.\n-An in-depth case study that goes beyond accuracy metrics, providing deeper insights into model behavior and interaction quality.\n-Proposed approach implicitly addresses (and casually solves?) the question clarification problem by optimizing for long-term goal achievement, naturally encouraging the model to seek more clarifications when needed, as demonstrated in the paper.\n\nreasons to reject\n-The improvements on simulated experiments of tab 1 (35% to 36-38% BLEU, 12.5 to 15) are small between prompt engineering and the proposed method (using PPO or DPO), raising doubts about real impact. With overall small performance improvement and a model size ≤8B parameters, the validation of the method is not 100% convincing to me. What would be the topline obtained w/ gpt4-o and the same prompt engineering for instance ?\n-It's unclear whether the improvements stem specifically from the multi-turn-aware reward (with w>0, regardless of whether the reward is based on helpfulness, intrinsic, extrinsic, or a mix) or from the reward modification itself (replacing helpfulness with extrinsic + intrinsic rewards). or alternativelly is it the interaction between both factors that drives the gains?", "other_comments_or_suggestions": "Typo:\n-caption of fig 2: Figure 2: Real examples from COLLABLLM and non-collaborative LLM fine-tuing \n=> fine-tuNing", "questions_for_authors": "-MediumDocEdit-Chat: task performance is evaluated using BLEU, which measures similarity between the extracted document and the original articles.\nHow is the document extracted? It's unclear what exactly is being generated here.\nIs BLEU the right metric for this task? Why not also use LLM judges for a more qualitative assessment?\n-Interactivity (ITR): engagement is evaluated using an LLM judge (Claude-3.5-Sonnet) with scores rescaled to [0,1]\nBut how exactly is this scoring performed? The methodology needs more clarity.\n-Figure 4: why does ITR performance decrease when the forward sampling window size increases from w=2 to  w=3? This behaviour seems counterintuitive to me. What could explain it?\n-What about optimizing helpfulness (as assessed by the LLM evaluator) using w>0? Is it feasible? If so, why was this approach not explored?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {"ethical_review_concerns": "none"}}, {"review_id": "gHjGpUtO2T", "reviewer": "Reviewer_mUmH", "summary": "While state-of-the-art Large Language Models (LLMs) trained with RLHF are good at following the instructions from users, this paper argues that they are often ``passive responders'' where they only passively respond to ambiguous or open-ended user requests. To address this limitation, this paper proposes to train LLMs with multi-turn aware utility through a conversation-level reward and a forward sampling process. The conversation-level reward is composed of an extrinsic reward of task completion and intrinsic reward that prioritizes user experiences. Experiments have shown that in three simulated tasks, CollabLLM (trained with either PPO or DPO) is able to achieve better performances compared to prompting baselines. A large-scale user study is also carried out and it is shown that CollabLLM can indeed enhance the user satisfaction over multiple turns.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their approval and rigorous comments! Here we address the remaining concerns:\n\n---\n\n### **[Experimental Designs] \"It would be nice to have additional discussions about using a stronger model as the user simulator.”**\n\nGreat catch! A user simulator should follow the language style of previous user turns while exhibiting typical user behaviors like evolving needs and limited background knowledge. This requires LLMs to role-play users with **basic understanding of real-world user traits and follow instructions effectively** [1,2,3].\n\nThough not motivated by self-training, we initially tried using `Llama-3.1-8B` as our user simulator to reduce latency. Unfortunately, it performed poorly, frequently getting \"confused\" and solving problems as an assistant instead of a user. This observation is the same even for `Llama-3.1-70B`. \n\nWe think this raises an interesting research problem - *while we have increasingly superior LLM assistants trained to solve problems, we lack user models that learn from real-world user behaviors*. Building better user models can be valuable running simulations for real-world applications.\n\nTherefore, \"self-play\" was difficult to implement within the scope of this work, but this points to potential future work. We've added discussion in future work directions.\n\n---\n\n\nMoreover, to reviewer’s Question \\#3 - **“It would be helpful if the authors can provide some insight in terms of the limitations of applying LLMs as user simulators.”**. We discussed this in our response to `Reviewer 2AZE` (please see the section's beginning), given space constraints. We've added the discussion in the paper to provides insights. \n\n---\n\n\n### **[Relation To Broader Scientific Literature]**\n\n**(1) “(add) prior literature in the main text.”**\n\n\nYes we agree. We will and should have space in the final version to accommodate the Related Work section in the main text! \n\n---\n\n\n**(2) “it would be valuable if the authors can provide some quantitative comparisons with prior \"Multiturn training for LLMs”**\n\nThanks for this suggestion! Previously we looked into MTPO (Shani et al. 2024), but unfortunately we didn’t find an existing implementation. Another relevant method in the category, ArCHer (Zhou et al. 2024), is a hierarchical multiturn RL approach that requires the training of three LLMs: 1) utterance-level Q-function and 2) utterance-level value function, and 3) a token-level actor that maximizes the prediction of the Q-model, which exhibits high complexity in our setup. Moreover, this contributes to a more task-specific training with learning the token-by-token policy within each turn, while CollabLLM offers a more intuitive and generalizable method for multiturn interaction with a single model. \n\n---\n\n\n### **[Essential References Not Discussed] \"(compare with) works that use user simulators to improve LLMs”**\n\nWe further added the following content in the related work section:\n```\nRecent works employ user simulators to enhance dialogue systems [4,5] and LLMs [6,7,8]. Tseng et al. improve both dialogue systems and simulators through reinforcement learning during their interactions. Recently, Hong et al. leverage LLMs to create diverse synthetic dialogues with varying user personas, then train smaller dialogue models to optimize conversational outcomes. CollabLLM differs in leveraging user simulators in forward sampling to account for long-term effect in both offline and online training. \n```\n\nIt is also worth mentioning that our main contribution comes from the key intuition in making the model be aware of future outcomes and prioritize responses with higher long-term impact. The other components are fundamentally aimed at computing long-term effects, with the use of a user simulator for forward sampling being just one, albeit minor, contribution of ours. \n\n---\n\n## **Summary**\n\n\nWe sincerely hope our answers mitigates your concerns about (1) the user simulator models and their difference between real users (2) the comparison with works within \"Multiturn training for LLMs\" and works using user simulators and. \n\nGiven our answers, we would very much appreciate it if you could reconsider your evaluation. Thank you very much!\n\n\n---\n\n\n## **Reference**\n\n[1] Park et al. Generative Agent Simulations of 1,000 People. ArXiv.\n\n[2] Wang et al. User Behavior Simulation with Large Language Model-based Agents. ArXiv. \n\n[3] Yoon et al. Evaluating Large Language Models as Generative User Simulators. NAACL 2024.\n\n[4] Shi et al. How to Build User Simulators to Train RL-based Dialog Systems. EMNLP 2019.\n\n[5] Tseng et al. Transferable Dialogue Systems and User Simulators. ACL-IJCNLP 2021.\n\n[6] Hong et al. Zero-Shot Goal-Directed Dialogue via RL on Imagined Conversations. ArXiv.\n\n[7] Hu et al. Unlocking the Potential of User Feedback: Leveraging Large Language Model as User Simulator to Enhance Dialogue System. CIKM 2023.\n\n[8] Faltings et al. Interactive Text Generation. EMNLP 2023.", "claims_and_evidence": "In general, claims in the paper are well supported through empirical evidence, both through simulated experiments of three tasks and a conducted user study with Mechanical Turks. I particularly appreciate the inclusion of the user study with more than 200 participants that show the positive generalization from model-simulated user to real users.", "methods_and_evaluation_criteria": "Do proposed methods and/or evaluation criteria (e.g., benchmark datasets) make sense for the problem or application at hand?\n\nYes, the proposed evaluations are well-suited for the problem.", "theoretical_claims": "No theoretical claims made in the paper.", "experimental_designs_or_analyses": "The experiment designs make sense in general, and I appreciate the ablation of the effect of reward mechanisms in Figure 4 and Figure 6, and the zero-shot generalization experiment in Table 2. However, it seems strange why a different model GPT4-o is used as the user simulator compared to the original model Llama-3.1-8B-Instruct. It would nice to have some additional discussions in terms of the effect of using a stronger model as the user simulator, and I would be curious to see if it is possible for the model to improve with \"self-play\" without relying on a stronger model as a user simulator.", "relation_to_broader_scientific_literature": "I find a detailed discussion of the paper's contribution with the broader literature missing from the main text. Although Appendix B mentions related works, I believe that it is important to address the connection of the paper to the prior literature in the main text. Additionally, in the related works section, the paper mentions that \"However, these methods primarily rely on post-hoc trajectory-level data, learning from observed conversations rather than explicitly modeling the causal effect of individual responses on long-term task success (see Appendix A for further details).\", but the relative advantage of learning from real-user conversations and explicitly modeling the causal effect of individual responses seem to be missing from the paper as far as I can tell. While I understand that real-user conversations might be hard to obtain, it would be valuable if the authors can provide some quantitative comparisons with the methods used in the prior literature.", "essential_references_not_discussed": "Would be nice to have a comparison with other works that make use of user-simulators to improve LLMs, such as Zero-Shot Goal-Directed Dialogue via RL on Imagined Conversations (https://arxiv.org/abs/2311.05584).", "other_strengths_and_weaknesses": "See above", "other_comments_or_suggestions": "See above", "questions_for_authors": "My main questions are mentioned in the previous responses and I summarize them below:\n\n1) While I understand that real-user conversations can be hard to obtain, it may strengthen the paper and better situate the paper in the related literature if the authors can provide quantitaive comparisons with prior \"Multiturn training for LLMs\" as mentioned in the related works section.\n\n2) It would be nice to add discussions explaining the difference of this paper from other prior literature that uses LLMs as user simulators, e.g. ttps://arxiv.org/abs/2311.05584.\n\n3) It would be helpful if the authors can provide some insight in terms of the limitations of applying LLMs as user simulators. In my experience, I found instruction-tuned LLM as user simulators tend to be overly agreeable and always consent to the requests while real users can be harder to deal with. Curious if such issues are also observed in the experiments.\n\n4) Would be nice to se", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper addresses an essential problem: improving the LLMs' multi-turn conversational capability. The key strengths of this paper as identified by the reviewers include a method for generating data with a user simulator and multi-turn reward is important and technically sound.  The results are strong, and the authors compare the proposed method against strong baselines. The paper additionally introduces three public benchmarks for multi-turn conversation research.

The authors did substantial work to address all reviewer concerns.
Decision: Accept (oral)