Learning to Clarify: Multi-turn Conversations with Action-Based Contrastive Self-Training
Authors: Maximillian Chen
Ruoxi Sun
Tomas Pfister
Sercan O Arik
Venue: ICLR 2025 Poster (2025)
Keywords: dialogue system
conversation modeling
reinforcement learning
mixed-initiative interaction
LLM
RLHF
domain adaptation
data-efficient learning
clarification questions
Primary area: applications to computer vision, audio, language, and other modalities
TL;DR: Data-efficient contrastive RL tuning approach for improved conversation modeling in mixed-initiative interaction.

Abstract: Large language models (LLMs), optimized through human feedback, have rapidly emerged as a leading paradigm for developing intelligent conversational assistants. However, despite their strong performance across many benchmarks, LLM-based agents might still lack conversational skills such as disambiguation -- when they are faced with ambiguity, they often overhedge or implicitly guess users' true intents rather than asking clarification questions. Under task-specific settings, high-quality conversation samples are often limited, constituting a bottleneck for LLMs' ability to learn optimal dialogue action policies. We propose Action-Based Contrastive Self-Training (ACT), a quasi-online preference optimization algorithm based on Direct Preference Optimization (DPO), that enables data-efficient dialogue policy learning in multi-turn conversation modeling. We demonstrate ACT's efficacy under data-efficient tuning scenarios, even when there is no action label available, using multiple real-world conversational tasks: tabular-grounded question-answering, machine reading comprehension, and AmbigSQL, a novel task for disambiguating information-seeking requests for complex SQL generation towards data analysis agents. Additionally, we propose evaluating LLMs' ability to function as conversational agents by examining whether they can implicitly recognize and reason about ambiguity in conversation. ACT demonstrates substantial conversation modeling improvements over standard tuning approaches like supervised fine-tuning and DPO.

Published as a conference paper at ICLR 2025
LEARNING TO CLARIFY: MULTI-TURN CONVERSATIONS WITH ACTION-BASED CONTRASTIVE SELF-TRAINING
Maximillian Chen Ruoxi Sun', Tomas Pfister', Sercan Ö. Arik
1 Google *Columbia University
maxchen@cs co sollun lumbi..ed (rucxis s, tpf stter, soari ( @google.ccmm
ABSTRACT
Large language models (LLMs), optimized through human feedback, have rapidly emerged as a leading paradigm for developing intelligent conversational assistants. However, despite their strong performance across many benchmarks, L.LM-based agents might still lack conversational skills such as disambiguation when they are faced with ambiguity, they often overhedge or implicitly guess users' true intents rather than asking clarification questions. Under task-specific settings, high-quality conversation samples are often limited, constituting a bottleneck for LLMs' ability to learn optimal dialogue action policies. We propose Action-Bas Based Contrastive Self-Training (ACT), quasi-online preference optimization algorithm based on Direct Preference Optimization (DPO), that enables data-efficient dia- logue policy learning in multi-turn conversation modeling. We demonstrate ACT's efficacy under in data-efficient tuning scenarios, even when there is no action label available, using multiple real-world conversational tasks: aabular-roonded question-answerte machine reading comprehension, and AmbigSQL, novel task for disambiguating information-sede seeking requests for complex SQL generation towards data analysis agents. Additionally, we propose evaluating LLMs ability to function as conversational agents by examining whether they can implicitly recogmize and reason about ambiguity in conversation. ACT demonstrates substan- tial conversation modeling improvements over standard tuning approaches like supervised fine-tuning and DPO.
INTRODUCTION
Conversations offer a natural and effective way for humans and intelligent systems to collabo- rate (Amershi et al., 2019; Lemon, 2012). The impressive capabilities of large language models (LLMs) have powered the rapid development of many generalist conversational assistants such as ChatGPT! and Gemini (Gemini Team et al., 2023), which present an opportunity for users to verbalize their need for assistance on complex tasks. However, the promises of conversational interfaces also come with the complexities of language. Human conversation riddled with ambiguity, whether be due to humans' tendency to underspecify (Zipf, 1949) or even due to syntactic errors (Messer, 1980). Moreover, disambiguation becomes even more important in complex domains where can be difficult multi-turn process to achieve common ground (Beers et al., 2006). As it stands, existing L..--powered conversational agents continue to struggle with modeling ambiguity (Liu et al., 2023), and tend to exhibit unwanted behavior such as overhedging (Ouyang et al., 2022) or generating responses which represent a "guess" of the user's intent (Deng et al., 2023a) (see Figure 1).
One of the primary reasons that LLMs may exhibit unwanted conversational behaviors is that their language modeling objective during pre- traininin or supervised fine-tuning (SFT) is not directly aligned with this goal (Ouyang et al., 2022). While approaches like Ouyang al. (2022) propose LLM "alignment" using post-training approaches like reinforcement learning from human feedback (RLHF) (Christiano al., 2017), existing models still struggle with conversational tasks spanning
Work done during an internship at Google. htaps///ooema.ccmmmloo/hggggg

---

Published as a conference paper at ICLR 2025
What the recorded inveatment for Americas?
Recorded Investments
Leae Recevables
Americas
EMEA
JAsia Pacifiic
Incorrect Assumation: 53,419 User's need nol me!
Possible Information
3,419
1,186
Overaaggng: Risk of infommation overladd
O O
Seaking Indentis:
Laiss Rebelvebles
The value of Loase Receivables was 3,419 and the value of Lown Receivables wes 6,726.
Recelvables
State both
6.726
Indant Cladficatio
Average both
What kind recorded Investment are you asking about?
Figure 1: Simplified example of ambiguity present at cabular-groundee conversational question answering based on Deng al. (2022). A conversational agent should recognize when there is ambiguity and ask clarifying question towards a more accurate final answer.
multiple turns (Wang et al., 2023). This is partly due to the fact that existing approaches do not directly optimize for pragmatic skills (e-g. Bai al. (2022)). Moreover, there is often high variance in the target distribution of a particular use case, so it is imperative that downstream adaptation approaches can effectively steer LLM policies. Given large-scale in-distribution training data, this may be feasible with standard SFT or RLHF. But, dialogue policy learning can be particularly challenging given limited data (Chen et al., 2022a; Dong et al., 2023) and collecting high-quality conversational datasets can be difficult for reasons such annotation costs and privacy concerns (Chen et al., 2023a).
This motivates the design of conversational adaptation approach for LLMs which is more closely aligned with the goal of modeling ac- tions in multi-tury conversation. We focus
0.65-
Action-based Conttrastive Self-Training
Carification Reasoning (DROP F1)
on improving LLMs' abilities to implicitly se- lect conversational strategies in ambiguous con- texts, and propose an approach called Action- Based Contrastive Self-Training (ACT).ACT is a sample-efficient, quasi-online Direct Prefer- ence Optimization algorithm (Rafailov al., 2024) which focuses on contrasting the differ- ences between an agent's possible pragmatic conversational actions. We demonstrate ACT's sample-efficient performance on a diverse range of
Sx data efficiency
Supervised Iine-Tnning
0.55
clarification reascring:
10% 1 action aptimization
Gemini Pro (Prompting)
0.45
Number of Conversations
Figure 2: ACT greatly outperforms standard tuning approaches in data-efficient settings for conversa- tional modeling, exemplified here PACIFIC.
conversational tasks: (i) aubular-gruunde question answering, (ii) ma- chine reading comprehension, and (iii) text-to- SQL generation, demonstrating large improvements compared to standard adaptation approaches (see Figure 2). Our work highlights the necessity of considering action-based preferences for conversa- tional tasks, and we propose a workflow for evaluating LLMs' ability to recognize and reason about ambiguity conversation.
2 RELATED WORK
2.1 IIXEI-INITIATIV CONVERSATIONAL. AGENTS
Neural approaches to building mixed-initiative conversational agents typically consist of two components: an understanding and planning module (e.g., a binary prediction task to determine whether to ask a clarifying question or provide an answer), and generation module which can be controlled at pragmatic level using the output of the planning module (Chen et al., 2017; 2022b; Qian al., 2022; Yu, 2017) (e.g., forming an utterance which follows the predicted action).
Generation Many existing works focus on novel training methodologies to improve conditional generation as complement to planning, with approaches such as multi-objective SFT (Chen et al., 2022b; Wen et al., 2016) or introducing specialized embeddings for control codes (Keskar et al., 2019). LLMs have vastly improved performance in pragmatically-comme coottrllll generation (Chen et al., 2023b), but all these approaches still depend on conversational planning. Planning remains a difficult task natural interaction is not deterministic and often requires long-horizon planning.
2

---

Published as a conference paper at ICLR 2025
Planning The planning task can be viewed stochastic Markov Decision Process (Wang et al., 2020b; Yang et al., 2021; Yu et al., 2023) in which some dialogue state is drawn from a potentially unknown distribution, given the previous dialogue state and an imposed action. However, the action itself is not literally presented to the interacting parties; rather, an action is a low-dimensional representation of the pragmatic intent carried by a given dialogue utterance (i.e., dialogue act Sadek (1991); Stolcke et al. (2000); Wu et al. (2023); Yu & Yu (2021)). As such, training planning modules often requires complex long-horizon reasoning and simulation to model the responses and intents of each interacting party. Such efforts have examined combining neural models with search algorithms (Cheng et al., 2022: Väth et al.. 2023: Yu et al., 2023) and simulation (Deng et al., 2023c; Wang et al., 2020a; Yu et al., 2023). However, such modular approaches can incur high computational overhead (Yu et al., 2023) and may result in error propagation while not directly optimizing for response quality itself. We propose directly optimizing dialogue action planning as an implicit subtask of response generation in muxxddiniiiiativ conversation contexts, as we discuss in Sec. 3.2.
2.2 LEARNING FOR LLM ALIGNMENT
The current paradigm of LLM training for downstream use cases consists of three phases: pre- training, supervised fine-tuning (SFT) for instruction-folloiing and tuning for alignment with human preferences (Tunstall et al., 2023; Rafailov et al., 2024; Lee et al., 2023; Ouyang et al., 2022). We primarily focus the phase of tuning for alignment. These approaches typically start with initial policy model obtained by conducting SFT on a target task (TSFT), before performing tuning (often with RL), using contrastive preference examples (most commonly collected through human feedback (Ouyang et al., 2022) or a similar proxy like LL ----eeerrrteed feedback (Lee et al., 2023)). In the case of online algorithms like PPO, reward model is first fit over the preference examples so that it could be used for RL optimization (Ouyang et al., 2022). Such algorithms have certain advantages which may benefit the Markov Decision Process-like nature of conversations namely, diverse search space as opposed to fixed dataset, flexible reward functions, and broader policy exploration. However, PPO notoriously difficult to tune, and offline algorithms such as DPO (Rafailov et al.. 2024), SLiC (Zhao et al., 2023), and IPO (Azar et al., 2024) have become widely adopted as an LLM adaptation approach because they bypasses explicit reward modeling and thus only require one set of hyperparameters to optimize (Huang et al., 2024; Rafailov et al., 2024; Zhao et al., 2023; Zheng et al., 2023) while still achieving similar empirical results given a fixed preference dataset.
Many of our contemporaries also question the limits of fully offline preference learning algorithms and have examined "online" variants of them (Guo et al., 2024; Xu et al., 2023; 2024b). Yuan et al. (2024) proposes iterative DPO, and Chen et al. (2024) proposes variant where ground-truth responses are considered winning, and responses sampled from the previous iteration of a policy model are considered losing. Pang et al. (2024) applies variant of iterative DPO to optimize externalized reasoning chains. Our work differs from these in that we are proposing a novel approach to customize LLMs for specific conversational settings, in particular, multi-turn conversational settings. While other works look at applying DPO to conversations in general (e.g. Sun al. (2024)), their focus is still on single-turn response optimization. ACT considers multi-turn trajectories for preference optimization, and to our knowledge, our work is the first paper to consider contrastive learning on the basis of conversational actions.
On-Policy DPO
3 METHODS
3.1
PROBLEM SETUP
We consider the task of tuning an LLM to function as a mixed-initiativv conversational agent. Through a series of dialogue interactions with a user, the LLM is expected to assist the user by eventually providing a correct response to their request. Unlike the common agent interaction setting where users completely control the flow of interaction with the expectation that agents may autonomously complete tasks such as online shopping (Liu et al., 2024), mixed-initiaaive agents should understand how to redirect the flow of the interaction (Allen et al., 1999) through the execution of conversational actions or strategies such as clarifying questions (Chu-Carroll, 2000; Peng et al., 2018).
Notation Consider a goal-oriented conversational environment. Let TT, be an LLM's policy parameterized by f at timestep é 2 0, with Kref being the reference policy model (i.e., Tref + Too). Let D be dataset consisting of conversations. Let each conversation in D contain dialogue turns, through which a user is requesting one or more pieces information from an agent. The turn state of conversation (the observed utterances and actions given by each interacting party) timestep
3

---

Published as a conference paper at ICLR 2025
On-Poliay Polloyy Response Sampling
Trajectory Simulation and Evaluation
Pollcy Update
Scanario A Wrong Implict.ctionn
Pronpt (yl: "Shae za isfercatio stoe:
pland the Bobineen B-22
Resporue Reaporme
Compat oaparien
Opra
Wilninn dg):
O
Repprme thyft infermation
ieplich
Scararia Rt: insamnet Simutated Outsame
Losing Hlrpont balt
Lookty Actin Annie
ARIAITT ang... page
Sunepted Rorpame byy Arryes bokking for kethertgg
Stexbrd Outoex
Restace Lowinp Resparos with Simukandd Trqpetory
Inforazaiion Goel (igh SE: urg....
X
Whath
Scorarto B2: Correct Strudisted Outcorns
supereal +
Pakky at ttop
Chenster
Action
Sieplaed Oetine
Replace Wivsiig fixaponue with
Sinutator
Pollcy Stop! n1A
Repent Procasa Until Convergance
Figure 3: Overview of the tuning phase of ACT. For each initial contrastive pairing from Dyref (constructed as in Sec. 3.2. we sample an on-policy response from the model being tuned. After evaluating the sampled response's trajectory, update the contrastive pairing by either replacing the existing winning or losing response. The model policy is updated using the objective in Eq. 1.
i can be represented by t.. Implicitly, each t. is part of trajectory which ends when the user's question expressed at an earlier timestep j I i is answered. Any turn 4 can first be broken down into two primary components: p; and r;, where Pt can be a prompt at i, consisting of any task-specific information (e.g. a SQL database schema, tabular data or retrieved passages) combined with any existing dialogue context, and r; is the ground truth system-side response at Next, we can let gi be the goal response which resolves 4 ''s implicit trajectory, i.e., the answer to the user's original question after any possible clarification turns. the single-turn trajectory case, 9: < T.. Each r; implicitly expresses an action, ag, where exists in the latent Action Space S of a particular task and ag can be inferred by some Action Annotation Agent G Thus, we can formally represent turn state I, using the tuple (pi, Fi dz). For the datasets considered in our experiments, S = [CLAIIFY, ANSWER] (although the method can be extended to broader action space). We assume access to a controllable generation model (M), Action Classifier (A) and model which can be controlled to function as a User Simulator (U). As we discuss in Sec. 3.2, M is used for preference data creation whereas and U used during tuning and evaluation. We illustrate this notation in Fig. A4.
User Simulators Works such as Deng et al. (2023c); Yu et al. (2023) directly prompt LLMs for gral-oriented tasks conditioned dialogue context and task objectives. Our implementation of U is inspired by this setup. We first prompt an LLM to summarize the user's information-seeking goal. Then, we form another prompt using this summary along with the current dialogue context to simulate a user response. Prompting with this goal summary allows for more flexibility than directly providing the user simulator with the ground truth information objective. We provide details on our implementation of U in Appendix H.
Action Classifiers In the datasets we considered, the possible actions are to either "clarify" or "directly answer" a question. We directly use few-shot in-context learning as action classifier A. We provide details on our in-context examples for A in Appendix G.
3.2 ACT: ACTION-BASHE CONTRASTIVE SELF-TRAINING
One of the north stars in developing intelligent conversational models is the ability to automatically produce responses which take actions which lead to the highest probability of conversational suc- cess (Wu et al., 2023; Zhao et al., 2019). We propose ACT, an approach that adapts generic LLMs for dialogue generation and models action planning extension of the DPO algorithm which maintains its case of use of offline method while incorporating the flexible exploration found during online learning. ACT relies on a few intuitions. 1) Contrastive preferences are an intuitive medium for demonstrating the pragmatic differences between the implicit actions of "winning" and "losing" dialogue responses. 2) Conversational improvements require multi-turn optimization, which are difficult to express using only single-turn contrast pairings. 3) The
implicit subtask. ACT is quasi-online
'The gold standard for full label supervision in the context of fixed dataset is the scenario in which G may be well-designed human annotation framework such crowdsourcing. However, at inference time or in settings without label supervision, the implicit action must be inferred by other means such as classification.
4

---

Published as a conference paper at ICLR 2025
gradient of the objective of DPO-like algorithms (see Eq. 2) is weighted based on the log probabilities assigned to the winning and losing responses. and by construction, on-policy response sampling yields high-probability token sequences. ACT is summarized in Fig. 3. ACT consists of two phases: action-based contrast dataset construction (Alg. 1) and contrastive self-training (Alg. 2).
3.2.1 CONSTRUCTION OF PREFERENCE DATA
The preference dataset primarily consists of con- trastive winning-losing action pairs, as shown in Alg. 1. That is, for each conversation turn 4; in dataset D. we can construct Dpref consisting of augmented tuples. We add rejected action af which is sampled from S \ a, winning response
Algorithm 1 Building Contrastive Action Pairs
input Dataset D. Conditional generation model M, Ac- tion Space S, Action Annotation Agent G
1: Initialize empty dataset preff- for conversation turn
2:
do
Let a; C(pi.rr) D Infer Contextual Action
Yw, < rú, and Yt, which is losing response
4:
Let aj = \ 0z b Determine Rejected Action
sampled using M. Given that each df is pre- defined when constructing Dyreff we use a high capacity LLM (Chen et al., 2023b) rather than tuning a smaller one or asking crowdworkers for losing response construction (more details in Appendix F).
5:
Let Yuoi =
6:
Sample yts Par(-lpi, a().
Let t = (pe, Is. as ais Buss yts.
Add f22 to Dpref
output Dpref
Action optimization for unlabeled conversations "in-the-wild" Obtaining gold-standard ambigu- ity annotations may not always be possible. In such settings, one can obtain pseudo-labe) supervision using a classifier the Action Annotation Agent G rather than human annotation. We discuss details and analyze performance in Sec. 5.4. Depending on the data, it may be appropriate to introduce an initial preprocessing step which involves inferring user satisfaction similarly to Shi et al. (2024).
3.2.2 SELF-TRAINING USING ON-POLICY CONVERSATION TRAJECTORY SIMULATION
As in DPO training, we continuously sample batches from Prref- Although each conversation turn in each batch j has default winning (yws) and losing (yua) response, we also sample an on-policy response yi from Te, We use A to determine whether the implicit action of y; matches the inferred action a, of the ground truth response. If the implicit action of y; is incorrect, we set Vi Y/ If it does match ag then we simulate the outcome gf of the trajectory resulting from ys using U and Te,- If the trajectory outcome of fails to meet task-specific heuristics (e.g.. low semantic similarity an incorrect execution), we set yo to the entire simulated trajectory resulting from (e.g.. "Are you looking for..." "What is the average total number... ... " + "SELECT max ... in Figure 3). Otherwise, we set Ywu to the simulated trajectory (e.g. "Are you looking for..." + "What is the average total number... " + "SELECT avg ... in Figure 3).
3.2.3 CONTRASTIVE RL TUNING FOR ALIGNMENT
After constructing the up-to-date po-te winning Yos and losing Wh pairing at turn i through simulation (See. 3.2.2), we update the policy model (Ta) using the DPO training objective (Rafailov et al., 2024),
which is as follows (we ignore the i iterator for simplicity):
(Tro Fref) (pp [log (Blog
(y p) 3 log To p)
)
(1)
Kref (yw (p)
Tree pp)
where p is a prompt consisting of concatenation between task info and conversation history {ø1, //,.... Ti-1, W5-1,;; 2; with each 2; and y: representing observed user-side and sistem-side utterances turn i; Yo and y are the designated "winning" and "losing" responses or trajectories as set in Sec. 3.2.2; Wref is the initial reference policy model; and /3 is hyperparameter that regularizes the ratio between Ty and Tref- The gradient of this objective given follows:
V VoLoro(TO: Tref)
o (Rolp. vi Ro(p, vw))) Va log (2w pp) Va log π ( p)] (2)
- -B
Classiffing yi 's action optimizes the following: argmax Es Pa(as (tlalppi,3.. "The user (denoted M;+1) is sampled according to u4+1 Pc((.pp.3s))
5

---

Published as a conference paper at ICLR 2025
Algorithm 2 ACT: Action-Based Contrastive Self-Training
input Initial Policy Model Re.- Action Contrast Dataset Drref, Number of Batches B, Action Classificr A, User Simulator U. Task Heuristic H. Heuristic Tolerance
1: for conversation turn & in batch by sampled from Dreff where 0 S & do b Sample response from the current model policy
2: Sample y, ~ Po,(-lp) (-lp.)
if Action A(y)) # Action a; then
4: Set yu y/
b Implicit pragmatic action does not match ground truth
5: else
6: Initialize Trajectory Add y, to Trajectory
8: while A(ys) + ANSWER do
9:
Clarification Answer = (o(p: ys)
b Simulate User Clarification
10:
Add Clarification Answer to Trajectory
11:
Ve41 Pao (P: 04)
Simulate next policy response
Add v.L11 to l'rajectory
13:
if H H(Traeecory outcome, Ground Truth Outcome g:) > then
14:
Let Yws = Trajectory
Reward acceptable trajectory outcome
15:
else
16:
Let yn Trajectory + (f) until convergence (eq 2)
Penalize bad trajectory outcome
17:
output men
(ulp) for a given policy model π and reference model Trafs as proven under the assumptions in Rafailov et al. (2024). The intuition behind the objective is that the gradient of the loss function would increase the likelihood of winning responses yo Y,, and would decrease the likelihood of losing responses E Yy, with each example being weighed by the magnitude of how incorrectly the implicitly defined reward model ranks the paired responses.
where R(p.y) y) 9 log *(yl))
4 EXPERIMENTAL SETUP
ACT is a sample-efficien approach to adapt an LLM to a conversational action policy. We are primarily concerned with learning optimized implicit selection for agent-side clarification question asking, and we thus evaluate ACT as a tuning approach for three complex conversational information-seeking tasks. As base model for tuning experiments, et al., 2023) which has been instruction tuned on UltraChat and aligned to human preferences on UltraFeedback (Cui et al., 2023; Ding et al., 2023; Tunstall al., 2023).
Zephyr B, version of Mistral 7B (Jiang
Table 1: Experimental results on PACIFIC's public evaluation set. ACT achieves the strongest performance compared all tuning approaches across every condition every metric. Tuning-basee adaptation strategies are not given any in in-context examples at inference time, whereas ICI. baselines include in-context conversation examples using the same strategy.
Adaption Setting
Action-level
Content-level
Base Model
Approach
Conversations
Macro F1 f
Turn F1 f
Traj. F1
Post-Clarify Fl
Gemini Standard
81.4
59.7
58.7
Claude Sonnet
Standard ICL
10
71.9
43.7
42.0
28.5
Gemini Pro
SFT
50
71.2
51.8
45.7
9.9
Gemini Pro
SFT
100
75.2
64.3
54.6
8.5
Gemini Pro
SFT
250
88.0
67.4
59.3
10.2
Zephyr 7B-8 8
SFT
50
69.0
57.8
61.3
43.5
Zephyr 7B-8
IRPO
50
67.7
59.1
56.7
34.4
Zephyr 7B-3
ACT (ours)
50
82.2
62.8
61.9
57.2
Zophyr 7B-3
SFT
100
82.3
58.6
60.3
49,9
Zephyr 7B-3
IRPO
84.5
60.4
$5.2
Zephyr 7B-8
ACT (ours)
100
86.0
65.0
62.0
57.4
Zephyr 7B-3
SFT
250
86.9
65.1
63.3
56.7
Zephyr 7B-3
IRPO
250
85.4
64.9
58.4
40.3
Zephyr 78-3
ACT (ours)
250
89.6
68.1
65.7
62.0

---

Published as a conference paper at ICLR 2025
Table 2: Abg-CoQA test set evaluation results. Although Claude Sonnet achieves the highest Action-level performance when prompted with in-distribution in-context conversation examples, does not result in improved multi-turn goal completion. ACT combines on-policy sampling with multi-turn simulation to achieve the best multi-turn goal completion ability on all data settings.
Adaptation Setting
Action-level
Content-level
Base Model
Approach
Conversations
Macro F
Turn Similarity
Traj. Similarity
Gemini Pro
Standard ICL
10
55.5
67.0
72.2
Claude Sonnet
Standard ICL
10
66.0
50.1
54.3
Zephyr 7B-3
SFT
50
44.6
53.3
64.2
Zephyr 7B-8
ACT (ours)
50
52.3
66.2
68.8
Zephyr 7B-8
SFT
100
52.6
63.1
69.4
Zephyr 7B-8
ACT (ours)
100
51.1
69.5
71.4
Zephyr 7B-8
SFT
250
53.5
64.0
66.2
Zephyr 7B-8
ACT (ours)
250
53.3
72.5
75.1
4.1 DATASETS
We investigate three mixed-initiatin initiative conversation tasks in which user interacts with assistant to retrieve some information. our setup of each task, a user asks a query which may or may not be underspecified. The assistant is tasked with providing a response which may either be clarifying question or an attempt to directly answer the user's query. For each task, we synthesize the initial rejected responses by prompting Gemini Ultra as the conditional generation model, M. ACT is evaluated on a diverse set of datasets spanning various domains; tabular conversational QA, conversational QA for machine reading comprehension, and conversational text-to-SQL. generation.
4.1.1 PACIFIC: CONVERS QA FOR TABULAR DATA
PACIFIC is task for proactive conversational question answering grounded on mixture of tabular and textual financial data (Deng et al., 2022). This may involve generating the correct words from a given span, from multiple spans, or providing correct arithmetic expression. The official evaluation for PACIFIC uses a numeracy-focused token overlap metric called DROP F1.
4.1.2 ABG-COQA: CONVERSATIONAL QA FOR MACHINE READING COMPREHENSION
Abg-CoQA is conversational question answering dataset for disambiguation machine reading comprehension (Guo et al., 2021). As there are no arithmetic expressions, we use embedding- semantic distance with SentenceBERT (Reimers & Gurevych, 2019) as an evaluation metric, which has been used to more flexibly measure question-answering performance (Risch et al., 2021).
4.1.3 AMBIGSQL: AMBIGUOUS CONVERSATIONAL TEXT-TO- TE-S-L GENERATION
AmbigSQL is our new task for SQL-grounded conversational disambiguation. We systematically perturbed unambiguous queries from Spider, a popular single-turn text-to-SQL benchmark (Yu et al., 2018), resulting in paired training examples which can be easily incorporated into contrastive RL tuning. Each trajectory evaluated by whether the final proposed SQL. query matches the ground truth query's s execution result.
Our motivation in constructing AmbigSQL. stems from the idea that disambiguation can lead to improved task performance. We prompt an LLM to introduce three types of ambiguous information requests. Those which the requested information ambiguous (e.g., "Show details about singers ordered by age from the oldest to the youngest"), those in which the requested population is ambiguous (e.g., "Which ones who live in the state of Indiana?": see Table A12), and finally, those in which the requested presentation of results is ambiguous (e.g. "Show name, country, age for all singers ordered by age"" see Table A 13). We found that constructing a SQL query for an underspecified request with and without clarifications demonstrating the necessity of clarifying questions. We provide additional details in Appendix C.
result in performance gap of up to 45.8% (see Table A14),
4.2 EVALUATION SETUP
We conduct evaluations of ACT's ability to reason about ambiguity in conversation to better accom- plish conversational goals along two dimensions.
7

---

Published as a conference paper at ICLR 2025
Table 3: AmbigSQL test set evaluation. Zephyr tuned with ACT is able achieve the strongest task performance within each data setting. There are especially large performance improvements in post-clarification SQL. execution match when data resources are more scarce.
Adaptation Setting
Action-level
Content-level
Base Model
Approach
Conversations
Accuracy
Macro FI
Execution Match
PC Execution Match
Gemini
Standard ICT.
10
72.1
70,9
63.5
75.2
Claude Sonnet
Standard ICL
10
68.5
63.8
66.5
72.4
Zephyr 7B-9
SFT
50
77.4
77.4
21.9
13.9
Zephyr 7B-9
IRPO
50
91.0
91.0
27.8
30.8
Zephyr 7B-3
ACT (ours)
50
80.8
80.7
43.6
38.1
Zephyr 7B-9
SFT
100
97.2
97.2
43.3
34,3
Zephyr 7B-3
IRPO
100
96.2
96.1
45.0
37.0
Zephyr 7B-8
ACT (ours)
100
99.2
99.3
48.0
49.6
Zephyr 7B-8
SIT
250
99.8
99.7
51.0
50.7
Zephyr 7B-3
IRPO
97.0
97.1
49.7
45.6
7ephyr 7B-99
ACT (ours)
250
99.9
99.8
52.3
53.0
Zephyr 7B-3
SFT
14,000 (All)
99.8
99.8
63.1
60.4
Agent task performance: We evaluate whether ACT improves multi-turn task completion capabili- ties. PACIFIC and Abg-CoQA are initially proposed only with static single-turn evaluations. We mirror this by conducting turn-level evaluation where we compare the model's response to the ground truth utterance given in response to the user's query, using the task-specific heuristics given Sec. 4.1. Since we are specifically concerned with improving LLMs' multi-turn capabilities, we additionally propose a multi-turn evaluation scheme for the trajectory outcomes in all three tasks considered. While the sampled response from an LLM is clarifying question, we simulate user response and re-sample another response from the evaluated LLM until it attempts to answer the original query. We evaluate this outcome against the user's ground truth information-seeking goal. We use A and U for simulation as described in Sec. 3.2.2 for ACT, and use the heuristics defined in Sec. 4.1. An example is illustrated in Fig A5. In PACIFIC and AmbigSQL, we also compute task performance the simulated responses in which the model has previously asked any clarifying questions, in order to get a more fine-grained measure of the model's ability to reason about its own clarification questions. Details of each evaluation metric for each task are provided in Appendix D.
Implicit ambiguity recognition: To help further understand an agent's 's multi-turn task completion ability, we consider "dialogue act accuracy" (Chen et al., 2023b). Assuming access to ground-truth ambiguity labels, given a contextually-ambiee question, otherwise, should attempt to provide the requested information. We primarily consider Macro F1 since PACIFIC and Abg-CoQA have highly imbalanced classes.
user request, model should generate clarifying
4.3 BASELINES
Prompting baselines We compare our tuning approaches with smaller models against various prompt-based approaches for multiple frontier LLMs: Gemini 1.5 Pro. Gemini 1.5 Flash, Claude 3.5 Sonnet, and Claude 3.0 Haiku". The results for Gemini Flash and Claude Haiku are included in Appendix B due to space constraints. We use 10 conversations as in-context examples, with three different prompting frameworks: i.) "Standard" which uses the same instruction formatting used for tuning; ii.) chain-of-thought reasoning (Wei et al., 2022); and fii.) "Proactive MIPrompt", the prompting baseline in Deng et al. (2023c), which is combination of the mixed-initiative prompting approach used in Chen et al. (2023b) and Proactive Prompting (Deng et al., 2023b). We provide a detailed description of each style with examples in Appendix E.
Tuning baselines We compare ACT with supervised fine-tuning (SFT) as well as other off- off-po and on-policy approaches to DPO-based alignment. For SFT, we use the ground truth responses for each dataset's training split. As for DPO-based alignment, an on-policy variant called Iterative Reasoning Preference Optimization (IRPO) was recently proposed and has gained traction for improving model performance in reasoning tasks such as arithmetic. We have thus evaluated IRPO on our two quantitative reasoning tasks, PACIFIC and AmbigSQL. A popular off- policy approach is to sample responses from two high capacity models, with Y,, coming from whichever model is
Wee access each I.I.M through Vertex AI: https: //cl///cluu googlgooleee. le. com coo/vertex-ee //ocs//
8

---

Published as a conference paper at ICLR 2025
Table 4: Examining ACT on PACIFIC with unlabeled conversational data. We assume access to action labels and instead use 0-shot Gemini Pro as the source of action label supervision.
Task Adaptation Environment
Action-level
Content-leve
Buse Model
Pramework
Action Supervision
Tuning Ex.
Macro FI f
Turn Fl
Traj. Fl
Post-Clarify F!
Zephyr 7B-3
SPT
69.0
57.8
61.3
43.5
Zephyr 7B-8
ACT
Crowdsourced
82.2
62.8
61.9
57.2
Zephyr 7B3-3
ACT
Pseudo-labeled
50
80.1
62.4
61.1
54.7
Zephyr 7B-8
SFT
NA
100
82.3
58.6
60.3
49,9
Zephyr 7B-8
ACT
Crowdsourced
100
86.0
65.0
62.0
57.4
Zephyr 7B-0
ACT
Pscudo-labeled
100
84.8
63.5
61.5
56.1
Zephyr 7B.3
86.9
65.1
63.3
56.7
Zephyr 7B-3
ACT
Crowdsourced
250
89.6
68.1
65.7
62.0
Zephyr 78-3
ACT
Psendo-laheled
250
89.0
68.1
64.9
61.0
of higher capacity (henceforth DPO-Dist; see Mitra et al. (2023); Mukherjee et al. (2023); Xu et al. (2024a)). We present DPO-Dist results in Appendix B.
EXPERIMENTAL RESULTS
To emulate real-world scenarios with limited data, we evaluate ACT as tuning approach in different scenarios with limited conversation samples across set of diverse tasks.
5.1 CONVERSATIONAL QA WITH TABULAR GROUNDING
In Table 1, we see that across all three data-efficient settings considered, ACT achieves the strongest performance across all metrics compared both SFT and IRPO, which has the advantage of additional test-time computation (Snell et al., 2024; Pang et al., 2024). In particular, ACT achieves up to a 19.1% relative improvement over SFT when measuring the tuned model's ability to implicitly recognize ambiguity (from 69.0 to 82.2 Macro F1) given only 50 conversations as tuning data. We also observe that ACT has greatly improved data efficiency compared adapter- based SFT with Gemini Pro, with a relative improvement of as high as 35.7% in multi-turn task performance (from 45.6 to 61.9 9 in terms of trajectory-leve DROP F1). Additionally, tuning with ACT in these limited data settings grants the model the ability to match or outperform frontier LLMs used with in-context learning despite having zero in-context examples during inference. Overall, we find that on-policy learning and multi-turn trajectory simulation are crucial for improved multi-turn goal completion.
5.2 COIVERS QA FOR MACHINE READING COMPREHENSION
Our results for Abg-CoQA are presented in Table In all three data settings, we observe that ACT achieved the strongest performance in terms of task-specific metrics (notably, in terms of trajectory-level embedding similarity). However, in the setting with 100 and 250 conversations, Zephyr tuned with SFT slightly outperforms ACT in terms of implicit action recognition, although action-level performance primarily helps to contextualize clarification reasoning ability. We discuss this point further in Appendix A. Our approach leads to the strongest turn-level and trajectory-level task performance in all conditions, indicating improved multi-turn reasoning.
5.3 CONVERSATIONAL TEXT-TO-SQL GENERATION
We find that although the prompting baselines do not achieve high Action Accuracy, the bench- marked frontier I.MSs can achieve relatively strong downstream performance terms of execution match. In contrast, tuning Zephyr with both SFT and ACT results rather high Action Accuracy but lower text-to-SQL performance compared to the frontier LLMs. We observe that holistically, ACT achieves the largest relative performance improvements in multi-turn task performance compared to other tuning approaches, although the downstream SQL generation ability of larger models is much greater than that of smaller models. This is primarily due to the SQL generation benefiting greatly from scale (Sun et al., 2023b). It is possible that multi-turn performance larger models can be improved further if ACT applied, as even able yield larger performance improvements than baseline approaches for quantitative reasoning such as IRPO.
5.4 ACT TN-THE- WILD: LEARNING WITHOUT DIALOGUE ACTION SUPERVISION
Although we have ambiguity labels in the tasks considered here and use them for supervision in Tables 1-3, also demonstrate that it is possible to perform action-based tuning in the absence

---

Published as a conference paper at ICLR 2025
Table 5: Ablation study of various conditions using PACIFIC's 50 conversation setting.
Macro Fl
Tum FI
Trj.. Fl Post-Clarify FI
Action Importance
ACT
w/ Random Actions
63.2
55.3
58.7
32.8
Ablation of ACT sabcomponents
ACT
w/o on-policy sampling
74.8
61.5
59.1
40.5
ACT
sampling simulation
81.4
60.8
50.1
ACT (full)
82.2
62.8
61.9
57.2
ACT with unaligned foundation models
Gemma 2B SFT
57.7
38.0
40.5
17.0
Gemma 2B ACT
62.7
42.6
44.0
24.8
Mistral 7B SFT
57.7
53.8
51.4
27.7
Mistral 7B ACT
75.7
58.1
$7.6
31.9
of action-label supervision. We pre-existing LLM, Gemini 1.5 Pro, as zero-shot action annotator to re-label the ground truth Assistant-side turns on the PACIFIC corpus. We find that there is astonishingly high agreement (98.5%) with the ground truth action labels. Our results in terms of both Action-level and Content-level metrics reflect that there is nearly no empirical difference in performance. This highlights the potential of ACT being highly effective for adaptation to "in-the-wild" settings with smalls amount of unlabeled conversational data.
5.5 ABLATION STUDIES
Are action-based preferences necessary? One of the key factors of ACT is that the contrastive pairs highlight differences between conversational actions. In Table 5 ("ACT w/ Random Actions"), we additionally examine the importance of action selection by randomly sampling both the winning and losing action when constructing the preference pair, and observe this underperforms normal ACT.
Do we need on-policy sampling? Table 5 ("ACT w/o on-policy sampling"), we examine the importance of on-policy sampling by evaluating normal off off-policy DPO on the dataset constructed in Sec. 3.2.1. While we do observe some improvements over SFT (e.g.. from 69.0 to 74.8 Macro FI), the overall improvements may be due to the fact that the off-policy negative responses are not guaranteed to lie in the language manifold of the policy model, and distribution shift may be too difficult to overcome with off-policy learning (Guo et al.. 2024).
much larger when using on-policy sampling as with full ACT. This
Is trajectory simulation necessary? ACT is better-aligned with multi-turn conversations due to its on-policy trajectory simulation. Without multi-turn simulation, our approach can be viewed similarly to on-policy DPO variants like Pang et (2024), but with a cnnversation-speeiff reward signal which accounts for conversation actions and task heuristics. In Table 5 ("ACT w/ sampling w/o simulation"), we find that this trajectory-level simulation critical to improving multi-turn performance, especially the policy model's 's ability to reason about its own clarification questions.
ACT model agnostic? The base model in our main experiments, Zephyr, is obtained by aligning Mistral. In Table ("ACT with unaligned foundation models") we observe a performance gap of 6.5 Action Fl and 4.3 Trajectory Fl after ACT tuning for the two models. However, our results demonstrate ACT can improve performance regardless of pre-existing alignment with human feedback, although it can help as an improved model imitialization. Overall, we find that improving base model performance with ACT is model agnostic.
CONCLUSION
We propose ACT, a model agnostic quasi-online contrastive tuning approach for sample-efficient conversational task adaptation, along with a workflow for evaluation of conversational agents. We demonstrate encouraging evidence that ACT highly effective for task adaptation the limited data regime, even when there ACT with existing sophisticated tuning approaches for complex tasks like text-to-SQL generation, as well as generalization large-scale data and multi-task environments.
action labels available. Future work may also consider combining
10

---

Published as a conference paper at ICLR 2025
REFERENCES
Rishabh Agarwal, Avi Singh, Lei M Zhang, Bernd Bohnet, Luis Rosias, Stephanie Chan, Biao Zhang. Ankesh Anand, Zaheer Abbas, Azade Nova, al. Many-shot in-context learning. arXiv preprint arXiv:2404. 1018, 2024.
James E Allen, Curry I Guinn, and Eric Horvitz. Mixed-initiative interaction. IEEE Intelligent Systems and their Applications, 14(5):14-23, 1999.
Saleema Amershi, Dan Weld, Mihaela Vorvoreanu, Adam Fourney, Besmira Nushi, Penny Collisson, Jina Suh, Shamsi lqbal, Paul Bennett, Kori Inkpen, et al. Guidelines for human-ai interaction. In Proceedings of the 2019 chi conference on human factors in computing systems, pp. 1-13, 2019.
Anthropic AI. The claude model family: Opus, sonnet, haiku. Claude-3 Model Card, 1, 2024.
Mohammad Gheshlaghi Azar, Zhaohan Daniel Guo, Bilal Piot, Remi Munos, Mark Rowland, Michal Valko, and Daniele Calandriello. A general theoretical paradigm to understand learning from human preferences. International Conference on Artificial Intelligence and Statistics, pp. 4447-4455. PMLR, 2024.
Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, et al. Training helpful and harmless assistant with reinforcement learning from human feedback. arXiv preprint wrXiv:224.008800 2022.
Pieter Beers, Henny PA Boshuizen, Paul A Kirschner, and Wim Gijselaers. Common ground, complex problems and decision making. Group decision and negotiation, 15:529-5555 556,,,55,,00 2006.
Senthilkumar Chandramohan, Matthicu Geist, Fabrice Lefevre, and Olivier Pietquin. User simulation in dialogue systems using inverse reinforcement learning. In Interspeech 2011, pp. 1025-1023 2011.
Maximillian Chen, Alexandros Papangelis, Chenyang Tao, Andy Rosenbaum, Seokhwan Kim, Yang Liu, Zhou Yu, and Dilek Hakkani-Tur. Weakly supervised data augmentation through prompting promp for dialogue understanding. In NeurlPS 2022 Workshop on Synthetic Data for Empowering ML Research, 2022a.
Maximillin Chen, Weiyan Shi, Feifan Yan, Ryan Hou, Jingwen Zhang, Saurav Sahay, and Zhou Yu. Seamlessly integrating factual information and social content with persuasive dialogue. In Proceedings of the 2nd Conference of the Asca-Pacific Chapter of the Association for Computational Linguistics and the 12th International Joint Conference Natural Language Processing (Volume 1: Long Papers), pp. 399-413, 2022b.
Maximillian Chen, Alexandros Papangelis, Chenyang Tao, Seokhwan Kim, Andy Rosenbaum, Yang Liu, Zhou Yu, and Dilek Hakkani-Tur Places: Prompting language models for social conversation synthesis. In Findings of the Association for Computational Linguistics: EACI. 2023, pp. 2023a.
Maximillian Chen, Xiao Yu, Weiyan Shi, Urvi Awasthi, and Zhou Yu. Controllable mixed-initiativn di- alogue generation through prompting. In Proceedings of the 61 Annual Meeting of the Association for Computational Linguistics ( (Volume 2: Short Papers), pp. 951-966, 2023b.
Yun-Nung Chen, Asli Celikyilmaz, and Dilek Hakkani-Tur. Deep learning for dialogue systems. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics: Tutorial Abstracts, pp. 8 8-14,22 2017.
Zixiang Chen, Yihe Deng, Huizhuo Yuan, Kaixuan Ji, and Quanquan Gu. Self-play fine-tuning converts weak language models to strong language models. arXiv preprint arXiv:2401.0133. 2024.
Yi Cheng, Wenge Liu, Wenjie Li, Jiashuo Wang, Ruihui Zhao, Bang Liu, Xiaodan Liang, and Yefeng Zheng. Improving multi-tum emotional support dialogue generation with lookahead strategy planning. In Yoav Goldberg. Zoritsaa Kozareva, and Yue Zhang (eds.), Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pp. 3014-3026.

---

Published as a conference paper at ICLR 2025
Abu Dhabi, United Arab Emirates, December 2022. Association for Computational Linguis- tics. doi: 10. 3666:////222..mmlmmmmmaaa 195. URL SaRLLttts://cllattholle. org//022. aun1p-main.. 195.
Paul F Christiano, Jan Leike, Tom Brown, Miljan Martic, Shane Legg, and Dario Amodei. Deep reinforcement learning from human preferences. Advances in neural information processing systems, 30, 2017.
Jennifer Chu-Carroll. Mimic:
An adaptive mixed initiative spoken dialogue system for information queries. In Sixth Applied Natural Language Processing Conference, pp. 97-104, 2000.
Ganqu Cui, Lifan Yuan, Ning Ding, Guanming Yao, Wei Zhu, Yuan Ni, Guotong Xie, Zhiyuan Liu, and Maosong Sun. Ulrraeedback: Boosting language models with high-quality feedback. arXiv preprint arXiv:2310.01377 2023.
Yang Deng, Wenqiang Lei, Wenxuan Zhang, Wai Lam, and Tat-Seng Chua. PACIFIC: towards proactive conversational question answering over tabular and textual data in finance. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, EMNLP 2022, 2022.
Yang Deng, Wenqiang Lei, Minlie Huang, and Tat-Seng Chua. Goal awareness for conversational ai: proactivity, non-collaborativitt, and beyond. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume Tutorial Abstracts), pp. 1-10, 2023a.
Yang Deng, Lizi Liao, Liang Chen, Hongru Wang, Wenqiang Lei, and Tat-Seng Chua. Prompting and evaluating large language models for proactive dialogues: Clarification, target-guided, and non-collaboration. In Findings of the Association for Computational Linguisties: EMNLP 2023, pp. 10602-10621, 2023b.
Yang Deng, Wenxuan Zhang, Wai Lam, See-Kiong Ng, and Tat-Seng Chua. Plug-and-ple policy planner for large language model powered dialogue agents. In The Twelfth International Conference on Learning Representations, 2023c.
Ning Ding, Yulin Chen, Bokai Xu, Yujia Qin, Shengding Hu, Zhiyuan Liu, Maosong Sun, and Bowen Zhou. Enhancing chat language models by scaling high-quality instructional conversations. In The 2023 Conference on Empirical Methods in Natural Language Processing, 2023.
Guanting Dong, Hongyi Yuan, Keming Lu, Chengpeng Li, Mingfeng Xue, Dayiheng Liu, Wei Wang, Zheng Yuan, Chang Zhou, and Jingren Zhou. How abilities in large language models are affected by supervised fine-tuning data composition. arXiv preprint arXiv:2310.05492, 2023.
Dheeru Dua, Yizhong Wang, Pradeep Dasigi, Gabriel Stanovsky, Sameer Singh, and Matt Gardner. Drop: A reading comprehension benchmark requiring discrete reasoning over paragraphs. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Com- putational Linguistics: Human Language Technologies, Volume I (Long and Short Papers), pp. 2368 2378, 2019.
Gemini Team, Rohan Anil, Sebastian Borgeaud, Yonghui Wu, Jean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, et al. Gemini: family of highly capable multimodal models. arXiv preprint arXiv:2312.118000 2023.
Gemma Team, Thomas Mesnard, Cassidy Hardin, Robert Dadashi, Surya Bhupatiraju, Shreya Pathak, Laurent Sifre, Morgane Rivière, Mihir Sanjay Kale, Juliette Love, et al. Gemma: Open models based gemini research and technology. arXiv preprint 2xXiv:2403.0829. 2024.
Meiqi Guo, Mingda Zhang, Siva Reddy, and Malihe Alikhani. Abg-coqa: Clarifying ambiguity in conversational question answering. In 3rd Conference on Automated Knowledge Base Construction. 2021.
Shangmin Guo, Biao Zhang, Tianlin Liu, Tianqi Liu, Misha Khalman, Felipe Llinares, Alexandre Rame, Thomas Mesnard, Yao Zhao, Bilal Piot, et al. Direct language model alignment from online ai feedback. arXiv preprint 20X.v:2402.04792 2024.
12

---

Published as a conference paper at ICLR 2025
Shengyi Huang, Michael Noukhovitch, Arian Hosseini, Kashif Rasul, Weixun Wang, and Lewis Tunstall. The n+ implementation details of rlhf with ppo: A case study on tl; dr summarization. arXiv preprint arXhv:2403.1703/ 2024.
Albert Q Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, et al. Mistral 7b. arXiv preprint arXiv:23/0.0668: 2023.
Nitish Shirish Keskar, Bryan McCann, Lav R Varshney, Caiming Xiong, and Richard Socher. Ctrl: A conditional transformer language model for controllable generation. arXiv preprint arXiv: 2019.
Florian Kreyssig, Iñigo Casanueva, Pawel Budzianowski, and Milica Gasic. Neural user simulation for corpus-based policy optimisation of spoken dialogue systems. In Proceedings of the 19th Annual SIGdial Meeting on Discourse and Dialogue, pp. 60 60-69, 2018.
Harrison Lee, Samrat Phatale, Hassan Mansoor, Kellie Lu, Thomas Mesnard, Colton Bishop, Victor Carbune, and Abhinav Rastogi. Rlaif: Scaling reinforcement learning from human feedback with ai feedback. arXiv preprint arXiv:2309.00267 2023.
Oliver Lemon, Conversational interfaces. In Drive-riven Methods for Adaptive Spoken Dialogue Systems: Computational Learning for Conversational Interfaces, pp. 1-4. Springer, 2012.
Sergey Levine, Aviral Kumar, George Tucker, and Justin Fu. Offline reinforcement learning: Tutorial, review, and perspectives on open problems. arXiv preprint arXiv:2005.0166 2020.
Alisa Liu, Zhaofeng Wu, Julian Michael, Alane Suhr, Peter West, Alexander Koller, Swabha Swayamdipta, Noah A Smith, and Yejin Choi. We're afraid language models aren't model- ing ambiguity. In The 2023 Conference on Empirical Methods in Natural Language Processing. 2023.
Xiao Liu, Hao Yu, Hanchen Zhang, Yifan Xu, Xuanyu Lei, Hanyu Lai, Yu Gu, Hangliang Ding, Kaiwen Men, Kejuan Yang, et al. Agentbench: Evaluating Ilms agents. In The Twelfth International Conference on Learning Representations, 2024.
Donald K Messer. Six common causes of ambiguity. Technical Writing Teacher, 7(2):50-52, 1980.
Arindam Mitra, Luciano Del Corro, Shweti Mahajan, Andres Codas, Clarisse Simoes, Sahaj Agarwal, Xuxi Chen, Anastasia Razdaibiedina, Erik Jones, Kriti Aggarwal, al. Orea 2: Teaching small language models how to reason. arXiv preprini 2r.iv:2311.11000 2023.
Subhabrata Mukherjee, Arindam Mitra, Ganesh Jawahar, Sahaj Agarwal, Hamid Palangi. and Ahmed Awadallah. Orca: Progressive learning from complex explanation traces of gpt-4. arXiv preprint arXiv:2306.02707, 2023.
Erik Nijkamp, Bo Pang, Hiroaki Hayashi, Lifu Tu, Huan Wang, Yingbo Zhou, Silvio Savarese, and Caiming Xiong. Codegen: An open large language model for code with multi-turn program synthesis. In The Eleventh International Conference on Learning Representations, 2023.
Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. Advances in neural information processing systems, 35:27730- 27744, 2022.
Richard Yuanzhe Pang. Weizhe Yuan, Kyunghyun Cho, He He, Sainbayar Sukhbaatar, and Jason Weston. Iterative reasoning preference optimization. arXiv preprint arXhv:2404.19755 2024.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. Pytorch: An imperative style, gegeorerformance deep learning library. Advances in neural information processing systems, 32, 2019.
13

---

Published as a conference paper at ICLR 2025
Baolin Peng, Xiujun Li, Jianfeng Gao, Jingjing Liu, and Kam-Fai Wong. Deep dyna-q: Integrating planning for task-completion dialogue policy learning. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 2182-2192, 2018.
Kun Qian, Satwik Kottur, Ahmad Beirami, Shahin Shayandeh, Paul A Crook, Alborz Geramifard, Zhou Yu, and Chinnadhurai Sankar. Database search results disambiguation for task -rientdee dialog systems. In Proceedings of the 2022 Conference of the North American Chapter of the
Association for Computational Linguisties: Human Language Technologies, pp. 1 1158-1 1173, 2022.
Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language model is secretly a reward model. Advances in Neural Information Processing Systems, 36, 2024.
Nils Reimers and Iryna Gurevych. Sentence-bert: Sentence embeddings using siamese bert-networks. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLE) pp. 3982-3992, 2019.
Julian Risch, Timo Möller, Julian Gutsch, and Malte Pietsch. Semantic answer similarity for evaluating question answering models. In Proceedings of the 3rd Workshop on Machine Reuding for Question Answering, pp. 157, 2021.
M David Sadek. Dialogue acts are rational plans. In The Structure of Multimodal Dialogue; Second VENACO Workshop, 1991.
Pararth Shah, Dilek Hakkami- Tur, Bing Liu, and Gökhan Tür. Bootstrapping a neural conversational agent with dialogue self-play, crowdsoureing and on-line reinforcement learning. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 3 (Industry Pupers), pp. 41-51, 2018.
Taiwei Shi, Zhuoer Wang, Longqi Yang, Ying-Chun Lin, Zexue He, Mengting Wan, Pei Zhou, Sujay Jauhar, Xiaofeng Xu, Xia Song, et al. Wildfeedback: Aligning lims with in-situ user interactions and feedback. arXiv preprint arXiv:2408. 15549, 2024.
Charlie Snell, Jaehoon Lee, Kelvin Xu, and Aviral Kumar. Scaling Ilm test-time compute optimally can more effective than scaling model parameters. arXiv preprint drXiv:2408.03314, 2024.
Andreas Stolcke, Klaus Ries, Noah Coccaro, Elizabeth Shriberg, Rebecca Bates, Daniel Jurafsky, Paul Taylor, Rachel Martin, Carol Van Ess-Dykema, and Marie Meteer. Dialogue act modeling for automatic tagging and recognition of conversational speech. Computational linguistics, 26(3): 339-373 2000.
Ruoxi Sun, Sercan Arik, Rajarishi Sinha, Hootan Nakhost, Hanjun Dai, Pengcheng Yin, and Tomas Pfister. Sqlprompt: In-context text-to-sql with minimal labeled data. In Findings of the Association for Computational Linguistics: EMNLP 2023, pp. 542-550. 2023a.
Ruoxi Sun, Sercan O Arik, Hootan Nakhost, Hanjun Dai, Rajarishi Sinha, Pengcheng Yin, and Tomas Plister. Sql-palm: Improved large language modeladaptation for text-to-sql. arXiv preprint vrXiv:2306.0077 2023b.
Yuchong Sun, Che Liu, Kun Zhou, Jinwen Huang, Ruihua Song, Wayne Xin Zhao Fuzheng Zhang, Di Zhang, and Kun Gai. Parrot: Enhancing multi-turn instruction following for large language models. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). pp. 9729-9750, 2024.
Matthew Toles, Yukun Huang, Zhou Yu, and Luis Gravano. Pragmatic evaluation of clarifying questions with fact-level masking. arXiv preprint arXiv:2310.11577 2023.
Lewis Tunstall, Edward Beeching, Nathan Lambert, Nazneen Rajani, Kashif Rasul, Younes Belkada, Shengyi Huang, Leandro von Werra, Clémentine Fourrier, Nathan Habib, et al. Zephyr: Direct distillation of Im alignment. arXiv preprint arXiv:2310. 16944, 2023.
14

---

Published as a conference paper at ICLR 2025
Dirk Väth, Lindsey Vanderlyn, and Ngoc Thang Vu. Conversational tree search: A new hybrid dialog task. In Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics, pp. 1264-1280,, 2023.
Sihan Wang, Kaijie Zhou, Kunfeng Lai, and Jianping Shen. cosk-completion dialogue policy learning via Monte Carlo tree search with dueling network. Bonnie Webber, Trevor Cohn, Yulan He, and Yang Liu (eds.). Proceedings of the 2020 Conference Empirical Methods in Natural Language Processing (EMNLP). pp. 3461-3477 Online, November 2020a. Association for Computational Linguistics. doi: 18.18/53/v 1053//11222000 http:2200000002mmmm main.m.2.....
URL https: aa htt///aaaannnooooooo org/
2020. emnlp-main. 278.
Sihan Wang, Kaijie Zhou, Kunfeng Lai, and Jianping Shen. Task-completion dialogue policy learning via monte carlo tree search with dueling network. In Proceedings of the 2020 conference on empirical methods in natural language processing (EMNLP), pp. 3461-3471, 2020b.
Xingyao Wang, Zihan Wang. Jiateng Liu, Yangyi Chen, Lifan Yuan, Hao Peng, and Heng Ji. Mint: Evaluating Iims in multi-turn interaction with tools and language feedback. In The Twelfth International Conference on Learning Representations, 2023.
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. Chain-of-thought prompting elicits reasoning in large language models. Advances in neural information processing systems, 35:24824-24883 248377, 2022.
Tsung- Wen, Milica Gasic, Nikola Mrksic, Lina M Rojas- Barahona Pei-Hao Su, Stefan Ultes, David Vandyke, and Steve Young. Conditional generation and snapshot learning in neural dialogue systems. arXiv preprint arXiv:1606.03333 03.35666..3....... 2016.
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Amthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander M. Rush. Transformers: State-of-the-at natural language Empirical Methods in Natural Language Pro- cessing: System Demonstrations, pp. 38 45, Online, October 2020. Association for Computational Linguistics. URL https: www. ac web. org/ ant holooyyyy/////ooooooooooooooooot 2020. emnnp-deemde 6.
processing. In Proceedings of the 2020 Conference
Qingyang Wu, James Gung, Raphael Shu, and Yi Zhang. Diacttod: Learning generalizable latent dialogue acts for controllable task -rieeted dialogue systems. Proceedings of the 24th Annual Meeting of the Special Interest Group Discourse and Dialogue, pp. 255-267, 2023.
Canwen Xu, Corby Rosset, Luciano Del Corro, Shweti Mahajan, Julian McAuley, Jennifer Neville, Ahmed Hassan Awadallah, and Nikhil Rao. Contrastive post-training large language models on data curriculum. In Findings of the Association for Computational Linguistics: NAACL 2024, 2024a.
Jing Xu, Andrew Lee, Sainbayar Sukhbaatar, and Jason Weston. Some things more cringe than others: Preference optimization with the pairwise cringe loss. arXiv preprint arXiv:23 16682, 2023.
Shusheng Xu, Wei Fu, Jiaxuan Gao, Wenjie Ye, Weilin Liu, Zhiyu Mei, Guangju Wang, Chao Yu, and Yi Wu. Is dpo superior to ppo for I1m alignment? a comprehensive study. arXiv preprint orXiv:2404.10711 2024b.
Jingxuan Yang. Si Li, and Jun Guo. Multi-turn target-guided topic prediction with monte carlo tree search. In Proceedings of the 18th International Conference on Natural Language Processing (ICON), pp. 324-3344 2021.
Dian Yu and Zhou Yu. Midas: A dialog act annotation scheme for open domain humanmachine spoken conversations. In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume. Association for Computational Linguistics, 2021.
Tao Yu, Rui Zhang, Kai Yang, Michihiro Yasunaga, Dongxu Wang, Zifan Li, James Ma, Irene Li, Qingning Yao. Shanelle Roman, et al. Spider: A large-scale human human-Labeled dataset for complex and cross-domant semantic parsing and text-to-sql task. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pp. 399111,,,, 2018.
15

---

Published as a conference paper at ICLR 2025
Tao Yu, Rui Zhang, He Yang Er, Suyi Li. Eric Xue, Bo Pang, Xi Victoria Lin, Yi Chern Tan, Tianze Shi, Zihan Li, et al. Cosql: A conversational text-to-sql challenge towards cross- dossan natural language interfaces to databases. In Proceedings of the 2019 Conference Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNIP)) 2019.
Xiao Yu, Maximillian Chen, and Zhou Yu. Prompt-based monte-carlo tree search for goal-oriented dialogue policy planning. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pp. 7101-7125, 2023.
Zhou Yu. Situated intelligent interactive systems. PhD thesis, Carnegie Mellon University, 2017.
Weizhe Yuan, Richard Yuanzhe Pang. Kyunghyun Cho, Sainbayar Sukhbaatar, Jing Xu, and Jason Weston. Self-rewarding language models. arXiv preprint arXiv:2401.10020 2024.
Tiancheng Zhao. Kaige Xie, and Maxine Eskenazi. Rethinking action spaces for reinforcement learning in end-to-end dialog agents with latent variable models. In Proceedings of NAACL-HLT, pp. 1208-1218, 2019.
Yao Zhao, Rishabh Joshi, Tianqi Liu, Misha Khalman, Mohammad Saleh, and Peter Liu. Slic-hf: Sequence likelihood calibration with human feedback. arXiv preprint arXiv:2305.10425, 2023.
Rui Zheng, Shihan Dou, Songyang Gao, Yuan Hua, Wei Shen, Binghai Wang, Yan Liu, Senjie Jin, Qin Liu, Yuhao Zhou, et al. Secrets rlhf in large language models part Ppo. arXiv preprint arXiv:2307.04994, 2023.
George Kingsley Zipf. Human behavior and the principle of least effort. 1949.
16

---

Published as a conference paper at ICLR 2025
Prompt (pg):
You helpful Assistant answering some questions from a User about a Story. Sometimes the User will ask clear questions but other times the User will ask questions which are umbiguous or underspecified. [Story]
Chosen Response (Nw)): Do you know what materialism would be in contrast for singular explanations reulity for the categories that all philosophics are said to fall into!"
phenomenal
Materialism is form philosophical monism which holds that matter is the fardumental substance nature, and that all phenomens, incliding mostal phenomenn are identical with material interactions.
Chosen Action (a): CLARIFY
Materialism is closely related to physicalism, the view that all that uxists ultimately physical. Philowohical physicalism has evalval from materialiam sciences incopponti more soghisticared notions of physicality than more ondintry matter, such spocetime, physical energies forces, dark matter, so This ferm "physicaliem" preferred over 'matteinlism" by some, while others use the terms as if they are synanytauu.. Materialism belongs the
Rejected Response (fg)) singular
the discoveries the physica!
explanations of phannoenaa reality, materialism contrasts with idealism, neutral monism, and spiritualism.
ontology such, different from antlogical theories based un dualism or pluralism. For singular explanations of the phenemenal reality. materiaismm would be contrist to idealism, neutral monism. and spiniualism.
Rejected Action (a,): Answer
Information Goal (g)) In the context of the
primary philosophical categories, materialism contrasted with idealism.
Despite the large sumber of philosaphical schools and subiie all philosophies said to fall into two peimary categories, which other: Idealism, and materilism.fa [a] The basic proposition these two categories pertains to the nature of reality, and the primary distincrion between
mances between
contrast each
the way they fundamental questions: 'what does reality consist of?" and "how does it originate To idcalists, spint mind or the objects mind (ideas) peimary, and matter spcondary. To materialists, muttrr primary, and mund or spirit secordsry, the product of mutter acting upon matter. [Conversation
Gold Trajectory:
Assistant: Do you want to know what materialism would in contrast to for singalar explanations
phenomenal reality for the categories that all philosophies
User: What would be secondary to them?
said to fall into?
Assistant: mailer is primary, and mind or spiril or ideas are sccondar
Lsser: For the categories that all philosophies are said to fall into.
User: What does the basic proposition of these two
Assistant, In the context of the two primary philosophical categories, materialism is contrasted with idealism."
categuries pretuin to?? Assistant; nature of reality
User: What would materialism be in contrast to Ascidaatt
Figure A4: Example of a contrastive pairing constructed for RL tuning with Abg-CoQA (Guo et al., 2021). The notation used is as described in Section 3.1.
A LIMITATIONS, ETHICAL CONSIDERATIONS, AND BROADER IMPACTS
A.1 DISCUSSION OF LIMITATIONS
We assume that the clarification questions are appropriately timed. However, crowdsourced con- versation datasets are often noisy (Chen et al., 2023b), and relying on noisy annotations or token sequences may result in suboptimal learned policies (from the perspective of asking unnecessary clarifying questions, as well as generating disfluent language). Depending on the source of data, it may be necessary do an additional preprocessing stage which one infers whether an action is useful or not. Shi et al. (2024) infers user satisfaction given model responses in-the-wild, whereas Yu et al. (2023) ranks dialogue actions using Monte-Carlo Tree Search.
Label noise can also affect the implicit action recognition evaluation, which assumes that an action in a benchmark task is "optimal." a corpus like PACIFIC with high inter-annotator agreement (0.62), this is reasonable assumption. However, we observe inconsistency in Abg-CoQA which may be a result of the low inter- annotator agreement (0.26) reported in Guo et al. (2021). Recent work has demonstrated the promise of many-shot in-context learning Agarwal et al. (2024) compared to supervised tunint at the trade-off of inference-time latency. Yet, Table A6 indicates that even with greatly increased number of in-context conversation examples (e.g. 50, 100, and 250), the
17

---

Published as a conference paper at ICLR 2025
Medelas Rerponse and User Simuletor's Atsponse Input Prompt
Scenario: Modal mdks clartyying questier: tejectory sinubrtion neaded
Proceet "Whtt
Lass
Whe on
neorded Lrvornsee
moaerror adtin shoar)
Evsleane Madely Rengasse
for Mittooe"
Uwer Sivekaoor
Groasd Trush Ropone knd
Trajectory
Infernation Gast The thcndacullase rocivabla $3,419
DROPFI:
Eraiaeted Model
Scenarto: Model attemper reseive User'' requeet with en armovr
Ivsteate Made''s Trajectery Reuatttio:
Stally
$3,155
Truth Eijjectry Bosslatier: $ta19
MusISA
Figure A5: Trajectory-level content evaluation using the example scenario from Figure 1. Trajectory-level evaluation seeks measure the extent to which candidate LLM can interact with "User" to reach target information goal. The "interactive" evaluation of a given instance continues until the candidate LLM attempts resolve the User's request by providing a direct answer. The candidate trajectory resolution is scored using downstream task metrics. In this example, DROP F1 is used following the task metrics for PACIFIC.
Table A6: Analysis of the impact of additional data on Abg-CoQA. Additionaly many-shot examples do not necessarily improve implicit action recognition performance. ACT tuning with Zephyr 7B greatly outperforms many-shot Gemini performance.
Adaptation Setting
Action-level
Content-level
Base Model
Approach Conversations
Macro FI
Turn Similarity
Traj. Similarity f
Gemini Pro Zephyr 7B-3
ICL
50
56.4
64.5
68.9
ACT (ours)
50
52.3
66.2
68.8
Gemini Pro
ICL
100
59.2
67.0
72.0
Zephyr 7B-3
ACT (ours)
100
51.1
69.5
71.4
Gemini Pro Zephyr 7B-3
ICL
250
58.8
66.0
71.1
ACT (ours)
250
53.3
72.5
75.1
downstream disambiguation ability does not improve uniformly. We thus posit that in such scenarios, it is more important that for such corpora, multi-tur task completion is more important measure of a model. We do find that even with 250 in-context examples, tuning a smaller model with ACT and the same conversation samples has the potential to outperform frontier models with many-shot examples.
ACT also makes use of task-specific heuristics. While this was intentional since success criteria can vary greatly across domains, this may also require more customization and engineering expertise/- effort. Our overall approach tuning and evaluation also makes heavy use of existing LLMs. We prompt Gemini for purposes such as Action Classification or User Simulation, but such approaches are not perfect and may occasionally result in unwanted behavior. These prompting approaches similarly may require substantial customization efforts. We also realize that not all researchers may have access commercial L.MMs due to researchers for financial or privacy reasons.
Our study also focuses specifically on the limited data regime. We believe that such contexts (e.g.. when the target user population is unknown; when conversational data cannot be collected due to privacy concerns; when conversational system is in its early stages and collecting abundant data for development iteration is not feasible; etc.) would benefit the most from focused adaptation designed to fundamentally teach conversational skills approaches such ACT. As such, in paper, this was the focus all of our experiments. is not clear how much our findings would generalize in settings in which there is abundance of training data whose distribution closely matches the target
18

---

Published as a conference paper at ICLR 2025
distribution. Intuitively, if much more in in-distribttion data is made available, the performance of unaligned objectives like SFT would start to catch up to the performance of focused approaches.
Is ACT online learning? Levine et al. (2020) defines offline reinforcement learning using fixed dataset of experiences, whereas online reinforcement learning relies on interacting with an environment in real-time. Additionally, Guo et al. (2024) defines on-policy sampling in contrastive RL tuning as settings where both the winning and losing responses are sampled from the policy model. In our case, during tuning, we sample single response from the policy model.
As such, we define ACT as quasi-onling contrastive RL tuning approach. ACT does rely on action-based preference dataset, as is common in fully-offline reinforcement learning. However, ACT continuously samples responses form the policy model in order to update the contrast pairing with good bad trajectories. Overall, has both dynamic and static components, so refer to it as quasi-online. ACT also specifically is different from fully online DPO where both the winning and losing responses are sampled (i.e. in Guo al. (2024)) because our focus on conversational actions. There is no guarantee that sampling response from the policy model twice will result in differing actions, unless you change the prompt. However, in that case, you would no longer be computing the DPO objective.
By nature of the domains considered, the extent which ACT allows for online exploration is also limited. As previously mentioned, our experiments are constrained by the fact that there is an objective right/wrong target answer. For instance, if the target answer is an arithmetic expression as is common in PACIFIC, there are fairly limited number of unique trajectories (when inspected in terms the number of tokens) that will arrive at that particular expression. such cases, the trajectory sampled from the policy model during ACT tuning may not be any different from the offline trajectory found in the training data.
A.2 ETHICAL CONSIDERATIONS
Usage and Safety We do condone the use of our work for any unethical, unlawful, and/or harmful applications. In our work, we do not release any new model artifacts or web-scraped data, but we do not specifically introduce any model guardrails as a part of ACT. However, our implementation of ACT relies on other LLMs such as Gemini to produce user simulation. Gemini is released with safety guardrails in place, but these inference-time guardrails may not be available when using open-source LLMs instead. We advise that any deployments of models tuned with ACT should consider adding safety guardrails at inference-time.
initial preference dataset, and to perform
Hallucinations One commonly documented concern with using LLMs is their tendency to halluci- nate answers Assistant QA contexts. A solution is to provide an LLM with information from a retriever (i.e., retrieval-augmenter generation). Two of the datasets we use, PACIFIC and Abg-CoQA, mirror this setting by performing grounded QA using a mixture of long-context textual passages and tabular data. It follows that ACT could be further studied in combination with approaches for improved retrieval-auee augmennte generation.
Our evaluation criteria in this paper are also rather restrictive towards hallucinations. In PACIFIC, use a token-level metric (DROP FI); in Abg-CoQA we evaluate a candidate response's 's semantic similarity with ground truth answer; in AmbigSQL we execution match, which is fully objective metric. As such, it is difficult to perform well on PACIFIC and Abg-CoQA if model consistently hallucinates answers, and in AmbigSQL,, a "hallucinated" response would not consist of the appropriate SQL code.
A.3 BROADER IMPACTS
There is an abundance of modern conversational assistants. ACT seeks to improve multi-turn conversational experiences, and thus, it can be used to improve many applications used by potentially millions of users around the world. However, the popularity of conversational assistants also creates an increased risk of misuse. Some people may develop conversational for unethical applications such as misinformation generation, or gray areas such as optimizing dialogue generation for content which is not suitable for the workplace. As discussed above, do condone the of ACT' for any
19

---

Published as a conference paper at ICLR 2025
Table A7: Experimental results PACIFIC's public evaluation set with additional results using Gemini Flash and Claude Haiku. ACT achieves the strongest performance compared to all tuning approaches across every condition in every metric. Tuning- fuuning-based adaptation strategies given any in-context examples at inference time, whereas inference-timg adaptation strategies are prompted with 10 in-context conversation examples using the same strategy.
Adaption Setting
Action-level
Content-leval
Base Model
Approach
Conversations
Macro FI
Tum FI
Traj. FI f
Post-Clarify FI
Gemini Pro
Standard Prompt
10
81.4
59.7
58.7
49.7
Gemini Pro
Chain-of-Though
10
86.3
66.3
17.1
19.2
Gemini Pro
Proactive MIPrompt
10
78.9
63.4
61.1
18.9
Gemini Flash
Standard Prompi
58.8
58.7
Gemini Flash
Chain-of-Thouee Thouught
10
77.1
62.0
16.9
20.0
Gemini Flash
Proactive MIPrompt
10
76.8
64.0
62.0
24.4
Claude Sonnet
Standard Prompt
10
71.9
43.7
42.0
28.5
Claude Sonnet
Chain-of-Thouge
10
80.0
37.2
13.0
Claude Sonnet
Proactive MIPrompt
10
74.9
47.2
45.9
7.6
Claude Haiku
Standard Prompt
10
46.9
26.4
26.2
Claude Haiku
Chain-of-Thhen Thought
10
23.7
12.0
2,9
Claude Haiku
Proactive MIPrompt
18.2
Gemini
51.8
45.7
9,9
Gemini Pro
SFT
100
75.2
64.3
$4.6
8.5
Gemini Pro
SFT
250
88.0
67.4
59.3
10.2
Zephyr 7B-3
SET
50
69.0
57.8
61.3
43.5
Zephyr 7B-8
DPO-Dist (Pro Flash)
50
75.5
61.7
55.7
30.8
Zephyr 7B-8
DPO-Dist (Sonnet v. Haiku)
62.0
56.3
Zephyr 7B-8
IRPO
50
67.7
59.1
$6.7
34.4
Zephyr 7B-8
ACT (ours)
50
82.2
62,8
61,9
$7.2
Zephyr 7B-3
SET
100
82.3
58.6
60.3
49.9
Zephyr 7B-8
DPO-Dist (Pro v. Flash)
100
68.8
53.3
53.3
31.7
Zephyr 7B-8
DPO-Dist (Sonnet v. Haiku)
59,0
53.7
Zephyr 7B-8
IRPO
100
84.5
60.4
55.2
38.2
Zephyr 7B-8
ACT (ours)
100
86.0
65.0
62.0
$7.4
Zephyr 7B-8
SET
250
86.9
65.1
633.3
56.7
Zephyr 7B-8
DPO-Dist (Pro v. Flash)
250
65.6
53.6
$4.1
30.9
Zephyr 7B-8
DPO-Dist (Sonnet v. Haiku)
43.3
38.6
Zephyr 7B-8
IRPO
250
85.4
64.9
58.4
40.3
Zephyr 7B-8
ACT (ours)
250
89.6
68.1
65.7
62.0
unethical, harmful, unlawful applications, and we suggest the usage of safety guardrails for any deployments.
B ADDITIONAL EXPERIMENTAL RESULTS
As described Section 4.3, we include additional prompting experiments using Gemini 1.5 Flash and Claude 3.0 Haiku using Standard Prompting, Mhain-of-Thougee Prompting, and Proactive Mixed- Initiative Prompting. We also include additional comparisons against an off-policy DPO baseline, DPO-Dist, which involves distilling from two LLMs of differing sizes. The full results for PACIFIC are displayed in Table A7, the full results for Abg-CoQA are in Table A8, and the full results for AmbigSQL are in Table A9.
20

---

Published as a conference paper at ICLR 2025
Table A8: Abg- Abg-CoQA test set evaluation results with additional results using Gemini Flash and Claude Haiku. ACT outperforms SFT across all evaluations in all three data settings. However, Gemini Ultra achieves the strongest downstream task performance when prompted with in-distribution in-context conversation examples.
Adaptation Setting
Action-Jevel Macro FI
Content-
Base Model
Approach
Conversations
Turn Similarity
Traj. Similarity
Gemini Pro
Standard Prompt
10
55.5
67.0
72.2
Gemini Pro
Chain-of-Though
10
61.2
63.4
39.1
Gemini Pro
Proactive MIPrompt
10
55.5
63.3
33.3
Gemini Flash
Standard Prompt
10
52.6
62.5
67.4
Gemini Flash
Chain-of- Thoughtt
61.2
56.5
36.6
Gemini Flash
Proactive MIPrompt
58.1
61.7
36.1
Claude Sonnet
Standard Prompt
66.0
50.1
Claude Sonnet
Seiinof-Thhoughe
63.7
46.2
36.8
Claude Sonnet
Proactive MIPrompt
57.2
60.8
32.9
Claude Haiku
Standard Prompt
10
49.3
40.9
41.7
Claude Haiku
Mhin-offTThoughe
10
46.2
30.7
28.0
Claude Haiku
Proactive MIPrompt
10
45.2
34.5
31.4
Zephyr 7B-8
SFT
50
44.6
53.3
64.2
7ephyr 7B-8
DPO-Dist (Pro v. Flash)
50
46.9
$7.2
61.2
Zephyr 7B-8
DPO-Dist (Sonnet v. Haiku)
50
44.7
57.9
61.5
Zephyr 7B-8
ACT (ours)
50
52.3
66.2
68.8
7ephyr 7B-8
SFT
100
52.6
63.1
69.4
Zephyr 7B-8
DPO-Dist (Pro v. Flash)
100
47.8
61.9
67.1
Zephyr 7B-8
DPO-Dist (Sonnct v. Haiku)
100
44.8
62.0
66.4
Zephyr 7B-8
ACT (ours)
51.1
69.5
Zephyr 7B-8
SFT
250
53.5
64.0
66.2
Zephyr 7B-8
DPO-Dist (Pro v. Flash)
250
46.0
61.9
66.3
Zephyr 7B-8
DPO-Dist (Sonnet v. Haiku)
46.3
62.6
67.0
Zephyr 7B-3
ACT (ours)
250
53.3
72.5
75.1
Table A9: AmbigSQL test set evaluation with additional results using Gemini Flash and Claude Haiku. Zephyr tuned with ACT is able to achieve the strongest task performance within each data setting. There especially large performance improvements post-clarification SQL execution match when data resources are more scarce.
Adaptation Setting
Action-level Accuracy
Content-level
Base Model
Approach
Conversations
Execution Match
PC Execution Match
Gemini Pro
Standard Prompt
10
72.1
63.5
75.2
Gemini Hlash
Standard Prompt
66.2
Claude Sonnel
Standard Prompt
10
68.5
66.5
72.4
Claude Haiku
Standard Prompt
10
73.8
$7.3
65.3
Zephyr 7B-D
SFT
50
77.4
21.9
13.9
Zephyr 7B-3
DPO-Dist Dist (Pro v. Flash)
50
77.7
42.6
31.5
Zephyr 7B-3
DPO-Dist (Sonnet Haiku)
50
78.0
40.9
41.2
Zephyr 7B-3
IRPO
50
91.0
27.8
30.8
Zephyr 7B-3
ACT (ours)
50
80.8
43.6
38.1
Zephyr 7B-3
SFT
100
97.2
43.3
34.3
Zephyr 78-3
DPO-Dis
Flash)
45.3
Zephyr 7B-3
DPO-Dis (Sonnet Haiku)
100
99.8
47.8
44.8
Zephyr 7B-9
IRPO
100
96.2
45.0
37.0
Zephyr 7B-3
ACT (ours)
100
49.6
Zephyr 7B-3
SFT
250
99.8
51.0
50.7
Zephyr 7B-3
DPO-Dist Flash)
250
97.3
49.7
44.2
Zephyr 7B-3
DPO-Dist (Sonnet Haiku)
250
99.7
50.7
50.3
Zephyr 7B-5
IRPO
250
97.0
49.7
45.6
Zephyr 7B-3
(ours)
250
53.0
Zephyr 7B-9
SFT
(All)
60.4
21

---

Published as a conference paper at ICLR 2025
C
AmbigSQL: MODELING AmbigUITY IN CONVERSATIONAL TEXT-TO-SQL
Table A10: Overview of AmbigSQL, an ambiguous Text-to-SQL dataset synthesized from Spider.
Train
Dev
Test
Num. Unambiguous Requests
7,000
1,034
1,034
Num. Ambiguous Requests
7,000
1,034
1,034
Num. Unique Schemas
1,056
145
145
Types of Ambiguity
3
3
3
There is growing interest in using LLM-basee agents for coding tasks (Liu al., 2024). Particularly, due to the complexity of such tasks, multi-turn interactions which an agent able to clarify assumptions and user intents should intuitively help with goal completion (Nijkamp al., 2023). Despite this, there are few existing resources for multi-turn code generation tasks. One example is CoSQL, conversational text-to-SQUL task which also includes linguistic ambiguities (Yu et al., 2019), but the proposed task does not include agent-style interaction in which model must learn to ask clarifying questions. Upon our inspection of the dataset, there are also various inconsistencies related to the "system-side" clarification questions given in the dataset's s conversational contexts, which we highlight in Table All. As result, we propose AmbigSQL, our own synthetically constructed resource for ambiguous conversational text-to-SQL.
Table A11: Conversations CoSQL with noisy "clarification questions" (highlighted red). Example 1) is remnant of crowdsourcing in which the system- side party makes mention of the task guideline. Example 2) demonstrates a system-side clarification question being asked prior to the user making any information requests. Example 3) The system-side clarification question makes reference to some prior database search result, but the execution feedback is not made accessible to the system during inference.
No. Interacting
Utterance
Party
User
Can you all singer ids that aren't 't present the song table?
Assistant
SELECT Name FROM singer WHERE S Singer_ID NOT IN Thanks!
User
1
Assistant
You should ask at least 3 questions
2
Assistant
Did you want the full name of makers and the number?
3
Assistant
Do you mean the address of the customer with first name Luis?
C.1 AMBIGSQL CONSTRUCTION
We start from Spider, popular single-turn text-to-SQL. dataset and benchmark. An overview of AmbigSQL is given in Table A10. Due to the nature of the single-turn requests Spider, each instance can be viewed as a conversation consisting of single state t (see notation defined in Section 3). f1 Pı contains any instructions, the database schema, and the user's request. r11 the correct SQL query which yields the information requested by the user. Ø1 = because the trajectory ends on the same turn due to the system yielding the correct query. a = SNSW ER because the only action possible is to provide SQL query.
Our desired result in constructing AmbigSQL. is to have a corpus which can be used to demonstrate to an LLM the linguistic differences between ambiguous and clear user requests. That is, given fixed database schema and a target SQL query, we want a pair of requests such that one requires asking a clarification question and one does not. This would also result in balanced dataset in which half of the requests require asking clarification questions, and half do not.
We focus three fundamental types of ambiguous information requests. Those in which the requested information is ambiguous (c.g.. "Show details about singers ordered by age from the oldest to the youngest"), those in which the requested population is ambiguous (e.g., "Which ones who live
22

---

Published as a conference paper at ICLR 2025
Table A12: In-context example given as part of prompt for creating information requests in which the target population is ambiguous. The format of the black text represents how a ground- truth request would be used to form the prompt for a target example. The blue text represents the content that would be synthesized from an LLM. We omit the database schema from the paper.
Database Schema Omitted]
The target SQL query is the following:
last_name cell_number FROM Professionale 'Indiana' UNION SELECT T1 r1.profossional_ic T1. r1.profeesionai__ GROUP BY T1. [1.professional. id HAVING count (x) 2
SELECT professional_id
WHERE state
T1. last__namee T1. f1.cell_number FROM Profossionals AS T1 JOIN Treatments AS T2 ON
= T2. 12.professioona_it
Here is a clear request that would correspond this SQI.querry
"Which professionals live in the state of Indiana or have done treatment on more than 2 treatments? List his or her id, last name and cell phone."
Here is the same request converted into ambiguous format by underspecifying the target columns:
"Which ones live in the state Indiana or have done treatment on more than 2 treatments?"
Here is an appropriate clarifying question to recover the clear request from the ambiguous request: "Are you asking about the Professionals?"
in the state of Indiana?"; see Table A12), and finally, those in which the requested presentation of results is ambiguous (e.g. "Show name, country, age for all singers ordered by age": see Table A13).
We iterate through each of the examples in Spider, and use LLM (Gemini Ultra 1.0) to synthesize a perturbed version of each unambiguous query, along with an appropriate clarifying question. For queries which require some manipulation of results presentation, we prompt an LLM perturb the query such that the requested presentation style becomes ambiguous. Otherwise, we randomly select perturbation strategy: either masking the requested information (Table A13) or masking the requested population (Table A12). For each of these three strategies, we use five in-context examples demonstrating the appropriate masking process. The exact in-context examples are given in the attached code. Each of these ambiguous queries are thus associated with conversations containing "ground truth" states two timesteps, & and 12- P1 contains an ambiguous user request, 71 the synthesized clarification question, and accordingly, user request, T2 is the correct ground truth SQL query, and accordingly, 02 AN SW ER.
CLARIFY p2 contains the disambiguated
The code to create AmbigSQL will be released publicly. Each instance includes the database metadata included in Spider, but for all experiments used in this paper, the database schema is linearized into the format used for prompting in Sun et al. (2023a;b).
Table A13: In-context example given as part of prompt for creating information requests in which the target columns are ambiguous. The format of the black text represents how a ground- truth request would be used to form the prompt for a target example. The blue text represents the content that would be synthesized from an LLM. We omit the database schema from the paper.
Database Schema Omitted]
The target SQL query the following:
SELECT professional_ie
last_name cell_number FRCM Professionale 'Indiana' UNION SELECT T1 pr.oresssioonl
WHERE state
T1. $1.1ast_name
n.ceellnumber FROM Professionals AS T1 JOIN Treatments AS T2 ON f1.professional
12.profess prrroffessioa_____ id GROUP BY pr.prffessi onal_id iid HAV ING count (+) > 2
Here a clear request that would correspond this SQL query:
"Which professionals live the state of Indiana have done treatment on more than treatments? List
his or her id, last name and cell phone."
Here is the same request converted into an ambiguous format by underspecifying the target columns:
"Which professionals live in the state of Indiana or have done treatment more than 2 treatments?" Here an appropriate clarifying question recover the clear request from ambiguous request: "Which information of the professionals do you want to know?"
23

---

Published as a conference paper at ICLR 2025
Table A14: Examination of ambiguity in AmbigSQL. Competitive high capacity LLMs struggle with producing "correct" SQL queries given only ambiguous user requests. Including disambiguation turns in the prompts greatly improves execution match.
Model
Ambiguous Request Execution Match
Execution Match with Clarification Turns
Gemini Pro
28.5
68.7
Gemini Ultra
31.2
77.0
C.2 EXAMINING CLARIFICATION NEED IN AMBIGSQL
We primarily are concerned with examining the extent to which clarification questions are necessary in providing the requested SQL queries. Inspired by the notion of "recovery" presented in Toles et al. (2023), we examine the performance in constructing each of the unique SQL queries with and without the gold clarification turns. Concretely, we first evaluated the Execution Match performance achieved by prompting LLMs with only the ambiguous versions of each user request, with each instruction including the instruction that the LLM must construct a SQL query. Then, we prompted LLMs to construct the same SQL queries, but given the disambiguation turns as conversation history (i.e., with context consisting the original ambiguous request, the clarification question, and then non-ambiguous request).
We conducted this analysis on the test set using two competitive LLMs, Gemini Pro and Gemini Ultra, with the Execution Match tool from Spider and CoSQL (Yu et al., 2018; 2019). Our results are shown in Table A14. Given only an ambiguous request, both Gemini Pro and Gemini Ultra struggle to consistently construct the correct SQL query. However, given disambiguation turns, Execution Match improves dramatically, approximating the performance on the validation set given in Sun et al. (2023a:b).
C.3 EXAMPLES
Table A15 and Table A16 each contain pair of examples from AmbigSQL's test set. Each example contains the prompt which is provided to an L.M,, the immediate ground truth response to the user request provided as part of the prompt, and the resulting ground truth trajectory (for examples which include clarifying questions). The examples in each table are paired the top example directly taken from Spider and converted into conversational format, and the bottom example is the result of introducing ambiguity into the first example.
24

---

Published as a conference paper at ICLR 2025
Table A13: Examples from the AmbigSQI. test set using the "concert seieerrtsiggrrrre database. Black text: context provided to LLM prompt. Blue text. ground-truth single-turn response. Purple text: target trajectory which should be achieved through simulation.
[Instruction]
You are an Assistant having a conversation with a User who would like to retrieve some information from a SQL database with the schema provided below. If you are confident in the User's intent, you should provide the appropriate SQL query. Otherwise, you should ask the User a clarifying question. Database Schema Omitted]
Conversation]
User: How many singers do we have?
Assistant: SELECT count(*) FROM singer
[nstruction]
You are an Assistant having a conversation with a User who would like to retrieve some information from a SQL. database with the schema provided below. If you are confident in the User's intent, you should provide the appropriate SQL query. Otherwise, you should ask the User clarifying question. Database Schema Omitted]
Conversation]
User: Tell me about the singers.
Assistant: What specifically would you like to know about the singers? For example, would you like to know how many there are, or see list of their names?
User: How many singers do we have? Assistant:
SELECT count(") +) FROM singer
Table A16: Examples from the AmbigSQL test set using the "flight 2" database. Black text: context provided to an LLM as a prompt. Blue text: ground-truth single-turn response. Purple text: target trajectory which should be achieved through simulation.
[Inttructio]]
You are Assistant having conversation with User who would like to retrieve some information from SQL. database with the schema provided below. If you confident in the User's intent, you should provide the appropriate SQL query. Otherwise, you should ask User a clarifying question. Database Schema Omitted]
Conversation]
User: Return the number of airports.
Assistant: SELECT count(*) FROM AIRPORTS
[Inttructio]]
You are an Assistant having a conversation with User who would like to retrieve some information from a SQL database with the schema provided below. If you are confident in the User's 's intent, you should provide the appropriate SQL query. Otherwise, you should ask the User a clarifying question. Database Schema Omitted]
[onversation]
User: How many are there?
Assistant: Could you please specify which table you are referring to when you ask 'How many are there?"
User: Return the number of airports.
Assistant: SELECT count(*) FROM AIRPORTS
25

---

Published as a conference paper at ICLR 2025
D ADDITIONAL DETAILS ON EVALUATION
We define the following action-level evaluation metrics:
Accuracy: Percentage correct implicit actions
Macro Fl: Unweighted Average of Fl for each action
We define the following content-level evaluation metrics:
Turn-level DROP F1: Average immediate response DROP F1 Deng al. (2022); Dua et al.
(2019)
Trajectory-level DROP F1: Average trajectory-outcome DROP F1 Deng et al. (2022); Dua et al. (2019)
Post-Clarification DROP F1: Average DROP FI Deng et al. (2022); Dua et al. (2019) of responses which follow agent clarification turns
Turn-level Similarity: Immediate response embedding similarity
Trajectory-level Similarity: Trajectory outcome embedding similarity
Trajectory-level Execution Match: Percentage of trajectory outcomes with correct execution results
Dost-Clarification Execution Match: Percentage of trajectory outcomes with correct execu- tion results out those that which contain clarification turns
PACIFIC As described in Section PACIFIC is conversational question-answertn dataset in which the final answers may involve generating the correct words from given span, from multiple spans, or providing a correct arithmetic expression. As such, the authors propose using DROP Fl as the official evaluation metric. The way DROP F1 is used in the original paper Deng et al. (2022) is analogous our aforementioned "Turn-level DROP F1." However, as this does not fully represent model's conversational reasoning abilities, we additionally evaluate LLMs in the PACIFIC environment using Macro F1, Trajectory-level DROP F1, and Dost-Clariiication DROP Fl. Concretely, the evaluation for some LLM is as follows. Assume we have some example with prompt p. winning action a, ground truth response т, and trajectory-level information goal g. We sample candidate response from the LLM: from each response y according to Lines 6- in Algorithm 2 and obtain trajectory outcome g'. The aforementioned action-level metrics are computed using the implicit actions of each y with each ground truth implicit action a. Turn-level DROP F1 is computed between all sampled responses y and all ground truth responses r, and Trajectory-level DROP F1 is computed over all simulated trajectory outcomes g' and all ground truth information goals g. Post-Clasn most-Clarification Fl is defined as Trajectory-Ievel Fl for only the subset of trajectories which include clarification turns.
Po(-lp). We then simulate the trajectory resulting
Abg-CoQA As previously mentioned in Section 4, Abg-CoQA is conversational question- answering dataset for machine reading comprehension, Thus, we use embedding similarity Risch et al. coherent and diverse responses which may be scored lowly
(2021) it allows for producing by criteria such as token-level F1 score. In the original paper, language models are only evaluated in terms of QA performance Guo et al. (2021), rather than their ability disambiguate requests. Thus, for our evaluation, we remove all clarification turns from the prompt and require the LLM to produce clarifying questions on its own. However, unlike the other tasks considered in this paper, each ambiguous request is paired with all of the possible trajectories (i.e., the reasons why the request considered ambiguous). We thus perform an evaluation for every ground truth trajectory, so that is impossible for an LLM to achieve a high cumulative trajectory-leve score simply by getting lucky guessing the user's intent.
Concretely, the evaluation for some L.M æ is as follows. Assume we have some example with prompt p. winning action a, ground truth response r, and the set trajectory-level information goals G. For every individual trajectory-level goal g G, we sample candidate response from the LLM: y ~ Po(-lp) then simulate the trajectory resulting from each response y according to Lines 6-12 in Algorithm/ 2 and obtain trajectory outcome g' As with PACIFIC, compute Macro F1 using the implicit actions of each y with each ground truth implicit action a. We compute Turn-level similarity for each unique p between sampled responses y and ground truth responses We compute
26

---

Published as a conference paper at ICLR 2025
Table A17: "Standard" in-context example given as part of a prompt to an LLM for response generation in PACIFIC. The purple text is the overall system instruction. The blue text is what an LLM would have to produce at inference time.
You are an Assistant answering questions from a User. You should either attempt to answer the question or ask clarifying question if there is any ambiguity.
[Table and Passage Omitted]
User; How much does the company expect to make contributions to the defined plans during the next fiscal year?
Assistant: ('What kind of defined plans you asking about?"]
User: The defined benefit plans and the defined contribution plan respectively. Assistant: ['$5.1 million', '50.6 million')
User: What is the value of the expected defined contribution plan as a percentage of the defined benefit plan during the next fiscal year?
Assistant: 0.6/5.1 11.76
User: How much would change if there 1 1% increase and 1 % decrease respectively in the discount rate?
Assistant: ('What kind of change are you asking about?"]
User: The pension.
Assistant: ['$(3,,155), '$49.361"]
User: What the difference in that amount if the discount rate had 1% increase and a I % decrease?
Assistant: 49,361 - 145) = 88506
User: How much would the pension change if there is a % increase and a I % decrease respectively?
Assistant: ['Which rate of increase and decrease are you talking about?"]
User: The future salary growth.
Assistant: ['$7,572', "$(6,919']
User: What the difference between them?
Assistant: 7,572 (6,919) 14491
Trajectory-level similarity over all simulated trajectory outcomes g' and all ground truth information goals g.
is ambiguous text-to-SQL generation task. By design, the labels are balanced, only use Accuracy for action-level metrics. The primary objective is to produce SQL queries which result in correct executions, we evaluate LLMs in terms of Trajectory-level and Post-Clarificatte Execution Match. For each instance in the test set, we sample a response from an LLM and compute the trajectory as specified in Lines 6-12 in Algorithm 2. Trajectory-level Execution Match includes SQL queries which are constructed with or without clarification turns, whereas Cost-Clarification Execution Match only looks at SQL queries which are constructed following clarification turns.
AmbigSQL
E DESCRIPTION OF IN-CONTEXT LEARNING BASELINES
We use several in in-context learning baselines with frontier LLMs in Section 4. For each condition, randomly sample 10 conversation examples from each task's s 250-instance data pool, and apply one of the following prompting frameworks.
Standard Prompting We simply provide in-context examples that are structurally identical to the inputs used for model tuning. Our format is similar to the formats used in Chen et al. (2023b); Deng et al. (2023b) and we provide an example in Table A17.
Chain-of-Thoughh Prompting We integrate the popular reasoning framework, Sfain-of-thougee prompting (Wei et al., 2022) into our aforementioned "Standard" conversational prompt format. Effectively, we ask an L.M to do end-to-end dialogue generation by having first produce a reasoning chain which states whether the current context is ambiguous or not. We provide example in Table A18.
Proactive Mixed-Initiative Prompting Following the baselines given in Deng et al. (2023c), apply the Proactive Prompting framework Deng et al. (2023b) mixed with the Mixed-Initiative Prompting style Chen et al. (2023b). Ultimately, the LLM conditions the possible set of actions,
Initiative
27

---

Published as a conference paper at ICLR 2025
Table A18: "Chain-of-Thouue in-context example given as part prompt an LLM for response generation in PACIFIC. The purple text is the overall system instruction. The blue text is what an LLM would have to produce at inference time.
You are an Assistant answering questions from a User. You should either attempt to answer the question or ask clarifying question if there is any ambiguity. [Table and Passage Omitted]
User: What is the value of the expected defined contribution plan as a percentage of the defined benefit plan during the next fiscal year?
Instruction: If the user's question is ambiguous, ask appropriate clarifying question. Otherwise, directly the user's question using the information from the passage context and the table. Let's think step step.
Reasoning: The user's question not ambiguous. Assistant: 0.6/5.1 = 11.76
User: How much would change if there is a I % increase and a 1 % decrease respectively in the discount rate?
Instruction: If the user's question is ambiguous, ask appropriate clarifying question. Otherwise, directly answer the user's question using information from the passage context and table. Let's think step by
step.
Reasoning: The user's question was ambiguous. Assistant: ("What kind of change are you asking about?']
User: The pension.
Instruction: the user's question is ambiguous, ask an appropriate clarifying question. Otherwise, directly answer the user's question using the information from the passage context and the table. Let's think step by step.
Reasoning: The user's beer's question is not ambiguous. Assistant: (5(39,145) '$49,361"]
User: What the difference in that amount if the discount rate had 1% increase and 1 1% decrease?
Instruction: the user's question is ambiguous, ask an appropriate clarifying question. Otherwise, directly answer the user's question using the information from the passage context and the table. Let's think step by step.
Reasoning: The user's question is not ambiguous. Assistant: 49.361 (39,145) 88506 User: How much would the pension change if there is 1% increase and 1% decrease respectively? Instruction: the user's question is ambiguous, ask an appropriate clarifying question. Otherwise, directly answer the user's question using the information from the passage context and the table. Let's think step by step.
Reasoning: The user's question ambiguous. ('Which rate increase decrease are you talking about?")
Assistant:
along with interweaved natural language instructions that describe which actions correspond to existing dialogue turns. We provide an example in Table A19, We use this framework for each of the long-context dialogue corpora (PACIFIC and Abg-CoQA).
28

---

Published as a conference paper at ICLR 2025
Table A19: "Proactive Mixed-Initiative" in-context example given as part of a prompt to an LLM for response generation in PACIFIC. The purple text is the overall system instruction. The blue text what an LLM would have to produce at inference time.
You Assistant answering questions from User. You should either attempt answer the question ask clarifying question there any ambiguity. [Table and Passage Omitted] User: How much does the company expect to make contributions to the defined plans during the next fiscal year?
The user's last question ambiguous. The Assistant asks clarifying question.
Assistant: ('What kind defined plans are you asking about?"]
User: The defined benefit plans and the defined contribution plan respectively. The user's last question was unambiguous. The Assistant directly answers the question. Assistant: ['5.1mm Imillion' 006 million')
User: What is the value of the expected defined contribution plan as a percentage of the defined benefit plan during the next fiscal year?
The user's is last question was unambiguous. The Assistant directly answers the question.
Assistant: 0.6/5.1 11.76
User: How much would change if there 1 1% increase and 1 % decrease respcctively in the discount rate?
The user's last question was ambiguous. The Assistant asks a clarifying question.
Assistant: ['What kind of change are you asking about?']
User: The pension.
The user's last question was unambiguous. The Assistant directly answers the question.
Assistant: ['(39, 145), 49,361']
User: What is the difference in that amount if the discount rate had a 1% increase and a I % decrease?
The user's last question unambiguous. The Assistant directly answers the question.
Assistant: 49,361 (39,145) 88506
User: How much would the pension change there is a % increase a 1% decrease respectively?
Actions: ("Directly Answer", "Ask Clarification Question"]
Prompt: Given the task background and the conversation history, please use appropriate actions to generate the response.
Response: The user's last question ambiguous. The Assistant asks clarifying question. Assistant: [' ('Which rate increase and decrease are you talking about?']
29

---

Published as a conference paper at ICLR 2025
Table A20: In-context example given part of a prompt an LLM for action classification with PACIFIC. The blue text represents the action for this example.
TTable and Passage Omitted]
User: What was the total number of acquisitions FY19?
Assistant: 35 acquisitions
User: What are the benefits of active growth strategy?
Assistant: strengthens and scales the National Storage operating platform which drives efficiencies the business.
User: What was the total NLA?
Assistant: Which region are you asking about?
The last Assistant utterance is clarifying question.
Table A21: In-context example given as part of a prompt to an LLM for action classification with Abg-CoQA. The blue text represents the action for this example.
Passage Omitted]
User: And who in the group was anxious?
Assistant: Peppe
User: Was she well-rested?
Assistant: no
The last Assistant utterance is direct answer.
F CONDITIONAL GENERATION MODEL DETAILS
As mentioned Section 3.1, we make use of high capacity LLM a conditional generation model, M. For all experiments considered, we use Gemini Ultra to construct the initial action-based contrastive preference dataset. We follow the format of (2023b), rather than performing zero-shot inference, we use ten in in-context examples to adapt the I.M to the prompting structure. Concretely, for given input/output pair to used as an in-context example, we interweave each System-side utterance with narrative instruction that states that the next utterance is either a clarifying question or direct answer. This yields "control" over the pragmatic action of the generated utterance.
prompting" described in Chen al.
G ACTION CLASSIFIER DETAILS
Compared to recognizing whether request ambiguous or not, action classification (i.e., recognizing whether an existing utterance a question answer attempt) is considerably simpler. We directly prompt Gemini Ultra 1.0 with 10 in-context examples to serve as A (as per the notation given in Section 3.1).
Table A20 is an example of an in-context example used to demonstrate a system-side response with the "Clarify" action in PACIFIC ("Assistant: Which region are you asking about?"). All of the conversation history is made available in addition to the relevant table and passage in the prompt. We include contextual information available as part of the input to the Action Classifier, along with the last Assistant response which is an example of the classification target.
Table A21 an example of an in-context example used demonstrate a system-siee side response with the "Answer" action in Abg-CoQA ("Assistant: no"). As is common in this corpus, subset of the conversation is made available in addition to the relevant passage in the prompt. We include all contextual information available as part of the input to the Action Classifier, along with the last Assistant response which is an example the classification target.
Table A22 is an example of an in-context example used to demonstrate system-side response with the "Answer" action in AmbigSQL. In AmbigSQL, the information provided as input is the database schema, and the user's initial information request. This information is provided part of the input to the Action Classifier, along with the last Assistant response which is an example of the classification target.
30

---

Published as a conference paper at ICLR 2025
Table A22: In-context example given as part of a prompt an LLM for action classification with AmbigSQL. The blue text represents the action for this example.
Database Schema Omitted]
User: "Which professionals live in the state Indiana or have done treatment on more than treatments?
List his or her id, last name and cell phone." Here is the same request converted into an ambiguous format by underspecifying the target columns:
Assistant:
SELECT professional_i
last_ pant_name__e cell_number FROM Professions inals
WHERE state = ' "Iddiana' UNION SELECT T1. 22.professionaa_.
T1. 1a.ssstaame
T1.cell number FROM Professionais AS T1 JOIN Treatments AS T2 ON r2.professional
IL.professiona_
GROUP BY T1. rt.prrffesionai_] HAVING count (+)
The last Assistant utterance is direct answer.
H USER SIMULATOR DETAILS
In order iterate on system-side dialogue models, many works investigate the use of user simulator, U (as per the notation given in Section 3.1) (e.g. Chandramohan et al. (2011); Kreyssig et al. (2018); Shah et al. (2018)). At core, it requires response generation model which can robustly represent user-side intents. Traditionally, this involved training model specifically but due to the aforementioned control capabilities of modern LLMs Chen al. (2023b), we directly perform few-shot prompting for user simulation, following Deng et al. (2023c); Yu et al. (2023). For given conversation during training, we again prompt a high-capacity LLM with in-context examples to user intents. Then, conditioned on the generated information-seeking intents, we prompt the same LLM with the responses from the policy model to simulate the sub-goal trajectory. This allows U to ground the real intents of the user while retaining enough flexibility to respond coherently to some system-side response. In our work, we use Gemini Ultra 1.0 with three hand-crafted in-context examples for both intent summarization and user response generation.
user-side responses
identify the information-seekinn
Table A23: In-context example given as part of prompt to LLM for intent summarization with PACIFIC. The purple text represents the system prompt used for this task. The blue text represents the summarized information-seeking intents.
The following is conversation between a User and an Assistant. The User is asking some questions. Summarize what information the User is looking for.
Table and Passage Omitted]
User: What does Walletron deliver?
Assistant: patented mobile wallet technology.
User: How much did cost the company to acquire certain technology assets of RevChip, L.I. ("RevChip") and TranSend Integrated Technologies Inc. ("TranSend")?
Assistant: $7.0 million
User: What was pro forma revenue in 2019?
Assistant: $1,382,957
User: What was the change in its amount between 2018 and 2019?
Assistant: 21228
User: How about that in pro forma net income?
Assistant: -6425
[Information] The user wants to know: What technology Walletron delivers, 2. How much cost Walletron to acquire certain technology assets of RevChip. LLC and TranSend Integrated Technologies Inc., What the pro forma revenue was in 2019, What the change in pro forma revenue was between 2018 and 2019, and 5. What the change pro forma net income was between 2018 and 2019.
We provide one of our three handcrafted in-context examples of summarizing information-scekte seeking intent in Table A23. While the information summary is handerafted, the conversation and grounding data are taken directly from the PACIFIC. After intent summarization, U is given the existing conversation history along with instruction that includes the information
PACIFIC
31

---

Published as a conference paper at ICLR 2025
seeking intents and asked provide the next user response. This format is structurally the same as the three comulation-specinn in-context examples - an example is provided in Table A24.
Table A24: In-context example given as part of prompt to LLM for user simulation with PACIFIC. The purple text represents the system prompt used for this task, which contains the infornation--eekingg intents summarized from Table A23. The entire conversation is used for this example to demonstrate grounding between a conversation and intents.
The following conversation between User and Assistant. The User asking some questions. The user wants to know: What technology Walletron delivers, 2. How much cost Walletron to acquire certain technology assets of RevChip, LLC and TranSend Integrated Technologies Inc., 3. What the pro forma revenue was in 2019, 4. What the change in pro forma revenue was between 2018 and 2019, and 5. What the change pro forma income between 2018 and 2019.
[Table and Passage Omitted]
Assistant: $7.0 million
User: What was the pro forma revenue in 2019?
Assistant: $1,382,957
User: What was the change in its amount between 2018 and 2019?
Assistant: 21228
User: How about that in pro forma net income?
Assistant: -6425
Abg-CoQA We conduct user simulation for Abg-CoQA similarly to PACIFIC. Table A25 is one of three hand-crafted examples of intent summarization, and Table A26 is one of three hand-crafted examples used for simulating the final user response.
Table A25: In-context example given as part prompt an LLM for intent summarization with Abg-CoQA. The purple text represents the system prompt used for this task. The blue text represents the summarized information information-sceking intents.
Passage Omitted]
User: What was his ranking?
Assistant: General
User: Did someone else have horse fighters?
Assistant:
User: Who?
Assistant: you want
who had 2,500 horse fighters initially?
User: No, 1 want to know who had a considcrable force horse fighters west of him.
Assistant: General (Bishop) Polk.
Instruction: Summarize what information the user is looking for.
Summary: The user wants know: What his ranking was. 2. Whether someone clse had horse fighters. Who had considerable force of horse fighters west him.
AmbigSQL We conduct user simulation slightly differently for AmbigSQL. Rather than having separate step for knowledge-seeking intent summarization, we directly condition U on the target SQL query. As demonstrated in Table A27, the prompt includes the target SQL query, the original user request, and the clarification question sampled from the Assistant. In the example given in Table A27, the last line the ground truth disambiguated query, and in practice, this would be the generation target of U.
I QUALITATIVE EXAMPLES
In Table A28 we compare two trajectories produced by Zephyr 7B tuned on 50 conversations from PACIFIC using SET and ACT, respectively, given the same conversation context and tabular information. The SFT model provided incorrect answer to the user request, matching the "Incorrect Assumption" case in Figure 1. contrast, the model tuned using ACT asked a clarifying question before eventually producing the correct answer.
32

---

Published as a conference paper at ICLR 2025
Table A26: In-context example given as part of prompt to LLM for user simulation with Abg-CoQA. The purple text represents the system prompt used for this task, which contains the information-seeking intents summarized from Table A25. The entire conversation is used for this example to demonstrate grounding between a conversation and intents.
The following is snippet from conversation between an Assistant and a User who is asking some questions about the Story below. The user only wants know the following information: 1. What his ranking was. 2. Whether someone else had horse fighters. 3. Who had a considerable force horse fighters west of him. Passage Omitted
User: What was his ranking?
Assistant: General
User: Did somcone else have horse fighters?
Assistant: yes
User: Who?
Assistant: Do you
know who had 2,500 horse fighters initially? User: No. want know who considerable force horse fighters west of him.
Assistant: General (Bishop) Polk.
Table A27: In-context example given as part prompt to an LLM for user simulation with AmbigSQL. The purple text represents the system prompt used for this task, which contains ground truth SQL query goal. The entire conversation used for this example to demonstrate SQL query- grounded request disambiguation.
user is asking an assistant to retrieve some information from a SQI. datahase. The command that the assistant should ultimately return is as follows:
SELECT county FROM campuses where campus 'California State Universityy-Chioo
The assistant will ask some questions to clarify the user's intent. The user should respond with rephrased request that reflects their desired query.
User: what county?
Assistant: Are you asking for a list of all the counties in the database?
User: I'm looking for the county of the campus 'California State University-Chir Chicco"
Table A28: Comparison of trajectories using different tuning approaches. Left: snippet of tabular data from the PACIFIC evaluation set. Middle: trajectory resulting from the SFT model in response to the user query. Right: trajectory resulting from the ACT model. Red text: sampled from U.
Relevant Table Data
SFT
ACT
Year: 2019 2018
User: What were the total liabilities of IMFT?
User: What were the total liabilities of IMFT?
Total current liabilities 851 167
Assistant: $909
Assistant: Which year are you asking about?
User: 2018
Total Liabilities: $909 $1,305
Assistant: $1.305
33

---

Published as a conference paper at ICLR 2025
Table A29; Comparison of PACIFIC trajectories using differrnt tuning approaches. Left; snip- pet of from à passage from the PACIFIC test set. Middle: trajectory resulting from the SFT model in response to the user query. Right: trajectory resulting from the ACT model. Red text: sampled from U. Both the SFT and ACT model answered the question incorrectly.
SFT
ACT
Relevant Data
User. What the prodi(llss) after tux?
User: What was the profiilloss) after
Nei profitlloss) afler tax was S(9.8) million Net prolit/(oos) after tax: (9,819) 6,639 (248%)
Assisiant: 819,819) x
Assistant: Which year are you asking the amount of? User: FY19
$/9,819)
Table A30: Comparison of PACIFIC trajectories using different tuning approaches. Left; snip- pet of from passage from the PACIFIC test set. Middle: trajectory resulting from the SFT model in response to the user query. Right: trajectory resulting from the ACT model. Red text: sampled from U. Both the SFT and ACT model answered the question incorrectly, and neither asked clarification question.
SFT
ACT
Relevant Table Data Yoar: 2019 2018 Investments: 1,216.0 1,212.9
User: In which year the of Investments higher? Assästant: [2018]
User: In which year Assistant: [2018,2019] x
Investments higher?
Table A29 includes an erroneous trajectory produced by the ACT model in the 50 conversation setting of PACIFIC. In this particular example, both the provided passage and the provided table provided similar information regarding net profit/loss after tax. In the dataset, the ground truth answer selected a span from the passage, whereas the ACT model produced clarification question and then eventually an answer using tabular information. The SFT model directly assumed the intent of the user and similarly produced an answer from the tabular information.
Table A30 includes another pair of erroneous trajectories in the 50 conversation setting of PACIFIC. The requested information was accessible through the provided table data, but both the SFT model and the ACT model failed to produce the correct answer. This example not annotated as requiring a clarifying question, so it simply indicates that there is much room for improvement in terms of downstream task performance.
Table A31 includes a pair of trajectories resulting from SFT and ACT in the 50 conversation setting of MAQ.C The requested information is given in the portion of the passage in the left panel. The SET model asked an off-topic clarification question and failed to bring the qestioning on track. The ACT model successfully asked on-track clarification question and eventually produced the correct answer.
Table A31: Comparison of Abg-CoQA trajectories using different tuning approaches. Left: snippet of relevant passage from the Abg-CoQA test set. Middle: trajectory resulting from the SFT model in response to the user query. Right: trajectory resulting from the ACT model. Red text: sampled from U. Ground truth outcome: Meghan asked Lizzie, "Are you awake?"
MT
Natexant Pansage
Usen Wast Meghan ak?
User What di Mealen zik?
She yowrad, enaching like boy ca
Aucesone you moss when ene faiter!
Autidan: you mese the mominn on the sight bofor??
Her ester was soaks. "Lizza".
No
The night bekere.
Megiaas whiigeeedd you waase"
Ascastun: Lizcie only her person once, accordnn skory
Asaazan: Meghan adodl Lizne th was that momning.
J ADDITIONAL EXPERIMENTAL DETAILS
J.1 COMPUTE RESOURCES
We conduct all experiments using one Google Compute Engine Virtual Machine with 8x 80GB A GPUs. All of our tuning experiments full-model multi-GPU tuning with Deepspeed. Full ACT with 250 conversations (1.923 training examples) from PACIFIC took the longest amount of time and required approximately 16 hours run for 12 epochs.
34

---

Published as a conference paper at ICLR 2025
Beyond the experiments reported in the paper, there many preliminary experiments that took place as part of this overall research project that were ultimately directionally different from our final contribution.
J.2 TRAINING HYPERPARAMETERS
For all of our SFT experiments with Zephyr, Mistral, and Gemma, we tune the model for up epochs. We choose the best-performing model with learning rates from {le 4, 2e 5, le - 5) with the AdamW optimizer.
For our SFT experiments with Gemini Pro, we use the Vertex Al API and tune for up to 4 epochs with Adapter size of 4.
For all of our RL tuning experiments, we allow the model to train for up to 12 epochs, and select the checkpoint that results in the highest reward margin on the validation set (which is action-based preference dataset constructed as described Section 3.2 using each task's original validation set). For all experiments, we use batch size of 4, and maximum sequence length of 1, 280.
Hy pereeaarrsete for Equation 2
For experiments with Zephyr 7B PACIFIC, we achieve strongest results using 0.01 and a learning rate of 5e - 7. On AmbigSQL, we use B = 0.01 and learning rate of 5e 7. On AmbigSQI.,
0.5 and learning rate of 5e 7.
K
ASSETS USED
All resources used have been cited appropriately in the paper. In this section, we enumerate each of the existing artifacts used in our work along with their license.
Existing Models
Gemma Gemma Team al. (2024): Gemma Open-Source License.
https: //ai.
google dwv/ htev//gema/teeee ttemms
Gemini 1.0 Ultra -gemini-1.0-ultae Gemini 1.5 Pro 1-5mini-.5-pro-001),, Gemini 1.5 Flash (gemini- 1. 1.5--fash--s001 (Gemini Team al., 2023): Accessed through the Google Cloud Vertex AI Platform. https:://// //tc:/cclll// clo..doooogoooo.... e. cwm.coo///////ooo com/products /gemi n ?hl=en
Claude 3.5 Somnet, Claude 3.0 Haiku (Amthropic AI, 2024); Accessed through the Google Cloud Vertex AI Platform. https: //cloud http://oooooooooo...oooo google. cow/products//eeeeeeeoeeeeeeeeee hl=en
MiniLM--66v22 (Reimers & Gurevych, 2019): Apache 2.0. https://huggingface bo//senttenee-trrnnffreetttttt
Mistral 7B-v0.1 (Jiang et al., 2023): Apache 2.0.
mistrallai/Missrrl-1-2 1
Zephyr 7B-3 (with Mistral 7B Base Model) (Tunstall al., 2023): MIT Open-Source License. http:::/huggi
co /t/////ggngaaeeee///eeeeee zephyrr Soerrr---bbbttte beta
Existing Datasets
Abg-CoQA (Guo al., 2021): MIT Open-Source License. https://github.com
BeiqiGuo// /tw////C//22111 -1b---22--AA---0
PACIFIC (Deng et al., 2022): MIT Open-Source License. https://github.r com/ in
Spider (Yu al., 2018): CC BY-SA 4.0. https://yale-1fly. io/ spider dei
Existing Algorithms and Software
hhtpps: ://:://aa htt/////..ddd/.. spogle..d gemini -sse-use--- supervia aed-uuinnn
/occs//////////oo/oo// /ode 1
35

---

Published as a conference paper at ICLR 2025
Direct Preference Optimization (Rafailov al., 2024): CC
Google Cloud Pipeline Components: Apache 2.0. https://clloud.ooggee..ooo com/
vertex-ai/docee
HuggingFace Transformers (Wolf al., 2020): Apache 2.0. https://itthub.co
PyTorch (Paszke et al., 2019): PyTorch Open Source License.
com/
Vertex AI SDK: Apache 2.0. https:/ / http://ooodooooog..... wooglee.cmmveertexaa////ooo
python- /dw/use--ereeeeeeeeeeeeeeeeeeeee weeree-eee--ee------ ai-python- sdk
36


Reviews:
[{"review_id": "eQ9Cm8agww", "reviewer": "Reviewer_GK1A", "summary": "This paper proposes a novel online optimization algorithm: action-based contrastive self-training, which facilitates data-efficient dialogue policy learning for modeling multi-turn conversation. The authors demonstrate their method on multiple real-world conversational tasks: tabular-grounded QA, machine reading comprehension and AmbigSQL. They also propose a way to evaluate LLM's ability to function as conversational agents. Their method ACT achieves substantial improvements than DPO and other baselines.", "questions": "None", "limitations": "", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. This paper proposes a feasible way to optimizing multi-turn conversation modeling for LLMs, which overcome some obstacles faced by existing methods.\n2. Their proposed method ACT is sample-efficient based on online DPO, and achieves good performance on three real-world conversational tasks. \n3. The paper is well-written and the experiments are convincing.", "weaknesses": "This paper conducts experiments on limited applications of dialogue tasks; there are also many variants for DPO based methods, however, the authors have not compared them as baselines.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "slzJrj8Jvp", "reviewer": "Reviewer_g3nF", "summary": "The paper introduces Action-Based Contrastive Self-Training (ACT), an algorithm designed to enhance large language models' (LLMs) ability to handle ambiguity in multi-turn conversations. Current LLM-based conversational agents often fail to ask clarifying questions when faced with ambiguous input, opting instead to overhedge or guess the user's intent. ACT addresses this by enabling data-efficient dialogue policy learning through contrasting different conversational actions, even without explicit action labels, using a preference optimization approach based on Direct Preference Optimization (DPO).\n\nThe authors demonstrate ACT's effectiveness across several real-world tasks, including tabular-grounded question answering, machine reading comprehension, and AmbigSQL—a new task focused on disambiguating complex SQL queries. Their experiments show that ACT outperforms standard tuning methods like supervised fine-tuning and DPO, particularly in scenarios with limited data. Additionally, the paper proposes a workflow for evaluating LLMs' ability to recognize and reason about ambiguity in conversations. The results suggest that incorporating action-based preferences via ACT enhances LLMs' performance in modeling multi-turn dialogues and improves their conversational skills.", "questions": "see above", "limitations": "", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. This paper is clear in writing. It is easy to understand the proposals and experiments. \n2. Diversified tasks of the evaluation datasets. I recognize the efforts author made to experiment on different tasks (e.g. SQL, tableQA etc.)", "weaknesses": "1. The biggest weakness of the paper should be the choice of weak baselines and this japardizes the contributions claims of the paper. \nOne paper on the top of my head is https://arxiv.org/pdf/2404.19733. This paper has many identical settings: online sampling and a heuristic to filter trajectories. I believe this paper should be a baseline to compare. \nMaybe the authors have similar ablation comparisons in Table 6. I would like to see some clear discussions on the baselines and ablations. \n\n\n2. Related to weakness 1, this paper doesn't discuss DPO variants built for tasks that require multi-step trajectories. This paper is proposing a new DPO algorithm but it has a very small section to discuss relevant DPO works. I would suggest the authors check into DPOs for conversational tasks or reasoning tasks that require COT. This will provide a more justified view of the paper contributions.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "EJ7orKDaPq", "reviewer": "Reviewer_Qb6Q", "summary": "The authors present ACT, a framework for fine-tuning LLMs to respond better to ambiguous queries. The approach relies on online preference optimization, using a DPO-like objective, where the user responses and action classifications are provided by some other LLM user simulation and a trained action classifier, and the agent responses are provided by the model under tuning. The authors design a particular, detailed approach to generating conversational examples from which the model learns, and show that it is highly effective and with relatively few examples enables mid-size (~7B parameters) LMs to outperform very large LLMs even with 10 examples in their prompts. Finally, the authors provide a new dataset of ambiguous queries that are resolved to SQL queries, again showing that the ACT framework provides better results by learning to ask clarifying questions.\n\nUpdated ratings after reading author responses. In particular the presentation has improved and the paper is easier to understand. Thank you to the authors for their efforts.", "questions": "Figure 3 was critical to my understanding of your approach. It would help the text to label figure 3 with the notation used in the text, and to refer to specific parts of the figure in the text to better connect the two. Without the figure I would have been completely lost in section 3. \n\nI found all of the sections on \"conversations in the wild\" difficult to follow and kind of disconnected from the rest of the text. Can you clarify what specific problem you are trying to address with these tests?\n\nIn Algorithm 2 you say (line 14-16) that the _trajectory_ is assigned to the winning or losing labels--but in the text it is just the sampled y_j that is assigned to these depending on the conversation outcome. These should me made consistent with each other.\n\nI would like to see details of the user simulator U and action classifier A in the main text. Making your text less wordy and shortening the tables to just the key results will help make room to discuss these critical pieces more. The other results from the tables, and things like the ablations, can be expounded on more fully in the appendices. That's what appendices are for: extra results and things that don't turn out to be as interesting. \n\nYou could also omit most or all of the DPO-Dist results from the main text--it never gets close to the best results. You can mention this in the main text and then detail it more in the appendices.\n\nLine 304: How do you guarantee that the responses from Gemini Ultra model M _should_ be rejected? Won't it often come up with a good response?\n\nLine 313: I admit I haven't focused on this aspect for some time, but surely there's something better than SentenceBERT for evaluation now?\n\nLine 354: why switch metrics between datasets? Why not just stick with F1, even though your AmbigSQL dataset is balanced? It seems like being consistent would help us get a better sense of how challenging (or not) the AmbigSQL dataset is. \n\nLines 425-427: ACT does better than frontier LLMs when it has 250 examples to learn from. I assume these are generated examples, and therefore you don't need human labeling. It makes me wonder (1) how much does this depend on the models you use being updated with human labeled data (since, if I understand correctly, the success of the method does depend on prior model alignment) and (2) why not try with more examples than 250?\n\nLine 448: It seems odd to me that ACT outperforms Zephyr SFT on the overall trajectory outcome but not action recognition. Does the A model show bias somehow to prefer asking clarifying questions? \n\nLine 527: this paragraph can just be a single sentence--just tell us prior alignment was helpful and how much, don't need to discuss more.\n\nFinally, there wasn't much (any?) discussion of the limitations of user simulations. In particular, I always worry that humans are less predictable than simulations--how might this affect your results?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "First and foremost, the authors present extensive comparisons showing that their approach leads to better conversational outcomes, both in terms of recognizing the need to clarify a user prompt, and in resolving the user's information need. The difference is sometimes quite large, with 10-20% improvements over SFT or a modified DPO approach. It is a significant result, showing that off-policy learning techniques can be bested by even this quasi-online learning approach. While not alone in finding this, the authors make an important contribution to showing the limits of off-policy learning and the result is timely as more and more effort is focused on trying to get better performance out of more reasonably sized models. The addition of the new AmbigSQL data set should prove useful as it is more easily evaluated automatically.", "weaknesses": "The paper is probably over-stuffed with results, and is generally very wordy making it difficult to read. It is saved by the clear algorithms and figures, but the text should be shortened considerably. This would also allow for more discussion of the AmbigSQL data set, which is a significant contribution, but which is barely described in the text. As an example, I found that the first three sentences of the paragraph at L350 (\"Implicit Ambiguity Recognition\") all say essentially the same thing. This could be shortened to make room for more informative text. Similarly, I don't know that we need to see every possible result in the main results tables. Focus on the main results/models to tell your story, and omit the rest to make the tables easier to read. I didn't find the ablation experiments useful in this case--you could have summarized the entire section in one short paragraph. \n\nThe AmbigSQL data might be too easy to learn, as some models achieve near perfect performance on it.\n\nThe paper is novel, but not exceptionally so. One thing you could do after shortening the rest of the paper is to consider the use of user simulators more, and find parallels between earlier work and yours. I think the novelty of the paper lies mostly in the exact implementation of data generation for preference optimization, summarized in Algorithm 1 and Figure 3, and it would be good to focus on that specifically.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "yCrZz08vKP", "reviewer": "Reviewer_thTU", "summary": "The paper proposes Action-Based Contrastive Self-Training (ACT), to improve large language models' (LLMs) conversational skills, particularly in handling ambiguity and learning optimal dialogue policies. The paper shows results in three conversational tasks, including tabular-grounded question answering, machine reading comprehension, and text-to-SQL generation, highlighting the importance of considering action-based preferences for conversational tasks.", "questions": "- Why is the SFT results reported in Table 1 and 2 do not improve with more data? Can you try a cross validation split?", "limitations": "", "rating": 5, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 3, "strengths": "Originality \nThe paper proposes a novel DPO algorithms for handle ambiguity, which to the best of my knowledge is novel in this settings.\n\nClarity \nThe paper is overall clear (Algorithm 2 is pretty helpful to understand the overall process).", "weaknesses": "Significance\nLow overall significance. The experiments focus on a specific sub-domain of dialogue systems (SQL-based), and a particular skill, which larger language model already posses. Moreover, the results focus on a single LLM, and report dubious SFT performance. This diminish the significance of the proposed approach.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)