Learning to Clarify: Multi-turn Conversations with Action-Based Contrastive Self-Training

Authors: Maximillian Chen, Ruoxi Sun, Tomas Pfister, Sercan O Arik
Primary area: applications to computer vision, audio, language, and other modalities
Keywords: dialogue system, conversation modeling, reinforcement learning, mixed-initiative interaction, LLM, RLHF, domain adaptation, data-efficient learning, clarification questions

Abstract:
Large language models (LLMs), optimized through human feedback, have rapidly emerged as a leading paradigm for developing intelligent conversational assistants. However, despite their strong performance across many benchmarks, LLM-based agents might still lack conversational skills such as disambiguation -- when they are faced with ambiguity, they often overhedge or implicitly guess users' true intents rather than asking clarification questions. Under task-specific settings, high-quality conversation samples are often limited, constituting a bottleneck for LLMs' ability to learn optimal dialogue action policies. We propose Action-Based Contrastive Self-Training (ACT), a quasi-online preference optimization algorithm based on Direct Preference Optimization (DPO), that enables data-efficient dialogue policy learning in multi-turn conversation modeling. We demonstrate ACT's efficacy under data-efficient tuning scenarios, even when there is no action label available, using multiple real-world conversational tasks: tabular-grounded question-answering, machine reading comprehension, and AmbigSQL, a novel task for disambiguating information-seeking requests for complex SQL generation towards data analysis agents. Additionally, we propose evaluating LLMs' ability to function as conversational agents by examining whether they can implicitly recognize and reason about ambiguity in conversation. ACT demonstrates substantial conversation modeling improvements over standard tuning approaches like supervised fine-tuning and DPO.

Decision: accept (poster)
Reviewer scores: 6, 6, 8, 5

Reviews and discussion:
<thread>
creation date:
1730494187870

thread_type:
official_review

<thread_object>
creator: reviewer_thtu

<content>
summary:
The paper proposes Action-Based Contrastive Self-Training (ACT), to improve large language models' (LLMs) conversational skills, particularly in handling ambiguity and learning optimal dialogue policies. The paper shows results in three conversational tasks, including tabular-grounded question answering, machine reading comprehension, and text-to-SQL generation, highlighting the importance of considering action-based preferences for conversational tasks.

strengths:
Originality 
The paper proposes a novel DPO algorithms for handle ambiguity, which to the best of my knowledge is novel in this settings.

Clarity 
The paper is overall clear (Algorithm 2 is pretty helpful to understand the overall process).

weaknesses:
Significance
Low overall significance. The experiments focus on a specific sub-domain of dialogue systems (SQL-based), and a particular skill, which larger language model already posses. Moreover, the results focus on a single LLM, and report dubious SFT performance. This diminish the significance of the proposed approach.

questions:
- Why is the SFT results reported in Table 1 and 2 do not improve with more data? Can you try a cross validation split?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730659288317

thread_type:
official_review

<thread_object>
creator: reviewer_qb6q

<content>
summary:
The authors present ACT, a framework for fine-tuning LLMs to respond better to ambiguous queries. The approach relies on online preference optimization, using a DPO-like objective, where the user responses and action classifications are provided by some other LLM user simulation and a trained action classifier, and the agent responses are provided by the model under tuning. The authors design a particular, detailed approach to generating conversational examples from which the model learns, and show that it is highly effective and with relatively few examples enables mid-size (~7B parameters) LMs to outperform very large LLMs even with 10 examples in their prompts. Finally, the authors provide a new dataset of ambiguous queries that are resolved to SQL queries, again showing that the ACT framework provides better results by learning to ask clarifying questions.

Updated ratings after reading author responses. In particular the presentation has improved and the paper is easier to understand. Thank you to the authors for their efforts.

strengths:
First and foremost, the authors present extensive comparisons showing that their approach leads to better conversational outcomes, both in terms of recognizing the need to clarify a user prompt, and in resolving the user's information need. The difference is sometimes quite large, with 10-20% improvements over SFT or a modified DPO approach. It is a significant result, showing that off-policy learning techniques can be bested by even this quasi-online learning approach. While not alone in finding this, the authors make an important contribution to showing the limits of off-policy learning and the result is timely as more and more effort is focused on trying to get better performance out of more reasonably sized models. The addition of the new AmbigSQL data set should prove useful as it is more easily evaluated automatically.

weaknesses:
The paper is probably over-stuffed with results, and is generally very wordy making it difficult to read. It is saved by the clear algorithms and figures, but the text should be shortened considerably. This would also allow for more discussion of the AmbigSQL data set, which is a significant contribution, but which is barely described in the text. As an example, I found that the first three sentences of the paragraph at L350 ("Implicit Ambiguity Recognition") all say essentially the same thing. This could be shortened to make room for more informative text. Similarly, I don't know that we need to see every possible result in the main results tables. Focus on the main results/models to tell your story, and omit the rest to make the tables easier to read. I didn't find the ablation experiments useful in this case--you could have summarized the entire section in one short paragraph. 

The AmbigSQL data might be too easy to learn, as some models achieve near perfect performance on it.

The paper is novel, but not exceptionally so. One thing you could do after shortening the rest of the paper is to consider the use of user simulators more, and find parallels between earlier work and yours. I think the novelty of the paper lies mostly in the exact implementation of data generation for preference optimization, summarized in Algorithm 1 and Figure 3, and it would be good to focus on that specifically.

questions:
Figure 3 was critical to my understanding of your approach. It would help the text to label figure 3 with the notation used in the text, and to refer to specific parts of the figure in the text to better connect the two. Without the figure I would have been completely lost in section 3. 

I found all of the sections on "conversations in the wild" difficult to follow and kind of disconnected from the rest of the text. Can you clarify what specific problem you are trying to address with these tests?

In Algorithm 2 you say (line 14-16) that the _trajectory_ is assigned to the winning or losing labels--but in the text it is just the sampled y_j that is assigned to these depending on the conversation outcome. These should me made consistent with each other.

I would like to see details of the user simulator U and action classifier A in the main text. Making your text less wordy and shortening the tables to just the key results will help make room to discuss these critical pieces more. The other results from the tables, and things like the ablations, can be expounded on more fully in the appendices. That's what appendices are for: extra results and things that don't turn out to be as interesting. 

You could also omit most or all of the DPO-Dist results from the main text--it never gets close to the best results. You can mention this in the main text and then detail it more in the appendices.

Line 304: How do you guarantee that the responses from Gemini Ultra model M _should_ be rejected? Won't it often come up with a good response?

Line 313: I admit I haven't focused on this aspect for some time, but surely there's something better than SentenceBERT for evaluation now?

Line 354: why switch metrics between datasets? Why not just stick with F1, even though your AmbigSQL dataset is balanced? It seems like being consistent would help us get a better sense of how challenging (or not) the AmbigSQL dataset is. 

Lines 425-427: ACT does better than frontier LLMs when it has 250 examples to learn from. I assume these are generated examples, and therefore you don't need human labeling. It makes me wonder (1) how much does this depend on the models you use being updated with human labeled data (since, if I understand correctly, the success of the method does depend on prior model alignment) and (2) why not try with more examples than 250?

Line 448: It seems odd to me that ACT outperforms Zephyr SFT on the overall trajectory outcome but not action recognition. Does the A model show bias somehow to prefer asking clarifying questions? 

Line 527: this paragraph can just be a single sentence--just tell us prior alignment was helpful and how much, don't need to discuss more.

Finally, there wasn't much (any?) discussion of the limitations of user simulations. In particular, I always worry that humans are less predictable than simulations--how might this affect your results?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730690681367

thread_type:
official_review

<thread_object>
creator: reviewer_g3nf

<content>
summary:
The paper introduces Action-Based Contrastive Self-Training (ACT), an algorithm designed to enhance large language models' (LLMs) ability to handle ambiguity in multi-turn conversations. Current LLM-based conversational agents often fail to ask clarifying questions when faced with ambiguous input, opting instead to overhedge or guess the user's intent. ACT addresses this by enabling data-efficient dialogue policy learning through contrasting different conversational actions, even without explicit action labels, using a preference optimization approach based on Direct Preference Optimization (DPO).

The authors demonstrate ACT's effectiveness across several real-world tasks, including tabular-grounded question answering, machine reading comprehension, and AmbigSQL—a new task focused on disambiguating complex SQL queries. Their experiments show that ACT outperforms standard tuning methods like supervised fine-tuning and DPO, particularly in scenarios with limited data. Additionally, the paper proposes a workflow for evaluating LLMs' ability to recognize and reason about ambiguity in conversations. The results suggest that incorporating action-based preferences via ACT enhances LLMs' performance in modeling multi-turn dialogues and improves their conversational skills.

strengths:
1. This paper is clear in writing. It is easy to understand the proposals and experiments. 
2. Diversified tasks of the evaluation datasets. I recognize the efforts author made to experiment on different tasks (e.g. SQL, tableQA etc.)

weaknesses:
1. The biggest weakness of the paper should be the choice of weak baselines and this japardizes the contributions claims of the paper. 
One paper on the top of my head is https://arxiv.org/pdf/2404.19733. This paper has many identical settings: online sampling and a heuristic to filter trajectories. I believe this paper should be a baseline to compare. 
Maybe the authors have similar ablation comparisons in Table 6. I would like to see some clear discussions on the baselines and ablations. 


2. Related to weakness 1, this paper doesn't discuss DPO variants built for tasks that require multi-step trajectories. This paper is proposing a new DPO algorithm but it has a very small section to discuss relevant DPO works. I would suggest the authors check into DPOs for conversational tasks or reasoning tasks that require COT. This will provide a more justified view of the paper contributions.

questions:
see above
</content>
</thread_object>

</thread>
<thread>
creation date:
1731424567050

thread_type:
official_review

<thread_object>
creator: reviewer_gk1a

<content>
summary:
This paper proposes a novel online optimization algorithm: action-based contrastive self-training, which facilitates data-efficient dialogue policy learning for modeling multi-turn conversation. The authors demonstrate their method on multiple real-world conversational tasks: tabular-grounded QA, machine reading comprehension and AmbigSQL. They also propose a way to evaluate LLM's ability to function as conversational agents. Their method ACT achieves substantial improvements than DPO and other baselines.

strengths:
1. This paper proposes a feasible way to optimizing multi-turn conversation modeling for LLMs, which overcome some obstacles faced by existing methods.
2. Their proposed method ACT is sample-efficient based on online DPO, and achieves good performance on three real-world conversational tasks. 
3. The paper is well-written and the experiments are convincing.

weaknesses:
This paper conducts experiments on limited applications of dialogue tasks; there are also many variants for DPO based methods, however, the authors have not compared them as baselines.

questions:
None
</content>
</thread_object>

</thread>
<thread>
creation date:
1732238150438

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Response to All Reviewers

comment:
We would like to extend our sincere gratitude to the reviewers for their insightful feedback on  our paper. We are glad to see all reviewers recognize the value of this work and unique contribution of our work from the perspectives of clarifying question optimization and exploring the limits of off-policy learning. The helpful suggestions we have received have helped us to greatly strengthen our paper. To summarize, we have run the following experiments and made the appropriate changes to our manuscript on OpenReview:

---

1. Following Reviewer g3nF and Reviewer GK1A’s suggestion, we have run additional experiments comparing ACT to Iterative Reasoning Preference Optimization on our quantitative reasoning tasks, finding that ACT consistently outperforms IRPO in multi-turn evaluations. We have updated Table 1 and 3 in the manuscript with these results. 

2. Following Reviewer Qb6Q’s suggestion, our other baselines (e.g. prompting variants) have been moved to our Additional Experiments Section (Appendix Tables A7-9).

3. Following Reviewer g3nF’s suggestion, we have added an additional description on the differences between ACT and existing on-policy variants of DPO which are not focused on conversational modeling improvements.

*Addressing the above points has helped us clarify the differences between our work and existing works which propose generic approaches to on-policy DPO.*

4. Following Reviewer thTU and Reviewer GK1A’s feedback, we have clarified that we do not only focus on a single dialogue domain. We have also run an additional experiment to demonstrate that ACT can generalize beyond a binary action space, to non-conversational actions, even though this is not the focus of our paper.

*This has helped us to reiterate that our work is not specific to SQL-based dialogue tasks, and demonstrated that ACT has the potential to generalize beyond optimizing the clarification action.*

5. Following Reviewer thTU’s feedback, we have demonstrated that even larger LLMs do not currently excel at multi-turn disambiguation and that this ability can be greatly improved through specialized tuning.

6. Following Reviewer thTU’s feedback, we have clarified that our results do not focus on a single LLM. We have run prompting experiments with multiple LLMs and we have run comparisons for ACT tuning with different base models.

*This has helped us to highlight the significance of our work, which is a general tuning approach to adapt LLMs to complex conversational settings which may require skills such as disambiguation.*

7. Following Reviewer Qb6Q’s suggestion, we have added a discussion on user simulators to the main text in Sec 3.1.

8. Following Reviewer Qb6Q’s suggestion, we have shortened the description of Agent task performance in Sec 4.2

9. Following Reviewer Qb6Q’s suggestion, we have shortened the description of Implicit Ambiguity Recognition in Sec 4.2.

10. Following Reviewer Qb6Q’s suggestion, we have added more details on AmbigSQL in Sec 4.1.3 of the revised manuscript.

11. Following Reviewer Qb6Q’s suggestion, we have shortened our discussion of the ablations.

12. We have merged our initial Qualitative Analysis section with the extended section in the Appendix (see Appendix Section I).

13. Following Reviewer Qb6Q’s suggestion, we have updated the notation in Figure 3.

*Incorporating these helpful suggestions have greatly improved the clarity, quality, and positioning of our paper.*

---

We are also appreciative of the reviewers’ acknowledging the various strengths of our paper, particularly:

Reviewer g3nF and GK1A acknowledged the *breadth of our experiments on three diverse real-world conversational tasks.*

Reviewer GK1A and thTU acknowledged the *novelty and significance of our proposed method for multi-turn modeling*. 

Reviewer Qb6Q acknowledged the *soundness* of our work, our *contribution to the growing literature exploring the tradeoffs of off-policy and on-policy learning*, and the *novelty* of our AmbigSQL dataset.

Reviewer thTU, GK1A, and g3nF acknowledged the *quality and clarity of our writing*.

---

In conclusion, we would like to express our appreciation to the reviewers for their invaluable insights which have been immensely helpful in refining our paper. We have responded in detail regarding these points to each review individually, and we are committed to incorporating each point of feedback in our revisions.
</content>
</thread_object>

</thread>
<thread>
creation date:
1733171965929

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Thank you to all reviewers!

comment:
Dear Reviewers,

As the discussion period comes to a close today, we would again like to express our sincere gratitude to each of you for your valuable insights and feedback! The feedback and discussion has greatly helped to strengthen our experiments, paper framing, and overall completeness. If there are any remaining questions or concerns you may have about our submission, we are happy to do our best to address them in the remaining time.

Thank you again for your time and efforts!

Kind regards,

The Authors
</content>
</thread_object>

</thread>


Metareview:
metareview:
This paper presents Action-Based Contrastive Self-Training (ACT), a framework for fine-tuning LLMs to address ambiguity in multi-turn conversations. The approach leverages a preference optimization objective, building on DPO, to contrast different conversational actions without requiring explicit action labels. ACT is demonstrated on tasks including tabular-grounded question answering, machine reading comprehension, and AmbigSQL, a new dataset designed for disambiguating SQL queries. The paper also introduces a workflow for evaluating LLMs' ability to handle ambiguity in dialogues.

The strengths of the paper are:

1. The paper introduces a novel online DPO method, ACT, which demonstrates sample efficiency and effectively addresses challenges in multi-turn conversational modeling for LLMs.
2. The authors provide experimental results across various conversational tasks, including SQL and TableQA, showing some performance improvements, e.g., in some cases 10–20% over existing methods like SFT and modified DPO.
3. The inclusion of the new AmbigSQL dataset could be a valuable resource to the field, enabling more straightforward automated evaluation and furthering research on handling ambiguity in conversational tasks.

As the paper has been extensively extended during the author response period, I will focus on the concerns that seem to remain in the current version:

1. While the authors provided clarifications on their DPO method and highlighted its focus on conversational modeling improvements, the method of the paper doesn't seem very creative or original. It is yet another variant of DPO aimed at a particular goal (clarifying question optimization).

 2. Experimental results: The most compelling results seem to be on the PACIFIC dataset, but this dataset is small and domain-specific (finance). The models evaluated in this work are quite small scale, not going beyond 7B. The experiments shown in the paper rely on 250 conversations or less, but without data scaling curves, it is more difficult to appreciate the contribution of the proposed method. As sample training size is small for all the experiments, it would be important to show mean and variance across different runs.

During the reviewer–AC discussion period, most reviewers sided with accepting the paper, but the dissenting reviewer kept their rating at 5 and expressed concerns about the experiments (see weakness 2, which is mostly a paraphrase of their points). I think this reviewer makes valid points, and the authors should seriously consider their concerns for the next version of the paper. However, I think these concerns are not serious enough to override the other reviewers' consensus to accept, so I recommend acceptance.