Few-shot Dialogue Strategy Learning for Motivational Interviewing via Inductive Reasoning (2024)

Abstract: We consider the task of building a dialogue system that can motivate users to adopt positive lifestyle changes: Motivational Interviewing. Addressing such a task requires a system that can infer how to motivate a user effectively. We propose DIIR (Dialogue Strategy Learning via Inductive Reasoning), a framework that is capable of learning and applying conversation strategies in the form of natural language inductive rules from expert demonstrations. Automatic and human evaluation on instruction-following large language models show natural language strategy descriptions discovered by DIIR can improve active listening skills, reduce unsolicited advice, and promote more collaborative and less authoritative responses, outperforming various demonstration utilization methods.

heading: 1 Introduction
text: We study Motivational Interviewing (MI), where a conversational agent assists users in finding motivation and reducing ambivalence in adopting positive lifestyle changes such as quitting drug abuse and seeking medical assistance (Moyers et al., 2016). A successful MI dialogue system must be able to infer how to best motivate a user to adopt positive changes, which is a nontrivial task. For example, as shown in Figure 1, one way to convince a user to workout is to educate the user about its benefits, yet immediately offering suggestions without understanding the context can risk creating resistance and confronting users (Brehm, 1981). While prior work studied rule-based dialogue systems for MI (Welch et al., 2020; Park et al., 2019) and MI dialogue response rephrasing (Welivita and Pu, 2023), another important problem is to build dialogue systems that automatically discover and learn MI skills.
In this work, we introduce DIIR (Dialogue Strategy Learning via Inductive Reasoning)1, a framework for aligning LLMs to dialogue behavior of
1https://github.com/zhouhanxie/DIIR
professional MI interviewers given a few demonstration dialogues. DIIR does not require training and instead employs LLMs to generate a set of natural language dialogue strategy descriptions, such as “when the user is hesitant about change, the interviewer should ask open questions to learn more about the user’s concern" by analyzing demonstration dialogues. These strategies are hypotheses about how professional interviewers respond in various situations, and are reused at inference time as instructions for LLMs to mimic MI interviewers. This process of identifying underlying principles from observations and generalizing them for inference is a form of inductive reasoning (Feeney and Heit, 2007).
Different from common practice that uses available demonstrations as in-context-learning (ICL) examples (Brown et al., 2020; Xu et al., 2023a), strategies descriptions discovered by our framework explicitly state the desired behavior for the model, such as “ask open questions" and their
13207
applicable situations, such as “when the client is hesitant about change". These automatically generated strategy descriptions are a more precise nudge to the LLM being instructed than ICL examples. Existing work that employs language models for generating and reusing natural language strategies from observations (Majumder et al., 2023; Nottingham et al., 2023; Wang et al., 2024a) (i.e. Inductive Reasoning with LLMs) requires interactive environments to verify and improve the generated statements. In comparison, our method learns from a static demonstration dialogue dataset which is more commonly available than interactive environments for dialogue models.
To evaluate DIIR, we automate and adapt a set of metrics derived from clinical psychology literature for evaluating human MI practitioners. We experiment with DIIR on a publicly available MI dataset. Automatic and human evaluation show strategies discovered by DIIR can help instruction-following downstream language models produce more collaborative, respectful, and least resistance-inducing responses with as few as five annotated dialogues of expert demonstrations. Our contributions are as follows: (1) To our best knowledge, we are the first to work to build and evaluate a dialogue system that automatically learns from dialogue data for MI. (2) We develop DIIR, a framework that learns and applies dialogue strategies as natural language inductive rules from MI experts for only a few dialogues. (3) We automate evaluations from the clinical psychology literature on MI to facilitate future research on MI.

heading: 2 Dialogue Strategy Learning via Inductive Reasoning
text: The goal of an MI interviewer is to convince a client to make a positive change of interest, such as to start working out. Given a dialogue history h consisting of previous user-system turns, an ideal MI dialogue system produces the next system response a that best convinces the user to make a positive change. We solve this optimal response generation problem by using LLMs to analyze demonstration dialogues and generate dialogue strategy descriptions stating how to respond in various situations, such as “When the user is hesitant about change, ask open questions.” These descriptions can then be reused to improve an instructionfollowing LLM to promote demonstration-aligned dialogue behavior, such as asking open questions
Algorithm 1 ENHANCESTRATEGY Input: Dialogue history h, gold response a∗, generator LLM
G, discriminator LLM D, executor LLM E, maximum number of trials N Output: Strategy e in natural language, description of its applicable situation k
1: e← "" 2: for i in N do 3: a′ ← E.generate_response (h, e) 4: if D.confirms_is_similar (a′, a∗) then 5: break 6: e← G.improve_strategy (h, e, a∗, a′) 7: k ← G.describe_situation (h) 8: return e, k
when the client is hesitant. We now describe DIIR’s learning and inference process (see Appendix D for step-by-step concrete examples). Learning Dialogue Strategy by Analyzing Demonstration Dialogues Given a dialogue history h and a gold interviewer response a∗, a corresponding dialogue strategy description e should contain sufficient information such that an LLM could closely reproduce a∗ given h and e. Intuitively, one could instruct an LLM to analyze the gold response and its corresponding dialogue history to generate descriptions containing information about “When <description about h>, one should <description about a∗>,” which can then be used as strategy descriptions. We denote this LLM that produces strategy descriptions as the strategy generator G.
However, there is no guarantee that G always generates valid strategy descriptions. To address this issue, we propose to employ a discriminator LLM D and an executor LLM E in place of the interactive environment in previous works to give feedback during the strategy statement generation process. The intuition is that to verify a dialogue strategy such as “When the user is hesitant about change, ask open questions", we could instruct the executor E to generate a response given the dialogue history where a user is hesitant following this this instruction, and check whether the generated response matches the gold interviewer response.
This verification process eliminates two sources of error: the generated strategy is incorrect or is correct but is not detailed enough to instruct an LLM to reproduce the groundtruth response. Existing work typically addresses this statement-verification problem by verifying the statements in interactive environments such as MineCraft (Nottingham et al., 2023). However, there are no environments to ver-
ify statements about dialogue strategies. The process for generating and verifying a strategy description given a context-and-gold-response pair is as shown in Algorithm 1. Concretely, given a hypothesized dialogue strategy e (initialized to be an empty string), the executor generates a response a′ given the dialogue history h and the strategy e. If the discriminator checks and marks a′ as similar to gold response a∗, we accept the strategy e as valid. Otherwise, we instruct the generator to regenerate a better strategy given the dialogue history, current strategy, generated response, and the gold response. This generateand-verify process repeats until a valid strategy description has been found or a fixed number of search steps N has been reached.
Finally, to be able to reuse this strategy in the future, we prompt the generator to generate a description of the situation k such as “the user is hesitant about change.” This situation description can be derived from e since the dialogue strategy already contains information about when the strategy is applicable, but we use an LLM to generate it in practice for ease of implementation. We repeat the process introduced above for all context-response pairs available in the demonstration dialogues to create suitable strategies for various scenarios.
Inference via Strategy Reuse After learning, DIIR accumulates a set of strategies {ei} and their corresponding applicable situations {ki}. Thus, given an inference-time dialogue history hinference, we can simply generate a description of the situation kinference using the generator LLM, and then use kinference to retrieve the most suitable strategy ei by identifying ki that is most similar to kinference. We then instruct the executor to generate a response given hinference and the retrieved ei. In practice, we use a text embedding model Φ to encode all situational descriptions and find the top ten most suitable demonstration situation by maximizing the dot product of Φ(ki) and Φ(kinference) following Lewis et al. (2020); Guu et al. (2020). After this, we instruct the Generator LLM to pick a strategy that is most appropriate to the dialogue history from the ten retrieved strategies and follow its description. This process is illustrated in Figure 1.

heading: 3 Experiments
text: Models and Baselines We demonstrate DIIR’s few-shot learning ability by learning from 5 di-
alogues using GPT3.5 and GPT4, which mimics data-scarce scenario where only a few highquality demonstration dialogues are available. We chose GPT3.5 and GPT4 as backbone models due to their strong reasoning ability, similar to various recent works on LLM-based reasoning (Wang et al., 2024b; Majumder et al., 2023; Nottingham et al., 2023; Yu et al., 2023; Qiu et al., 2024), and find open source models less performant (see Appendix E for additional discussions). For baselines, we consider three variants of in-context learning (ICL) in addition to the vanilla LLM: ICL-RAND with 5 random demonstrations, ICL-KNN (Xu et al., 2023a) with 5 demonstrations with most similar dialogue context to the inference-time context computed using a text embedding model, and ICLALL with all demonstration dialogues.
Dataset We experiment on the AnnoMI dataset (Wu et al., 2022), an MI dataset transcribed from YouTube videos with 133 MI transcripts for various topics. The dataset also comes with quality labels for each dialogue, where 110 dialogues are labeled as high-quality demonstrations from good MI interviewers, and the remaining 33 dialogues are labeled as low-quality demonstrations. We consider the high-quality dialogues only since the goal of DIIR is to learn conversation strategies from MI experts. We randomly sample 5 dialogues with 112 turns for learning and 80 dialogues with 1192 turns for our primary evaluation on GPT3.5. For high-cost evaluation experiments involving GPT4 or human annotators, we further randomly sub-sample 10 dialogues with 148 turns from the primary evaluation data.

heading: 4 Evaluation
text: For automatic evaluation, we evaluate the alignment of our model by obtaining dialogue action counts with a dialogue act classifier trained on crowd-sourced annotations (Welivita and Pu, 2022). We report the following behavior-count-based evaluation adopted from the Motivational Interviewing Integrity Treatment (MITI; Moyers et al. (2016)) and Motivational Interviewing Skill Code (MISC; Miller et al. (2003)) used to evaluate professional interviewers from clinical psychology literature. Ratio of MI-guideline-inconsist Behavior (%MIi) The ratio of MI-inconsistent behaviors (confront, direct, warn, and advice without permission) in model responses. Complex over Simple Reflections (C/S) The

heading: Method %MI-i↓ C/S↑ R/Q↑ %AL↑ %NA↑
text: ratio of complex over simple reflections in model responses, indicating how often the model produces in-depth feedback to the client. Reflection over Questions (R/Q) The count of reflections over questions in the models’ responses. Percent of Active Listening Behaviors (%AL) The ratio of the sum of Questions and Reflections among all behavior counts, showing how often does the modal employ active listening skills. Percent of Non-authoritatitive Behaviors (%NA) The frequency of responses that are not confront, warn, direct, advice (with or without permission), or give information.
We verify the effectiveness of our automatic evaluation by running the evaluation on high and lowquality interviewer demonstrations in the AnnoMI dataset (Wu et al., 2022), results show our method can effectively tell apart high versus low quality demonstrations (see Appendix C for additional discussions and details of the dialogue act classifier).
For human evaluation, we evaluate our models’ performance against in-context baselines by asking human evaluators on Amazon Mechanical Turk to read four context-response pairs produced by both models and choose the one that is best aligned with a description of MI (more details in Appendix C). This results in 37 groups of evaluations between each pair of opponent models, where each group is rated by an annotator.

heading: 5 Results and Analysis
text: Our main results are as shown in Table 1. Due to space constraints, we report the best performance among the four baselines (base model and three ICL variants) for each dimension under the name BESTBASE, and show detailed performance in Table 4 in the Appendix. Although BESTBASE has an unfair advantage due to combining best
baseline results for each dimension, our method outperforms the baselines in all categories except the Reflection-to-Question ratio, sometimes outperforming the ground truth responses. Meanwhile, MI guideline defines an R/Q ratio of 1.0 to be good and 0.5 to be satisfactory (Moyers et al., 2016), showing that our model maintains competitive performance in this dimension.
Overall, our method produces fewer MIinconsistent responses, provides more in-depth reflections, and better leverages active listening skills and the MI spirit of not assuming expert roles. These trends are coherent with human evaluation results, as shown in Table 2.

heading: 6 Related Work
text: Motivational Interviewing in NLP Prior work in NLP about MI primarily studied the task of annotating (Welivita and Pu, 2022; Tanana et al., 2015) and predicting (Pérez-Rosas et al., 2017; Huang et al., 2018) dialogue actions. Recently, there are attempts on building systems for evaluating (Min et al., 2022) and rewriting (Welivita and Pu, 2023) responses for MI but does not focus on response generation. A few pioneering works build MI dialogue systems with hard-coded rules or dialogue templates (Welch et al., 2020; Park et al., 2019) but focus on specific domains such as COVID-19. To the best of our knowledge, we are the first to build an MI dialogue system that automatically learns from expert demonstrations.
LLMs for Generating and Reusing Natural Language Strategies Generating hypothesis from observations and reusing the generated hypothesis with LLMs have shown a promising direction for various tasks, such as planning (Majumder et al., 2023; Nottingham et al., 2023; Wang et al., 2023), prompt-generation (Wang et al., 2024b), and code generation (Wang et al., 2024a). This is a form of inductive reasoning (Wang et al., 2023) where LLMs draw conclusion from obvervations. However, prior works require interactive environments
for feedback which are unsuitable for learning from static datasets, while our work use an LLM for feedback and lifts the requirement for interactive environments.

heading: 7 Conclusion
text: We propose DIIR, a method for few-shot aligning large language models to MI techniques. Our method infers natural language strategies from expert demonstrations and reuses these strategies at inference time. Automatic and human evaluation show our method is better aligned with the behavior of motivational interviewing experts, produces more in-depth reflections, and better leverages active listening skills, outperforming a variety of demonstration-based methods.

heading: 8 Limitations
text: We note that our framework is developed for Motivational Interviewing and benefits from domainspecific knowledge such as MI-specific dialogue actions. Meanwhile, our framework assumes access to a strong instruction following language model, which is usually restricted by API access or computational resources. Finally, we note that while we adopt clinical psychology literature verified metrics and human evaluation to measure the performance of DIIR, the ideal evaluation is deploying MI dialogue systems for continued intervention.

heading: 9 Ethical Concerns
text: Following previous work on conversational persuasion (Wang et al., 2019), we note Motivational Interviewing shares similarities with persuasion dialogues. While the technique for Motivational Interviewing is developed for inducing positive changes in users, system deployers should carefully analyze the specific topic for Motivational Interviewing to eliminate risks of harm. Meanwhile, when deploying a system equipped with MI abilities, the user should be informed about the system’s identity so that they can make an informed choice when interacting with such a system.

heading: 10 Acknowledgments
text: We would like to thank Tim Althoff, Jessica Echterhoff, Yilan Chen, and the anonymous reviewers for their valuable comments. This work is funded by the SONY Research Award Program.

heading: A Building the Dialogue Act Classifier
text: Following prior work in the Computer Science and NLP community on annotating motivational interviewing dialogues (Shah et al., 2022; Welivita and Pu, 2022; Pérez-Rosas et al., 2016; Tavabi et al., 2021), we train a dialogue classifier to map sentences to motivational interviewing behavior codes. We train the dialogue act classifier on a publicly available dataset (Welivita and Pu, 2022) with annotated dialogue actions specific to MI, since the AnnoMI dataset does not contain fine-graned dialogue action labels. This dataset contains single turn interactions only, and thus are suitable for training dialogue act classifiers only but not the dialogue model itself. We fine-tune two variants of pretrained mask language models, BERT (Vaswani et al., 2017) and Mental-BERT (Ji et al., 2022), a fine-tuned version of the base BERT model on psychotherapy dialogues. We use a 0.6−0.2−0.2 trainvalidation-test split. We fine-tune both models using the Huggingface Transformers Library (Wolf et al., 2020) for a maximum of 10 epoch with early stopping at a learning rate of 5e−5. While the abso-
lute accuracy of both BERT 2 and Mental-BERT 3 are not optimal (Recall at 1 at 0.60 and 0.61, respectively), we fine the Recall at 5 out of 12 classes are high (at 0.95). Thus, we use the classifier to retrieve N=5 relevant labels given a input sentence, and use a GPT-3.5-turbo to decide the final label, following recent findings in NLP that combining an LLM with small classifier for classification has superior performance to both base models (Xu et al., 2023b).

heading: B Other Experiment and Implementation Details
text: For DIIR’s learning process, we set the maximum number of optimization steps to 3, and break the reference game between the generator and the discriminator afterwards regardless of success. To enhance the feedback quality of the discriminator, we also use the same dialogue act classifier as described in Appendix A to provide distant labels of what potential actions are in the ground truth response (this means predicting the string labels of dialogue actions, and append this to the instruction for the LLM).
For all experiments, the specific model names we used in OpenAI is gpt-3.5-turbo-1106 and gpt-4-1106-preview. These models are available as of December 2023. For the off-the-shelf retriever for retrieving strategies, we use a publicly available light-weight model 4 and retrieve usiang dot product scores. All experiments are conducting using 1 RTX A6000 GPU. Learning and Inference of DIIR on the specified data size finishes in 3 hours and 1.5 hours, respectively. The trainable component of the dialogue act classifier (based on BERT) contains 12-layer, 768-hidden, 12-heads, and 110M parameters. The retriever model contains 12-layer, 384-hidden, 12-heads, 33M parameters.

heading: C Details on Evaluation

heading: C.1 Discussion on Evaluation Metrics
text: We provide discussions on the derived metrics in this section. In particular, the evaluation process in MISC evaluates a interviewer on three dimensions: (1) Acceptance: being non-judgemental and do not direct or confront the user, (2) Empathy (different from the definition as used in Empathetic
2https://huggingface.co/bert-base-uncased 3https://huggingface.co/mental/mental-bert-base-uncased 4https://huggingface.co/sentence-transformers/all-
MiniLM-L6-v2
Dialogues in NLP): showing genuine interest in understanding the client’s situation and thoughts with active listening skills, and (3) Motivational Interviewing Spirit: the interviewer should be collaborative rather than assuming expert roles. In practice, we found all models rarely direct or confront the user using our dialogue action classifier, trivially satisfying the Acceptance criteria. We use %AL as a proxy for the Empathy criteria, and %NA as a proxy to measure the Motivational Interviewing Spirit criteria. These metrics are proven to reduce client resistance and improve intervention outcome in MI literature (Moyers et al., 2016).
C.2 Verifying the Effectiveness of Automatic Evaluation
to validate our evaluation, we run our classifierbased evaluation pipeline on 231 datapoints (dialogue context + gold response) of good vs. bad demonstration of MI interviewers (the binary high v.s. low quality labels are available in the original AnnoMI dataset). Bad demonstrations here means the response are produced by interviewer that does not follow MI guideline. We use 231 pairs from each class (i.e. 231 high and 231 lowquality demonstrations) since that is the number of all bad demonstrations available, and we downsampled good demonstrations to balance classes. Specifically, the desired behavior here is our evaluation pipeline should be able to tell apart good versus bad MI interviewers.
As shown in Table 3, our evaluation pipeline does indicate that high quality MI interviews are much better than low quality ones - this shows that our evaluation method can tell apart good vs bad MI interviewers.

heading: C.3 Human Evaluation Details
text: We use Amazon Mechanical Turk for our human evaluation5. The specific prompt we use is “Which response does not claim an expert role and make overt and
5https://www.mturk.com/
frequent attempt to persuade personB, doesn’t push for persuation when giving factual information, and act more like a companion that show genuin interest in understanding personB’s thoughts and situation? Pick the one that you thinks is best described by the style above". To unsure quality response, we request works to have an lifetime approval rate of 98% or above, and filter out all response that failed an attention check (for example, picking an animal that is not often kept as pet from ragdoll cat, golden retriever, elephant, and gold poodle). We do not impose any constraints (other than English-speaking), and pay at least 0.1 dollar per task (8 responses).

heading: D Concrete Example of DIIR’s Learning and Inference Process

heading: D.1 Learning
text: We provide a concrete example of the learning process in the “Inferring Strategy” section in Table 7. The comments denoted in “##" in the table should make it self-contained, but we provide more details here. The input to our model is the dialogue state (dialogue history h). First, the generator (GPT-3.5-Turbo) generates an Initial Response “It is normal to. . . ” prompted with the dialogue context; this is just a vanilla response generated by the LLM. Then, the discriminator model (another LLM) checks and gives feedback to the generator LLM on its flaws compared to the ground truth, in this case “when the client acknowledges the difficulty. . . ”. The generator will then be prompted to regenerate the response based on Dialogue State as well as the feedback from the discriminator model. In this case, the regenerated response is “It sounds like it’s been. . . ” At this point, the discriminator model will be prompted to check whether the response is matching ground truth - here, both the generated response “It sounds like it’s been. . . ” and the gold response “It’s something that’s always on your mind” are reflecting on what the client said, and thus the discriminator marked the process as complete. We then save this instruction “when the client acknowledges the difficulty. . . ” as a stategy decription e for reuse at inference time. The discriminator might not always generate ideal feedback on the first trial, so we prompt the discriminator model to refine the feedback and

heading: Method %MI-i↓ C/S↑ R/Q↑ %AL↑ %NA↑
text: repeatedly try the new feedback prompt on the generator until success (or some predefined maximum number of refinement steps reached).
D.2 Inference
We provide a concrete example of inference in Table 8. The comments denoted as “##" should make the table self contained, but we provide more details here. The input to the model is the Dialogue State (dialogue history h). After this, we prompt an LLM to generate a dialogue situation description of the dialogue state, for example, “The client is hesitant and unsure about changing yet". Recall that at training time we assign a dialogue situation description like the one above to each training dialogue state, so at inference time we can use a dense retriever to pick out the training data that has the most similar client mental states to the current one. We then prompt a separate instance of LLM with the current dialogue state (context) and the N=10 natural language strategies associated with the retrieved dialogue contexts, and let it determine the most relevant one, i.e. the reranking step. In the example shown in Table 5, the selected strategy is “when the client seems hesitant and uncertain about making a positive change. . . " We give this strategy to the generator model and let it improve the response, producing the final output “It’s important to consider the potential risks. . . ".

heading: E Discussion on Open Source Model
text: In our exploration phase, we test DIIR on open source models, but find they cannot consistently follow the generated strategy statements. This is

heading: Method %MI-i↓ C/S↑ R/Q↑ %AL↑ %NA↑
text: coherent with recent findings that LLMs are better at generation than understanding (Qiu et al., 2024; West et al., 2024). We provide the performance of Mistral6 in Table 5, as shown in results, our method is comparable to various baselines on a weaker-instruction following model (than GPT3.5 and GPT4).

heading: F Definition of Dialogue Actions
text: We show definition of each dialogue action as in Table 6.

heading: G Prompting Details
text: We document the prompts used in this work for reference as follows. All placeholders in the prompts are wrapped in “<” and “>” marks. For example, dialogue context placeholder is denoted as “<Dialogue Context>.”
G.1 Instruction for LLM to Generate Situation Description k
We ask the LLM to describe the situation with the prompt in Table 10. For a better inductive bias, we ground the generation in the scope of stages of change model (Prochaska and Velicer, 1997) to help the model better summarize relevant client mental states by simply pasting the definition of stages into the prompt.
G.2 Instruction for Other LLM Components
We report the instructions for the generator, descriminator, and executor LLMs in Table 11,Table 12, and Table 13, respectively. For the ease of implementation, the descriminator LLM reuses prompts for the generator LLM, with an additional question that prompts the model to check whether the executor’s response is close enough to the ground truth.
6https://huggingface.co/mistralai/Mistral-7B-v0.1

heading: H License Statement
text: The code base of this work is open sourced under CC BY license.

heading: Inferring Strategy...

heading: Example of Learning w/ DIIR+GPT3.5 Dialogue State h

heading: Example of Inference w/ DIIR+GPT3.5 Dialogue State h
text: Example of Learned strategy descriptions w/ DIIR+GPT3.5

heading: Prompt for the executor LLM E

heading: Prompt for the descriminator LLM D

heading: Prompt for the generator LLM G

heading: Prompt for Generating Situation Description k

heading: Future Work
text: Finally, to be able to reuse this strategy in the future, we prompt the generator to generate a description of the situation k such as “the user is hesitant about change. ” This situation description can be derived from e since the dialogue strategy already contains information about when the strategy is applicable, but we use an LLM to generate it in practice for ease of implementation. We repeat the process introduced above for all context-response pairs available in the demonstration dialogues to create suitable strategies for various scenarios. Inference via Strategy Reuse After learning, DIIR accumulates a set of strategies {ei} and their corresponding applicable situations {ki}. Thus, given an inference-time dialogue history hinference, we can simply generate a description of the situation kinference using the generator LLM, and then use kinference to retrieve the most suitable strategy ei by identifying ki that is most similar to kinference. We then instruct the executor to generate a response given hinference and the retrieved ei. In practice, we use a text embedding model Φ to encode all situational descriptions and find the top ten most suitable demonstration situation by maximizing the dot product of Φ(ki) and Φ(kinference) following Lewis et al. (2020); Guu et al. (2020). After this, we instruct the Generator LLM to pick a strategy that is most appropriate to the dialogue history from the ten retrieved strategies and follow its description. This process is illustrated in Figure 1