Dynamic Prompting: Large Language Models for Task Oriented Dialog
Authors: Jan Nehring, Akhil Juneja, Adnan Ahmad, Roland Roller, Dietrich Klakow (German Research Center for Artificial Intelligence, Technische Universität Berlin, Saarland University, 2024)
Topic: Chatbots and Dialogue Systems

Abstract
Large Language Models show impressive results in many different applications, most notably in the context of question-answering and open dialog situations. However, it is still an open question how to use those models for task-oriented dialogs such as booking or customer information systems, and such. In this work, we propose Dynamic Prompting, an architecture for task-oriented dialog, integrating the benefits of Large Language Models and showcasing the approach on the MultiWOZ 2.2 dataset. Our architecture leads to a high task success rate, provides sensible and specific answers, and is resistant to hallucinations. Further, we show that Dynamic Prompting is able to answer questions that were not anticipated by the dialog system's designer and that it can correct several types of errors and other characteristics of the system.

Introduction
Task-Oriented Dialog Systems (TODS) assist users in completing a task within a conversation [1], for instance, in the context of customer information and bookings (train/restaurant). In an applied setting with real users, it is important that those systems provide correct answers, tasks can be quickly solved, and lead ideally to high user satisfaction. To ensure this, TODS often provide a high level of control over its dialog management and answer behavior for system developers. Existing solutions normally either manually implement a dialog manager to control the complete interaction, or train it on large amounts of dialog interactions [2, 3, 4, 5]. In contrast, Large Language Models (LLMs) are very good at open-domain dialog and provide fluent and convincing messages in different styles. However, those answers might be misleading and even false (hallucination) [6, 7, 8]. In task-oriented dialog, the model could possibly ‘break out’ of the given dialog task. Using LLMs for task-oriented dialog is still in its infancy. Madotto et al. [9] used LLMs for the whole pipeline of Natural Language Understanding, Dialog State Tracking, Dialog Policy and Natural Language Generation. Hudeček and Dusek [10] expand on this idea by evaluating the abilities of LLMs to generate complete task-oriented multi-turn dialogs. They used LLMs for NLU and DST also but, unlike our work, they used a static prompt. Other approaches to LLMs for task-oriented dialog are presented by Cao [11], Hu et al. [12], Wei et al. [13], Li et al. [14]. To address those limitations and concerns, we propose Dynamic Prompting, a technique to combine a traditional task-oriented dialog system pipeline with the benefits of LLMs. Showcased and tested in the context of restaurant booking, we present the advantages and limitations of our approach.

Conclusion
We presented Dynamic Prompting, a technique integrating LLMs for task-oriented dialog. The results show high sensibility and specificity values, which indicate that the system answers on point and does not deviate from the dialog’s goal. The relatively low Prompt Extraction Performance and Response Slot Accuracy values still result in excellent task success. The high values in the performance metrics Prompt Instruction Performance and Information Extraction Performance indicate that the LLM follows the task-oriented guidance of the dynamic prompts. The Information Extraction Performance of 0.98 shows that the system could very well reuse the database information embedded in the prompt in the JSON format. In addition, our system shows various ways to correct errors, such as NLU errors, user requests not anticipated by the designer of DS, and errors in the format of the database entries. Moreover, the generated system answers are more diverse (Section 3.1.4) and more polite (Section 3.1.2) than the human-generated responses in the dataset. We would like to examine these qualitative results in future research in a more quantitative way. Overall, we find that the widespread problem of hallucinations in LLMs is not an issue in our system as long as we present the correct information to the LLM. As soon as the user asks the system for information that is not present in the prompt, such as the booking reference numbers, the LLM starts to hallucinate. Although we assess the system’s performance solely on the restaurant domain, the dynamic prompting method can be extended to other domains in the MultiWOZ 2.2 dataset, such as hotel, taxi, and train. Expanding to new domains will require updating the prompt generation module to accommodate new intents and state values, ensuring smooth integration with these additional domains.