Large Language Models as Zero-shot Dialogue State Tracker through Function Calling (2024)

Abstract: Large language models (LLMs) are increasingly prevalent in conversational systems due to their advanced understanding and generative capabilities in general contexts. However, their effectiveness in task-oriented dialogues (TOD), which requires not only response generation but also effective dialogue state tracking (DST) within specific tasks and domains, remains less satisfying. In this work, we propose a novel approach FNCTOD for solving DST with LLMs through function calling. This method improves zero-shot DST, allowing adaptation to diverse domains without extensive data collection or model tuning. Our experimental results demonstrate that our approach achieves exceptional performance with both modestly sized open-source and also proprietary LLMs: with in-context prompting it enables various 7B or 13B parameter models to surpass the previous state-of-the-art (SOTA) achieved by ChatGPT, and improves ChatGPT’s performance beating the SOTA by 5.6% average joint goal accuracy (JGA). Individual model results for GPT-3.5 and GPT-4 are boosted by 4.8% and 14%, respectively. We also show that by fine-tuning on a small collection of diverse task-oriented dialogues, we can equip modestly sized models, specifically a 13B parameter LLaMA2Chat model, with function-calling capabilities and DST performance comparable to ChatGPT while maintaining their chat capabilities. We have made the code publicly available.1

heading: 1 Introduction
text: Recent years have seen the rapid development of large language models (LLMs) that have demonstrated exceptional natural language understanding and generation capabilities. The integration of LLMs into industry applications, particularly as
1https://github.com/facebookresearch/FnCTOD †Work undertaken while interning at Meta. *Correspondence authors: zekunli@ucsb.cs.edu, and
pacrook@meta.com.
conversational assistants, is a notable trend. Finetuned with conversations between users and assistants, these models are further aligned with human preferences to enhance their ability to deliver fluent, helpful, and polite responses to user inquiries. Notable examples include proprietary systems such as ChatGPT2 and Claude3, as well as open-source models such as LLaMA2-Chat (Touvron et al., 2023), Vicuna (Chiang et al., 2023), Baichuan (Baichuan, 2023).
The primary focus of these chat-tuned LLMs has typically been on responding in general contexts. However, for another important type of conversation, task-oriented dialogues (TOD), the model is required to extract the intentions of users at each turn of the conversation, represented as slot-value pairs of per-domain predefined schemas; a process known as Dialogue State Tracking (DST). The challenge lies in the model’s ability to accurately summarize user needs over multiple turns of conver-
2http://chatgpt.openai.com/ 3https://www.anthropic.com/index/introducing-claude
8688
sation and also strictly adhere to a domain-specific ontology. The most direct solutions (Hosseini-Asl et al., 2020; Peng et al., 2020; Su et al., 2021) necessitate training on curated domain-specific annotated data, a process that is notoriously costly and labor-intensive. Despite efforts in data augmentation (LI et al., 2020) and automated dataset creation using GPT-3 (Li et al., 2022), these methods struggle to generalize to unseen domains. To achieve zero-shot DST for unseen domains, prior approaches usually involved domain transfer methods (Campagna et al., 2020; Lin et al., 2021a; Zhao et al., 2022). While such approaches are trained on alternative domains, they still require domains with matching annotation schema, and their performance has been far from satisfactory.
LLMs exhibit remarkable capabilities for tackling various tasks without the need for task-specific fine-tuning, making them suited for zero-shot DST. However, while there have been initiatives to leverage ChatGPT for zero-shot DST (Hu et al., 2022; Hudeček and Dušek, 2023; Heck et al., 2023; Chung et al., 2023), these methods tend to treat DST as a standalone task rather than chat completion, which the models, especially chat-tuned models, are more proficient in. They usually take the whole conversation as input along with detailed instructions to generate in domain-specific formats. This setup poses challenges due to the long task context and specific output requirements. Consequently, this works exclusively with advanced ChatGPT or Codex models but fails with less powerful LLMs (Hudeček and Dušek, 2023).
In this work, we introduce a novel approach FNCTOD, to address zero-shot DST with LLMs. Our method seamlessly integrates DST as a part of the assistant’s output during chat completion. Specifically, we treat the schema of each taskoriented dialogue domain as a specific function, and DST for this domain as the process of “calling” the corresponding function. We thus instruct LLMs to generate function calls along with the response in the assistant’s output. To achieve this, we
convert the domain schema into function specifications, which include the function’s description and required arguments, and incorporate them into the system prompt of the LLM. Additionally, we integrate these function calls into the assistant’s output within the dialogue context.
As shown in Figure 1, experimental results on the MultiWOZ benchmark (Budzianowski et al., 2018) represent a significant milestone. Our approach is the first that, without further fine-tuning, enables modestly sized open-source LLMs (7B or 13B parameters) to achieve comparable or superior performance compared to previous state-of-the-art (SOTA) prompting methods that relied exclusively on advanced proprietary LLMs such as ChatGPT and Codex (Hudeček and Dušek, 2023; Heck et al., 2023; Chung et al., 2023). Furthermore, our approach beats the previous zero-shot SOTA by 5.6% Av. JGA, firmly establishing a new standard. It improves ChatGPT performance; beating previous individual best results for GPT-3.5 and GPT-4 by 4.8% and 14%, respectively.
Additionally, we show that by fine-tuning a 13B LLAMA2-CHAT model using a collection of 7,200 task-oriented dialogues — consisting of 200 randomly selected dialogues covering 36 diverse domains, from heterogeneous TOD datasets — we can equip it with function-calling DST abilities comparable to ChatGPT while still maintaining its response generation capabilities.
The comparison with prior studies is summarized in Table 1 and Figure 1. Our contribution is threefold: (1) Demonstration that the FnCTOD approach achieves outstanding performance with both open-source and proprietary LLMs through in-context prompting: enables open-source 7–13B models to surpass the previous SOTA achieved by ChatGPT, and enhances GPT-4’s performance by 14%, establishing a new SOTA. (2) Bridging the zero-shot DST performance gap between opensource models and ChatGPT by fine-tuning on a small collection of diverse dialogues. (3) Showing that function calling DST capabilities can be
integrated into existing chat-tuned LLMs while preserving response capabilities.

heading: 2 Related Work

heading: 2.1 Dialogue State Tracking
text: DST is an essential, yet challenging task in the construction of TOD systems. Its primary purpose is to extract and track user goals at each turn throughout the conversation. The tracked dialogue state is usually represented in the slot values of the predefined schema for specific domains. This requires the slot values to adhere closely to the domain-specific schema. Consequently, previous methods have relied on the collection and annotation of domainspecific dialogues for model training (Lee et al., 2019; Wu et al., 2019; Heck et al., 2020; HosseiniAsl et al., 2020; Peng et al., 2020; Lin et al., 2020) However, obtaining training data is notoriously expensive, even with data augmentation methods (LI et al., 2020) and methods that utilize GPT-3 to automatically simulate such data (Li et al., 2022). Furthermore, these approaches are limited to handling only the domains covered in the training data.
To address zero-shot DST in unseen domains, previous cross-domain transfer strategies based on small models typically leverage extra dialogue corpora in similar domains (Wu et al., 2020; Lin et al., 2021b; Su et al., 2021) or redefine DST in terms of alternative tasks, e.g., Q&A (Lin et al., 2021c) or summarization (Shin et al., 2022) to find additional data. Despite these efforts, their overall zero-shot performance remains relatively low.

heading: 2.2 Leveraging LLMs for Dialogue Tasks
text: LLMs (Brown et al., 2020; Chowdhery et al., 2023; OpenAI, 2023) have demonstrated remarkable capabilities in handling various tasks without further fine-tuning. Recent chat/instruction-tuned models further exhibit impressive performance in conversational contexts (Touvron et al., 2023; Chiang et al., 2023; Yang et al., 2023). However, current chat models primarily focus on general conversation, often omitting task-oriented dialogues (TOD). TOD differs from general conversation in that it requires models to not only generate responses but also track dialogue states according to domain-specific schemas. While ChatGPT has shown effectiveness in response generation for TOD tasks (Li et al., 2024), its performance in zero-shot DST, as explored in recent research on prompting approaches (Hu et al., 2022; Bang et al.,
2023; Hudeček and Dušek, 2023; Heck et al., 2023; Zhang et al., 2023; Chung et al., 2023), is unsatisfactory and therefore remains as a open challenge.

heading: 2.3 Tool Use within LLMs
text: Early work on tool use (Parisi et al., 2022; Schick et al., 2023) and the recent launch of GPT-4 plug-in and function calling features (OpenAI, 2023), have highlighted the importance of function calling for LLMs, encouraging follow-up work (Patil et al., 2023; Shen et al., 2023; Li et al., 2023a). Commonly integrated tools include web browsers, calculators (Cobbe et al., 2021), translation systems. We are the first to utilize this tool usage/function calling capability to solve the challenging DST task in TOD with LLMs, bridging the gap between general conversation and task-oriented dialogues.

heading: 3 Background

heading: 3.1 Chat-tuned LLMs
text: Chat-tuned LLMs are models specifically finetuned to interact with users in a conversational manner. This category encompasses proprietary models such as ChatGPT and Claude, as well as open-source models such as Vicuna (Chiang et al., 2023), LLaMA2-Chat (Touvron et al., 2023), and Baichuan (Yang et al., 2023). These chat models typically start as base models that are further finetuned with a dialogue format, enabling them to function effectively as conversational agents. As depicted in Figure 2, the dialogue format typically features three distinct roles within two components: (1) the system role in the system prompt section, which defines the assistant’s roles, responsibilities, and expected behaviors; and (2) the user and assistant roles in the dialogue context section, encompassing their conversation. The model is typically tasked to produce the assistant’s responses to the user’s input. These chat models are primarily designed to generate helpful, detailed, and friendly responses to general user inquiries, rather than handling task-specific conversations as in TOD.

heading: 3.2 DST Task Formulation
text: In TOD, at each turn of conversation, the task of DST is to summarize the dialogue state St given the dialogue context Ct = {A1, U1, · · · , At, Ut}, where Ut and At represent the user utterance and assistant response at the t-th turn. For simplicity, we will omit the turn index t in subsequent discussions. The dialogue state S is a set of slot-value
the model is prompted to determine the function to be called (function name) . Following this, the specifications of the predicted function/domain, along with optional demonstration examples, are incorporated into the system prompt. This guides the model to generate function arguments and subsequently the response .
pairs:
S = {(s1,D1 , v1,D1), · · · , (si,Dj , vi,Dj )}, (1)
where si,Dj is the i-th slot in the Dj domain, and vi,Dj is its tracked value. Each domain Dj corresponds to a set of slots for a specific service, API call, or database query, such as restaurant reservations. In the case of the restaurant domain, the slots might include “restaurant-food”, “restaurantarea”, “restaurant-pricerange”, etc. We use SDj to denote the tracked slots for domain Dj .

heading: 4 Approach
text: Our method redefines DST as function calling, treating each domain as a distinct function, and the slot values within the domain as its arguments. As shown in Figure 2, this paradigm is represented in chat-tuned models by embedding function specifications within system prompts, as shown in Figure 3. The model is tasked with generating function calls followed by a response, as shown in Figure 4. We provide more details of our approach below.
DST as Function Calling In our formalization, DST is conceptualized as function calling. Each domain Dj is modeled as a unique function Fj , with the associated slot values serving as arguments. Consequently, at each turn of the conversation, the DST task transforms into identifying the correct function Fj and its arguments SDj :
<fn_call> Fj(s1,D1 = v1,D1 , s2,D1 = v2,D1 ,
· · · , si,Dj = vi,Dj ) </fn_call>, (2)
where “<fn_call>” and “</fn_call>” are special tokens. In practice, we use “<function_call>”
and “</function_call>” to represent them, and generate the function calls in JSON format. Some examples of function calls generated within a conversation are shown in Figure 4.
Dialogue Prompt Format As shown in Figure 4, we incorporate this function calling as an integral part of the conversation. At each turn of the conversation, given the current conversation context,
consists of the function calls and response .
the chat model is tasked with first generating the function call, followed by the response. To achieve that, we convert the domain schema into function specifications, using JSON, and include them in the system prompt within the dialogue prompt, as shown in Figure 3. By doing so, the model can be aware of the supported functions and the correct generation format to call the function.
Function Call Decomposition As outlined, the model is tasked with predicting not only the function to call (i.e., function name) but also generating arguments for the predicted function. Providing detailed specifications for all the supported functions would make the prompt very lengthy. To streamline this process and minimize the prompt length, we split the whole generation process into two consecutive steps: Function Selection and Argument Generation. As shown in Figure 2, during each turn of the conversation, the model first selects a function Fj from all supported functions. In this step, we provide concise function descriptions rather than detailed specifications in the system prompt and instruct the model to generate only the selected domain/function, surrounded by the special tokens “<domain>” and “</domain>”. Subsequently, we include the detailed specification of only the chosen function Fj in the system prompt,
prompting the model to generate the corresponding arguments Fj . This approach not only simplifies the task but also significantly reduces the prompt length, enhancing both efficacy and efficiency.
In-context Prompting Since the current opensource models are not specifically fine-tuned to generate function calls, there is no guarantee that the model could always generate the correct formats. To address that, we also include in-context example conversations as shown in Figure 4, along with the specification of the predicted function in the system prompt. We manually selected a few demonstration examples for each supported domain.
Prompt-based Fine-tuning To illustrate equipping an open source model with function calling capabilities, eliminating the need for demonstration examples, we fine-tune a LLAMA213B-CHAT model using a collection of heterogeneous task-oriented dialogue datasets, including WOZ (Mrkšić et al., 2016), CamRest676 (Wen et al., 2016b,a), MSR-E2E (Li et al., 2018), TaskMaster (Byrne et al., 2019) and SchemaGuided Dialogues (SGD) (Rastogi et al., 2020). Note that we deliberately exclude the entire target dataset. From these datasets, we choose 36 distinct domains with high-quality annotations. Instead of using all the data in those datasets, we randomly sampled 200 dialogues from each domain across the datasets, totaling 7,200 dialogues for training. This small sample size proved sufficient.
We convert these dialogues into our designed dialogue prompt. Specifically, we incorporate the specifications of all functions invoked in each conversation’s system prompt. Our loss calculation focused solely on the function calling aspect of the assistant’s generation. We refrained from finetuning the response generation component, in consideration of the LLMs’ existing competence in producing coherent responses and the scarcity of function-calling examples in our dataset. Our finetuned model is dubbed FNCTOD-LLAMA2-13B.

heading: 5 Experiments

heading: 5.1 Experimental Setup
text: Dataset and Metrics We evaluate on the widelyused task-oriented multi-domain dataset MultiWOZ 2.1 (Budzianowski et al., 2018; Eric et al., 2020). We used the 1,000 dialogues in the test split and measured joint goal accuracy (JGA), which measures the percentage of turns for which all slot values are correctly predicted. This test set spans 5
evaluate ChatGPT , and our fine-tuned model via zero-shot prompting, and open-source models via few-shot (5-shot) prompting. In addition to per-domain JGA and slot F1 scores, we report the macro-averaged JGA of these five domains (Average JGA), and also the multi-domain JGA (Overall JGA). The baseline results are directly taken from their respective works. The best performances in each column are in bold.
domains, with each conversation potentially covering multiple domains. Baselines We compare our approach with two distinct approaches: (1) Cross-domain transfer approaches, which involve training on MultiWOZ with one domain excluded and then evaluating on the held-out domain. This category includes TRADE (Wu et al., 2019), MA-DST (Kumar et al., 2020), TransferQA (Lin et al., 2021b), T5DST (Lin et al., 2021c), and D3ST (Zhao et al., 2022). (2) Previous prompting approaches that have only shown efficacy with advanced proprietary models, include IC-DST (Hu et al., 2022) using Codex, (Heck et al., 2023) and InstructTODS (Chung et al., 2023) using ChatGPT (GPT-3.5/4).
Evaluated Models We evaluate our method on proprietary ChatGPT and various open-source models. For ChatGPT, we evaluated the versions of GPT-3.5-Turbo (gpt-3.5-turbo-1106) and GPT4 (gpt-4-1106-preview), both of which are already equipped with function calling capabilities. Regarding open-source models, we assessed several widely recognized chat-tuned models of varying sizes, including the 7B parameter model ZEPHYR-7B-BETA (Tunstall et al., 2023), the
*IC-DST requires in-domain data to train the retriever for example selection, making it not strictly zero-shot DST.
7B and 13B versions of VICUNA-V1.5 (Chiang et al., 2023), the 7B, 13B, and 70B versions of LLAMA2-CHAT (Touvron et al., 2023), as well as the 13B parameter model BAICHUAN2-13BCHAT (Baichuan, 2023).
Additionally, we evaluate our fine-tuned model FNCTOD-LLAMA2-13B. It’s worth noting that unlike these domain transfer baselines, our model is trained exclusively on 7,200 dialogues from datasets other than MultiWOZ, making the setup more realistic and challenging.
Inference Details For both ChatGPT and our fine-tuned FNCTOD-LLAMA2-13B, which have been equipped with function-calling capabilities, we perform zero-shot prompting, excluding incontext examples in the system prompt. For the other open-source models, we perform few-shot prompting using five examples (5-shot) by default. It’s worth noting that the shot in zero/few-shot prompting refers to the number of in-context examples used when prompting the models, whereas the shot in zero-shot DST refers to the number of in-domain examples seen in the training data.

heading: 5.2 Zero-shot DST Evaluation
text: Table 2 presents the zero-shot DST performance comparison, with observations summarized below.
Our approach empowers moderately-sized opensource models to surpass previous SOTA results achieved with advanced ChatGPT. Previous prompting approaches showed promising results exclusively with advanced proprietary models but underperformed with less advanced models (Hudeček and Dušek, 2023). Our approach is the first to enable moderately sized open-source models to achieve comparable or superior performance compared to previous SOTA results obtained with advanced ChatGPT and Codex. Specifically, the 7B parameter ZEPHYR-7B-BETA and 13B parameter BAICHUAN2-13B-CHAT models outperform the previous SOTA. This significant advancement marks a milestone in the practical application of LLMs for DST and TOD.
Our approach significantly improves ChatGPT’s performance over previous prompting approaches. The efficacy of our approach is verified by improvements of 4.8% (Average JGA) for GPT-3.5, and 14% for GPT-4, compared to previous reported results with each of these models. Our result with GPT-4 beats the previous SOTA prompting approach using Codex by 5.6% Avergage JGA.
Our fine-tuned 13B parameter model matches the performance of ChatGPT. It is evident that our fine-tuned FNCTOD-LLAMA2-13B significantly improves over its base model LLAMA213B-CHAT and achieves a performance comparable to ChatGPT. This demonstrates that we can easily equip moderately sized open-source LLMs with function-calling capabilities and zero-shot DST performance comparable to ChatGPT, marking an exciting advance in bridging the gap between opensource and proprietary models.

heading: 5.3 Zero-shot End-to-End TOD Evaluation
text: In practical settings, a TOD system queries a knowledge base or API using the tracked dialogue states to ground responses. We perform an end-to-end evaluation of both DST and response generation, which is a more realistic and challenging setting. Our FNCTOD approach enables the generation of both dialogue states, i.e., function calls, and responses in the assistant’s output. This contrasts with the prompting methods that typically treat DST as a standalone task. Consistent with the previous work on end-to-end zero-shot TOD evaluation (Hudeček and Dušek, 2023), we evaluated using the MultiWOZ 2.2 dataset (Zang et al., 2020)
with delexicalized responses. Our evaluation metrics include JGA for DST and Success rate for the generated response. Success measures the percentage of dialogues in which the user’s goals were fully met (Budzianowski et al., 2018). The results are presented in Table 3.
Compared to previous prompting approaches, by enabling both zero-shot DST and response generation (Hudeček and Dušek, 2023), the superiority of the FnCTOD approach becomes more evident. Specifically, all open-source models evaluated using our approach outperform ChatGPT’s results achieved by (Hudeček and Dušek, 2023), except for LLAMA2-7B-CHAT. In addition, the results show that the fine-tuned model FNCTOD-LLAMA213B retains its ability to generalize and generate informative responses in a zero-shot TOD setting.

heading: 5.4 Ablation Studies
text: Impact of function call decomposition Our twostep decomposition approach can yield benefits in terms of both efficacy and efficiency. To demonstrate this, we compared the accuracy and cost with and without the two-step decomposition. The “without decomposition” condition includes the full specification of all domains/functions in the prompt, and requires the LLM to generate the full function call, including the function name and arguments in one step. This comparison is conducted on ChatGPT and our fine-tuned FNCTOD-LLAMA213B, which supports zero-shot prompting. Since ChatGPT is API-accessible, we compare its API calling cost which is calculated based on the number of tokens in the prompt. The results in Table 4 demonstrate both the effectiveness and efficiency of our decomposition approach.
In addition, to investigate the impact of cascading, two-stage errors due to incorrect function (name) prediction, we conduct experiments to com-
pare the performance with oracle and predicted functions. The results are shown in Table 5. As can be seen, while errors do impact the overall performance, the significance depends on the prediction accuracy. Many LLMs, especially ChatGPT (GPT-3.5), demonstrate very high prediction accuracy considering potential label noise. Performance with oracle and predicted functions are largely comparable. Improving the function prediction accuracy could further boost the performance of our approach.
Impact of function specifications In addition to directly including function specifications in JSON within the prompt, we experimented with translating the data into more human-readable natural language descriptions (refer to the comparison in Figure 7 in the Appendix). Figure 5 presents a comparison between using the JSON format di-
rectly (json) and converting it into natural language descriptions (text). The results indicate that the models perform similarly with both methods of function specification, indicating the high degree of flexibility of our approach in function specification customization.
Impact of the unified dialogue prompt In our approach, we seamlessly integrated function calls into the assistant’s output in the conversation context, which could also serve as demonstration for the current turn’s generation. To investigate its effect, we show the performance with and without function calls (w/ and w/o prev) in Figure 5. The results emphasize the effectiveness of embedding function calls within the conversation context.
Impact of varying numbers of in-context examples We assessed the performance of various open-source models, which were not originally trained for function call generation, with different numbers of in-context examples, ranging from 0 to 5. We note that using more than five examples might surpass the context-window capacity (such as 4096 tokens) for some models. The findings are illustrated in Figure 6. The results indicate that the models perform significantly better when in-context examples are utilized compared to zeroshot prompting. There is a consistent performance improvement as the number of examples increases, across most domains and models.
Impact of fine-tuning data sizes Our results indicate that with as few as 200 samples per domain, totaling 7,200 dialogues across 36 domains, we were able to fine-tune a LLAMA2-13B-CHAT model to match the zero-shot DST performance of ChatGPT. We explored the model’s performance with varying numbers of samples, ranging from 100 to 400 per domain. The results, depicted in
Table 6, show that optimal performance is achieved with 200 samples per domain. We speculate that beyond this point, the number of training samples leads to the model over-fitting to domains in the training data and, therefore, less effective at zeroshot generalization. However, we anticipate that increasing the data size through increased diversity of datasets would lead to further improvements.

heading: 6 Conclusion
text: We introduce a new approach to tackle the challenging task of zero-shot DST with LLMs, enabling them to handle both general conversations and task-oriented dialogues in diverse domains without the need for additional data collection. Our experimental results on MultiWOZ demonstrate that our approach not only delivers exceptional performance in advanced ChatGPT models (setting a new benchmark) but also across a range of moderately sized open-source LLMs. Furthermore, we demonstrate that we can fine-tune the open-source model LLAMA-2-13B-CHAT using only 7,200 training samples from 36 diverse domains, resulting in FNCTOD-LLAMA2-13B, which achieves function calling, zero-shot DST performance comparable to ChatGPT.

heading: 7 Limitations
text: In this work, we propose a novel approach to solve zero-shot DST with LLMs. Our approach achieves outstanding performance with various LLMs, both modestly-sized open-source and advanced proprietary LLMs, setting the new state-of-the-art. However, it is important to recognize that the current accuracy may still not be good enough for the practical deployment of such zero-shot systems. We anticipate that with further advancements in the NLU and NLG capabilities of base LLMs, our approach could achieve even greater performance levels. In addition, our approach can handle both the DST and response generation task in TOD. We evaluate DST with the well-established metric JGA with results suggesting the strong zero-shot DST performance of our approach. For the response evaluation, due to the current lack of a more realistic evaluation setting for response generation in TOD, we evaluated delexicalized responses as this is widely used in prior work. This setting and associated metrics have some known shortfalls in terms of being able to game-the-metrics with nonnatural responses as well as presenting a data mismatch with how LLMs are trained. In the era of LLMs, we advocate for the development of more realistic evaluation approaches for full-natural-languageresponse generation in TOD. Additionally, while this work concentrates DST and response generation, the two critical tasks in TOD, our approach can also be extended to include other complex tasks and handle various scenarios in dialogues as in the LangChain framework. We plan to explore these extensions in future research.

heading: A Appendix
text: A.1 Evaluation Details Model and Inference Details We evaluated two versions of ChatGPT and six leading chat/instruction-tuned LLMs representing varying sizes and instruction-following and conversational capabilities. The six evaluated open-source models include: ZEPHYR-7B-BETA (Tunstall et al., 2023) is an instruction-tuned version of Mistral-7B (Jiang et al., 2023), which is the leading model among its size on the AlpacaEval leaderboard (Li et al., 2023b). VICUNA-7B-V1.5 and VICUNA-13BV1.5 (Chiang et al., 2023) are LLAMA-2 models fine-tuned on user conversations with ChatGPT. LLAMA2-7B-CHAT and LLAMA2-13B-CHAT are chat-tuned versions of LLAMA2 models with varying sizes (Touvron et al., 2023). BAICHUAN213B-CHAT is also a LLAMA2-13B model further fine-tuned on extensive corpus (Baichuan, 2023). We utilized checkpoints available on Huggingface4. The specific paths for these models are detailed in Table 8. For inference, the temperature was fixed as 0.3, top_p as 0.2, and max_tokens as 128. For each test case, we conducted a single inference run. All inferences were executed on a cluster equipped with eight 48G NVIDIA RTX A6000 GPUs.
End-to-end Evaluation Setup Our end-to-end evaluation of task-oriented dialogues follows wellestablished standards in the literature. The traditional TOD computational process contains multiple steps, of which this work focuses on two: (1) dialogue state tracking (DST) which tracks the mentioned slots in order to query a back-end knowledge source, e.g., database or API, and (2) response generation (NLG) wherein the model generates a response given the context and query results. Following the literature, our end-to-end evaluation includes the evaluation of DST with JGA (our main results) at the turn level, and the evaluation of generated responses (NLG) using “Success rate” at the dialogue level. The latter measures the percentage of dialogues in which the user’s goals were fully met. We do not use or evaluate dialog acts, as they are not necessary for response generation.
The responses of the TOD system should provide specific information about a set of entities as requested by the user, like a restaurant or hotel. As multiple entities may satisfy the user’s goal and it can be difficult to detect whether the response re-
4https://huggingface.co/models
lates to a matching entity, delexicalized responses have been used in previous literature to allow for easier measurement and to decouple NLG errors from retrieval errors. In a delexicalized response, the mentioned slot values are noted as [value_xxx]. For instance, a delexicalized response might be: “It is a restaurant that serves [value_food] food near the [value_area]. The address is [value_address] and the phone number is [value_phone].” This strategy facilitates easier automatic determination of whether the dialogue answers specific requests by the conversation’s end. If both, all entities of the type expected are mentioned, and provision of all required information about those entities is also detected, then the dialogue is considered successful (Budzianowski et al., 2018). This results in a binary “Success” metric per dialogue. Regardless of the user’s input, the evaluation is on whether the entities and information about those entities provided to the user by the end of the conversation satisfies their goal. For example, given a user goal at the beginning of a dialogue to obtain the phone number of a restaurant serving French cuisine, the dialogue is marked as a success if by the end the system has mentioned at least one restaurant and provided a restaurant phone number. While JGA indicates whether the correct search for a restaurant serving French food was attempted.
A.2 Training Details
Training Data For constructing our fine-tuning dataset, we selected five high-quality, multi-turn TOD corpora, excluding MultiWOZ, as detailed in Table 9. Each dataset encompasses one or multiple domains. We excluded several domains with lowquality annotations, retaining a total of 36 domains. For our fine-tuning, we exclusively sampled data
from the training sets of these datasets to constitute our training data.
Hyperparameters We fine-tuned the LLaMA-213b-Chat checkpoint from Hugginface.5. We utilize Low Rank Approximation (LoRA) (Hu et al., 2021) and limited our fine-tuning to the parameters in the q_proj and v_proj modules. Further details about the fine-tuning hyperparameters can be found in Table 7. The fine-tuning was conducted on 4 A6000 48GB GPUs.
A.3 Supplementary Results
In this section, we present the detailed figures from the experimental results depicted in Table 3, Table 5, Figure 5, and Figure 6. Specifically, additional results from the end-to-end evaluation on MultiWOZ 2.2 are presented in Table 10 (complementing Table 3). Further results from the ablation studies with predicted and oracle functions are presented in Table 11 (complementing Table 5).
A.4 Details of Prompt Formatting
Conversation Context To format conversations within the prompt, we adopted the specific chat format for each LLM evaluated, as used in their respective instruction tuning steps.6
System prompt In our evaluation, we utilized the following system prompt template:
System prompt
You are a task-oriented assistant. You can use the given functions to fetch further data to help the users.
<FUNCTIONS> {Function Specifications} </FUNCTIONS>
<EXAMPLES> {Example Conversations} </EXAMPLES>
The parts surrounded in brackets and highlighted in blue serve as placeholders and are replaced with specific function specifications and example conversations related to that function/domain. The example part is only employed for few-shot prompting with the models not fine-tuned for functioncalling.
5https://huggingface.co/meta-LLaMA/ LLaMA-2-13b-chat-hf
6https://github.com/lm-sys/FastChat
Function Specifications For the function specification within the system prompt section of the prompt, we adhere to ChatGPT’s format. To enhance model comprehension, we also experimented with translating the JSON format into a natural language description to include in the system prompt. An example illustrating both the JSON format and its corresponding natural language description for a specific domain is depicted in Figure 7.
Full Prompt Combining all components, an example of the full dialogue prompt is displayed in Figure 8. For clearer illustration, we adopt a more human-readable dialogue format not including the special tokens used in model-specific dialogue formats.
components such as the function specifications in the system prompt, the function call , and the response in the assistant’s output are highlighted for clarity. It’s important to note that for easier comprehension, we’ve adopted a more human-readable dialogue format, and have simplified some function arguments for clearer visualization. In practice, we use the specific chat format tailored to each evaluated model, and the LLaMA2-Chat format is applied for fine-tuning the LLaMA2-Chat model.

heading: Future Work
text: We plan to explore these extensions in future research