Venue: NAACL 2024

# FLAP: Flow-Adhering Planning with Constrained Decoding in LLMs

Shamik Roy  Salilk Sengupta Daniele Bonadiman Saab Mansour Arshit Gupta

{royshami, sailiks, dbonadim, saabm, arshig}@amazon.com

WSA AI Labs

###### Abstract

Planning is a crucial task for agents in task oriented dialogs (TODs). Human agents typically resolve user issues by following predefined workflows, decomposing workflow steps into actionable items, and performing actions by executing APIs in order; all of which require reasoning and planning. With the recent advances in LLMs, there have been increasing attempts to use them for task planning and API usage. However, the faithfulness of the plans to predefined workflows and API dependencies, is not guaranteed with LLMs. Moreover, workflows in real life are often custom-defined and prone to changes; hence, adaptation is desirable. To study this, we propose the problem of faithful planning in TODs that needs to resolve user intents by following predefined flows and preserving API dependencies. To solve this problem, we propose **FLAP**, a **Flow**-**A**dhering **P**lanning algorithm based on constrained decoding with lookahead heuristic for LLMs. Our algorithm alleviates the need for finetuning LLMs using domain specific (plan/dependency) data, enables quick adaptation to predefined flows, and outperforms other decoding and prompting-based baselines. Further, our algorithm empowers smaller LLMs (\(\approx 7\text{B}\)) to perform at par larger LLMs (\(\approx 30\text{B-40B}\)).

+
Footnote †: _Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers)_, pages 517–539 June 16-21, 2024 ©2024 Association for Computational Linguistics

## 1 Introduction

A successful task oriented dialog (TOD) often involves interaction with internal or external world through tools or APIs. For example, as shown in Fig. 1, booking a flight through a conversation may involve searching for flights, getting payment information, and so on. API usage is a difficult task for chatbot agents as it involves various reasoning and planning. Beyond predicting the correct API to use in a particular scenario, it requires task decomposition and resolution of inter-API dependencies (e.g., searching for flights requires extracting airport codes) Zhang et al. (2021); Prasad et al. (2023); Wang et al. (2023); Lu et al. (2023b). Moreover, there may exist customized workflows to follow for API usage, such as "confirm before placing order" or "recommend add-ons before checking out", which can conveniently be defined using natural language (NL) instructions.

Recent advances in LLMs have enabled researchers to consider LLMs for task-planning Huang et al. (2022); Valmeekam et al. (2023) and API usage Qin et al. (2023b); Patil et al. (2023). In planning, studies have primarily focused on open-ended scenarios and emphasized mostly on goal attainment Shridhar et al. (2020); Jansen (2020) without considering any instructions on the process Valmeekam et al. (2024), while works on API usage have mostly considered the appropriateness of API calls Wang et al. (2023). However, considerations about API dependencies, and the faithfulness of generated API calls to predefined workflows or instructions are understudied.

In this paper, we propose a novel problem of zero-shot faithful planning with LLMs in TODs. As input, we provide (1) the domain definition in terms of APIs and their descriptions, (2) NL flows to fulfill various intents in the domain, and (3) a user query. The task is to generate a plan that fulfills the user's query by using the domain-specific APIs while adhering to the specified flows and API dependencies (SS2).

Prior works attempted to teach LLMs to use

Figure 1: An agent-user interaction scenario. Agent plans how to resolve the user query by following given natural language workflows and API dependencies.



APIs by pretraining them with large amount of API related data (Schick et al., 2023; Yang et al., 2023; Qin et al., 2023b). However, in practice, API signatures and the NL flows to use them are customized to use cases and are prone to change (e.g., due to change in business policy). Hence, quickly adapting the LLMs to new flows and API dependencies is required. Unfortunately, pretraining LLMs cannot guarantee faithful planning with custom-defined APIs and flows which may significantly differ from the pretraining data (Blodgett et al., 2020; Bommasani et al., 2021); we experimentally verify this in SS4.1. To alleviate the issue, we propose **FLAP**, a **F**low **A**dhering **P**lanning algorithm that uses constrained decoding of LLMs with lookahead heuristic (Och et al., 2001; Haghighi et al., 2007; Lu et al., 2022a; Huang et al., 2024) (SS4.2). FLAP generates faithful plans to predefined flows and API dependencies, requires no pretraining, and is applicable on top of any autoregressive LLM regardless of its pretraining mechanism.

Given previous studies have shown that thoughts enhance the reasoning capabilities of LLMs (Wei et al., 2022) (even in the context of API usage (Yao et al., 2022)), FLAP prompts the LLMs to generate plans consisting of thoughts and corresponding APIs. FLAP aligns the generated thoughts to pre-defined NL flows, enabling thoughts to act as an intermediary (and address the vocabulary mismatch) between NL flows and APIs. In addition, FLAP captures the relations among NL flow steps and APIs using dynamic dependency graphs; these are used to score different dimensions related to API and flow dependency adherence in the lookahead heuristic function. Finally, FLAP also enforces structural constraints using lookahead heuristics.

To quantify the efficacy of current LLMs with promising prompt strategies, and our proposed approach for _faithful planning_, we construct a novel dataset comprising \(4\) domains, \(13\) intents and flows, \(64\) APIs, and \(260\) user queries, resembling real life use-cases (SS3). We find that existing open-source LLMs and prompting techniques lack faithful planning capabilities and our proposed constrained decoding based algorithm, FLAP, improves the faithfulness of plans by reducing various errors, such as API and flow step dependency violations, redundant/lacking API in generated plan, etc. We also demonstrate that smaller LLMs (7B) with FLAP can achieve comparable performance against much larger LLMs (\(40\)B) (SS5).

## 2 Task Formulation

In this section, we define the novel task of faithful planning in task oriented dialogues (TODs) to resolve queries by adhering to predefined natural language (NL) flows and API dependencies.

### TOD Agent Scenario

In a typical TOD system, an agent is equipped with the following elements.

**Domain:** An agent usually has access to a specific domain, \({\cal D}\) (e.g., trip planning, banking, and so on).

**Intents:** In a domain \({\cal D}\), an agent has capabilities to handle a set of intents, \({\cal I}\). For example, an agent form the domain "trip planner" may only be able to book a hotel, search flights, and so on, however, unlikely to be able to create a bank account.

**Flows:** Agents are often provided with a sequence of natural language instructions, defined as "flows", \({\cal F}\) (e.g., "recommend add-ons before checking out"). Agents are expected to follow these flows _step-by-step_ to resolve intents. Flows are typically defined by businesses and are prone to change.

**APIs:** Agents have access to a set of internal APIs, \({\cal A}\), to interact with the system (typically through GUI) and trigger actions based on the inputs from the user. For example, if a user is looking to book a car in Chicago, the agent may trigger the API, FindRentalCar(location="Chicago").

### Faithful Planning for Resolving Intent

In real life, when a user expresses an intent, an agent has to execute a sequence of APIs to fulfil it. Successful fulfillment of the user intent depends largely on the reasoning and planning capability of the agents because of the following challenges.

**Identifying and following NL flows:** Given an intent in \({\cal I}\), the execution steps need to be strictly consistent with a predefined flow in \({\cal F}\). Hence, the agents need to first identify the correct flow in \({\cal F}\) and then adhere to it.

**Task decomposition and dependency resolution:** APIs in \({\cal A}\) may have dependencies on other APIs in \({\cal A}\), and a flow step may require calling multiple APIs. For example, GetAirports(city="Chicago") is needed to be called before FindRight(airport="ORD"), to execute the flow step "suggest flights". Hence, the agents have to decompose flow steps and preserve API dependencies.

Grounded in such challenges, we define the novel task of faithful planning for TOD agents (Fig.

 

1). Given, a domain \({\cal D}\), a set of flows \({\cal F}\) (for resolving intents \({\cal I}\)), a set of APIs \({\cal A}\) (with parameters and descriptions), and a user query \({\cal Q}\); the task is to generate a plan \({\cal P}\) consisting of a sequence of APIs to fulfil the user query while obeying the flow steps in \({\cal F}\) and preserving API dependencies in \({\cal A}\).

In real world scenarios, agents may need to dynamically update the plan based on the situation (say, the user starts with query \({\cal Q}\) and then wants to do \({\cal Q^{\prime}}\) as well). As a first step towards faithful planning in TODs, we study static planning (user does not change mind) in this paper, and leave dynamic (run-time) planning as a future work.

### Zero-shot Planning with LLMs

We study the capabilities of LLMs in zero-shot planning by following predefined flows and preserving API dependencies in TODs. For that, we instruct a base LLM to act as a customer care agent and in context, we provide the domain definition consisting of APIs, their input-output parameters and descriptions, and NL flows for resolving intents in the domain. Given a user query in context, the LLM is instructed to plan how to resolve the query by using the given APIs and following the appropriate flow. The LLMs are expected to resolve API dependencies from their input-output parameters. The prompt structures are shown in SSB. We study the problem in the following settings by varying the nature and complexity of the task.

**All _vs._ relevant flow in-context:** We experiment by providing either all flows or only the flow related to the user query in context. The former setting is more complex as the LLM has to first identify the relevant flow and then follow it to plan.

**Reasoning and acting:** Recent studies have shown that step-by-step thinking and acting improves the end goal achievement Wei et al. (2022); Yao et al. (2022). Inspired by these works, we study the problem of plan generation in a setting where the LLMs are prompted to generate a thought and then an API in the plan to help it rationalize its selection of the API.

### Evaluation Metrics

#### Number of edits to fix the plan:

A generated plan is required to contain _only_ the steps and APIs from the gold plan. We define number of edits as the sum of additions and deletions of steps/APIs in the generated plan to make it contain the same steps/APIs as the gold plan. The lower the number of edits, the better the generated plan.

**Inconsistent steps/APIs:** A generated plan is required to obey flow steps and API dependencies. Hence, we measure the percentage of steps or APIs in the generated plan that do not follow the dependency structure (e.g., a step/API is generated before its parent steps/APIs).

**Fine-grained metrics:** We quantify other fine-grained metrics such as API hallucination, API repetition and parsability of the generated plans.

## 3 Data Collection

In this section, we describe the data collection procedure for studying faithful plan generation.

**Domain Creation.** Previous datasets have developed domains, flows, and APIs for tackling intents in TODs. We find, none of the existing datasets is usable as it is in our problem. For example, the STAR Mosig et al. (2020) and STARv2 Zhao et al. (2023) datasets contain APIs (without flows) to resolve user queries, however, dependencies among the APIs are sparse, e.g., STAR contains only \(10\) dependencies among \(25\) APIs. In real life, API dependencies are more dense, hence, these datasets are not very useful for our study where the goal is to study the ability of LLMs to preserve dependencies. The ABCD dataset Chen et al. (2021) contains flows for a few tasks, however, the related APIs needed to perform the steps in the flows are not given, hence, it is also not suitable for our study. Hence, inspired from these two datasets, we create in total, \(4\) domains, \(13\) flows (similar to ABCD) for resolving \(13\) distinct intents in these domains, and \(64\) APIs (similar to STAR) to execute different flow steps. We also annotate the gold plans to

\begin{table}
\begin{tabular}{l|c|c|c|c} \hline \multirow{2}{*}{**Domains:**} & \multicolumn{1}{c|}{**Trip**} & \multirow{2}{*}{**Nonactive**} & \multirow{2}{*}{**Ranking**} & **Restaurant** \\  & & & & **\& Ride Book** \\ \hline \# of intends/flows & 3 & 3 & 3 & 4 \\ \# of steps per inter/flow & 5-6 & 4-5 & 3-5 & 4 \\ \# of APIs model per step & 1-2 & 1-2 & 1-3 & 1-4 \\ \# of APIs in domain & 13 & 15 & 14 & 22 \\ \# of relationships among APIs & 13 & 13 & 15 & 19 \\ \# of user queries & 60 & 60 & 60 & 80 \\ \hline \end{tabular} 
\begin{tabular}{l l} \hline \multirow{2}{*}{**Intects in all domains**} & \multirow{2}{*}{**Interactions in all domains**} & \multirow{2}{*}{**Randomization**} & **Roadernat** \\ block can be seen here by users, named + open access, target graph + box network, host node, \\ ul. batch depth & reverse, add member + lens, created processing & accsed customers, accsed traffic \\ \hline \end{tabular}
\end{table}
Table 1: Test data statistics (a) and example flow (b).

 

resolve the \(13\) intents. The statistics of the dataset and examples are presented in Tab. 1.

**Test Query Generation.** We aim to study plan generation in response to a user query in our created domains. To collect diverse user utterances related to the intents in Tab. 0(a), we utilize the generative power of pretrained LLMs. We prompt an open-source LLM, Falcon-4UB-instructAlmazrouei et al. (2023)1, to generate user queries related to the intents. We prompt the LLM to write paraphrases of a query with a specific intent. Prompt structure and generated examples are shown in SSA.2. We manually go through the generated queries and discard any queries that are not related to the intent in the prompt. We find that \(94\%\) of the generated queries are correct. In this manner, we generate \(20\) user queries per intent resulting in \(260\) queries in total. We use this dataset for studying flow-constrained plan generation given a user query.

Footnote 1: We experimented with other open source LLMs and found Falcon to be the best in generating diverse utterances.

## 4 Constrained Decoding for Planning

In this section, first, we run an ablation on plan generation in various settings as explained in SS2.3, to understand the complexity of the task. Then we explain our proposed constrained decoding based algorithm for faithful plan generation.

### Ablation

To understand the complexity of zero-shot faithful planning in different settings, we run a zero-shot prompting-based ablation with a number of open-source LLMs pretrained with/without tools and of different parameter size. We use SantacoderAllal et al. (2023), FalconPenedo et al. (2023), MptTeam (2023), Mistral-v8.1Jiang et al. (2023), KoalaGeng et al. (2023), VicunaChiang et al. (2023), LlamaTouvron et al. (2023), ToolAlpacaTang et al. (2023), and ToollLaMAQin et al. (2023b). The prompt structures for generating plans with and without thoughts are shown in SSB. The results for plan generation in the two settings are summarized in Tab. 2. Our main observation is that all LLMs fail heavily in zero-shot planning regardless of the setting, especially LLMs fail to preserve the API and flow step dependencies in plans. For example, the average percentage of inconsistent APIs in the two settings, ranges from \(22.6\%\) to \(49.6\%\). This finding is in line with previous studies on LLMs' capability in task planning and tool usageRuan et al. (2023); Valmeekam et al. (2022). We categorize our main observations below.

**Effect of thoughts.** Performance improves in flow faithfulness for almost all LLMs when prompted to generate thoughts and APIs compared to when prompted to generate only APIs in plan. This is consistent with the findings of Yao et al. (2022). However, with thoughts, API repetition and halluci

\begin{table}
\begin{tabular}{l l|c c c c||c c c|c c c} \hline \hline  & & \multicolumn{5}{c||}{**All flows in context**} & \multicolumn{5}{c}{**Only relevant flow in context**} \\ \cline{3-17}  & **With** & **Avg \% of** & **Avg \# of edit for** & **Avg \% of inconsistent** & **Avg \% of** & **Avg \# of edit for** & **Avg \% of inconsistent** \\
**LLMs** & **Thought** & **API Hallmo.** & **Steps (s)** & **APIs (s)** & **APIs (s)** & **API Hallmo.** & **Steps (s)** & **APS** (s)** & **APS** (s)** & **APS (s)** \\ \hline santacer-1.1b & No & 1.2\(\times\) & 30.3 & 25.9 & 15.4\% & 33.1\% & 1.8\% & 29.6 & 25.8 & 15.3\% & 34.1\% \\  & Yes & \(13.0\%\) & 11.8 & 13.0 & 18.2\% & 34.1\% & 20.0\% & 9.5 & 11.5 & 19.5\% & 31.2\% \\ \hline \multirow{2}{*}{toolAplace-7b} & No & \(10.8\%\) & 4.6 & \(5.1\) & \(20.4\%\) & 49.6\% & 7.0\% & 4.4 & 4.7 & 17.8\% & 49.2\% \\  & Yes & \(19.0\%\) & 4.7 & 5.3 & 17.6\% & 44.8\% & 8.0\% & 3.0 & 3.5 & 10.6\% & 47.6\% \\ \hline \multirow{2}{*}{falcon-7b-inst.} & No & \(6.1\%\) & 47.5 & 40.3 & 16.6\% & 27.7\% & 6.6\% & 40.8 & 38.1 & 17.4\% & 23.9\% \\  & Yes & \(19.0\%\) & 9.7 & 11.4 & 24.0\% & 38.8 & 9.0\% & 9.7 & 9.2 & 21.5\% & 37.9\% \\ \hline \multirow{2}{*}{mpt-7b-inst.} & No & \(0.2\%\) & 16.2 & 13.4 & 14.7\% & 33.3\% & 0.2\% & 13.6 & 11.5 & 13.5\% & 35.6\% \\  & Yes & \(4.0\%\) & 13.9 & 12.4 & 11.9\% & 29.6\% & 3.0\% & 9.9 & 8.9 & 9.1\% & 33.4\% \\ \hline \multirow{2}{*}{nistral-7b-inst.} & No & \(2.4\%\) & 2.9 & 3.0 & 10.6\% & 37.2\% & 1.4\% & 3.1 & 3.1 & 8.0\% & 30.6\% \\  & Yes & \(0\%\) & 2.8 & 2.8 & 3.1\% & 40.3\% & 1.0\% & 2.7 & 2.6 & 2.8\% & 34.4\% \\ \hline \multirow{2}{*}{kola=13b} & No & \(6.3\%\) & 37.6 & 31.4 & 10.5\% & 24.0\% & 4.2\% & 29.4 & 23.8 & 11.3\% & 26.2\% \\  & Yes & \(10.0\%\) & 8.8 & 9.1 & 9.0\% & 24.0\% & 9.0\% & 7.4 & 7.9 & 9.3\% & 22.6\% \\ \hline \multirow{2}{*}{vicuna-13b} & No & \(2.4\%\) & 4.1 & 4.3 & 15.1\% & 35.2\% & 2.4\% & 3.9 & 4.1 & 14.2\% & 31.7\% \\  & Yes & \(4.0\%\) & 4.0 & 4.3 & 6.7\% & 35.5\% & 2.0\% & 3.1 & 3.4 & 5.4\% & 35.0\% \\ \hline \multirow{2}{*}{llana-13b} & No & \(2.4\%\) & 26.8 & 20.9 & 13.2\% & 31.7\% & 2.0\% & 24.7 & 19.1 & 11.3\% & 29.4\% \\  & Yes & \(4.0\%\) & 9.2 & 8.7 & 4.7\% & 31.0\% & 3.0\% & 6.6 & 5.6 & 4.0\% & 38.4\% \\ \hline \multirow{2}{*}{mpt-3bb-chst} & No & \(0.5\%\) & 3.3 & 3.4 & 9.1\% & 33.4\% & 0.2\% & 3.5 & 3.3 & 6.3\% & 33.3\% \\  & Yes & \(1.0\%\) & 2.7 & 2.7 & 5.7\% & 32.5\% & 1.0\% & 2.1 & 2.1 & 4.6\% & 34.3\% \\ \hline \multirow{2}{*}{falcon-4db-inst.} & No & \(0.9\%\) & 8.1 & 7.2 & 18.0\% & 38.3\% & 0.9\% & 9.0 & 7.9 & 17.9\% & 35.5\% \\  & Yes & \(5.0\%\) & 5.4 & 5.4 & 9.0\% & 36.4\% & 7.0\% & 4.0 & 4.1 & 8.7\% & 25.7\% \\ \hline \end{tabular}
\end{table}
Table 2: Zero-shot plan generation results using prompting-based approaches with Greedy decoding (average over all plans). Results with other metrics and standard deviation can be found in §D.

 

nation increase and plan parsability (SSD) decreases a bit. On some test examples, we notice that the LLMs enter into a repetitive loop with thoughts (examples in SSE.1), in line with well-known phenomenon in greedy and beam-search decoding Vijayakumar et al. (2016); Shao et al. (2017).

**Effect of parameter size.** Performance in most of the metrics improve in larger LLMs and smaller LLMs are prone to hallucination. We observe that different models of the same parameter size perform differently. For example, the 7B version of MPT is better than the 7B version of Falcon in most of the metrics and Mistral-7B is better than other 7B models and it performs as per 30B-40B models.

**Effect of pretraining.** We observe that LLMs pretrained on tools, e.g., ToolAlpaca fails heavily in API consistency and results in high API hallucination. With ToolLlama, we find that it fails to follow instructions and generates non-parsable responses (hence, not reported). These findings suggest that pretraining LLMs with tool data makes them heavily biased towards the pretraining data.

**All flows_vs._ relevant flow in prompt.** When only the relevant flow to the user query is given in-context, the performance improves with almost all LLMs indicating that LLMs are more confused when multiple flows are given in-context.

### Proposed Algorithm

We now introduce **FLAP**, a **F**low **A**dhering Planning algorithm based on constrained decoding with lookahead heuristic (Fig. 2). In FLAP, we first convert the flow steps and APIs to dependency graphs. At decoding time, we keep track of these dependency graphs to enforce the dependencies as constraints during next token generation.

#### 4.2.1 Constructing Dependency Graphs

For each domain, we construct dependency graphs for flow steps and APIs. The API dependency graph is constructed based on the input-output parameter dependencies among APIs. For example the APIFindRight(inputs:airport; output-s:flights) is dependent on the API GetAirport (inputs:city; outputs:airport) for the input parameter 'airport'. Hence, in the API dependency graph 'GetAirport' will be a parent of 'Findflight'. Similarly for flow steps, we construct a separate flow-step dependency graph. While generating a plan, we maintain a list of the steps and APIs that are executed in these dependency graphs. At each planning step, only non-executed/under-execution APIs/steps in the dependency graphs with parents in the executed list are considered permitted.

#### 4.2.2 Constrained Decoding of Plans with Lookahead Heuristic

The traditional text generation problem can be formulated as, given an input sequence of tokens \(x\) = \(\{y_{1},y_{2},\cdots,y_{t-1}\}\), the next token generator predicts the next token \(y_{t}\) by solving the equation:

\[y_{t}=\operatorname*{arg\,max}_{y^{\prime}_{t}\in Y}P(y^{\prime}_{t}|x)\]

Here, \(Y\) is the set of all possible generations and \(P(y^{\prime}_{t}|x)\) is the likelihood of the token \(y^{\prime}_{t}\) given the

Figure 2: An example state of FLAP, our proposed lookahead heuristic-based constrained decoding algorithm for faithful planning, in the domain “Trip Planning” for a query related to “book flight”. Here, \(\uparrow\), \(\downarrow\), and \(\leftrightarrow\) indicate high, low, and mediocre alignment scores, respectively. The selected path based on the heuristic scores is highlighted.

 

input sequence \(x\). In constrained decoding, the likelihood function takes the form \(P^{\prime}(y_{t}^{\prime}|x)=P(y_{t}^{\prime}|x)\) + \(H(y_{t}^{\prime}|x)\), where \(H(y_{t}^{\prime}|x)\) is the score based on constraint satisfaction by generating \(y_{t}^{\prime}\). However, often it is difficult to determine the constraint satisfaction score by only looking at the next token. The lookahead mechanism comes handy here where the constraint satisfaction score for a candidate next token, \(y_{t}^{\prime}\) is estimated by looking at the estimated future generations if \(y_{t}^{\prime}\) is selected. Hence, instead of \(H(y_{t}^{\prime}|x)\), the heuristic function takes the form \(H(y_{t}^{\prime},\cdots,y_{t+L}^{\prime}|x)\), where \(L\) is lookahead length.

For constrained plan generation, we prompt LLMs to generate thoughts and corresponding APIs. During generation, for each candidate next token, we lookahead till the end of one thought+API and score the candidate based on their constraint satisfaction. Hence, our heuristic function takes the form \(H_{c}=H([thought][api]|x)\). \([thought][api]\) may contain some already generated tokens; we also consider those when scoring. We score each \([thought][api]\) in the aspects below.

Generated thought to permitted flow step alignment (\(H_{th:step}\)).In our approach, thoughts are natural language sentences describing the next step to be taken. Ideally, they should correspond to the flow steps. Hence, we score the generated thoughts based on their semantic similarity to the permitted flow steps in the flow dependency graph. A flow step is permitted, if all of its parent flow steps are already executed or the flow step is under execution (e.g., flow step consisting of multiple API calls). First, we estimate the intended flow step, \(\hat{s}\) by a thought, \(th\) by obtaining its semantically closest flow step using the formula below.

\[\hat{s}=\operatorname*{arg\,max}_{s\in S}sim(th,s)\]

Here, \(S\) is the set of all flow steps in the domain and \(sim()\) is semantic similarity scorer. Then we calculate \(H_{th:step}\) using the following equations.

\[H_{th:step} =\begin{cases}\alpha\times sim(th,\hat{s}),&\text{if $\hat{s}$ in $S_{p}$}\\ 0,&\text{if $\hat{s}$ in $S_{n}$}\end{cases}\] \[\alpha =\begin{cases}\alpha_{a},&\text{if $\hat{s}$ is in a flow}\\ \alpha_{b},&\text{if $\hat{s}$ is deviating from flow}\\ \alpha_{c},&\text{if $\hat{s}$ is same as the last step}\end{cases}\]

Here, \(S_{p}\) and \(S_{n}\) are permitted and not permitted flows steps at the current thought, respectively. We assign different weights to \(H_{th:step}\) based on the intended flow step \(\hat{s}\). If the intended flow step \(\hat{s}\) belongs to the flow that the model has been following so far, we assign the weight \(\alpha_{a}\). If the model deviates from the flow it has been following, we assign the weight \(\alpha_{b}\). In single intent use-cases, the model is required to follow only one flow, hence, setting \(\alpha_{a}>\alpha_{b}\) prevents the model from deviating from the flow it has been following. If the model generates a flow step that is the same as the step it intended in its previous thought, we assign a weight \(\alpha_{c}\). One step may involve executing multiple APIs. Hence, setting \(\alpha_{c}>\alpha_{a}\) encourages the model to complete the current step before moving to the next.

Generated API to permitted APIs alignment (\(H_{api:api}\)).We score the generated API based on their alignment to the permitted APIs. Permitted APIs at a step are the non-executed APIs whose parents are already executed. We calculate \(H_{api:api}\) using the following equations.

\[\hat{a} =\operatorname*{arg\,max}_{a\in A}sim(a,\overline{a})\] \[H_{api:api} =\beta\times sim(\overline{a},\hat{a})\] \[\beta =\begin{cases}\texttt{[0,1)},&\text{if $\hat{a}\notin A_{p} \cup A_{c}$ or $\overline{a}\notin A$}\\ 1,&\text{if $\hat{a}\in A_{p}$}\\ 0,&\text{if $\hat{a}\in A_{c}$}\end{cases}\]

Here, \(\overline{a}\) is the generated API, \(A\) is the set of all APIs in a domain, \(A_{p}\) and \(A_{c}\) are permitted APIs and already called APIs, respectively, and \(A_{p}\cap A_{c}=\emptyset\). When APIs from none of the sets \(A_{p}\) and \(A_{c}\) are generated, the API is either hallucinated or generated from the non-permitted list (\(A_{n}\)). In both of the cases, we assign a lower weight than 1 in the alignment score to make it a soft constraint while making it zero will enforce a hard constraint against such cases. We keep this weight open to tuning for different LLMs because of their varying degree of hallucinations and errors.

Generated thought to user intent alignment (\(H_{th:in}\)).The plan should correspond to the user intent in the query. Hence, we score the generated thought, \(th\) based on its semantic similarity with the user query, \(in\), using, \(H_{th:in}=sim(th,in)\).

Generated thought to generated API alignment (\(H_{th:\overline{api}}\)).We generate thought to resonate the API usage. Hence, the generated API, \(\overline{a}\) should be relevant to the generated thought, \(th\). We align the generated thought to the generated API using the score, \(H_{th:\overline{api}}=sim(th,\overline{a}_{d})\), where \(\overline{a}_{d}\) is the given 

description of the generated API, \(\overline{a}\). When \(\overline{a}\) is a hallucination, we set \(\overline{a}_{d}=\overline{a}\).

Structural constraint (\(F_{st}\)).For practical use-cases, it is convenient if the LLMs generate plans in a pre-defined format. It helps automatic parsability of the generated plans. Hence, we introduce a function, \(F_{st}\) that accounts for the structural consistency of the generated plan. It operates on the combined heuristic score, \(H^{\prime}_{c}\) = \(a\times H_{th:step}\) + \(b\times H_{api:\overline{api}}\) + \(c\times H_{th:in}\) + \(d\times H_{th:\overline{api}}\) (\(a\), \(b\), \(c\), \(d\) are scaling factors) as follows.

\[F_{st}(H^{\prime}_{c})=\begin{cases}H^{\prime}_{c},&\text{if thought+API follows format}\\ 0,&\text{otherwise}\end{cases}\]

We use \(H_{c}\) = \(F_{st}(H^{\prime}_{c})\) as the final heuristic score for a candidate token, \(y^{\prime}_{t}\) and the final scoring function for next token selection is defined as follows.

\[S(y^{\prime}_{t}|x)=(1-\lambda)\times P(y^{\prime}_{t}|x)+\lambda\times H^{y^{ \prime}_{t}}_{c}\]

The parameter \(\lambda\) controls how much weight is given in the LLM logits, \(P(y^{\prime}_{t}|x)\), and the heuristic score \(H^{y^{\prime}_{t}}_{c}\). We use top-\(k\) beams based on \(P(y^{\prime}_{t})\) for the next token generation using constrained decoding. An example of our approach is shown in Fig. 2.

## 5 Experimental Evaluation

### Experimental Setting

We evaluate FLAPs performance in faithful planning on the dataset described in SS3, by comparing it with prompting-based baselines and different decoding methods. Note that, the approach with thoughts is similar to ReAct (Yao et al., 2022). Other approaches either depend on multiple modules, require finetuning, or tuned for a different task setting, hence, not directly applicable in our task. Based on our ablation (in Tab. 2), we apply FLAP on two LLMs in 7B parameter range, Mistral-7b-instruct and Mpt-7b-instruct. We leave application of FLAP on larger LLMs as future work because of their high resource requirement. We demonstrate the structure of plan, ([thoughtt]...[API]...), through dummy start steps in the prompts (Fig. 6, SSB) and enforce it through \(F_{st}\) during decoding. We use a pretrained sentence transformer (Reimers and Gurevych, 2019) to calculate semantic similarities (\(sim()\)). We report hyper-parameter tuning and other experimental details in SSC.

\begin{table}
\begin{tabular}{l l|l c c c|c c c c|c c}  & & **With** & **Avg count per plan** & **\% Plan** & **Avg * d / APIs in plan** & **Avg * e / APIs in plan** & **Avg * f / APIs in plan** & **Avg * f / APIs in plan** & **Avg * f / APIs in plan** & **Avg * f / APIs** (i) \\ \hline \multirow{4}{*}{**LLM**} & **Decoding Strategy** & **Through** & **Threshed Targets** & **Arts** & **Proactive** & **Program** & **Avg * d / APIs in plan** & **Avg * g / APIs in plan** & **Avg * f / APIs in plan** & **Avg * f / APIs** (i) & **Apfs * f / APIs** (j) \\ \cline{2-11}  & **Geredy Scath** & No. & 17.3 & 108V & 11.68 & 0.22 & 16.2 & 12.4 & 14.75 & 33.3\% \\  & **Geredy Scath** & Yes & 16.2 & 16.1 & 97.57 & 38.14 & 4.06 & 11.9 & 12.4 & 11.95 & 29.6\% \\  & **Bon Search** & Yes & 9.9 & 9.9 & 10.87 & 5.55 & 3.00 & 7.6 & 7.7 & 13.9 & 41.0\% \\  & **Nucleic Sampling** & Yes & 12.3 & 12.4 & 76.95 & 25.35 & 4.00 & 3.8 & 9.5 & 13.85 & 31.7\% \\ \cline{2-11}  & **BLAP1 fold up at single thought to (eq., item)** & 10.3 & 10.4 & 100V & 1.35 & 0.00 & 5.5 & 5.5 & 6.30 & 1.3\% \\  & **TAR-2 label soft up at single thought to (eq., appl)** & Yes & 9.3 & 9.4 & 100V & 3.00 & 4.00 & 5.7 & 5.6 & 4.34 & 1.6\% \\  & **FLAP1-3** joint step along thought to (eq., numgt)** & Yes & 9.2 & 9.4 & 100V & 1.6 & 0.00 & 5.0 & 5.0 & 6.13 & 1.3\% \\  & **FLAP1-3** joint step at split might brought to (eq., numgt)** & Yes & 9.3 & 9.4 & 100V & 3.6 & 0.00 & 5.0 & 5.0 & 6.13 & 1.3\% \\  & **FLAP1-3** joint step at split might brought to (eq., numgt)** & Yes & 9.3 & 9.4 & 100V & 3.6 & 0.00 & 5.5 & 5.4 & 4.8\% & 2.1\% \\ \cline{2-11}  & **Greedy Scath** & No & - & 4.8 & 100V & 0.35 & 2.4 & 2.9 & 3.0 & 10.65 & 32.7\% \\  & **Geedy Scath** & No & 4.7 & 4.8 & 90.6 & 2.9 & 4.00 & 2.8 & 2.8 & 13.5 & 40.8\% \\  & **Bonnet Search** & Yes & 3.6 & 3.8 & 98.1\% & 0.27 & 0.00 & 2.9 & 3.0 & 2.4\% & 38.9\% \\  & **Norde Sampling** & Yes & 5.7 & 5.9 & 93.6\% & 5.8 & 2.00 & 3.0 & 3.0 & 7.55 & 34.3\% \\ \cline{2-11}  & **Tra-1** part of all eligible thought to (eq., item)** & Yes & 6.6 & 7.1 & 100V & 0.00 & 0.05 & 2.5 & 2.5 & 6.3\% & 4.6\% \\  & **PLAP1-3** joint step at slightly outside to (eq., appl)** & Yes & 6.1 & 6.1 & 100V & 0.15 & 0.00 & 2.3 & 2.3 & 5.1\% & 5.5\% \\  & **PRAP1-3** joint step at single thought to (eq., appl)** & Yes & 6.3 & 6.4 & 100V & 0.04 & 0.00 & 2.0 & 2.0 & 6.2\% & 3.6\% \\  & **RLAP1-3** joint step at single thought to (eq., appl)** & Yes & 6.4 & 6.4 & 100V & 0.55 & 0.00 & 2.5 & 2.5 & 5.6\% & 7.5\% \\ \hline \end{tabular}
\end{table}
Table 3: Comparison of our proposed approach, FLAP, with baselines in zero-shot faithful plan generation. Here, the numbers (FLAP.#) indicate different ablation versions of FLAP. Structural constraint is applied in all versions of FLAP. Results with standard deviations can be found in Tables 13 and 14 in SSD. As reference for the “Avg count per plan” column, we note, there are \(6.84\) APIs per gold plan.

 



### Experimental Results

The experimental results are summarized in Table 3. First, we observe that the Beam Search and Nucleus Sampling Holtzman et al. (2019) methods do not improve the overall performance compared to the Greedy Search method. Rather both Beam Search and Nucleus Sampling result in fewer number of parsable plans compared to Greedy Search. Note that Beam Search and Nucleus Sampling explore a wider search space compared to Greedy Search, and Nucleus Sampling increases diversity; we observe that without constraints, it results in more deviation from the instructions and often decreases performance in different metrics. Hence, in the rest of this section, we compare our models performance with "Greedy Search with thoughts". We also observe that different baselines and our approach show similar comparative trends in the two settings (all flows in context vs. relevant flow in context). Hence, we focus our rest of the discussion on the more difficult setting where all flows are given in the prompt (Tab. 3a).

In case of Mpt, different versions of our algorithm, FLAP, outperform the baselines by a great margin. In case of Mistral, the average percentage of inconsistent steps increases a bit (by \(2\text{-}4\%\)) with FLAP, however, performance improves in all other metrics, especially the inconsistent APIs decreases from \(40.3\%\) to \(3.6\%\). A similar trend is observed in the relevant flow in-context setting (Tab. 3b).

We observe that Mistral is good at following flows even without FLAP in the difficult all flows in-context setting, however, it fails to decompose when one flow step requires multiple API calls. Hence, without FLAP, the average API count per plan is lower (\(4.8\)) than the average number of APIs in gold plans (\(6.84\)). As a result, the avg. % of inconsistent steps is low (\(3.1\)%), however, % of inconsistent API is very high (\(40.3\%\)). With FLAP, the decomposition capability improves and the model generates plans with number of APIs (\(6.1\text{-}6.7\)) closer to the gold reference value (\(6.84\)) with fewer number of edits and much less inconsistent APIs (\(3.6\text{-}7.7\%\)), however, with a trade off of slightly increased (by \(2\text{-}4\)%) inconsistent steps. We present qualitative examples in Tab. 4.

We perform ablation by dropping different components of the heuristic scoring function in FLAP. We observe, different components of the heuristic score affect the corresponding aspects they are designed for, especially in the difficult setting where all flows are present in-context. For example, in case of Mpt, dropping \(H_{th:step}\) (thought to flow step alignment) results in higher step dependency violation (FLAP.1). Another component \(H_{th:in}\) (thought to intent alignment) is designed to keep the model focused on the flow related to the user intent and we observe an overall lower number of edits required when this component is added (FLAP.1, FLAP.3, FLAP.4), implying this component helps the model generate a plan related to the user intent. However, the effect of these components is not clear in case of Mistral. We conjecture, as Mistral is already good at following flows, the effect is lower. The effect of \(H_{th:\overline{api}}\) that makes generated thoughts and APIs coherent, is more qualitative and we find that dropping these component (FLAP.3) results in inconsistent thought and API pairs even if the generated API is correct in the plan. Examples of such errors are presented in Tab. 5. Finally, enforcing structural consistency (\(F_{st}\)) during decoding makes the plans parsable \(100\%\) of the time.

In Fig. 3, we present the comparative performance of LLMs of different size and smaller 7B LLMs with FLAP. We observe that FLAP on top of Mistral-7b outperforms and Mpt-7b performs at par larger LLMs.

## 6 Related Work

While API usage and planning have been studied as separate concepts in literature, we argue that these two are interdependent. Recently, different approaches have been proposed to teach LLMs to use the correct API, however, the faithfulness of API usage to their dependencies is understudied Ruan et al. (2023). For correct API usage, one line of research considers pretraining of LLMs with large API data that are mostly augmented using self-instruction Schick et al. (2023); Patil et al. (2023); Qin et al. (2023); Yang et al. (2023); Gao et al. (2023); Tang et al. (2023); Li et al. (2023). In some cases, authors considered modular approaches and

\begin{table}
\begin{tabular}{l l}  & Query: Can be used, Split Before Data Exchange in Sea Evaluation. \\ \hline
**Generalized Data Release Project:** & **Project-1/2** \\ \end{tabular}
\end{table}
Table 4: Generated plans by Mistral. It often fails in step decomposition without FLAP.



relied on feedback from humans, other LLMs, or API execution outputs to tune LLMs Song et al. (2023); Qin et al. (2023); Qiao et al. (2023); Lu et al. (2023); Prasad et al. (2023); Chen et al. (2023). Another line of work focused on improving the reasoning mechanism Qian et al. (2023) for API usage via prompting to generate thoughts Yao et al. (2022), consider API documentation Hsieh et al. (2023), etc. Unfortunately, pretraining-based methods are difficult and prohibitively costly when adaptation to new APIs is required and prompting-based approaches cannot reliably improve reasoning and planning which are key to faithful API usage (SS4.1).

In literature, planning is defined as decomposition of the end task and identifying intermediate executable actions or reasoning steps to achieve the goal where the decomposition is open-ended and does not require obeying instructions Jansen (2020); Huang et al. (2022); Ahn et al. (2022); Wu et al. (2022); Lu et al. (2022); Yu et al. (2023); Brahman et al. (2023). However, in many applications, such as customer care service or physical robots, planning needs to strictly adhere to predefined instructions Reijers (2003); Sengupta et al. (2019). To achieve this, we draw inspiration from decoding approaches Holtzman et al. (2019); Meister et al. (2020) that enable enforcement of lexical and parametric constraints during generation Hokamp and Liu (2017); Lu et al. (2021); Krause et al. (2021); Yang and Klein (2021); Huang et al. (2024). We extend the idea to enforce constraints related to predefined instructions.

## 7 Conclusion

In this paper, we study the novel problem of faithful planning for performing tasks in TODs, that adheres to predefined flows and preserves API dependencies. To solve the task, we propose **FLAP**, a constrained decoding algorithm based on lookahead heuristic. FLAP outperforms prompting-based baselines with different decoding methods and applying FLAP on two smaller LLMs we achieve comparable performance to larger ones. Our approach can be leveraged in accurate planning for various downstream applications such as next step prediction in conversations, user simulation, flow-following conversation data augmentation for training TOD models, and so on.

Figure 3: Comparison among different LLMs in flow step (top) and API (bottom) errors during plan generation in the setting where all flows are given in context.

\begin{table}

\end{table}
Table 5: Qualitative error analysis when the generated thought to generated API alignment component (\(H_{th:api}\)) in the heuristic scoring function of FLAP is dropped. The models results in incoherent thoughts and APIs at steps if \(H_{th:api}\) is dropped; such steps are highlighted in red.

 



### Limitations

We identify the following limitations of our study.

Simplifying Assumptions:In our study, we make the simplifying assumption that the user query can be resolved using the given APIs and workflows in context. However, that may not be the case in real life and users may have out-of-domain (OOD) queries. To tackle OOD intents, we can easily adapt our model by adding one additional workflow such as "cannot help", and instruct it to be selected if no other workflow is similar to the user query. Our constructed dataset also do not contain cases where multiple plans are valid based on the given workflows and APIs, however, our algorithm can be applicable in such cases without any modification.

Static Planning:In this paper, we study static planning. However, in real life, the agents may need to modify their plans in between a conversation because conversations are by nature dynamic. Some possible reasons for plan adaptation may be, based on API execution feedback, user changes their mind in between, and so on. Adapting such static plans based on the situation in dynamic conversations can be an interesting future work. We also focus only on API selection by following predefined flows and API dependencies. The correctness of parameter filling for the APIs is left as a future work as it is applicable mostly in the dynamic planning setting.

Runtime:We observe a higher runtime for plan generation using constrained decoding (SSC.3). This is partly because of the inefficient deployment of the LLMs in the existing implementations (we use Huggingface implementations) and partly because of our resource constraint (discussed in details in SSC.3). We intend to work on more efficient implementation of our algorithm by potentially parallelizing computations and implementing algorithmic optimizations (e.g., lookahead after every \(n\) tokens generation).

Usage of Open Source LLMs:We did not use any closed source LLMs for our research because of various reasons. Firstly, our algorithm requires the logits from the LLMs which may be difficult to obtain using closed source LLMs. Secondly, they might be changing inside the box and it will be difficult to replicate the results, hence, harming the benchmarking procedure. Thirdly, closed source LLMs are costly as they are accessible through paid APIs only. Due to resource constraint, we experimented with LLMs of at most 40B parameter size. However, as described in the paper, our approach can be applied on top of any autoregressive LLM.

## Ethics Statement

In this paper, we present all implementation and dataset details to replicate the study (partially in the Appendix). All the relevant datasets, publicly available LLMs, and other models used in this paper are publicly available for scientific research and are cited adequately. All the results are reported with standard deviations and necessary error analyses are done (some parts in the Appendix), to provide the readers an idea about the potential error patterns and risks related to using our proposed models.

## Acknowledgements

We gratefully acknowledge James Gung, Yi-An Lai, Nikolaos Pappas, and the members of the AWS AI Labs for providing valuable feedback at different stages of this work. We extend our special thanks to Justin Sun for his help in setting up different LLMs and Don Bennett for providing valuable feedback on writing. We would also like to thank the anonymous reviewers for their insightful comments.

## References

* Ahn et al. [2022] Michael Ahn, Anthony Brohan, Noah Brown, Yevgen Chebotar, Omar Cortes, Byron David, Chelsea Finn, Chuyuan Fu, Keerthana Gopalakrishnan, Karol Hausman, et al. 2022. Do as i can, not as i say: Grounding language in robotic affordances. _arXiv preprint arXiv:2204.01691_.
* Allan et al. [2023] Loubna Ben Allan, Raymond Li, Denis Kocetkov, Chenghao Mou, Christopher Akiki, Carlos Munoz Ferrandis, Niklas Muennighoff, Mayank Mishra, Alex Gu, Manan Dey, et al. 2023. Santacoder: don't reach for the stars! _arXiv preprint arXiv:2301.03988_.
* Almazrouei et al. [2023] Ebtesam Almazrouei, Hamza Alobeidli, Abdulaziz Albashamis, Alessandro Cappelli, Ruxandra Cojocaru, Merouane Debbau, Etinne Geffinet, Daniel Heslow, Julien Launay, Quentin Malaric, Badreddine Noune, Baptiste Pannier, and Guilherme Penedo. 2023. Falcon-40B: an open large language model with state-of-the-art performance.
* Blodgett et al. [2020] Su Lin Blodgett, Solon Barocas, Hal Daume III, and Hanna Wallach. 2020. Language (technology) is

power: A critical survey of "bias" in NLP. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, Online, pp. 5454-5476. External Links: Link Cited by: SS1.
* R. Bommasani, D. A. Hudson, E. Adeli, R. Altman, S. Arora, S. von Arx, M. S Bernstein, J. Bohg, A. Bosselut, E. Brunskill, et al.2021. On the opportunities and risks of foundation models. arXiv preprint arXiv:2108.07258. Cited by: SS3.1.
* F. Brahman, C. Bhagavatula, V. Pattakin, J. D. Hwang, X. Li, H. J. Arai, S. Sanyal, K. Sakaguchi, X. Ren, and Y. Choi (2023)Plasma: making small language models better procedural knowledge models for (counterfactual) planning. arXiv preprint arXiv:2305.19472. Cited by: SS1.
* D. Chen, H. Chen, Y. Yang, A. Lin, and Z. Yu (2021)Action-based conversations dataset: a corpus for building more in-depth task-oriented dialogue systems. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pp. 3002-3017. Cited by: SS1.
* Z. Chen, K. Zhou, B. Zhang, Z. Gong, W. Xin Zhao, and J. Wen (2023)Chatcot: tool-augmented chain-of-thought reasoning on\'chat-based large language models. arXiv preprint arXiv:2305.14323. Cited by: SS1.
* W. Chiang, Z. Li, Z. Lin, Y. Sheng, Z. Wu, H. Zhang, L. Zheng, S. Zhuang, Y. Zhuang, J. G. Boisca, and E. P. Xing (2023)Vicuna: an open-source chatbot impressing gpt-4 with 90%* chatgtp quality. Cited by: SS1.
* S. Gao, Z. Shi, M. Zhu, B. Fang, X. Xin, P. Ren, Z. Chen, and J. Ma (2023)Confocus: iterative tool learning from introspection feedback by easy-to-difficult curriculum. arXiv preprint arXiv:2308.14034. Cited by: SS1.
* X. Geng, A. Gudibande, H. Liu, E. Wallace, P. Abbeel, S. Levine, and D. Song (2023)Koala: a dialogue model for academic research. , 

the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_, pages 4288-4299.
* Lu et al. (2023b) Yining Lu, Haoping Yu, and Daniel Khashabi. 2023b. Gear: Augmenting language models with generalizable and efficient tool resolution. _arXiv preprint arXiv:2307.08775_.
* Lu et al. (2022b) Yujie Lu, Weixi Feng, Wanrong Zhu, Wenda Xu, Xin Eric Wang, Miguel Eckstein, and William Yang Wang. 2022b. Neuro-symbolic procedural planning with commonsense prompting. In _The Eleventh International Conference on Learning Representations_.
* Meister et al. (2020) Clara Meister, Tim Vieira, and Ryan Cotterell. 2020. Best-first beam search. _Transactions of the Association for Computational Linguistics_, 8:795-809.
* Mosig et al. (2020) Johannes E. M. Mosig, Shikib Mehri, and Thomas Kober. 2020. STAR: A Schema-Guided Dialog Dataset for Transfer Learning. _arXiv e-prints_.
* Och et al. (2001) Franz Josef Och, Nicola Ueffino, and Hermann Ney. 2001. An efficient a* search algorithm for statistical machine translation. In _Proceedings of the ACL 2001 Workshop on Data-Driven Methods in Machine Translation_.
* Patil et al. (2023) Shishir G Patil, Tianjun Zhang, Xin Wang, and Joseph E Gonzalez. 2023. Gorilla: Large language model connected with massive apis. _arXiv preprint arXiv:2305.15334_.
* Penedo et al. (2023) Guilherme Penedo, Quentin Malartic, Daniel Hesslow, Ruxandra Cojoercaru, Alessandro Cappelli, Hamza Alobeidli, Baptiste Pannier, Ebtesam Almazrouei, and Julien Launay. 2023. The RefinedWeb dataset for Falcon LLM: outperforming curated corpora with web data, and web data only. _arXiv preprint arXiv:2306.01116_.
* Prasad et al. (2023) Archiki Prasad, Alexander Koller, Mareike Hartmann, Peter Clark, Ashish Sabharwal, Mohit Bansal, and Tushar Khot. 2023. Adapt: As-needed decomposition and planning with language models. _arXiv preprint arXiv:2311.05772_.
* Qian et al. (2023) Cheng Qian, Chi Han, Yi R Fung, Yujia Qin, Zhiyuan Liu, and Heng Ji. 2023. Creator: Disentangling abstract and concrete reasonings of large language models through tool creation. _arXiv preprint arXiv:2305.14318_.
* Qiao et al. (2023) Shuofei Qiao, Honghao Gui, Huajun Chen, and Ningyu Zhang. 2023. Making language models better tool learners with execution feedback. _arXiv preprint arXiv:2305.13068_.
* Qin et al. (2023a) Yujia Qin, Shengding Hu, Yankai Lin, Weize Chen, Ning Ding, Ganqu Cui, Zheni Zeng, Yufei Huang, Chaojun Xiao, Chi Han, et al. 2023a. Tool learning with foundation models. _arXiv preprint arXiv:2304.08354_.
* Qin et al. (2023b) Yujia Qin, Shihao Liang, Yinting Ye, Kunlun Zhu, Lan Yan, Yaxi Lu, Yankai Lin, Xin Cong, Xiangru Tang, Bill Qian, et al. 2023b. Toollmi: Facilitating large language models to master 16000+ real-world apis. _arXiv preprint arXiv:2307.16789_.
* Reijers (2003) Hajo A Reijers. 2003. _Design and control of workflow processes: business process management for the service industry_, volume 2617. Springer.
* Reimers and Gurevych (2019) Nils Reimers and Iryna Gurevych. 2019. Sentence-bert: Sentence embeddings using siamese bert-networks. In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_, pages 3982-3992.
* Ruan et al. (2023) Jingqing Ruan, Yihong Chen, Bin Zhang, Zhiwei Xu, Tianpeng Bao, Guoqing Du, Shiwei Shi, Hangyu Mao, Xingyu Zeng, and Rui Zhao. 2023. Tptu: Task planning and tool usage of large language model-based ai agents. _arXiv preprint arXiv:2308.03427_.
* Schick et al. (2023) Timo Schick, Jane Dwivedi-Yu, Roberto Dessi, Roberta Raileanu, Maria Lomeli, Luke Zettlemoyer, Nicola Cancedda, and Thomas Scialom. 2023. Toolformer: Language models can teach themselves to use tools. _arXiv preprint arXiv:2302.04761_.
* Sengupta et al. (2019) Sailk Sengupta, Zahra Zahedi, and Subbarao Kambhampati. 2019. To monitor or to trust: observing robot's behavior based on a game-theoretic model of trust. In _Proceedings of the Trust Workshop at AAMAS_.
* Shao et al. (2017) Yuanlong Shao, Stephan Gouws, Denny Britz, Anna Goldie, Brian Strope, and Ray Kurzweil. 2017. Generating high-quality and informative conversation responses with sequence-to-sequence models. In _Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing_, pages 2210-2219.
* Shridhar et al. (2020) Mohit Shridhar, Jesse Thomason, Daniel Gordon, Yonatan Bisk, Winson Han, Roozbeh Mottaghi, Luke Zettlemoyer, and Dieter Fox. 2020. Alfred: A benchmark for interpreting grounded instructions for everyday tasks. In _2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_, pages 10737-10746. IEEE Computer Society.
* Song et al. (2023) Yifan Song, Weimin Xiong, Dawei Zhu, Cheng Li, Ke Wang, Ye Tian, and Sujian Li. 2023. Restgpt: Connecting large language models with real-world applications via restful apis. _arXiv preprint arXiv:2306.06624_.
* Tang et al. (2023) Qiayou Tang, Ziliang Deng, Hongyu Lin, Xianpei Han, Qiao Liang, and Le Sun. 2023. Toolalpaca: Generalized tool learning for language models with 3000 simulated cases. _arXiv preprint arXiv:2306.05301_.
* Team (2023) MosaicML NLP Team. 2023. Introducing mpt-7b: A new standard for open-source, commercially usable llims. Accessed: 2023-03-28 

Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothee Lacroix, Baptistie Roziere, Naman Goyal, Eric Hambro, Faisal Azhar, et al. 2023. Llama: Open and efficient foundation language models. _arXiv preprint arXiv:2302.13971_.
* Valmeekam et al. (2024) Karthik Valmeekam, Matthew Marquez, Alberto Olmo, Sarath Sreedharan, and Subbarao Kambhampati. 2024. Planbench: An extensible benchmark for evaluating large language models on planning and reasoning about change. _Advances in Neural Information Processing Systems_, 36.
* a critical investigation. In _Thirty-seventh Conference on Neural Information Processing Systems_.
* Valmeekam et al. (2022) Karthik Valmeekam, Alberto Olmo, Sarath Sreedharan, and Subbarao Kambhampati. 2022. Large language models still can't plan (a benchmark for llms on planning and reasoning about change). In _NeurIPS 2022 Foundation Models for Decision Making Workshop_.
* Vijayakumar et al. (2016) Ashwin K Vijayakumar, Michael Cogswell, Ramprasaath R Selvaraju, Qing Sun, Stefan Lee, David Crandall, and Dhruv Batra. 2016. Diverse beam search: Decoding diverse solutions from neural sequence models.
* Wang et al. (2023) Shufan Wang, Sebastien Jean, Sailik Sengupta, James Gung, Nikolaos Pappas, and Yi Zhang. 2023. Measuring and mitigating constraint violations of inconstext learning for utterance-to-API semantic parsing. In _Findings of the Association for Computational Linguistics: EMNLP 2023_, pages 7196-7207, Singapore. Association for Computational Linguistics.
* Wei et al. (2022) Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. 2022. Chain-of-thought prompting elicits reasoning in large language models. _Advances in Neural Information Processing Systems_, 35:24824-24837.
* Wu et al. (2022) Te-Lin Wu, Alex Spangher, Pegah Alipoormolabashi, Marjorie Freedman, Ralph Weischedel, and Nanyun Peng. 2022. Understanding multimodal procedural knowledge by sequencing multimodal instructional manuals. In _Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_, pages 4525-4542.
* Yang and Klein (2021) Kevin Yang and Dan Klein. 2021. Fudge: Controlled text generation with future discriminators. In _Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_, pages 3511-3535.
* Yang et al. (2023) Rui Yang, Lin Song, Yanwei Li, Sjjie Zhao, Yixiao Ge, Xiu Li, and Ying Shan. 2023. Gpt4tools: Teaching large language model to use tools via self-instruction. _arXiv preprint arXiv:2305.18752_.
* Yao et al. (2022) Shunyau Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik R Narasimhan, and Yuan Cao. 2022. React: Synergizing reasoning and acting in language models. In _The Eleventh International Conference on Learning Representations_.
* Yu et al. (2022) Xiao Yu, Maximillian Chen, and Zhou Yu. Prompt-based monte-carlo tree search for goal-oriented dialogue policy planning.
* Zhang et al. (2021) Yi Zhang, Sujay Kumar Jauhar, Julia Kiseleva, Ryven White, and Dan Roth. 2021. Learning to decompose and organize complex tasks. In _Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_, pages 2726-2735.
* Zhao et al. (2023) Jeffrey Zhao, Yuan Cao, Raghaw Gupta, Harrison Lee, Abhinav Rastogi, Mingqiu Wang, Hagen Soltau, Izhak Shafran, and Yonghui Wu. 2023. Anytod: A programmable task-oriented dialog system.

## Appendix A Dataset Creation

### Domain Creation

The flows and corresponding APIs for each domain are shown in Tab. 6. The API definitions in all domains, their input/output parameters, and descriptions are shown in Tab. 7.

### Test Utterance Generation

The prompt for generating test queries related to the intent "book flight" in the domain "Trip Booking" is shown in Fig. 4. Similar prompts were used for the generation of queries for all other intents. Some generated user queries related to various intents are shown in Tab. 8.

## Appendix B Prompt Structures for Generating Plans

The prompt structure for generating plans consisting of only APIs (no thoughts) is shown in Fig. 5. The prompt structure for planning with thoughts and APIs is shown in Fig. 6.

## Appendix C Experimental Details

### Hyper-parameter Tuning

In this section, we discuss the hyper-parameter selection process in FLAP, our proposed constrained decoding based algorithm in SS4.2.

In our dataset, all user queries are single intent, hence, we set \(\alpha_{c}>\alpha_{a}>\alpha_{b}\) (\(\alpha_{a}=0.5\), \(\alpha_{b}=0.1\), and \(\alpha_{c}=1\)). We empirically determine \(\beta=0.1\) when the APIs are hallucinated or violates the dependency, i.e., a soft alignment of generated APIs to permitted APIs 

To tune the values of the weight on the heuristic score (\(\lambda\)) and beam size (\(k\)), we run an ablation with a subset of the data (\(5\) user utterances per intent) in Tab. 1a, in the setting where only relevant subflow is given in-context, using Mistral-7b-instruct. We experiment with \(k=[5,10,15]\) and \(\lambda=[0.3,0.5,0.7]\). The results are summarized in Tab. 10. We find that \(k=10\) and \(\lambda=0.7\) result in overall best performance, especially in terms of API dependency violation (refer to "per plan avg % of inconsistent APIs"). Hence, we report the results with these values.

We set the values of all the scaling factors (_a_, \(b\), \(c\), _d_) to \(1\) implying equal weight to all heurictic score components. We set the lookahead length, \(L=32\) in our experiments. During plan generation, if the API Finish() is generated, we consider that as end-of-plan (as per the flows in Tab. 6) and terminate the generation.

For the Beam Search baseline, we set number of beams to \(3\) and use no_repeat_ngram_size=10 to prevent the model from generating the same thought and API pair repeatedly. For the Nucleus Sampling baseline we use top_p=0.9.

### Infrastructure and Libraries

We use eight NVIDIA A10G Tensor Core GPUs for all of our experiments. We use Pytorch2 for

Figure 4: Prompt for generating test utterances for the intent “book flight” in the domain “Trip Booking”.

Figure 5: Prompt format for generating a plan consisting of APIs only (without thoughts) in the domain “Trip Booking”. The first two dummy APIs InitSystem(), Start() are provided in context to guide the model to follow the plan format.

 

all implementations. For all the LLMs used in this paper, we use their Huggingface3 implementations. We build our proposed constrained decoding algorithm on top of Huggingface generation module4. We used the Huggingface implementation of the all-MiniLM-L6-v25 sentence transformer for measuring semantic similarities (sim()).

Footnote 3: [https://huggingface.co](https://huggingface.co)

Footnote 4: [https://huggingface.co/docs/transformers/main_classes/text_generation](https://huggingface.co/docs/transformers/main_classes/text_generation)

Footnote 5: [https://huggingface.co/sentence-transformers/all-MiniLM-L6-v2](https://huggingface.co/sentence-transformers/all-MiniLM-L6-v2)

### Runtime Analysis

The runtimes of different models in plan generation are shown in Tab. 9. Note that our proposed constrained decoding algorithm is run on only one GPU of 24GB memory with a batch size of \(1\). This is one of the reasons of the high runtime. We explored parallelization techniques by deploying the LLMs in multiple GPUs to make the process faster. However, it was observed that deploying the LLMs (Huggingface implementation) in multiple GPUs made the inference time of the base LLMs even slower. Our intended future work includes more efficient deployment of the LLMs in potentially multi-GPU machines which is out-of-scope of this paper.

## Appendix D Additional Results

The ablation results with standard deviations over all test data points with the prompting-based plan generation using numerous open-source models in the settings where all flows are given in-context and only relevant flows are given in-context, can be found in Tab. 11 and Tab. 12, respectively.

The results with standard deviations for plan generation using our proposed constrained decoding algorithm, FLAP, in the setting where all flows are given in-context and only relevant flows are given in-context, can be found in Tab. 13 and Tab. 14, respectively.

## Appendix E Qualitative Error Analysis

### Model entering into repetitive loop with thoughts

Example of error when model enters into a repetitive loop when prompted to generate plans with thoughts, can be observed in Tab. 15.

### API hallucinations by pre-trained LLMs with tool data

API hallucination examples by Alpaca-7B are shown in Tab. 16.

Figure 6: Prompt format for generating a plan (with thoughts) in the domain “Trip Booking”. The first two dummy steps, e.g., thoughts related to InitSystem() and Start() are provided in context to guide the model to follow the plan format.

 



\begin{table}
\begin{tabular}{l|c} \hline
**Flow steps** & **Required API sequence** \\ \hline Start processing the requests from the customer & InitSystem, Start() \\ Suggent can to the customer & FindCharacter[O] \\ Confirm and create the trip & OutFront/CraterID(D) \\ Extract and and postfunctional offers & GetServerNewBeiM(), UpdateInfoO \\ Order the trip & OutFusionInformation(O), OrderTripO \\ Push processing request & Finish(O) \\ \hline  & **Domain: Trip Boxing. Intent: Book Flight** \\ \hline Start processing the requests from the customer & InitSystem(), Start() \\ Suggent Inputs to the customer & OutFront/CraterID \\ Conflict and create the trip & OutFusionInformation(O), OrderTripO \\ Order the trip & OutFusionInformation(O), OrderTripO \\ Finally processing request & Finish(O) \\ \hline  & **Domain: Triple Boxing. Intent: Book Hotel** \\ \hline Start processing the requests from the customer & InitSystem(), Start() \\ SegreNet labels to the consumer & FindHost(I) \\ Confirm and create the trip & OutFront/CresterID \\ Order the trip & OutFusionInformation(O), OrderTripO \\ Finals processing request & Finish(O) \\ \hline  & **Domain: Insurance, Intent: Easy Insurance** \\ \hline Start processing the requests from the customer & InitSystem(), Start() \\ Providate these facts in the business & GutExectt(), GutExectt() \\ Get customer details & GutEmgraphicDetailis() \\ Order the insurance & Overtermware() \\ Prust Processing request & Finish(O) \\ \hline  & **Domain: Insurance, Intent: Concept Insurance** \\ \hline Start processing the requests from the customer & InitSystem(), Start() \\ Validation insurance & GetInterAppel0, ValidationInversance() \\ Reconnt the refined method & GetInfStartMethod() \\ Confirm and cured the Insurance & ConfEr(), CenceLInverse() \\ Frush processing request & Finish(O) \\ \hline  & **Domain: Insurance, Intent: Add Member** \\ \hline Start processing the requests from the customer & InitSystem(), Start() \\ Vididate insurance & GetInternetDec10, ValidationInversance() \\ Update the insurance with new member information & AttMediaIntenseNumberInformation(), UpdateInstance() \\ finish processing request & Finish(O) \\ \hline  & **Domain: Banking, Intent: Open Account** \\ \hline Start processing the requests from the customer & InitSystem(), Start() \\ Check eligibility for requesting an account & GetContainerIncos(), GetDateOfFirst(B), CheckSignibilityO \\ Traffic the account type & GetRecourtDesurpose() \\ Open the account & ConfEr(), OpenAccount() \\ Praith processing request & Finish(O) \\ \hline  & **Domain: Banking, Intent: Report Problem** \\ \hline Start processing the requests from the customer & InitSystem(), Start() \\ Record the problem and entity the internal team & RootGradSue(), NotifyComplainDepartEvent() \\ Trust processing request & Finish(O) \\ \hline  & **Domain: Banking, Intent: Closed Transaction** \\ \hline Start processing the requests from the customer & InitSystem(), Start() \\Validate transaction details & GetTransactionsIdentification(), ValidatesTransaction() \\ Cancel the transaction & ConfEr(), CancellTransaction() \\ Frish preprocessing request & Finish(O) \\ \hline  & **Domain: Restaurant + Rule Book, Intent: Book Restaurant** \\ \hline  & **Domain: Restaurant + Rule Book, Intent: Book Restaurant** \\ \hline  & **Domain: Restaurant + Rule Book, Intent: Book Right** \\ \hline  & **Domain: Restaurant + Rule Book, Intent: Book Right** \\ \hline  & **Domain: Restaurant + Rule Book, Intent: Book Right** \\ \hline  & **Domain: Restaurant + Rule Book, Intent: Book Right** \\ \hline  & **Domain: Restaurant + Rule Book, Intent: Book Right** \\ \end{tabular}
\end{table}
Table 6: Flows and required APIs in all domains. We study faithful plan generation in a zero-shot setting. Hence, we additionally add the dummy first (InitSystem(), Start()) and last (Finish()) steps in each flow to guide the model by indicating the start and end of processing a request in the zero-shot prompt.





\begin{table}
\begin{tabular}{l|p{142.3pt}|p{85.4pt}|p{96.7pt}|p{108.1pt}}
**APIs** & **Input Parameters** & **Output Parameters** & **Description** \\ \hline \multicolumn{4}{c}{**Domain Trip Reading**} \\ \hline Confer(C) & None & configuration_station & centre log/data with the customer before creating a Nep \\ FindHandler(M) & None & car, id & exists run through the existing city \\ FitModel(C) & None & hotel_id & finds and arrives beyond his in a city \\ GetPartOf(C) & None & my, info & open payment information from one customer \\ Start(C) & init, status & first receiving customer request & instance processing customer requests \\ IntiSystem(C) & None & isit,status & initializes the system to start requesting customer requests \\ OrderInfo(D) & vriq,i,dspy,info,comfication,status & order\_status & place the bookings \\ OutScript(D) & offer\_id, ffp,iad & rrrrrrrrr & options at step up their object \\ FireHight(D) & airport\_node & flight\_id & revects furthers throftree the airport \\ GetExtForwardClassION(C) & car,id & offer\_id & ensures instances attached decision related to the car option \\ CreateFif(D) & flip\_id,ffp,iad & tlrg:id & creates a new type before ordering \\ Configuration\_status & current\_state & try\_id & finishes processing customer requests \\ FinishO & order\_starts & turner\_node & returns nearby urgent cycles based on the travelling city \\ GetApprots(C) & None & airport\_node & returns nearby urgent cycles based on the traveling city \\ \hline \multicolumn{4}{c}{**Domain Insurance**} \\ \hline \multicolumn{4}{c}{**Domain Insurance**} \\ \hline ConsolInference(C) & resource_id, conf



**Incents and Generated Queries in Domain Trip Booking**

**Book Car: Could you arrange for car rental for my Minneapolis trip from June 25th to July 3rd? Book Hotel: I'm looking to book a hotel room in Los Angeles from August 18th to August 25th. Book Flight: I need to fly from Miami to Toronto, can you please help me with that? Intents and Generated Queries in Domain Insurance**

**Buy Insurance:** I want to safeguard my family's health with an insurance policy.

**Cancel Insurance:** My current policy doesn't cover all the expenses, would you please cancel it? Add Member: Would you please add my autn to my mobile insurance policy?

**Intents and Generated Queries in Domain Banking**

**Open Account:** My wife and I are looking to open a joint bank account together. What steps do we need to take? Report Problem: My card is declined at the ATM. Can you help?

**Camcel Transaction:** The balance of my savings account has been altered by an unauthorized transfer of $500. I would appreciate if you could get that money refunded.

**Intents and Generated Queries in Domain Restaurant & Ride Book**

**Book Restaurant:** Could you assist me in making a reservation at a restaurant in Manhattan?

**Book Ridge:** I am in Washington D.C. looking for a ride to Fort Lauderdale.

**Camcel Restaurant:** I made a restaurant booking in Los Angeles this morning but I cannot make it due to an emergency. Can you cancel my booting?

**Camcel Ridge:** May I cancel my ride from Brooklyn to Queens?

\begin{table} \end{table}
Table 8: Examples of generated user queries for each intent.





\begin{table}
\begin{tabular}{l|c|c|c|c c}
**LLM** & **Methods** & **With** & \multicolumn{2}{c}{**Average Ranking For Plan (seconds)**} \\  & & **Throughs** & **All Horn in context** & **Relevant flow in context** \\ \hline \multirow{6}{*}{**Lift**} & Greedy Search & No & 7.94 & 5.4 \\  & Greedy Search & Yua & 14.07 & 11.3 \\  & Beam Search & Yua & 27.94 & 28.69 \\  & Nedicase Sampling & Yua & 17.74 & 17.52 \\  & FLAP1 (sol agi+ align length tag



\begin{table}
\begin{tabular}{l l c|c||c|c|c|c|c|c|c|c|c} \hline
**LLM** & **Variable** & **Parameter** & **Value** & **Value** & **value pair pair** & **S****Pma** & **Avg. of APIs in plan that are** & **Age of adult per** & **Per plus avg to** **s" dimensional** \\  & **Parameter** & **Values** & **Throughings** & **APIs** & **APIs** & **Period(1)** & **BABILITY(4)** & **Stopups (1)** & **APIs(2)** & **APIs(3)** & **APIs(5)** & **APIs(6)** \\ \hline
* & 38.71 \(\pm\) 0.95 & 30.00 & 19.84 \(\pm\) 37.01 & 1.21 \(\pm\) 9.43 & 29.58 \(\pm\) 4.68 & 25.94 \(\pm\) 39.08 & 15.42 \(\pm\) 14.02 & 33.06 \(\pm\) 24.21 \\  & Yes & 15.09 \(\pm\) 8.66 & 14.98 \(\pm\) 8.70 & 98.62 \(\pm\) 7.61 \(\pm\) 16.00 & 13.02 \(\pm\) 21.75 & 17.28 \(\pm\) 7.66 & 13.29 \(\pm\) 42.77 & 18.17 \(\pm\) 16.39 & 34.18 \(\pm\) 27.51 \\ \hline
* & 4,13.2 \(\pm\) 2.16 & 100.00 & 0.40 \(\pm\) 0.00 & 10.76 \(\pm\) 23.62 & 4.64 \(\pm\) 2.04 & 5.12 \(\pm\) 2.42 & 30.41 \(\pm\) 19.98 & 49.64 \(\pm\) 35.21 \\  & Yes & 4.88 \(\pm\) 4.26 & 98.48 \(\pm\) 4.29 & 98.38 \(\pm\) 32.32 & 1.39 \(\pm\) 21.29 & 1.47 \(\pm\) 4.07 & 5.34 \(\pm\) 32.32 & 17.65 \(\pm\) 17.77 & 41.88 \(\pm\) 25.15 \\ \hline
* & 55.57 \(\pm\) 56.10 & 100.05 \(\pm\) 45.49 & 8.60 \(\pm\) 9.07 & 47.48 \(\pm\) 39.88 & 44.19 \(\pm\) 45.91 & 16.01 \(\pm\) 20.66 & 27.67 \(\pm\) 20.48 \\  & Yes & 11.62 \(\pm\) 6.44 & 11.09 \(\pm\) 6.21 & 100.00 & 39.20 \(\pm\) 30.38 & 19.20 \(\pm\) 27.34 & 97.33 \(\pm\) 6.45 & 11.41 \(\pm\) 7.34 & 24.63 \(\pm\) 25.24 & 38.83 \(\pm\) 26.52 \\ \hline
* & 17.30 \(\pm\) 32.42 & 100.00 & 11.18 \(\pm\) 28.77 & 0.17 \(\pm\) 13.67 & 13.21 \(\pm\) 24.68 & 14.65 \(\pm\) 14.12 & 33.32 \(\pm\) 21.74 \\  & Yes & 16.27 \(\pm\) 17.00 & 18.08 \(\pm\) 7.49 & 76.50 \(\pm\) 18.34 & 10.46 & 13.92 \(\pm\) 23.62 & 12.83 \(\pm\) 7.68 & 11.68 \(\pm\) 21.69 & 27.62 \(\pm\) 23.69 \\ \hline
* & 4.78 \(\pm\) 1.70 & 0.02 \(\pm\) 30.42 & 2.36 \(\pm\) 7.04 & 2.87 \(\pm\) 17.55 & 3.02 \(\pm\) 17.79 & 10.69 \(\pm\) 18.34 & 37.23 \(\pm\) 25.28 \\  & Yes & 4.72 \(\pm\) 2.38 & 4.76 \(\pm\) 2.41 & 98.62 \(\pm\) 17.15 & 0.49 \(\pm\) 34.96 & 2.85 \(\pm\) 25.26 & 3.50 \(\pm\) 89.21 & 40.31 \(\pm\) 28.89 \\ \hline
* & 4.95 \(\pm\) 4.94 & 100.00 & 29.92 \(\pm\) 17.15 & 0.49 \(\pm\) 37.54 & 3.67 \(\pm\) 4.94 & 38.47 \(\pm\) 10.85 & 11.37 \(\pm\) 13.39 & 39.58 \(\pm\) 15.54 \\  & Yes & 1.206 \(\pm\) 4.15 & 12.2 \(\pm\) 4.21 & 95.31 & 24.37 \(\pm\) 30.11 & 10.00 \(\pm\) 20.40 & 8.76 \(\pm\) 5.23 & 90.08 \(\pm\) 5.52 & 90.02 \(\pm\) 11.68 & 24.01 \(\pm\) 19.39 \\ \hline
* & 5.94 \(\pm\) 2.65 & 100.00 & 0.57 \(\pm\) 27.75 & 2.38 \(\pm\) 6.99 & 4.12 \(\pm\) 2.51 & 4.29 \(\pm\) 25.54 & 15.08 \(\pm\) 16.02 & 35.22 \(\pm\) 20.51 \\  & Yes & 6.91 \(\pm\) 3.53 & 6.99 \(\pm\) 36.74 & 3.44 \(\pm\) 39.76 & 4.95 \(\pm\) 39.55 & 4.02 \(\pm\) 43.81 & 4.79 \(\pm\) 7.19 & 6.73 \(\pm\) 14.99 & 35.98 \(\pm\) 22.64 \\ \hline
* & 29.41 \(\pm\) 37.77 & 100.00 & 25.84 \(\pm\) 38.81 & 2.36 \(\pm\) 9.99 & 2.07 \(\pm\) 37.17 & 28.97 \(\pm\) 29.88 & 13.16 \(\pm\) 16.06 & 31.72 \(\pm\) 25.51 \\  & Yes & 10.21 \(\pm\) 6.95 & 10.31 \(\pm\) 6.79 & 86.85 \(\pm\) 27.32 & 4.02 \(\pm\) 12.77 & 9.18 \(\pm\) 7.12 & 6.68 \(\pm\) 7.05 & 4.68 \(\pm\) 10.61 & 30.41 \(\pm\) 27.56 \\ \hline
* & 5.71 \(\pm\) 19.33 & 100.00 & 1.72 \(\pm\) 5.81 & 0.53 \(\pm\) 39.34 & 2.36 \(\pm\) 23.35 & 3.55 \(\pm\) 20.08 & 9.08 \(\pm\) 14.36 & 33.41 \(\pm\) 19.78 \\  & Yes & 5.77 \(\pm\) 2.07 & 8.77 \(\pm\) 25.35 & 96.29 \(\pm\) 17.69 & 1.69 \(\pm\) 5.15 & 2.74 \(\pm\) 25.58 & 2.77 \(\pm\) 26.2 & 5.72 \(\pm\) 10.18 & 32.45 \(\pm\) 22.35 \\ \hline
* & 10.21 \(\pm\) 21.88 & 100.00 & 1.31 \(\pm\) 15.86 & 8.99 \(\pm\) 42.44 & 8.97 \(\pm\) 32.55 & 7.14 \(\pm\) 14.25 & 18.19 \(\pm\) 17.61 & 38.27 \(\pm\) 29.27 \\  & Yes & 7.6 \(\pm\) 4.83 & 7.75 \(\pm\) 4.96 & 100.00 & 10.25 \(\pm\) 21.52 & 5.0 \(\pm\) 1




\begin{table}
\begin{tabular}{l l c c c c|c c c c c c c}
**L13b** & **Decoding Strategy** & **With** & **Avg center file** & **Pla** & **Pla** & **Avg of aPlus in data set** & **Avg of a Grid for** & **Per plan up vs. dimensional** \\  & & & **Transit** & **Throughs** & **APIs** & **APIs** & 0.1862-0.97 & 0.1741.19 & **10.157.37** & 3.142.48 & 2.468 (14.12) & 3.13.23.23.24 \\  & Group Search & Yes & 1.62 \(\pm\) 1.79 & 0.67 & 0.70 & 0.5011.39 & 0.40 (1.04) & 1.925.63 & 1.285.76 & 1.281.54 (1.96) & 2.961.24 (2.80) \\  & Region Search & Yes & 1.91 \(\pm\) 1.82 & 9.53 \(\pm\) 1.87 & 0.70 & 0.50 \(\pm\) 1.17 & 0.37 (1.17) & 0.56 (+\) & 0.71 (+\) & 0.71 (+\) & 0.71 (+\) & 0.71 (+\) & 0.71 (+\) & 0.71 (+\) \\  & Stacking Sampling & Yes & 1.23 \(\pm\) 6.4 & 1.21 \(\pm\) 6.4 & 9.56 & 0.58 \(\pm\) 2.25 & 0.42 (1.03) & 1.28 (1.03) & 1.28 (1.03) & 1.28 (1.03) & 1.28 (1.03) & 1.28 (1.03) \\  & PLAPI (set = get \(\alpha_{\text{diff}}\) through t-test, missed) & Yes & 1.90 \(\pm\) 1.03 & 1.07 \(\pm\) 1.08 & 1.00 (1.03) & 0.56 (1.03) & 0.51 (1.04) & 5.47 (1.04) & 6.49 (1.03) & 6.49 (1.04) & 6.49 (1.04) \\  & PLAPI (set = get \(\alpha_{\text{diff}}\) through t-test, missed) & Yes & 1.92 \(\pm\) 1.43 & 0.77 (1.00) & 1.00 (1.04) & 0.06 (1.03) & 0.56 (1.04) & 5.41 (1.04) & 5.47 (1.04) & 6.49 (1.04) & 6.49 (1.04) \\  & PLAPI (set = get \(\alpha_{\text{diff}}\) through t-test, missed) & Yes & 1.92 \(\pm\) 1.43 & 0.77 (1.00) & 1.00 (1.04) & 0.06 (1.03) & 0.56 (1.04) & 5.40 (1.04) & 5.47 (1.04) & 6.49 (1.04) & 6.49 (1.04) \\  & PLAPI (set = get \(\alpha_{\text{diff}}\) through t-test, missed) & Yes & 1.92 \(\pm\) 1.53 & 0.77 (1.00) & 1.00 (1.04) & 0.06 (1.04) & 0.06 (1.04) & 5.48 (1.04) & 5.40 (1.04) & 6.49 (1.04) & 6.49 (1.04) \\  & PLAPI (set = get \(\alpha_{\text{diff}}\) through t-test, missed) & Yes & 1.92 \(\pm\) 1.53 & 0.78 (1.00) & 1.00 (1.04) & 0.06 (1.04) & 0.06 (1.04) & 5.41 (1.04) & 5.40 (1.04) & 6.49 (1.04) & 6.49 (1.04) \\  & PLAPI (set = get \(\alpha_{\text{diff}}\) through t-test, missed) & Yes & 1.92 \(\pm\) 1.54 & 0.77 (1.00) & 1.00 (1.04) & 0.06 (1.04) & 0.06 (1.04) & 5.41 (1.04) & 5.40 (1.04) & 6.49 (1.04) & 6.49 (1.04) \\  & PLAPI (set = get \(\alpha_{\text{diff}}\) through t-test, missed) & Yes & 1.92 \(\pm\) 1.53 & 0.78 (1.00) & 1.00 (1.04) & 0.06 (1.04) & 0.06 (1.04) & 5.41 (1.04) & 5.40 (1.04) & 6.49 (1.04) & 6.49 (1.04) \\  & PLAPI (set = get \(\alpha_{\text{diff}}\) through t-test, missed) & Yes & 1.92 \(\pm\) 1.54 & 0.77 (1.00) & 1.00 (1.04) & 0.06 (1.04) & 0.06 (1.04) & 5.41 (1.04) & 5.40 (1.04) & 6.49 (1.04) & 6.49 (1.04) \\  & PLAPI (set = get \(\alpha_{\text{diff}}\) through t-test, missed) & Yes & 1.92 \(\pm\) 1.54 & 0.77 (1.00) & 1.00 (1.04) & 0.06 (1.04) & 0.06 (1.04) & 5.41 (1.04) & 5.40 (1.04) & 6.49 (1.04) & 6.49 (1.04) \\  & PLAPI (set = get \(\alpha_{\text{diff}}\) through t-test, missed) & Yes & 1.92 \(\pm\) 1.54 & 0.77 (1.00) & 1.00 (1.04) & 0.06 (1.04) & 0.06 (1.04) & 5.41 (1.04) & 5.40 (1.04) & 6.49 (1.04) & 6.49 (1.04) \\  & PLAPI (set = get \(\alpha_{\text{diff}}\) through t-test, missed) & Yes & 1.92 \(\pm\) 1.54 & 0.77 (1.00) & 1.00 (1.04) & 0.06 (1.04) & 0.06 (1.04) & 5.41 (1.04) & 5.40 (1.04) & 6.49 (1.04) & 6.49 (1.04) \\  & PLAPI (set = get \(\alpha_{\text{diff}}\) through t-test, missed) & Yes & 1.92 \(\pm\) 1.54 & 0.77 (1.00) & 1.00 (1.04) & 0.06 (1.04) & 0.06 (1.04) & 5.41 (1.04) & 5.40 (1.04) & 6.49 (1.04) & 6.49 (1.04) \\  & PLAPI (set = get \(\alpha_{\text{diff}}\) through t-test, missed) & Yes & 1.92 \(\pm\) 1.54 & 0.77 (1.00) & 1.00 (1.04) & 0.06 (1.04) & 0.06 (1.04) & 5.41 (1.04) & 5.40 (1.04) & 6.49 (1.04) & 6.49 (1.04) \\  & PLAPI (set = get \(\alpha_{\text{diff}}\) through t-test, missed) & Yes & 1.92 \(\pm\) 1.54 & 0.77 (1.00) & 1.00 (1.04) & 0.06 (1.04) & 0.06 (1.04) & 5.41 (1.04) & 5.40 (1.04) & 6.49 (1.04) & 6.49 (1.04) & 6.49 (1.04) \\ \hline  & PLAPI (set = get \(\alpha_{\text{diff}}\) through t-test, missed) & Yes & 1.92 \(\pm\) 1.54 & 0.77 (1.00) & 0.06 (1.04) & 0.06 (1.04) & 0.06 (1.04) & 5.41 (1.04) & 5.40 (1.04) & 6.49 (1.04) & 6.49 (1.04) & 6.49 (1.04) \\  & PLAPI (set = get \(\alpha_{\text{diff}}\) through t-test, missed) & Yes & 1.92 \(\pm\) 1.54 & 0.77 (1.00) & 1.00 (1.04) & 0.06 (1.04) & 0.06 (1.04) & 5.41 (1.04) & 5.40 (1.04) & 6.49 (1.04) & 6.49 (1.04) & 7.20 (1.04) \\ \hline  & PLAPI (set = get \(\alpha_{\text{diff}}\) through t-test, missed) & Yes & 1.92 \(\pm\) 1.54 & 0.77



\begin{table}
\begin{tabular}{l|l} \hline
**Query: I want to go from San Diego to Dallas, can you please help me with that?** \\ \hline \multicolumn{3}{c}{**Generated plan by Not-7b-instruct using prompting with greedy decoding**} \\  & I can suggest flights to the customer. [API] FindFlight() \\  & I can suggest hotels to the customer. [API] FindHotel() \\  & I can suggest cars to the customer. [API] FindRentalCar() \\  & I can suggest insurances to the customer. [API] GetCarInsuranceDiscount() \\  & I can suggest flights to the customer. [API] FindFlight() \\  & I can suggest hotels to the customer. [API] FindHotel() \\  & I can suggest cars to the customer. [API] FindRentalCar() \\  & I can suggest insurances to the customer. [API] FindHotel() \\  & I can suggest flights to the customer. [API] FindRentalCar() \\  & I can suggest hisrates to the customer. [API] FatCartInfrastructureDiscount() \\  & I can suggest hortels to the customer. [API] FindHotel() \\  & I can suggest cars to the customer. [API] FindRentalCar() \\  & I can suggest insurances to the customers. [API] GetCarInsuranceDiscount() \\  & I can suggest flights to the customer. [API] FindFlight() \\  & I can suggest hotels to the customer. [API] FindHotel() \\  & I can suggest cars to the customer. [API] FindRentalCar() \\ \hline \end{tabular}
\end{table}
Table 15: Example of error when the LLMs enter into a repetitive loop of thought+API in Greedy decoding when prompted to generate thoughts and APIs.





\begin{table}
\begin{tabular}{l|p{142.3pt}|c} Generated Plan by ToolAlpaca-7B & Correct APIs \\ \hline
**Query: I would like to book a rental car for my visit to Las Vegas on June 28th.** & \\ [though] I need to suggest cars to the customer. [API] SuggestCars() & FindRentalCar() \\ [though] I need to confirm and create the trip. [API] ConfirmTrip() & Confirm(), CreateTrirp() \\ [though] I need to extract and add promotional offers. [API] ExtractPromotionAffers() & GetCarInsuranceDiscount(), UpdateTrirp() \\ [though] I need to order the trip. [API] OrderTrip() & GetPaymentInformation(), OrderTrip() \\ \end{tabular}
\end{table}
Table 16: API hallucination examples by ToolAlpaca. Hallucinated APIs are highlighted.