Measuring and Mitigating Constraint Violations of In-Context Learning for Utterance-to-API Semantic Parsing
Authors: Shufan Wang
Sébastien Jean
Sailik Sengupta
James Gung
Nikolaos Pappas
Yi Zhang
Venue: EMNLP 2023 Findings (2023)
Keywords: executable semantic parsing
task-oriented semantic parsing
utterance-to-API generation
in-context learning
TL;DR: We define and analyze various categories of constraint violations by LLMs in utterance-to-API generation, and examine common mitigation strategies with our proposed fine-grained metrics.

Abstract: In executable task-oriented semantic parsing, the system aims to translate users' utterances in natural language to machine-interpretable programs (API calls) that can be executed according to pre-defined API specifications.
With the popularity of Large Language Models (LLMs), in-context learning offers a strong baseline for such scenarios, especially in data-limited regimes.
However, LLMs are known to hallucinate and therefore pose a formidable challenge in constraining generated content. 
Thus, it remains uncertain if LLMs can effectively perform task-oriented utterance-to-API generation, where respecting the API's structural and task-specific constraints is crucial.
In this work, we seek to measure, analyze and mitigate such constraints violations. First, we identify the categories of various constraints in obtaining API-semantics from task-oriented utterances, and define fine-grained metrics that complement traditional ones. Second, we leverage these metrics to conduct a detailed error analysis of constraints violations seen in state-of-the-art LLMs, which motivates us to investigate two popular mitigation strategies-- Semantic-Retrieval of Demonstrations (SRD) and API-aware Constrained Decoding (API-CD). 
Our experiments show that these strategies are effective at reducing constraints violations and improving the quality of the generated API calls, but require careful consideration given their implementation complexity and latency.

Measuring and Mitigating Constraint Violations of In-Context Learning for Utterance-to-API Semantic Parsing
Shufan Wang'; Sébastien Jean Sailik Sengupta", James Gung,, Nikolaos Pappas², Yi Zhang
University of Massachusetts, Amherst a WWS AI Labs
shufanwang @ emus {sebjean, sailiks, gungj, nppappa, yizhngn @ @maazon.oom
Abstract
Utterance
Show way options auditorium
by 10 am via 1st ave
In executable task-oriented semantic parsing, the system aims to translate users' utterances in natural language to interprraate pro- grams (API calls) that can be executed accord- ing to pre-defined API specifications. With the popularity of Large Language Models (LLMs), in-context learning offers a strong baseline for such scenarios, especially in data-limited regimes (Hu al., 2022; Shin al., 2021). However, LLMs are known to hallucinate and therefore pose a formidable challenge in con- straining generated content (Parikh et al., 2020). Thus, it remains uncertain if L.LMs can effec- tively perform task-oriented Iiterance-to-APP generation where respecting API's structural and task-specific constraints is crucial. In this work, we seek measure, analyze and miti- gate such constraints violations. First, we iden- tify the categories of various constraints ob- taining APl-semanties from task-oriented utter- ances, and define fine-grainee metrics that com- plement traditional ones. Second, we leverage these metrics to conduct a detailed error analy- sis constraints violations seen in otate-of-the- art LLMs, which motivates us to investigate two popular mitigation strategies- Semantic- Retrieval of Demonstrations (SRD) and API- aware Constrained Decoding (API-CD). Our experiments show that these strategies are effec- tive at reducing constraints violations and im- proving the quality of the generated API calls, but require careful consideration given their implementation complexity and latency.
Semantic Parse
[INGET DIRECTIONS Show me one way
options to
[SLDESTINATION
[IN:GET LOCATION
[SL:CATEGORY LOCATION auditorium) ]]
SSLAATE TIME ARRIVAL by am
[SLPATH 1st ave
Representation
IN:GET DIRECTIONS
SLPAAH SLDATE TIME ARRIVAL SLIDESTINATION
1st
IN:GET_ LOCATION
SLICATEGORY LOCATION
auditorium
API DIRECTIONS DESTINATION = GET LOCATTON ( CATEGORY LOCATION "auditorium"), DATE_ TIME, ARRIVAL by am".
PATH "1st ave")
Figure I: nested example from the TOPv2 dataset consisting of an utterance, corresponding semantic parse, the extracted tree representation and the corre- sponding API call. We measure & improve the ability of Large Language Model (LLM) to generate the API call given an utterance with In-Context Learning (ICL).
to be able to interact with the real-world execu- tion environment by translating users' utterances to API calls, that executable according to pre- defined API documentation. For example, in Fig- ure 1, given the user's utterance Show me one way options to auditorium by 10am via Ist ave, the ex- ecutable -oisctreetted semantic parser should not only understand the semantic parse structures from the user's input, but more importantly, be able to generate the API call:
1 Introduction
In task-oriented dialog (TOD) settings, executable semantic parsers maps natural language utterances from human users machine-interpretaale mean- ing representations that help achieve users' goals. order to effectively carry out the user's com- mand, such task-oriented semantic parsers need
GET DIRECTIONS C DESTINATION = GET GET_LOCATION ( CATEGORY LOCATTONNN "auditorium" ), PATH="lstavc"))
where the parsed intents and slots represent func- tions and arguments in the API calls.
WWrk performed during an internship at AWS Labs. "oorresponding author
Existing work has tackled task-oriented seman-

---

tic parsing by using auto-regressive language mod- cls (Rongali et al., 2020; Mansimov and Zhang, 2022; Hu et al., 2022: Shin and Van Durme, 2022). Recent advances of pre-trained Large Language Models (LLMs), especially those trained with code, present an exciting opportunity to frame executable semantic parsing tasks as an API call generation problem. In Hu et al. (2022), Shin et al. (2021) and Shin and Van Durme (2022), LLMs prompted with examples are able generate code-like represen- tations based on user queries. In-context Learning (ICL) (Brown et al., 2020) is a particularly appeal- ing approach for semantic parsing tasks as it allows developers to (1) make use of LLM's transfer learn- ing abilities in low-resource settings with only a small collection of labelled training data, and (2) easily prototype -Lsed systems without need- ing re-training or fine-tuning.
errors based on these constraint violation metries reveal important failure modes, such the lack of examples pertaining to a particular intent in the ICL input prompt, dependence on particular task- specific keywords, scenarios when hallucination is likely, etc. and inspires us to develop mitigation strategies that reduces constraint violations.
We investigate two popular mitigation strategies- retrieval-basee augmentation (Pasupat et al., 2021; Gupta al., 2022) and constrained decoding (Shin and Van Durme, 2022). In section 5, we show that our Semantic- Rettieval of Demonstrations (SRD), despite its implementation simplicity, improves the semantic parsing quality in the ICL setting and can also reduce constraint violation to an extent. On the other hand, when task-specific APIs are available, our API-aware Constrained Decoding (API-CD) strategy can completely eliminate con- straint violations. An umalgamation of these two complementary approaches provides the best re- sults, but incurs some tradeoffs in efficiency. We analyze this trade-off, offering advice for practi- tioners. Finally, we plan to release the modified TOPv2 dataset to encourage further research in this direction. In summary, our key contributions are:
However, it has been shown that outputs from LLMs are prone to hallucination, lack controlla- bility and have high sensitivity to prompt design (Parikh et al., 2020; Raunak et al., 2021; Rashkin et al., 2021), even when these models are prompted with human-writtee instructions and selected ex- emplars (Liu al.. 2022). These weaknesses are concerning for executable semantic parsing, as gen- erated output needs to respect the task constraints specified by the API documents.
(1) Defining and introducing constraint violation metrics for generated API calls ($2).
(2) Analysis of ICL failure modes using proposed metrics and proposal of mitigation strategies ($3$4).
In this work, we investigate the question, can LLMs with in-context learning respect the task- specific constraints required for task-oriented se- mantic parsing? To the best of our knowledge, there is no existing work that categorizes and de- fines fine-graince task-specific constraints for task- oriented semantic parsing or systematically ana- lyzes constraints violations in LLM outputs in the ICL scenario. We begin by identifying and catego- rizing the task constraints in Section 2, focusing on the TOP-V2 V2 datuset (Chen et al., 2020), an ex- isting task-oriented semantic parsing dataset, and detecting constraints violations in the output of GPT-NeoX (Black et al., 2022), a state-of-the-art LLM trained on code.
(3) Showcasing efficacy of solutions to com- pletely eliminate constraint violations ($5).
Defining Constraints Violations
Recent TOD systems, such as Shu et al. (2022) and Andreas et al. (2020), take ssingle/multi-tumi ut- terances of users as input and generate executable programs (API calls) to achieve user goals. Such systems are expected to not only generate effec- tive programming codes but also respect the API constraints, relevant to the application domains, specified by developers. To study the task of gener- ating valid APT calls, we convert the popular TOD benchmark dataset TOPv2 (Chen et al., 2020) to an utterance-to-API format. Specifically, TOPv2 consists of pairs of user-utterance and semantic parse representing the users' goals; we process the dataset by converting the semantic parses to API calls (see Figure 1).
We observe that task specific constraint violation are ubiquitous when using ICL in practice (ranging from 3- 14.8%), while syntactic constraints are lower (ss 0.6%). In Section 3, we show that the proposed constraint violation metrics serve a use- ful diagnostic tool, providing a fine-grained view and complementing existing evaluation measures in (Mansimov and Zhang, 2022). Our analysis of
'While the data in (Shu et al., 2022) would have been appropriate for our study, it is small in size and unrelcased.

---

Note that the TOPv2 dataset contains many com- positional/nested queries, where a particular slot value is the result of another intent. This reflected under the dialog-to-Ae to-og API framework as nested calls, where the argument value to some function can be the return value from another function. Hence, an API call can be represented as flattened list of functional calls, which we now define as a flat- tened list representation (examples for flattened list representations are in appendix A.1).
To measure these constraints, we obtain the sets Vz. Vas and Via using the functions, argu- ments, and interactions the entire training data of TOPv2. We note that one should use API-specific documentation, when available, to define these more efficiently. Further, we defined Vy and V. as set, and represented Vfa as dictionary, In this work, we focus our attention to hard- constraint satisfaction, i.e. for every generated API call y, C. is either a 1 or 0. Thus, even if (say) 90% of y forms a valid abstract syntax tree, or has a function f that is a synonym of function names E Vs. it gets no brownie points. We feel this is reasonable given the high standards of API calls (e.g. even single character typo in function name results in compilation/run-tine failures). Let us consider the metrics for the following example.
Definition 2.1. A Allattened list representation of an API call y is a list of tuples Ai) where each fi is function (corresponding to the intent) present in the API call and A, is set of (a, v) pairs (corresponding to the slots and slot values). Each function fi is defined to be associated with all arguments a's from Ai.
[User Utterance]
For compositional/nested API calls, the val- ues U can be other functions (f), while for non- compositional APIs they are resolvable/grounded values. It follows that in the latter case, the flat- tened list has only one function call.
shaw my alarms for Lomerzow
Generated call] ALARMSS DATE "lomorrow"
C,
1
abstract syntax tree
C
0
x function name GET ALARMS
2.1 Task-specific Constraints
C. 1
arg name (DATE_
(tomorow)
Cja 0
incorrect function name implies
We define four categories of constraints associated with the generation of API calls. The first con- straint imposes a restriction on the overall output. Definition 2.2 (Structural Constraint Cs). An API call expression satisfies this constraint iff the ex- pression can be parsed into valid abstract syntax tree.
valid manctinn-rrgumentt pair V/a
Note how each individual input- provides a distinct constraint violation signature. To evaluate system's 's success at constraint satisfaction, we compute constraint violation rate, i.e. the per- centage of constraint violations for each constraint violation category over all the test examples. While existing metrics like exact-match, F1, and accuracy give a top-level view of the model's capability for dialog-to-APe scenarios, our metrics can identify more granular aspects; thereby giving more com- plete picture. For example, model scoring low on exact-match may still be great correcting identi- fying functions and argument names from few-shot data in prompts and simply be bad at structural constraints. In addition, the fine-grained diagnosis can help us design better solutions.
The other constraints impose task-specific re- strictions on the generated functions, the arguments, and the mutual interactions between them. Con- cretely, we first define a triple (V,, Va Vz), where (I) Vy represents the set of valid functions (f); (2) Va represents the set of valid arguments a, and (3) Vja is the set of valid pairs of functions (f) and its associated arguments (a). We now define these task spec-ipe constraints as follows.
Definition 2.3 (Functional Constraint Cÿ). An API call y violates Cf if there is any function f E y and f 4 Vj.
Analyzing Constraint Violation
Definition 2.4 (Argument Constraint C.). An API call y violates Ca if there is any argument E and & V..
As we noted above, the use of exact match and F1- scores (Mansimov and Zhang, 2022) can only pro- vide a bird's eye view of the model's failures. Un- fortunately, recently employed metrics like BLEU and Execution Match Ratio (EMR) for such scenar- ios (Shu et al., 2022) are also incapable provid- ing a magnified view. Further, these metrics can
Definition 2.5 (Function-argument Association Constraint Cfa). An API call y violates Cja if there is any associated function-arguman pair (f, a) E y and (f. a) & Vja-

---

Reasons for violation
Construint Violation Examples
Constraint Violation Analysis
Misunderstaddnng task semantics
[Utterance]]: show wrecks avaid on the interstate
The system simply the template CET_INFF TNFC_ that is ubiquilous in the training duta (and there- fore in the prompe). Althhogh it has seen GBT INPO_PRAPPeO PRAPEIC in the prompes, it fails make the semantic implicature from wrecks to truffic during generation, thereby resulting undelined funetion call GRT TNPO WRRCKS and violating all three iask specific constraints.
[Ground-truth):
violates
GET INFO GET_INFO, TRAFFIC OBSTRUCTION AVOID = wwrckss LOCATION GETLLOCATION CATE-
Cy. Ca. Cya
GORY LLLCCTIONNN =" the interstate" "))
[Prediction]:
IRT_ WRECKS (LOCAATIN - the interstate") )
Misunderstanding slot.
[Utterance]: 10 forecast please.
The system fails to realize that DATR_ TIME support 10 Day as an argumeat value, Hence, hallacinates new argument FORECAST CCCASTDAYSS to provide this information. Note that argument definition for DACK 188 is correct, and thus it only violates Cje.
s'arguments ussociated with task
[Groend-truth]:
SET_WEATIER (DATE_TIME= =IME=" 10 Day
">
violstes Cza
[Prediction]:
GET WEATHER (DATE_ TIME today CAST DAYS -" 10")
FORE-
Lack of relevant exam- ples the prompt violates Cy. Cys
[Utteranee]: show alanms for tomorrow
The GET ALARM has a small support in the train- ing Hence, sampling omitts the example prommpt. The model, ablivious to this function call. violates Cy generating, undefined function SHOK ALARSS.
[Ground-truth]:
GET ALARM { ALLARM_ NAME
GET_TIME GET, TIME (
OXTE_TIME =" for lomBTTUw "))
[Prediction]:
SHOW SHOW_ALARMS (DATE_TIM) =" "oomrrrww ")
Iubiility to handle compositionality
[Utterance]: add remänder feed spurky 6pm daily
Correctly returning the value to RECURR TNG B___DARR__TTT execution of a nested function call GET RER.EERRRRNGG.C.. DATE_TTME provided time. The models fuils to comprehend this nested call and flattens the information required by hallucinating arguments, such AMCUNT and PIRSOO JLML NDIU, associtted with the primary function call, therehy violating C. and Cyu-
[Ground-trutl]:
requires the
violates C.. Cja
CREATE_ REMINDER ( TODO =
feod
sparky
RECURRING DATE_ TIME GET RECURRING DATE, TIMR DATE,
with the
6 pm .FREQUENCY= =" daily ))
[Prediction]:
CREATE REMINDER TODO feed sparky
PERSON REMINDED me" ", FREQUENCY
duily AMOUNT== 6pm ")
Table 1: Error analysis of constraint violations made by the GPT-NeoX model on the TOPv2 dataset with ICL.,
often be difficult to interpret in Aialog-to-APl sce- narios. Hence, these metrics cannot help guide us to analyze particular test examples for insights in task-specific constraint violations, such as the lack of meaningful in-context examples, and the mod- els' ability comprehend general APl definitions and process compositionality. In this section, we first perform the exercise of analysing constraint vi- olations guided by our constraint violation metrics. Second, we leverage our analysis to hypothesize reasons for these violations.
from words in test utterances to existing function and arguments. While this could be due to lack of diverse vocabulary usage in the ICL prompts, enumerating all such semantic mappings in an ex- ample prompt may be unreasonable. In the second row, we observe that with the limited ICL exam- ples, the model is unable to comprehend the do- main of an argument/slot, but realizes the value '10 day' is important. Thus, it hallucinates a new argument and provides this value to it regardless of actual association constraints. This motivates us to consider constrained decoding strategies that restrict the generation space of function and argu- ment names thereby limiting hallucinations.
In Table 1, we highlight a few example violations along with the unique signature generated by our metrics and analyze the common reasons for such constraint violations.
Lack of meaningful in-context examples In the third row, we notice scenario that has more to do with the ICL paradigm as opposed to shortcom- ing of the LLM itself. While the model can learn from in in-oonext examples, lack of certain function (or argument) names in the prompts can derail the model to come up with incorrect (although seman- tically similar) function names. This motivates us to consider demonstration retrieval strategies.
Misunderstanding APIs and task semantics In the first row, we see an example where the gener- ated API call violates all of the task-specific con- straints. When we see high counts of this for a particular domain (eg. traffic) in TOPv2, we can quickly infer that the system fails to fully com- prehend the task semantics and is hallucinating function names, argument names, and making up associations between these hallucinated functions and arguments. In such scenarios, we often observe that the model is unable to make semantic jumps
Inability to handle compositionality In the fi- nal row, we highlight a failure mode that is ubiq- uitous in generated API calls across all domains

---

of TOPv2. Specifically, when dealing with compo- sitional queries (nested functional calls), the gen- erated API calls tend to flatten the call structure and bypass necessary nested API calls to correctly fulfil these function queries. In turn, these violate several task-specific API constraints. We further highlight these in our experiments ($5).
of meaning representation (instead of natural lan- guage or programming code) (Shin et al., 2021; Shin and Van Durme, 2022). The canonical form is then mapped to semantic parse expressions associ- ated with a pre-defined set of grammars written by humans. As we aim to generate API calls, instead of generating outputs that are then post-processed by human-written grammatical rules, we simply constrain the model generate valid expressions according the API constraints. For example, when the model generates the API call for a func- tion 2, we restrict the decoding algorithm to only generate arguments a tokens that are the dictio- nary indexed with x, Vfa(z]] We can also ensure we generate well-formed API calls by enforcing syntactic constraints (eg. parenthesis closure) and other task-specific constraints (function and argu- ments names). A minor detail is that the function and arguments names are often tokenized (by the pre-trained model's tokenizer) to a sequence of sub-tokens; thus, we need to, define all the valid se- quences of sub-tokens that form a valid function's 's (or argument's 's name. For this purpose, we use a dynamic programming approach with backtracking to extract all such possible sequences and restrict the LLMs generation according to the extracted sub-token sequences.
4 Mitigation Strategies
We study two popular mitigation strategies to re- duce constraint violations- (1) semantic retrieval approach for in-context example selection that im- proves the prompt design, and (2) an APl-aware constrained decoding strategy that constrains token generation based on APT information.
4.1 Semantic Retrieval of Demonstrations
Under the in-context learning approach, language models takes list of ouput- pairs in the prompt, referred to as in-context examples or demonstrations. In the context of our API genera- tion task, such demonstrations consist of pairs of an utterance and corresponding ground-truth API calls (u, a). In the examples shown in $3, these ex- ample demonstrations are randomly sampled from the same domain in the TOPv2 dataset and result in task-specific constraint violations. In this method, the key is to leverage the user's 's test utterance to strategically select the in-context demonstrations in the ICL prompt.
Experiments
Using the constraints defined in the previous sec- tion, we now benchmark the performance of in- context learning with large language models on constraint satisfaction. In addition, we use the pro- posed mitigation strategies, namely retrieval-based and buustraineddeecoiin--beerr
Consider user's utterance uy from the test dataset. We use a dense retriever model to obtain an embed- ding for the text u4- We then retrieve top-k (u4, a:) pairs from the training set based on the cosine sim- ilarity distance between u; and and arrange then in descending order to obtain the demonstrations for our prompt. For the dense retriever model, leverage SBERT that has been trained on one bil- lion sentence pairs using a contrastive learning ob- jective (Reimers and Gurevych, 2019).
and show their effectiveness in reducing the constraint violation rates compared to the basclines. Finally, we discuss important practical considerations that accompany the proposed mitigation strategies.
Large Language Model (LLM) To understand how well LLMs perform at generating API rep- resentation for utterances the TOPv2 datasets, we use the GPT-NeoX model (Black et al., 2022), which is a 20B parameter model trained on the Pile data consisting of > 825GB of raw text data (Gao al., 2020). Given the training data contains textual sources (eg. books, subtitles, emails, Wikipedia, etc.), and APl/code like data (eg. GitHub, stack exchange, hacker news), we find it befitting for our context. Other suitable LLM options for our setting were either under behind a
4.1.1 API-aware Constrained Decoding
In $2, we use the sets V,, Va, and Vja to define the set of possible values for function, arguments and associations for particular domain. Looking at the hallucinations during our analysis in $3, adapting the model's generation capabilities to adhere to the API-specification is proactive approach that can reduce constraint violations. We explore this direction in APl-aware constrained decoding.
Previous works have proposed a constrained de- coding algorithm that generates a "canonical form"

---

Sampler
Query Types
Constraints Violations Metrics (%) ())
Semantic Parsing Metrics (% (1)
C.
Cj
Ca
Cfa
Exact Match
Intent F1
Slot F]
Flat
0.63
7.6
2.77
9.6
34.7
75.4
45.3
Random
Compositional
0.67
12.8
3.36
15.4
7.0
61.8
34.8
All 0.64
9.8
3.02
12.1
22,9
68.0
39.4
SPIS-1
All
0.00
14.8
3.00
17.8
24.0
68.7
43.9
Table 2: Constraint violations metrics using ICL with GPT-NeoX on the TopV2 dataset. Using a SPIS-1 sampling strategy (that ensures coverage all functions and arguments) in the ICL examples improves semantic parsing metrics, but degrades constraint violation metrics (rows 3, 4). Also, models commit more constraint violations when dealing with compositionality (rows 1, 2).
paywall (eg. OpenAl Codex) smaller in scale (eg. GPT-Neo-2.7B, GPT-J-6B).
5.1 Constraint Violation with LLMs
In these experiments, we use two sampling strategies- random and SPIS-1 sampling- to sam ple 10 in-context examples from domain in the ICL prompt followed by a test utterance from the same domain. The SPIS-1 sampling ensures that every function and argument is represented at least once in the in-context examples, whereas the ran- dom sampler selects examples randomly from the training data. We construct the ICL prompt by concatenating the task descriptions, the in-context examples and the test query. For example, a prompt for the test query Driving directions to the Eagles game (from the navigation domain) looks as fol- lows:
Dataset To study the constraint violations from generating API calls in task-oriented settings, we use the TOD Semantic Parsing dataset TOP22 (Chen et al., 2020). The TOPv2 dataset contains pairs of utterances and aligned semantic represen- tations for diverse of source domains 3, The semantic representation can either be flat or com- positional in nature. For low-resource setting, the TOPv2 dataset restricts the training data using the samples per intent and slot label (SPIS) strategy (lower & scarce) in two domains- Reminder and Weather. For completeness of evaluation, we use the SPIS sampling method and obtain low-resource training datasets for all domains in TOPv2 (and will release the data for future research). For the datasets, we sample 200 examples from the test split for each of the eight domains TOPy2 (including 100 flat queries and 100 compositional queries in each domain).
#LTASK DESCRIPTION]
Follow the examples below and generate Calls from the users" ulterances
#LIN-CONTENT EXAMPLES]
Example 1:
User: What's traffic going
Saturday API Call: GET INFO INFO__RARAA { DATE_ TIME = "on Saturday
Example 2:
User: long the flight from Fort Lauderdale to Jamaica
Task Prompt Creation The in-context learning upproach allows language models to output predic- tions based on carefully constructed natural lan- guage text inputs (prompts). The prompt contains task description, followed by input-output demon- strations, and finally the test input, for which an output prediction is desired. We then ask the lan- guage model to take the prompt as input and output API calls for the test query by greedy decoding. While our task descriptions and test inputs remain consistent across prompts, consider different sampling strategies for composing in-context ex- amples the experiments below.
Call: ESTIMATED DURATION METHOD 'flight SOURCE = "Fort Lauderdale",
DESTINATION "Jamaica"
13
[10 examples]
#[TEST QUERY]
15
Example 11:
User: Driving dircctions
Eagles game
API Call:
Note that for this baseline, we do not consider any mitigation strategies. Table 2, we evaluate the corresponding output API call using the constraint violation metrics and traditional semantic parsing metrics.
In line with our observation in $3, observe higher rate of constraints violation for composi- tional queries in the first two rows of Table 2. In this case, the semantic parsing metrics offer a simi- lar conclusion, showcasing the poor performance of the model on nested queries compared to flat
"e run GPT-NeoX on a single machine with 8 V-100 GPUs, each 16GB memory, take hours com- plete in-context learning inference for our test examples. "alarm, event, messaging, music, navigation, reminder, timer, and weather

---

Sampler
Constraints Violations Metrics (%) ())
Semantic Parsing Metrics (% X1)
C.
Cy
Co
Cfa
Exact Match
Intent
F1
Random
0.64
9.80
3.02
12.10
22.9
68.0
39.4
SRD (ours)
+SPIS-5
0.00
3.49
1.00
6,20
37.7
81.4
52.2
+SPIS-10
0.20
2,60
0.70
4.84
40.6
82,4
55.7
+SPIS-25
0.10
1.84
1.10
4.34
43.2
86.0
58.9
+SPIS-50
0.10
1.10
1.00
2,88
44.4
86.7
60.4
+SPIS-100
0.40
1.20
0.89
2.96
49.6
88.2
63.5
Table Applying a query-basee Semantic Retrieval for Demonstrations (SRD) reduces both the constraint violation rates and improves the semantic parsing metrics. Increasing the training data subset from which SRD samples in-context examples improves results further.
queries.
lations with retrieval strategies start to reduce in magnitude after SPIS-50 although the increase in exact match continues at steady pace up to SPIS- 100. This indicates that constraints violations are more difficult to eliminate and necessitates the need for other mitigation strategies.
In contrast, observing the last two rows of the table (where we consider All query types); it shows the impact of different samplers, and therefore different set of in-context examples, on the in- context lcarning performance. Given SPIS-1's more comprehensive coverage functions and ar- guments compared to the random sampler, we ob- serve higher semantic parsing metrics (EMs, intent and slot Fls). However, this higher coverage does not necessarily equate to a lower constraint viola- tions rate- we notice higher violation rate for Cy and Cfa- Not only does this highlight the added di- agnostic value our metrics offer complementary to semantic parsing metrics, but also showcases that SPIS-1 sampling may not be sufficient to eliminate function name hallucination errors (see example in the third row of Table 1).
APl-aware Constrained Decoding (API-CD)
As shown in Table 4, our algorithm is able to elimi- nate all categories of constraint violations and im- prove on semantic parsing metries (although the magnitude of improvement is disproportionately less for the latter metrics). One may ask how can constraint violation metrics zero while semantic accuracy metrics are only a little better. For this, consider an example where the generated function name belongs to Vy but is not the correct function name as per the input text. In addition, we find that the constrained decoding algorithm is 20% slower than original decoding needs to extract at run- time the appropriate sub-sequences that form the well-formed function and/or argument names. Im- proving its efficiency can a promising future direction.
5.2 Efficacy of Mitigation Strategies
Semantic Retrieval of Demonstrations (SRD) Table shows that by simply applying semantic retrievals to obtain top-10 demonstrations exam- ples, we can reduce the constraint violations signif- icantly. We cannot expect the availability of the full training data in a few-shot setting, that is often the motivation for using ICL. Thus, we sample the top-10 10 demonstrations for semantic retrieval from a pool of SPIS sampled training set. While larger and more comprehensive training sets (higher SPIS) in- crease the effectiveness of our retrieval strategy in reducing constraint violations, we see gains even in the most extreme few-shot setting (SPIS-5). In addition to reducing constraint violations, the se- mantic retrieval is simple strategy to implement that only augments the input prompt. Further, also leads better performance according to the traditional semantic parsing metrics. However, we also observe that the reductions in constraint vio-
Combined Approaches (SRD + API-CD) Note that
combine our proposed methods seam- lessly as they are designed for different phases of ICL (1) SRD for prompt generation and, (2) API- CD for decoding.
the last row of Table 4, we observe that this approach yields the best results, where SRD primarily improves the Semantic Pars- ing Metrics (SPM) and API-CD improves the Con- straint Violation Metrics. Interestingly, the cases where API-CD improves the SPM are subset of the cases where SRD improves SPM. Thus, we see the same SPM numbers as the SPIS-5 row of Table 3.
For practitioners, we note that implementing SRD is both simple and incurs little latency com-

---

Sampling
Decoding
Constraints Violations Metries (%) (J)
Semantic Parsing Metrics (%)( (
C.
Cy
Ca
Cfa
Exact Match
Intent F1
Slot F1
Original
0.64
9.80
3,02
12.10
22.9
68.0
39.4
Random
API-CD (ours)
0.00
0.00
0.00
0.00
24.2
71.7
39.5
SRD + SPIS-5 (ours)
API-CD (ours)
0.00
0.00
0.00
0.00
37.7
81.4
52.2
Table 4: Applying API-aware Constrained Decoding (API-CD) eliminates all four categories of constraint violations (constraint violation rate = 0%) and improves the semantic parsing metrics. Combining both the semantic retrieval and constrained decoding reaps complementary benefits.
pared to random sampling. On the other hand, API- CD requires the API documentation to be fully available and can increase the latency costs by R 20% compared to the original decoding strat- egy. In the future, one can consider tighter in- tegration, where the knowledge obtained during the SRD phrase reduces the decoding output space during API-CD, thereby offering a speed-up. Un- Tortunately, the development complexity may out- weigh the latency cost benefits (and some semantic parsing metric improvement) in this case.
to use tools with external APIs to execute com- plex instructions. Schick et al. (2023) demonstrates that language models can learn to use external tools via simple APIs in self-supervised way by in-context learning. Similar our setup, the in- context learning approach (Brown al., 2020) has been considered for intent and slot filing (Yu al., 2021), semantic parsing (Shin et al., 2021), by pre- dicting English-like canonical representations first (Shin et al., 2021), meaning representations directly (Shin and Van Durme, 2022), and dialog dialog-to-AA to-API output (Shu et al., 2022). None of these works con- sider, like us, fine-graine constraint violation met- rics that can help decipher the failure modes of ICL in their settings. Hence, benefits of in-context example retrieval do not show the shortcomings of these approaches in mitigating constraint vio- lations (Shin et al., 2021; Liu et al., 2022; Rubin et al., 2022). While lexically constrained decoding has been beneficial in a wide variety of setting such as image captioning, machine translation, seman- tie parsing, and paraphrase generation (Anderson et al., 2017: Post and Vilar, 2018; Hu al., 2019; Li et al., 2020; Lertvittayakumjorn al., 202 12; Shin and Van Durme, 2022), dynamic adaptation of the generation lexicon by leveraging API defini- tions has not been investigated before.
6 Related Work
Generating semantic representation from natural language is beast with many heads (Zettlemoyer and Collins, 2012: Berant and Liang, 2014; Dong and Lapata, 2018; Wang et al., 2019). In our work, we look at a subclass of this problem where the in- put is a task-oriented utterance and the output takes the form of API call representations. Our input format aligns us closely to work in task-oriented semantic parsing (Gupta et al., 2018), task-oriented dialog (Mansimov and Zhang, 2022; Xuan, 2020), and intent classification and slot filling (Agha- janyan et al., 2020; Weld al.. 2022), while our output representation invites challenges similar to ones seen in generating executable semantic parses (Liang, 2016; Zhong et al., 2020), database queries (Berant and Liang, 2014; Li and Jagadish, 2014; Zhang et al., 2019; Yu et al., 2020; Choi et al., 2021), and AWS API calls (Shu et al., 2022). Our problem borrows the API-styled output, similar to Shu et al. (2022), but does not rely on an entire conversation or dialog as the input medium. To this extent, we enhance the existing TOPv2 dataset (Chen et al., 2020), leveraged in task-oriented set- tings, with API-styled output representations.
Conclusion
In this paper, we consider the problem of generat- ing an APl-call-styled representation given an input task-oriented utterance using in-context learning (ICL) approaches with Large Language Models. We propose a set of fine fineinnaine constraint viola- tion metrics that align with the API specifications and show that it lets diagnose finc- fine-grained fail- ure modes. Our analysis leads us investigate simple mitigation strategies than can be used along- side the ICL approach. query-based Semantic Retrieval of Demonstration ICL prompts helps obtain gains on traditional metrics while API-aware
With the recent popularity of Large Language Models, adapting them for various tasks with the abilitics to use different tools is becoming ubiqui- tous. Qin et al. (2023) fine-tunes language models

---

constraint decoding helps eliminate the proposed constraint violations. Coupling the two approaches with ICL yields the best of both worlds. Future works may focus on speeding up the mitigation approaches, and finding solutions that couple the upproaches in a tighter fashion.
Acknowledgements
We thank the AWS Al Lab team members for the feedback and discussion on the paper, and the anonymous reviewers for their helpful comments.
References
Limitations
Armen Aghajanyan, Jean Maillard, Akshat Shrivastava, Keith Diedrick, Michael Haeger. Haoran Li, Yashar Mehdad, Veselin Stoyanov, Anuj Kumar, Mike Lewis, and Sonal Gupta. 2020. Conversational semantic parsing. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP),, pages 5026-5033 Online. Association for Computational Linguistics.
Focus on the in-context learning approach We explored generating API representations with the in-context learning approach due to its wide ap- peals to practitioners in the low-resource settings. The in-context prompts allow the model to make use of examples in the low-resource settings, with- out the need to update model parameters. However, investigation in mitigating constraint violations is limited because when the practitioners are able to pre-train large language models, there may be other approaches, such as incorporating the constraints in the language model pre-training stage, that may help reduce constraint violations.
Peter Anderson. Basura Fernando, Mark Johnson, and Stephen Gould. 2017. Guided open vocabulary im- age captioning with constrained beam search. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 936- 945, Copenhagen, Denmark. Association for Compu- tational Linguistics.
Jacob Andreas. John Bufe, David Burkett, Charles Chen, Josh Clausman, Jean Crawford, Kate Crim, Jordan DeLoach, Leah Dorner, Jason Eisner, Hao Fang, Alan Guo, David Hall, Kristin Hayes, Kellie Hill, Diana Ho, Wendy Iwaszuk, Smriti Jha, Dan Klein, Jayant Krishnamurthy, Theo Lanman, Percy Liang, Christopher H. Lin, Ilya Lintsbakh, Andy Mc- Govern, Aleksandr Nisnevich, Adam Pauls, Dmitrij Petters, Brent Read, Dan Roth, Subhro Roy, Jesse Rusak, Beth Short, Div Slomin, Ben Snyder, Stephon Striplin, Yu Su, Zachary Tellman, Sam Thomson, An- drei Vorobev. Izabela Witoszko, Jason Wolfe, Abby Wray, Yuchen Zhang, and Alexander Zotov. 2020. Task-oriented dialogue dataflow synthesis. Trans- actions of the Association for Computational Linguis- tics, 8:556-571.
Focus on "hard" constraints Additionally, in this work, we focus on categorical constraints (e.g. structural constraints, function constraints, argu- ment constraints ...), which can be automatically detected without any ambiguity by checking the generated API calls against pre-defined dictionar- ies that stipulate the constraints according to API documents. Other forms of constraints, such as knowledge-driven constraints (Lervvittyakkumjrn et al., 2021b), which requires common-sense rea- soning, are not included in our study. Future works may investigate detecting and mitigation of con- straints that require external knowledge.
for
Jonathan Berant and Percy Liang. 2014. Semantic pars- ing via paraphrasing, In Proceedings of the 52nd An- nual Meeting of the Association for Computational Linguistics (Volume I: Long Papers), pages 1415- 1425.
Ethical Considerations
Sid Black, Stella Biderman, Eric Hallahan, Quentin An- thony, Leo Gao, Laurence Golding, Horace He, Con- nor Leaby. Kyle McDonell, Jason Phang, Michael Pieler, USVSN Sai Prashanth, Shivanshu Purohit, Laria Reynolds, Jonathan Tow, Ben Wang, and Samuel Weinbach. 2022. GPT-NeoX X-2B: An open- source autoregressive language model. In Proceed- ings of the ACI. Workshop Challenges & Perspec- rives Creating Large Language Models.
Our study makes use of the GPT-NeoX model, a large pre-trained language model trained on col- lections of internet text. Large language models like GPT-NeoX are known to generate text that may reflect and spread biases from their training data. Hence, we advise post-processing on the gen- erated outputs to remove generated content that is potentially offensive. Additionally, our constrained decoding strategy limits the output space and may therefore reduce the risk of using these language models.
Tom Brown, Benjamin Mann, Nick Ryder, Melanic Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, al. 2020. Language models are few-shot learners. Advances neural information processing systems, 33: 19711

---

Xilun Chen, Asish Ghoshal, Yashar Mehdad, Luke Zettlemoyer, and Sonal Gupta. 2020. Low-resource domain adaptation for compositional task- semantic parsing. In Proceedings of the 2020 Con- ference on Empirical Methods in Natural Language Processing (EMNLIP), pages 5090-5100, Online. As- sociation for Computational Linguistics.
Piyawat Lertvittayakumjom Daniele Bonadiman, and Saab Mansour. 2021b. Knowledge-drives slot con- straints for goal-oriented dialogue systems. In Pro- ceedings of the 2021 Conference of the North Amer- ican Chapter of the Association for Computational Linguisties: Human Language Technologies, pages 3407-3419, Online. Association for Computational Linguistics.
DongHyun Choi, Myeong Cheol Shin, EungGyun Kim, and Dong Ryeol Shin. 2021. Ryansql: Recursively applying sketch-based slot fillings for complex text- to-sql in cross-domain databases. Computational Linguistics, 47(2):309-332.
Fei Li and Hosagrahar Jagadish. 2014, Constructing an interactive natural language interface for relational databases. Proceedings of the VLDB Endowment, 8()):73-44
Zhongyang Li, Xiao Ding, Ting Liu, J. Edward Hu, and Benjamin Van Durme. 2020. Guided generation of cause and effect. In Proceedings of the Twenty- Ninth International Joint Conference on Artificial Intelligence, ICCA1-20, pages 3629-360 3629-3636. Interna- tional Joint Conferences Artificial Intelligence Organization. Main track.
Li Dong and Mirella Lapata. 2018. Coarse-to-fine de- coding for neural semantic parsing. arXiv preprint arXiv:1805.00 047933.
Leo Gao, Stella Biderman, Sid Black, Laurence Gold- ing. Travis Hoppe. Charles Foster, Jason Phang, Ho- race He, Anish Thite, Noa Nabeshima, et al. 2020. The pile: An 800gb dataset of diverse text for lan- guage modeling. arXiv preprint arXiv:2101.0002
Percy Liang. 2016. Learning executable semantic parsers for natural language understanding. Com- munications of the ACM, 59(9):68-76.
Sonal Gupta, Rushin Shah, Mrinal Mohit, Anuj Ku- mar, and Mike Lewis. 2018. Semantic parsing for task oriented dialog using hierarchical representa- tions. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 2787-2792 Brussels, Belgium. Association for Computational Linguistics.
Jiachang Liu, Dingham Shen, Yizhe Zhang, Bill Dolan, Lawrence Carin, and Weizhu Chen. 2022. What makes good in-context examples for GPT-3? Proceedings of Deep Learning Inside Out (DeelJ0 2022): The 3rd Workshop on Knowledge Extrac- tion and Integration for Deep Learning Architectures pages 100-1 14, Dublin, Ireland and Online. Associa- tion for Computational Linguistics.
Vivek Gupta, Akshat Shrivastava, Adithya Sagar, Armen Aghajanyan, and Denis Savenkov. 2022.
Elman Mansimov and Yi Zhang. 2022. Semantic pars- ing in task-oriented dialog with recursive insertion- based encoder. In AAAI.
RetroNLU: Retrieval augmented task-oriented se- mantic parsing. In Proceedings of the 4th Work-
shop NLP for Conversational AI, pages 184-196. Dublin, Ireland. Association for Computational Lin- guistics.
Ankur Parikh, Xuezhi Wang. Sebastian Gehrmann, Man- aal Faruqui, Bhuwan Dhingra, Diyi Yang, and Dipan- jan Das. 2020. ToTTo: A controlled table-to-text generation dataset. In Proceedings of the 2020 Con- ference on Empirical Methods in Natural Language Processing (EMNLP), pages 1173-1186, Online. As- sociation for Computational Linguistics.
J. Edward Hu, Huda Khayrallah, Ryan Culkin, Patrick Xia, Tongfei Chen, Matt Post, and Benjamin Van Durme. 2019. Improved lexically constrained decoding for translation and monolingual rewriting. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computa- tional Linguistics: Human Language Technologies, Volume I (Long and Short Papers), pages 839 -500, Minncapolis. Minnesota. Association for Computa- tional Linguistics.
Panupong Pasupat, Yuan Zhang, and Kelvin Guu. 2021. Controllable semantic parsing via retrieval augmen- tation. In EMNLP.
Matt Post and David Vilar. 2018 Fast lexically con- strained decoding with dynamic beam allocation for neural machine translation. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Hu- man Language Technologies, Volume 1 (Long Pa- pers), pages 1324, New Orleans, Louisiana. Association for Computational Linguisties.
Yushi Hu, Chia-Hsuan Lee, Tianbao Xie, Tao Yu, Noah A. Smith, and Mari Ostendorf. 2022. In- context learning for few-shot dialogue state tracking. ArXiv, 0a./22233.8566
Piyawat Lertvitayyakujjon, Daniele Bonadiman, and Saab Mansour. 202 Knowledge-drin driven slot con- straints for goal-oriented dialogue systems. In Pro- ceedings of the 2021 Conference of the North Amer- ican Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 3407-3419.
Yujia Qin, Shihao Liang. Yining Ye, Kunlun Zhu, Lan Yan. Yaxi Lu, Yankai Lin, Xin Cong, Xiangru Tang. Bill Qian, Sihan Zhao, Runchu Tian, Ruobing Xie, Jie Zhou, Mark Gerstein, Dahai Li, Zhiyuan Liu, and Maosong Sun. 2023. Toolllm: Facilitating large language models to master 16000+ real-world apis.

---

Hannah Rashkin, David Reitter, Gaurav Singh Tomar, and Dipanjan Das. 2021. Increasing faithfulness in snowledge-groundes dialogue with controllable features. In Proceedings of the 59th Annual Meet- ing of the Association for Computational Linguistics and the 11th International Joint Conference on Natu- ral Language Processing (Volume I: Long Papers), pages 104-718,m Online. Association for Computa- tional Linguistics.
Bailin Wang, Ivan Tito,, and Mirella Lapata. 2019. Learning semantic parsers from denotations with latent structured alignments and abstract programs. arXiv preprint 200.:4999004165.
Henry Weld, Xiaoqi Huang, Siqu Long, Josiah Poon, and Soyeon Caren Han. 2022. A survey of joint intent detection and slot filling models in natural language understanding. ACM Comput. Surv., 55(8).
Vikas Raunak, Arul Menezes, and Marcin Junczys- Dowmunt. 2021. The curious case of ballucinations neural machine translation. In Proceedings of the 2021 Conference of the North American Chap- ter of the Association for Computational Linguistics: Human Language Technologies. pages 1172-1183, Online. Association for Computational Linguistics.
Chaoting Xuan. 2020. Improving sequence- to io-sequence semantic parser for task oriented dialog. In Pro- ceedings of the First Workshop Interactive and Executable Semantic Parsing, pages 18 22, Online. Association for Computational Linguistics.
Dian Yu, Luheng He, Yuan Zhang, Xinya Du, Panupong Pasupat, and Qi Li. 2021. Few-shot intent classifi- cation and slot filling with retrieved examples. Proceedings of the 2021 Conference of the North American Chapter of the Association for Computa- tional Linguistics: Human Language Technologies. pages 734-749, Online. Association for Computa- tional Linguistics.
Nils Reimers and Iryna Gurevych. 2019. Sentence-bert: Sentence embeddings using siamese nert-networks. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing. Associa- tion for Computational Linguistics.
Subendhu Rongali, Luca Soldaini, Emilio Monti, and Wael Hamza. 2020. Don't parse, generate! se- quence to sequence architecture for task-oriented se-
Tao Yu, Chien-Sheng Wu, Xi Victoria Lin, Bailin Wang. Yi Chern Tan, Xinyi Yang. Dragomir Radev, Richard Socher, and Caiming Xiong. 2020. Grappa: grammar-augmmener augmented pre-training for table semantic parsing. arXiv preprint arXiv:2009. 13845.
mantic parsing. In The Web Conference 2020.
Ohad Rubin, Jonathan Herzig, and Jonathan Berant. 2022. Learning to retrieve prompts for in-context learning. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Tech- nologies, pages 2655-2671, Seattle, United States. Association for Computational Linguisties.
Luke Zettlemoyer and Michael Collins. 2012. Learn ing to map sentences to logical form: Structured classification with probabilistic categorial grammars. arXiv preprint arXiv: 1207. 200.11220. 1420.
Rui Zhang, Tao Yu, He Yang Er, Sungrok Shim, Eric Xue. Xi Victoria Lin, Tianze Shi, Caim- ing Xiong, Richard Socher, and Dragomir Radev. 2019. bhiiting-bsscs sqi query generation for cross- domain conext-dependen questions. arXiv preprint arXiv:1909.000 00786.
Timo Schick, Jane Dwivedi- Yu,, Roberto Dessi, Roberta Raileanu, Maria Lomeli, Luke Zettlemoyer, Nicola Cancedda, and Thomas Scialom. 2023. Toolformer: Language models can teach themselves to tools. ArXiv, ab$/2302.0476).
Richard Shin, Christopher Lin, Sam Thomson, Charles Chen, Subhro Roy, Emmanouil Antonios Platanios, Adam Pauls, Dan Klein, Jason Eisner, and Benjamin Van Durme. 2021. Constrained language models yield few-shot semantic parsers. Proceedings of the 2021 Conference on Empirical Methods in Natu- ral Language Processing, pages 7699-7715, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
Victor Zhong, Mike Lewis, Sida I Wang, and Luke Zettlemoyer. 2020. Grounded adaptation for zero- shot executable semantic parsing. arXiv preprint arXiv:2009.073%6
Richard Shin and Benjamin Van Durme. 2022. Few- shot semantic parsing with language models trained on code. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Tech- nologies, pages 5417-5425, Seattle, United States. Association for Computational Linguistics.
Raphael Shu, Elman Mansimov, Tamer Alkhouli, Niko- laos Pappas, Salvatore Romeo, Arshit Gupta, Saab Mansour, Yi Zhang, and Dan Roth. 2022. Daplog2api: Task-triented dialogue with api description and ex- ample programs. arxiv preprint arXiv:2212.09994

---

A Appendix
A.1 Flattened API Calls
Given an API call, we flatten it to list of function calls, where each function is represented by function name f and set of ungument-value pairs A (Table 5).
Users' Utterances Ground-Trru Truth API Calls
Flattened List Representation of the Ground-Truth API calls
[Utterance]]:
add reminder feed sparky 6pm daily
(F=CREATE, REMINDER, ;
fcround-truthj:
A1=
TODO:
"feed
sparky",
RECUR-
CREATE REMINDER (TODO "feed
RING RING_DATE DATE_TTIEEEEE TEEETTTEEETTRRRRRRINN DACCURRIGGDATTETTINGG )),
sparky", RECURRING DATE TIME
(/=GET RECURRING DATE TIME,
GET RECURRING DATE TIME DATE_ TIME DATE_TIME "6 pm". FREQUENCY "daily") )
A2= DATE_TIME: "6 pm", FREQUENCY: "daily" }),
[Utterance]:
What's traffic going be like Saturday
GET INFO TRAFFIC,
[Ground-truthy
1 DATE_ "on Saturday" })
GET GET_INFO_ LLL_II___TAAAAA TRAFFIC DATE_TIME= Saturday")
TIME = onn
Table 5: Examples of API calls and their flattened list representations


Reviews:
[{"review_id": "stsSUVjOsw", "reviewer": "Reviewer_56zM", "summary": "", "questions": "\n\n", "limitations": "", "rating": 3, "confidence": 3, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank reviewer 56zM for their insightful review!\n\nWe appreciate reviewer 56zM’s recognition of the value of our work in defining, analyzing and mitigating task-specific constraints violations in executable semantic parsing, which is not studied in previous works.\n\n> Weakness 1: This work only focused on in-context learning for one dataset. Results may not generalize to other datasets or different training methods.\n\n**In-context learning paradigm**: \nIn the scope of this work, we only focus on in-context learning. We acknowledge that other alternative learning paradigms may yield different outcomes. However, the in-context learning paradigm holds substantial importance and practical appeals in the context of executable semantic parsing because it facilitates transfer learning in low-resource semantic parsing scenarios and enables rapid prototyping without the need of re-training. Even though our emphasis is on in-context learning in this work, we hope our fine-grained evaluation framework of constraints violation analysis can inspire future research on executable semantic parsing, to **study the models’ ability to meet pre-specified constraints**, going beyond only relying on conventional evaluation metrics such as exact match.\n\n**Dataset**:\nWe use the TopV2 dataset since it is a popular task-oriented semantic parsing dataset, targeting the low-resource and few-shot learning setting. Also it can be readily converted to the utterance-to-API generation format. While an alternative viable dataset is presented by Shu et al 2022, we note that this dataset is small in size and not released yet.\n\n> Weakness 2: The study focuses on categorical constraints, specifically those that can be automatically detected by comparing generated API calls with predefined dictionaries, while excluding knowledge-driven constraints\n\nWe limit the focus of this work to categorical constraints (or hard constraints, where for each constraint satisfaction condition C, C is 0 or 1). Categorical constraints are crucial in executable semantic parsing since the generated output often needs to be machine-interpretable or executable according to predefined APIs. \nWe exclude the knowledge-driven constraints (soft constraints) from the scope of this study,  since studying such soft constraints may require more carefully curated datasets and benchmarks. We hope our work can inspire further investigations to follow our framework of formulating task-specific constraints-based metrics, rather than relying only on traditional string-match based metrics.\n\n### Reference:\nDialog2API: Task-Oriented Dialogue with API Description and Example Programs. Shu et al, 2022. arXiv:2212.09946", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "Addresses an important issue in applying in-context learning for executable semantic parsing - ensuring generated API calls respect task constraints and specifications. This is crucial for real-world deployment.\nProvides useful analysis and insights into why constraint violations occur in state-of-the-art LLMs using in-context learning.\nThe proposed metrics, analysis, and mitigation strategies could aid progress in applying LLMs to executable semantic parsing and other code/program generation tasks.", "reasons_to_reject": "The scope is quite narrow - focused solely on in-context learning for one dataset. Results may not generalize to other datasets or different training methods.\n\nThe study focuses on categorical constraints, specifically those that can be automatically detected by comparing generated API calls with predefined dictionaries, while excluding knowledge-driven constraints", "questions_to_authors": "\n\n", "paper_topic_and_main_contributions": "This paper focuses on measuring and addressing constraint violations in in-context learning for generating task-oriented semantic parsing utterance-to-API calls. \n1. Categorizing constraint types (structural, functional, argument, function-argument association) and creating metrics to gauge violations in generated API calls.\n2. Studying constraint violations in leading language models, highlighting issues like inadequate relevant prompts and challenges in handling compositionality.\n3. Introducing two solutions: semantic retrieval of demonstrations (SRD) for prompt formulation and API-aware constrained decoding (API-CD) during generation.\n4. Demonstrating that SRD and API-CD effectively reduce constraint violations, with the best outcomes arising from their combined use. The study also examines tradeoffs between complexity, efficiency, and effectiveness of these solutions.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "6ZTb1Ivj6V", "reviewer": "Reviewer_13ew", "summary": "", "questions": "See weakness section", "limitations": "", "rating": 3, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank reviewer 13ew for their insightful reviews, and also for recognizing the value of our work on defining task-specific constraints violations, and our error analysis on existing LLMs.\n\n> Is the presented list of constraints exhaustive? This work is focused on hard constraints.\n\nIn this work we focus on categorical constraints (or hard constraints). The presented list of constraints may not be exhaustive since we exclude the study of soft constraints (such as knowledge-driven constraints) from the scope of this work. However, we believe that the categorical constraints in this work are important in the context of executable semantic parsing, because these categorical constraints are crucial in ensuring that the generated outputs satisfy constraints from the pre-defined API and are therefore machine-interpretable (please also see the response to reviewer 56zM). \n\n> Weakness: The proposed strategies are specifically designed for in-context learning and are not generalizable to other learning paradigms.\n\nIn this work we only focus on in-context learning, which is particularly appealing for the executable semantic parsing in practice. While we acknowledge that different learning paradigms may yield different outputs, we hope that future work can adopt our evaluation framework, which takes into consideration task-specific constraints when evaluating model outputs (please also see the response to reviewer 56zM).", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. This work proposes API-CD which shows strong performance by achieving 0% constraint violation rate.\n\n2. In-context learning (ICL) is interesting approach to investigate for user utterances to API calls conversion.\n\n3. Interesting constraints are specified which are related to the overall output of the task\n\n4. The latest related work is referenced.\n\n5. Error analysis of different constraint violations by different LLMs is also discussed.\n\n6. Considering the in-context learning paradigm, this work is relevant.", "reasons_to_reject": "1. Is the presented list of constraints exhaustive? This work is focused on hard constraints.\n\n2. The proposed strategies are specifically designed for in-context learning and are not generalizable to other learning paradigms.", "questions_to_authors": "See weakness section", "paper_topic_and_main_contributions": "Executable task oriented semantic parsing is used to convert user's natural language utterances to executable API calls. LLMs can provide a strong baseline for this task but they have a tendency to hallucinate, posing a significant challenge to generating constrained content. This work is related to measuring, analyzing and mitigating the constraint violations while using ICL with LLMs. Authors propose to identify the categories of various constraints in obtaining API semantics from task oriented utterances along with defining fine-grained metrics. These metrics are used for error analysis of constraint violations by LLM. Finally, two popular mitigation strategies are investigated which are SRD and API-CD. This work experimentally shows that these strategies help reduce the constraint violations and improve the quality of generated API calls.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Missing_References": "NA", "Typos_Grammar_Style_and_Presentation_Improvements": "NA", "Justification_for_Ethical_Concerns": "NA"}, "extra_text": {}}, {"review_id": "m5dXenvsfN", "reviewer": "Reviewer_dJAL", "summary": "", "questions": "1. Why is the task of creating API calls different than the semantic parse task proposed in Chen et al. 2020. The can be deterministic transfer between the API calls and the semantic parse?\n\n2. Line 130, release the modified TOPv2 dataset - What are the modifications?\n\n3. Are the API definitions same for the entire trainings/test set ? If that is the case, why can't the constrained decoding techniques for semantic parsing not applicable here?", "limitations": "", "rating": 3, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank reviewer dJAL for their insightful reviews and for recognizing the value of our defined metrics on measuring constraints violations.\n\n> Weakness 1: The lack of new ideas for the mitigation strategies\n\nWe agree with reviewer dJAL that the investigated strategies do not introduce novelty. However, we clarify that we do not claim novelty on the mitigation strategies. Our work focuses on the following aspects.\n 1. We formally define task constraints and constraint violations in the utterance-to-API generation task; \n 2. We evaluate existing LLMs’ ability to satisfy task constraints and conduct a human analysis on the causes for constraint violations; \n 3. We investigate mitigation strategies chosen based on our error analysis. \n\nEven though the investigated strategies are not novel, we hope that future research can draw insights from Section 5, where we compare and offer advice on using these strategies in practice.\n\n> Question 1: Why is the task of creating API calls different than the semantic parse task proposed in Chen et al. 2020. The can be deterministic transfer between the API calls and the semantic parse?\n\nReviewer dJAL is correct that generating API calls is indeed equivalent to the executable semantic parsing task. Nonetheless we adopt the API call generation format for the following reasons:\n1. Given the surge in interest in “executable” semantic parsing (where pre-existing APIs are used to ground the generated output), the utterance-to-API format can make our definition on constraints and evaluation more widely applicable to approaches that operate on pre-existing APIs (e.g. tool-augmented LLMs)\n2. The utterance-to-API format (generating executable code based on natural language input) aligns more closely with the pre-training of state-of-the-art LLMs used, which are trained with code.\n\n\n> Question 2: Line 130, release the modified TOPv2 dataset - What are the modifications?\n\nWe modify the datasets by mapping intents and slots in the semantic parse representation to the corresponding functions and arguments in the API call representation. An example is provided in Fig 1, where a “semantic parse” (second row) can be modified to an “API call” (last row).\n\n\n> Question 3: Are the API definitions same for the entire trainings/test set ? If that is the case, why can't the constrained decoding techniques for semantic parsing not applicable here?\n\nYes, given a domain, the API definitions used remain consistent for the training / test set. We acknowledge that our constrained decoding strategy is indeed modified from constrained decoding techniques for semantic parsing, to satisfy pre-defined function and argument API constraints.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The paper introduces a set of metrics to assess constraint violations in the context of API semantic parsing, which includes API names, API arguments, and valid arguments within an API. These metrics prove to be highly valuable in understanding the performance of API semantic parsing systems. By quantifying constraint violations, the proposed metrics offer crucial insights into the effectiveness of the parsing process.\n\n2.  The paper presents two mitigation techniques to address these constraints. Firstly, the authors suggest fetching similar queries from the training set, which helps in reducing constraint violations. This approach leverages contextually relevant examples to improve API generation accuracy. Secondly, the paper proposes constrained decoding as a means to further mitigate constraint violations. This technique guides the generation process, leading to APIs that conform more closely to desired constraints. The effectiveness of these mitigation strategies validates the utility of the proposed metrics. \n\n3. The experimental results presented in the paper support the proposed metrics and mitigation techniques.", "reasons_to_reject": "1. The lack of new ideas for the mitigation strategies to remove the constraint violations. -The role of selecting relevant information for in-context learning is popular in the retrieval augmented generation paradigm. Constrained decoding is also a popular technique to reduce parsing errors in semantic parsing literature. \n", "questions_to_authors": "1. Why is the task of creating API calls different than the semantic parse task proposed in Chen et al. 2020. The can be deterministic transfer between the API calls and the semantic parse?\n\n2. Line 130, release the modified TOPv2 dataset - What are the modifications?\n\n3. Are the API definitions same for the entire trainings/test set ? If that is the case, why can't the constrained decoding techniques for semantic parsing not applicable here?", "paper_topic_and_main_contributions": "The authors present a set of improved metrics for measuring constraint violations in the domain of natural language to API semantic parsing. The research explores the use of these metrics within the context of in-context learning with large Language Models (LLMs). The main objective is to generate APIs that effectively solve tasks based on a given test natural language utterance and a limited set of training examples with ground-truth data.\n\nThe paper's contribution lies in its novel approach to evaluating constraint violations, which is crucial for refining the performance of natural language to API semantic parsing systems. By employing these metrics, the authors aim to enhance the accuracy and reliability of the API generation process.\n\nThe study introduces two potential solutions for mitigating constraint violations. Firstly, the authors suggest selecting in-context examples that closely align with the test utterance. This approach helps ensure that the generated APIs are relevant to the specific context and leads to more accurate results. Secondly, they propose the use of constrained decoding to guide the LLM generation towards producing valid API names and arguments. By imposing constraints during the decoding process, the authors anticipate an improvement in the adherence of generated APIs to the desired constraints.\n\nTo validate their proposed metrics and mitigation strategies, the authors conduct experiments on the TOPv2 benchmarks, a dataset introduced by Chen et al. in 2020. The results showcase the practical utility of the proposed metrics and the effectiveness of the mitigation techniques in real-world scenarios.", "excitement": 3, "reproducibility": 5, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Missing_References": "The related work looks solid.", "Typos_Grammar_Style_and_Presentation_Improvements": "Paper reads well."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper proposes metrics and subsequent mitigation strategies for constraint violations during utterance-to-api semantic parsing, focusing on LLMs and the in-context learning paradigm.

**Pros**: Reviewers agree on the relevance of the problem, and further, agree the paper provides valuable insights to the community, citing the proposed metrics, analysis, and mitigation strategies as having the potential to advance research.

**Cons**: Reviewers agree on a lack of generalizability and limited scope, citing limited datasets and application to only one learning paradigm. Author's rebuttal emphasizes a lack of available resources, and the importance of this particular paradigm, respectively. Reviewers acknowledge this argument, but the concerns remain. In the end, no reviewer finds the work particularly exciting - there is no champion.
Decision: Accept-Findings