Zero and Few-shot Semantic Parsing with Ambiguous Inputs
Authors: Elias Stengel-Eskin
Kyle Rawlins
Benjamin Van Durme
Venue: ICLR 2024 poster (2024)
Keywords: semantic parsing
text-to-code
ambiguity
NLP
calibration
Primary area: datasets and benchmarks
TL;DR: Ambiguity is pervasive in natural language but often is ignored when translating between natural and formal languages. We propose a new dataset and data framework for testing models on ambiguous language and test current models.

Abstract: Despite the frequent challenges posed by ambiguity when representing meaning via natural language, it is often ignored or deliberately removed in tasks mapping language to formally-designed representations, which generally assume a one-to-one mapping between linguistic and formal representations. 
We attempt to address this shortcoming by introducing AmP, a framework, dataset, and challenge for translating ambiguous natural language to formal representations like logic and code. 
We define templates and generate data for five well-documented linguistic ambiguities.
Using AmP, we investigate how several few-shot text-to-code systems handle ambiguity, introducing three new metrics.
We find that large pre-trained models perform poorly at capturing the distribution of possible meanings without deliberate instruction.
However, models are able to capture the distribution well when ambiguity is attested in their inputs. 
These results motivate a call for including ambiguity explicitly in datasets and promote considering the distribution of possible outputs when evaluating systems. We release our data and code.

Published as a conference paper at ICLR 2024
ZERO AND FEW-SHOT SEMANTIC PARSING WITH AMBIGUOUS INPUTS
Elias Stengel-Eskin' Kyle Rawlins? Benjamin Van Durme?
UNC Chapel Hill
Johns Hopkins University
ABSTRACT
Despite the frequent challenges posed by ambiguity when representing meaning via natural language, it is often ignored or deliberately removed in tasks mapping language to formally-designed representations, which generally assume one-to- one mapping between linguistic and formal representations. We attempt to address this shortcoming by introducing AMP, a framework, dataset, and challenge for translating ambiguous natural language to formal representations like logic and code. We define templates and generate data for five well-documented linguistic ambiguities. Using AMP, we investigate how several few-shot text-to-code systems handle ambiguity, introducing three new metries. We find that large pre-trained models perform poorly at capturing the distribution of possible meanings without deliberate instruction. However, models are able to capture the distribution well when ambiguity is attested in their inputs. These results motivate a call for including ambiguity explicitly in datasets and promote considering the distribution possible outputs when evaluating systems.
INTRODUCTION
Formalizing the meaning of natural language into symbolic representation has been attempted across a variety of domains, from philosophy and linguistics (Wittgenstein/ 1921] Montague 1970) to artificial intelligence (Winograd] 1972 Zelle & Mooney 1996). Attempts at formalization have often
faced shared challenge: many natural language statements have multiple possible meanings, i.e. they ambiguous. Past work (e.g. Zipf 1949 Piantadosi et al. (012)) has argued that this is natural feature of a communication system, resulting from competing pressures on speakers and listeners. Specifically, [Piantadosi et al. contend that ambiguity allows speakers to minimize their efforts. Rather than exactly specify their intended meaning (resulting in a long and expensive message), speakers can send shorter, cheaper messages and rely on listeners resolve any ambiguities. However, this resolution in turn relies on commonsense knowledge and conversational context: most speakers of English would infer from the utterance, "I ate spaghetti with a fork that someone used fork utensil, but commonsense knowledge would preclude this parse of "I ate spaghetri with meatballs" Similarly, conversational context can provide clues to help us choose between interpretations.
Language can be used not only to communicate with other people, but also to interact with AI agents. One common method for interaction is semantic parsing, whereby natural language translated into a formal and symbolic representation of its meaning (e.g. code, logic, graphs, etc.). However, human tools for ambiguity resolution may be unavailable to these non-human translation systems: models typically lack human-like commonsense knowledge and are missing conversational context. This could lead to miscommunications between humans and models. Since semantic parsing systems are used to perform real-world actions (e.g. modifying calendar, sending emails, controlling physical robots, etc.) ambiguity-bant based miscommunication in parsing could have real-world consequences.
Ideally, given ambiguous input, we would like our parsing models to capture distribution over interpretations with some uncertainty across plausible items in the distribution. This would allow robust handling of ambiguous utterances for example by enabling smart follow-up interactions (Stengel-Fikiin & Van Durmel 2023) getting us closer to the goal of using language as a general- purpose API for interaction. Given that semantic parsing systems are typically based on language models - which represent distributions over strings - combined with search procedure (c.g. beam
Data and code; httpge///ggtta...cc/eeeee///gggp
Contact: estengdca. unc. edu

---

Published as a conference paper at ICLR 2024
The boy saw the man with the telescope
3i Be. (s) is manly A telescope(s) soweaass agent(a, 2r A patient(a, 2)A instrament(e.a)
(exists (exists y (exists exists a (AND (boy x) (nan y) (telescope 2) (sau (agent x) (patient y) (änstrument a z)]))))
3r 3x.3y.3r..o..o
(exists x (exists (exists exists (exists (AND (boy x) (nan y) (telescope 2) (saw (agent x) (patient y) (have e) (agent e x) (patient z)
A man(f) A felescope(s)
A sow(a] A ag))d(u,ax)
A putieut(a..) have(e)
A agent(e, A patient(e,
>)>>))>
Figure 1: An example of prepositional phrase (PP) attachment ambiguity. The statement is compatible with two possible interpretations, represented visually, in first-order logic, and in Lisp format.
search) it could be that models already capture ambiguity. However, this hypothesis is hard to test given current semantic parsing datasets, which typically commit single interpretation for each utterance. To this end, we introduce an extensible framework and dataset for investigating ambiguity in semantic parsing. Our framework consists of templates covering five well-documented types of natural language ambiguity: prepositional -phrase attachment, scope and inverse scope, pronominal coreference, and conjunctions. For each type, our templates can generate large numbers of ambiguous and unambiguous utterances. Each ambiguous utterance is paired with two possible interpretations, or logical forms (LFs); see Fig. for an example. LFs can be represented as first-order-lgic (FOL) formulae or programs in Lisp. We use our framework to create benchmark dataset we call AMP (Ambiguous Parsing). Unlike past efforts which have grounded ambiguous utterances in answers to questions SStengll-skinn et al.. 2023), language inferences (Liu et al. 2023), videos (Berzak et al.
2013). or images (Mchrabi et al., 2022). we focus our dataset on semantic parses. This choice follows from several motivating factors. For one, semantic parsing has a long tradition of use in interactive
systems, including in robotics (Kate et al., 2005) Tellex et al., 2011 Artzi & Zettlemoyer 2013
Tellex et al. (2000). guestion-answertg
(Zelle & Mooney| 1996; Berant et al. [2013] Yu et al. 2018).
and digital assistants (Semantic Machines et al. [0200] Damonte et al. 2209)). )) Ensuring that these
systems capture ambiguity and that their confidence reflects appropriate uncertainty about the user's intent is crucial, as misunderstandings could have negative real-world consequences (Stengel- & Van Durme 2023). Furthermore, semantic parsing not only allows people to access computation, but also provides a way for models to use external tools: for example, simple forms of semantic parsing have been employed to augment large language models (LLMss) (Parisi et al. 2022] Schick et al., 2023 Mialon et al. 2023). Finally, long- form text-to-code shares many challenges with parsing.
Using our generated AMP data, we introduce a pair of challenging tasks designed for LLMs using in-context learning (ICL). In ICL, rather than explicitly training models to predict LFs, we provide models with instructive examples in a prompt, which is prepended to the test input. This parsing setting has become increasingly popular in semantic parsing (Shin et al.) 2021 [Shin & Van Durme 2022. Roy et al.) 2022). Our tasks aim to quantify how well existing models capture ambiguity and to provide framework for improving their ability predict multiple meanings. We develop 3 metrics to measure models' performance on ambiguity in two settings: zero-shot and mixed prompt.
In the zero-shot setting, we provide models with the "ingredients" to produce both possible derivations of a given ambiguity type, but we provide no examples of that ambiguity type; see Appendix A.2 for an example. In this unique compositional generalization challenge, the model must combine structures into novel derivations and also recognize that the structures can be selected and combined in two ways produce different derivations. We also annotate a subset of our data with crowdsoureed judgements, comparing these to our models' predictions. Models struggle to predict parses correctly in this setting. When they do compose parses correctly, although models and people tend to choose similar interpretations, models generally fail to predict both possible parses.
In the mixed prompt setting, we examine how model distributions and outputs change when varying the number of examples for each interpretation in the prompt. For each ambiguity type, we construct "mixed prompts" consisting of confficting examples. Some examples shown to the model pair utterances of ambiguity type with one kind of LF, and others pair the same kinds of inputs with the alternative LF. This setting is motivated by a case in which ambiguity might lead to conflicting annotations in training dataset; when examples are retrieved from that data construct prompt for
2

---

Published as a conference paper at ICLR 2024
Type Prep. phrase at- tachment (PP)
Ex. Input
L.oo
1.F
The man saw the boy with the telescope
de Jy. Hz 3o. Je. man(x) A
Jz. da..y...aa......... w...z.aaa.aan Ja.man())
boy(v) (sow(() (a) Aagent(a, x) patientla, A telescope(2) A have(e)
boy(y) A telescope(z) saw(())
agent(a, patient(a,
agent(e, y)
patient(e, 2) Interpretatio:
instrument(a, 2) Interpre-
the man saw the boy, who was holding a telescope.
tation: the man used a telescope lo see the boy.
Quantifier scope
every cow saw a dog
Jz.wy.3a.cow)())
dog(z) A saro(a) A agent(a, y) Interpretation:
(z.-y.)..coo)(z) A dog(y) saw(a) A agent(a, x) A patienl(a.g) (a.y) Interpretation: there may be more than one dog. Vz. be..y.da.cowfl saws(a) (a) A agent(a,y) y) patient(a,z) Interpretation: there may be more than one cow.
(Scope)
patient(a, x)
there is exactly one dog.
Reversed, or inverse
a cow saw every dog
Zz. N...y.a.co..l
dogly) A (gent(a,z) A
dog(z)
sano(a) patient(a, y)
scope
Interpretation:
(revs-
there is exactly one cow.
cope)
Pronoun coref-
Mary
saw the woman and she
lx. Ba. Je..aawo))))
Fz. 3a.3a. N...Ba.Be..ooo (e..oo)a((z)
sano(a) A agent(a, Mary) A patient(a, x.) A smiled(e) A
savc(a) A agent(a, Mary) A patient(a, A smaled(e) A agent(e, x) Interpretation: the woman smiled.
erence (bound)
smiled
agent(e, Mary) Interpretation:
Mary smiled.
Conjunction
the man drank and
He. man.)))
A
Ax. Re. tamman())
A
(conj.)
ale or swam
((drank(a) A ag,nt(a, x) A
(drank(a) A agent(a, x) A ((ate(e) A a))n (swam()) A agent(i, x))))
agend(e,x)) V (swarn (i) (x)en((i,z))) Inter- pretation; the man either drank and ate or he swam.
ale(e)
Interpretation; the man drank, and he either ate or swam.
Table I: Ambiguity types considered with example inputs and L.F. See Appendix A.1.1 for more description, including the lexical items used.
a test example, the resulting prompt will also contain conflicting parses. Here, our metries measure to what extent model represents the distribution in its input given conflicting evidence. Some models perform remarkably well here, aligning with the prompt distribution across ambiguity types. To our knowledge, this is the first study of in-context learning with confficting evidence.
2 METHODS
Data We introduce a dataset of ambiguous parses, where natural language examples are parsed into first-order logic (FOL). Further details the construction of our logical forms (LFs) can be found in Appendix A We can canonicalize our LFs,, so that logically equivalent formulae with varying syntax are treated identical: we transform LFs into binary trees, where nodes are ordered alphabetically, and we anonymize variables. Note that when prompting our model, we do use standard variable set and order, where the variables F, y, z are used for nouns, and a, e, i are used for events. Our canonicalization process also allows us to render our LFs in different formats. In addition to standard FOL format, we experiment with a Lisp format (cf. Fig. [.. For machine- readabiiity we always render logical connectives in plaintext, i.e. 3 becomes exists, A becomes AND, etc.. We consider five types of syntactic and semantic ambiguities, given Table
Models Semantic parsing tasks are often framed as sequence transduction, where a model learns to translate text into LFs by training on paired data (Dong & Lapatal 2016; Zhang et al., 2019). It has become clear that neural models can capture distributions they are trained on; thus, if we were to train on ambiguous data, it would not be surprising if the model captured ambiguity, and vice-versa. Rather than training models, we instead consider several models for in-context learning (ICL), focusing on large pre-trained autoregressive (AR) language models. We use the Codegen series of models (Nijkamp et al., 2022) 350 million (M), 2 billion (B), 6B, and 16B parameters which are based on the GPT-2 architecture (Radford et al. 2019) and are pre-trained on large amounts examine Llama- 13B (Touvron et al. 2023), an open-source AR transformer. To examine the impact of instruction tuning (Wei et al. 2022
of code and text We also use LLMs pre- treitrained on text; here,
consider Vicuna-13BB (Chiang et al. 2023). which uses prompts distilled from ChatGPT to
A We can also generate unambiguous examples, and can extend AMP to new ambiguity types/vocab items. Past work (Shin & Van Durme 2022) has shown that code pretraining improves over text pre-training on other ICI. semantic parsing tasks.
3

---

Published as a conference paper at ICLR 2024
instruction finettne Llama- 13B. All models above 350M were run at fp16 precision. In the zero-shot setting, we also consider OpenAl's apt-3.5-turbo. While few details about the model are known, is a large AR transformer model which has undergone both instruction tuning and fine-tuning from human feedback (Ouyang et al., 2022). It is often the most performant model; however, the API does not provide access to logit scores, precluding analyses of uncertainty. As such, we only use it in our zero-shot experiments, where the metric is securacy-based rather than uncertainty-batt based. For non-API models, we used constrained decoding (Shin et al. 2021 Shin & Van Durme 2022 Roy et al. 2022 to ensure the model only produces valid logical statements; see Appendix A.3 for details.
Computing probability under forced decode In our analyses, we would like to compare the probabilities the model assigns to LFg and LF, While one could compare the product of probabilities under the model for each LF, we find that in practice, this results in very low scores for either LF, We instead use Stengel- Eskinn & Van Durme 2022) sequence confidence estimate to obtain (o(LFo). renormalizing the end:
1. We obtain token-level probabilities under forced decode of L.oo and LF For each token & in an LF with tokens f .1...ty LN. we compute (o(ts]z; f1=-1), where f15-1 is the gold token prefix and is the input prompt.
2. We take p min N (44 1), the minimum probability across all tokens 9
3. We normalize the probabilities: Po(LF)) = PL.. +PLF;
Prro
and set Pa(LFi) = 1 - (LFF)
2.1 METRICS
Zero-shot metries In the zero-shot setting, aim to measure the degree to which the model captures both possible interpretations of an ambiguous statement. Intuitively, when given the "ingredients" to make both interpretations, model that robustly captures ambiguity should allocate some probability to both. We measure this by computing the proportion of elements for which the model has both interpretations in its top-k predictions. Note that we remain agnostic here to the exact probability of each interpretation; we aim instead to quantify whether it predicts both interpretations at all. Note also that as we increase k. this metric becomes less stringent. Let Tie be the top most probable predictions from the model under some sampling method (e.g. beam search), and I be an indicator function. The zero-shot metric ZM; is given by Eq. (L. This metric counts how often both I.Fs are found in the top & outputs averaged across examples E [1, NJ. It ranges from 0 to 100, and higher better, it indicates more examples have both LFs in their top & outputs.
(T[LFoo E TT) I[F, E Tx))
ZM
100
(I)
N
Few-shot metrics In the few-shot setting, we concerned about the level to which the model is capturing the distribution given in the prompt. A core assumption here is that an ideal model would perfectly capture the uncertainty in the given distribution. We consider metrics at two levels of granularity to evaluate this behavior. The first metric we consider measures model performance at the level of the dataset. Intuitively, as we sweep across ratios r E R, we expect the proportion of predicted LFs to match r. For example, when r = 0.10 (meaning that 10% of the prompt examples are I.Fo and 90% are I.F;) we would expect the model to produce I.Fo in roughly 10% of instances. Let y, be the predicted I.F for input instance rg Then the fewshot dataset metric FDM is given by Eq. (2). Intuitively, this measures the difference between the accuracy on each LF and the ratio of that LF: lower is better for FDM, which ranges from 1.0 to 0.0. The second metric measures model performance at the level of individual datapoints. the model is capturing the distribution in the prompt, then the probability assigned to LFo should roughly match r, e.g. if 0.10, the model should assign ((LF) & 0.10. The few-shot instance metric FIM is given by Eq. 33. FTM resembles Brier score (Brier et al. 1950) and measures the error between the predicted probability and the ratio; it also ranges from 1.0
to 0.0 and lower is better.
N
N
FDM-
Σ
Mm = LFa)) - - 1(
Σ Mlu = LFi]) - (1 (1-r))
(2)
I
FER
i=1
FIM
1
Σ FER
Σ
(Po(y: = LF))-r)? -)))
-
R
"We also experimented with averaging, which resulted in similar results. We chose min to he consistent with Stengel-Fssknn & Van Durme 2022).
4

---

Published as a conference paper at ICLR 2024
3 EXPERIMENT 1: ZERO-SHOT PARSING
For each ambiguity type, we construct a prompt that provides the ingredients for deriving both LFs. The order of the component sentences is shuffled to avoid biasing the model towards one interpretation the other. Crucially, the prompt contains no examples of the types of sentences being tested. For example, for PP attachment, the model is given an example of how to parse transitive verbs ("the boy saw the man "), instruments ("the boy saw with the telescope"), and possessives ("the boy with the telescope") "), each in isolation. To successfully generalize, the model has to overcome two challenges: first, it must compositionally generalize to compose the ingredients in the prompt into a valid derivation. Secondly, it must recognize the ambiguity and reflect both derivations in its output. For each ambiguity type, we test 200 examples. Prompt examples are given in Appendix A.2
3.1 ZERO-SHOT RESULTS AND ANALYSIS
In Fig. 2 smaller Codegen models (350M, 2B) struggle to predict either LF correctly. On some ambiguity types, larger Codegen models (6B, 16B) predict one LF correctly. However, most models fail to ever predict both L.Fs correctly; exceptions to this are conjunction and coreference ambiguities, where we some models correctly predicting LFo for some examples and LF, for others. GPT--3.5 does well at predicting LFi for PP and scope ambiguities, and predicts both LFs for coreference and conjunctions. Interestingly, while Llama-13B is unable to correctly predict either LF for any of the ambiguities, Vicuna-138 (Llama's 's instruction-turn suggesting that instruction tuning helps the model predict one LF correctly (though not to capture ambiguity). Separately, we find that predicting FOL generally outperforms Lisp; for example, the Codegen-2B model on scope predicts L.F1 correctly 18% of the time when using FOL, and only 11% when using Lisp; we report only FOL results moving forward.
variant) is comparable to Codegen- Maltegen--6BB 16B,
LFO Accuracy
LFI Accuracy
100
75
PP
revscope
conj
50
scope
bound
25
g.5-3.5 eg-350M cg-2B cg-6B 11-1BB vi-13B cg- spt-3.5 Figure 2: Zero-shot exact-match accuracy on ambiguity types. Cg = Codegen, LI Llama, Vi Vicuna. Models increasing in size from left to right.
og- 35-350M cg-2B cg-6B
11-13B vi-13B 16
These results are further underscored in Table 2 showing the ZM, values for all models. We see that all models tested perform very poorly on this metric, with most models scoring 0.0 in most settings. Qualitatively, we find that the model's 's top 5 outputs tend to include variations of the same LE. This finding aligns with the probability results seen in Fig. 3 where models tend to assign extreme probabilities to LFs on 3 out of 5 ambiguity types. Notable exceptions here are conjunction and bound pronoun types, where in Fig. models assign closer to 0.5 probability to each parse; we see this also reflected in Fig. 2 and Table 2. where models predict both LF, and LF; correctly some of the time. As a whole, these results underscore the difficulty of the compositional task we have proposed; while some models are able obtain high accuracy on one LF in isolation (GPT-3.5 predicts I,F, for PP attachment and scope almost perfectly) no model is able to consistently predict both interpretations.
We can also ask whether token-level confidence reflects the ambiguity in the space of possible parses. Examining task-oriented semantic parsing models, Sttenel--Ekkiin & Van Durme (2022) find that many models (including Codegen) are relatively well-calibrated the token level, meaning their confidence aligns with their average accuracy. Taking token-level probabilities as confidence scores, we follow their analysis and ask whether models are well-calibrated w.r.t. alternative parses. Specifically. we compare the model's 's confidence on tokens at the points where the predicted and alternative parse diverge. This is visualized for each ambiguity type in Fig. 4 Here, we take the first correctly- predicted correctly-predictedd LF (either LF,, or LF,)) from Codegen-16B, predicted via beam search with grammar-conee constraieed decoding (not forced decoding). We overlay the confidence onto the token as the background color (darker is more confident). Below each predicted parse, we give the alternative parse. For scope and inverse scope, the model assigns low confidence to the quantifier tokens at the start of the formula, which are in the reverse order in the alternative parse. Similarly, the tokens involving the quantified variables have lower confidence. We also see low confidence around the area of divergence for conjunction. However, pronominal coreference and PP attachment lack such
5

---

Published as a conference paper at ICLR 2024
Model
Pp
Scope
Revscope
Bound
Conj.
1.0
cg-350M
0.00
0.00
0.00
0.00
1.00
cg-2B
0.00
0.00
0.00
0.50
0.00
cg-6B
0.00
0.00
0.00
0.00
3,50
0.5
cg-16B
0.00
0.00
0.00
3,50
15.00
PP
bound
scope
11-13B
0.00
0.00
0.00
0.00
0.00
rovscoge
com
vi-13B
0.00
0.00
0.00
4.00
9.50
gpt-3.5
0.00
0.00
0.00
0.00
0.00
cg-350M
eg-28
cg-6B
1-13B
vi-13B
98-68
Table 2: ZM; for all models (eg = Codegen). Models
Figure 3: P(LFo) per model.
generally fail to predict both LFs.
ccape (w): CemA an MISKEE M sccate 2 Sorkel Mi prisenser)
DERArNNI IN WO powSTOR CJ AnD cantertin, 57
scone (bi: cxists forall exists a boy(s1 AND orj an) AN oeservadia) ADD agentca, 5) AND poticctia, al
revscape (a): Deteed D farntl D DesTTs R BaveR [ND Eventeriyt tM Exteettn bo consitaa ( 22 pattestin, MT
revecape (B): foratl a. essnts exssts ssyly! AND eventer(s) Me spianta) MD agestce, y) M0 potiertla,
(N):
EEREEC (quneiny tomntre pastentio. bory) (ostedee) basetin
(b):
muss intiin agent(a, satientia, Nay) Led agent II monie Ii txmatee
con) (al: BNICE
stapeees catta) AND dranicaa AR agentin, CR stals) AND agentre, slayed(1) AND agectis, snints
cond (bI: ussnts extnts
esiste
(a): WRISEE 1
camerater
cuntel
lopantii Netbonl 2:ll instrmmntion DJ Mill puttentio [otticc)
(b2: exists AESets
ecists
colerack) AN cM)()) MS ageeste, Wetsor) MD patseetis, ontstee) AND havete) MND sgentie, datslee) wa pattentle, xl
Figure 4: seoo per-token probability (darker is more probable) for each ambiguity type. Alter- native parse given below each predicted parse. Token probability sometimes reflects divergences between the parses.
interpretable confidence changes. These results are promising: for some ambiguity types, the model's token-level confidence reflects the alternative parse.
3.2
HUMAN VALIDATION
Fig. 2 indicates that models tend to produce one interpretation or the other - when models have non-zero accuracy on one interpretation, they tend have zero the other. Psycholinguistic research suggests that people have preferred interpretations (AnderBois et al.) 2012 Dwivedi) (2013); at the aggregate level, Fig. shows that models align with human preferences on scope ambiguities.
AnderBois et al. 2012) and Dwivedi (201)) also describe strong lexical effects in scope ambiguity, meaning that the choice of words in the example has an effect on the interpretation taken. In order to further examine how the models tested compare with these results, we annotate a subset of our validation examples with human interpretations and confidence scores. This allows us to compare model predictions to humans at an item-level in addition to an aggregate level.
Annotators were asked to choose between interpretations and provide a confidence score on a sliding scale, following the EASL protocol (Sakaguchi & Van Durme 2018); the confidence score was then converted to a probability (cf. Appendix A.5 for details). Since annotators are unlikely to know FOL or Lisp, each LF is shown as a statement that clearly indicates the interpretation (as in Table D. For example, for a PP attachment example like, "the boy saw the man with the telescope". the verbalized interpretations are "the boy saw the man, who had/was wearing telescope" and "the boy saw the man and used a telescope to do so". For each ambiguity type except conjunctions, we randomly select 20 examples from our development splits 5] Each example is annotated by 3 annotators.
We find that annotators disagree almost as often as they agree: 38 examples have disagreement while 42 have all 3 annotators agreeing. This is positive finding, indicating that examples are highly ambiguous. Fig. 5 (left) shows confidence scores (averaged across 3 annotators) for each item (sorted separately by mean confidence for each ambiguity type). There are broad preferences for all ambiguity types except PP attachment: bound and inverse scope tend to be matched to L.o,, and scope to LF1. The latter aligns with some past findings AnderBois al. (2012); Caramazza et al.) (1977 For PP attachment, some inputs are confidently parsed as LT% and others as LF
In Fig. 5 (right), we contrast the human results with the output of the Codegen- 16B model. While the probabilities generally match in direction to the human annotations (except for PP-attachment) we
"Conuunction ambiguitics were excluded due to difficulties verhalizing their interpretations fluently. 'Other work has found linear order to have negligible effect and pointed to additional factors influencing interpretations (Kurtzman & MacDonald 1993 Dwivedi 2013

---

Published as a conference paper at ICLR 2024
1.0
1.00
scope
revscope
0.75
PP
bound
0.75
P(LFD)
P(LFD)
0.50
0.50
ww awwddng
0.25
0.25
0.0
0.00
Figure 5: Per-example probabilities derived from humans (left) and cg-16B (right) on LFs. Examples are sorted by probability. Human probabilities vary according to vocabulary choice, but model probabilities generally do not.
do not see the same kind of item-level sensitivity. For most PP, scope, and inverse scope examples, we see the model assigning all examples the same extreme probability. For bound pronouns, we see more variation, with the model switching between LFg and LF1; however, the predictions are fairly extreme. These results suggest that the model is poorly-calibrater w.r.t. ambiguity at the item level.
4 EXPERIMENT 2: FEW-SHOT PARSING
Ambiguities may lead to similar inputs being paired with different logical forms. Increasingly, is common to retrieve examples from training set to compose a prompt for ICL. If that train- ing set has ambiguity in it. it is likely that the retrieved prompt would contain conflicting exam- ples, e.g. some examples pairing an utterance type with LFo and others pairing it with LF.. In our few-shot experiments, we seek to fill this gap by investigating how model confidence and accuracy change at different prompt ratios. Crucially, by investigating mixed prompts with biguous inputs, we are ensuring that the disagreement in the prompt is not due to simple mis- takes; one could imagine mixed prompt arising from noisy data, where instances are mislabeled. In such cases, a strong enough model may even learn to ignore mislabeled data in the prompt. However, the case of ambiguity, there are multiple legitimate interpretations.
bound
100
scope
For each ambiguity type, we construct prompts by pairing sentences of the same type with L.Fo in some cases, and LF, in others; we run 100 examples per type, per ratio. Each prompt contains 10 input-LF pairs, and a different prompt is constructed for each test sentence. We vary the number of LFo sentences in the prompt from to 10 increments (e.g. 0 100%) and shuffle the prompt sentences ensure that there is no positional bias.
75
25
25
50
75
100
Perc 1 prompt
Figure 6: Fewshot acc. increases according to the ratio of the LF in the prompt. LF; acc. (not pictured) decreases accordingly.
Few-shot results and amalysis Fig. shows the L.Fo
accuracy of models
increase the per- centage of LFo in the prompt for the Codegen-2B model. We see that for scope and inverse scope, the accuracy tracks almost perfectly with the percentage. For other ambiguities, the accuracy is correlated with the percentage but never reaches 100%. Table 3 shows the F/M and FDM scores for all models across all ambiguity types. Recall that FDM measures how well the model's accuracy aligns with the percentage of examples for each LF in the prompt: to obtain lower F FDM score, the model needs predict each LF at roughly the rate that it is seen in the prompt. Note that model that fails to predict either LF correctly will have high FDM score. For example, because Llama- 13B fails to predict any the LFFs correctly, has an FDM score of 1.00 for all ambiguities. For Codegen models, FDM generally improves with model size most ambiguity types. Overall, several models achieve fairly low FDM, especially on scope and inverse scope.
We also see that all FIM scores are relatively low. Because FIM uses the gold LF to extract the sequence probability, even models with poor accuracy on both LFs can have a fairly low FIM. In other words, FIM presents the model with forced choice between two parses, rather than evaluating the model's most probable generations like FDM and ZM do. For example, Llama-13B never produces correct LF under beam-search decoding with constraints, but when probabilities
7

---

Published as a conference paper at ICLR 2024
extracted with a forced decode of the gold LFs, they align fairly well with those in the prompt, and it often attains lower FIM scores than other models, including Vicuna. FIM remains fairly constant with model size: Codegen-168 16B is tied with smaller models on 4/5 ambiguity types. Taken together, these results indicates that the models tested are surprisingly good at capturing the distribution in the prompt. The low FDM some types indicates that overall, models like Codegen- 16B and Vicuna-13BP 13B produce LPO roughly at the rate that it appears in the prompt. However, we see that low FIM does not imply low FDM,, since low FDM requires the model be accurate.
Interestingly, the models seem to override the zero-shot tendencies seen in Fig. 3 where scope and inverse scope were strongly associated with one LF over the other. With direct evidence on how to parse scope sentences in the prompt evidence we did not provide in the sero setting the model produces the interpretations seen in the prompt, and is especially close to the prompt distribution for scope and inverse scope. These results are promising: given that models seem to capture mixed prompts well, it could be that ambiguity poses less of a challenge in settings where such mixed prompts can be constructed, i.e. settings with ambiguity in the training data.
Model
PP
Scope
Revscope
Bound
Conj.
FDM
FIM
FDM
FIM
FDM
FIM
FDM
FIM
FDM
FIM
cg-350M
0.76
0.08
0.19
0.05
0.36
0.06
0.62
0.16
0.60
0.06
cg-2B
0.51
0.09
0.19
0.03
0.18
0.03
0.48
0.10
0.39
0,05
cg-6B
0.45
0.08
0.21
0.05
0.16
0,06
0.43
0.10
0.41
0,06
cg- 16B
0.35
0.08
0.20
0.03
0.21
0.03
0.38
0.10
0.27
0,06
11-13B
1.00
0.06
1.00
0.04
1.00
0.04
1.00
0.06
1.00
0.05
vi-13B
0.50
0.08
0.17
0.05
0.20
0.06
0.26
0.07
0.35
0.09
Table 3: Few-shot metrics for all models (lower=better). FDM (Eq. 2 measures the extent to
which the model's accuracy across the whole dataset matches the percentage of that LF in the prompt. FIM (Eq. (3) measures how well the model's uncertainty captures the prompt's uncertainty.
RELATED WORK
Ambiguity has been a longstanding topic of interest in linguistics and psycholinguistics. Past work has argued that it is feature arising naturally from the trade-off between competing objectives (Zipi) 1949. Schutze 1995 Piantadosi et al. 2012). However, providing a systematic account of ambiguity in linguistics lies beyond the scope of this section.
Some work in NLP has focused on modeling ambiguity in visual contexts, where questions and statements have been paired with images or videos depicting situations they refer to. |Stengel-Eskin Eskin jet al. (2023) introduce a dataset of linguistically ambiguous questions about images as well as a model for question disambiguation, and Futeral et al. (2022) examine ambiguous source sentences in machine translation, providing disambiguating images. More akin to our work, [Bezaak et al. (2015) introduce corpus of syntactic, semantic. and pragmatic ambiguities with video interpretations. Follow-up work by Mchrabi et al. (2022) generates images ambiguous statements. We use many of the same ambiguities, but represent meaning with LFs instead of videos images. This is motivated in part by the relative ease of checking the correctness of a logical formula over, for example, an image. Ambiguity has been studied in more general tasks such as guestion-answering (Min et al. 2020), natural language inference (NLI) (Liu et al.] 2023). and coreference resolution (Yuan et al. 2023). where models have broadly been found lacking in their ability to resolve ambiguities. In parsing specifically. Rasmussen & Schuler (2020) introduce X-calculus dataset on 2,000 sentences of simple Wikipedia text, where roughly 50% contain quantifier scope ambiguity. We use synthetic data instead, giving us greater control and allowing us to examine more ambiguity types.
DISCUSSION AND CONCLUSION
In addition to semantic parsing's 's many downstream applications, has often been used measure models' compositional generalization abilities through synthetic benchmarks like COGS (Kim & Linzen 2020) and SCAN (Lake & Baroni 2018). The ability to generalize systematically and compositionally to unseen combinations is core component of human intelligence (Fodor & Pylyshyn 1988). Past efforts have generally assumed that there is a single correct LF for any given input, either implicitly (Lake & Baroni 2018) explicitly (Kim & Linzen 2020, Appendix H). This assumption is not borne out in natural language, where statements can be ambiguous and have multiple meanings. It is also violated in many common applications of semantic parsing, such as
8

---

Published as a conference paper at ICLR 2024
text-to-code, where there myriad ways of producing logically equivalent programs. Making this assumption reduces semantic parsing to syntactic parsing, since there is a 1-to-I mapping between syntax and meaning
| In future work, we hope to improve on the challenging and novel compositional task proposed in Section where all models struggle to capture both meanings.
Section 4 Joffers more hopeful takeaway: when ambiguity is present in the input, many models are able to capture the distribution of LFs. Of course, for ambiguity to be attested in the prompt, must exist in the data used construct the prompt. Furthermore, it needs be attested in the evaluation data for us to test for it. However, in most current datasets, parses are not annotated redundantly or exhaustively, i.e. inputs are paired with a single output. Given that annotators often disagree on ambiguous examples (cf. Section (.2) it is crucial to obtain multiple judgements, at least on evaluation data |3) This has recently become more common in other domains, such as NLI (Chen et al. 2020 Nie et al.) 2020; Pavlick & K Kwiatkowski 2019). Even when items are annotated redundantly, disagreement has often been discouraged or treated as noise. More recent work has begun to recognize that disagreement can arise for valid reasons (Pavlick & Kwiatkowski 2019) including ambiguity (Bhattacharya et al. 2019 Stengel-Eskin et al. 2023] Liu et 2023). To improve the handling of ambiguity, we advocate for extending redundant, ambiguity-ani aware annotation protocols (with attention to disagreement) from single-label tasks (e.g. QA, NL.) to complex, sequential outputs like semantic parsing. Improving both zero-shot generalization and data collection would help models capture the full range of utterance interpretations. This could lead to robust, interactive systems in which agents ask for confirmation or clarification on ambiguous examples (Stengel-Eskin & Van Durme, 2023), ultimately improving safety for critical systems.
Finally, ambiguous utterances are underspecified, i.e. they lack the requisite information to decide which interpretation is correct. Some past work carries underspecification into the meaning represen- tation: Copestake et al. (2005) introduces Minimum Recursion Semantics, which leaves noun- bracketing (similar to conjunction ambiguities) and scope underspecified in the target representation, allowing multiple interpretations to be recovered. Bos (2004) introduce a discourse representation that also maintain scope ambiguities for later resolution. We have opted for more fully-specified representation, placing the onus of resolution onto the parsing model rather than the representation.
Limitations Firstly, we are limited by committing to particular logical form. To test the parsing abilities of models under ambiguity, we are forced to choose a fixed meaning representation (MR) form, possibly including suboptimal abstractions and design choices. Motivated by /Wu et al. (2023). who find that arbitrary choices in an MR's construction can hamper compositional generalization, we have limited the number and difficulty of our choices, and mitigated their effect by offering two output formats. It is also important to point out some key differences between the models we test and the relevant psycholinguistic literature. Broadly, experiments indicate that people maintain multiple interpretations during online processing, later settling on one interpretation (Lackner & Garrett 1972] Rayner al. 1983 Filik al. 2004). Our models do not do online processing, and receive input as text, not audio. Similarly, we do not provide a conversational context, which people generally use to resolve ambiguity. Thus, our results should not be taken reflect how people process language.
Methodologically, we are limited by our use of a fixed set of English-only ambiguities. We hope to add more languages, lexical items, and ambiguity types via our extensible data framework. Many optimizations have could be made for ambiguity, c.g. decoding strategies that emphasize diversity (temperature sampling, sequentially decoding outputs, etc.) might result in better results for the ZM metric, since models lack output diversity. In our experiments we attempted to mimic how semantic parsing is commonly done in practice, without optimizing for ambiguity specifically.
By introducing a new benchmark for parsing under ambiguity, we are able to examine how modern semantic parsers handle cases where utterances can have multiple meanings. To this end, we introduced three new metrics for measuring the extent to which models capture the distributiion of meanings. While we find that models struggle to compose symbols without explicit guidance, we also find that they are sensitive the ambiguity when given mixed prompts, suggesting that having ambiguity in the training data may be a sufficient condition for capturing it in the output. This motivates our call for capturing ambiguity during annotation.
Conclusion
7 For example, Rudinger Van Durme 2014) found that one of the key features separating dependency- syntax and event-hased semantics was the ability to handle PP attachment ambiguitics.
Foor training data, we may he able to obtain diverse judgements instead, with examples of given ambiguity rype paired with different outputs, was done in Section 4

---

Published as a conference paper at ICLR 2024
7 ETHICS STATEMENT
By creating sentences templatically, we ensure that AMP does not contain any harmful texts, This protects our human annotators from exposure such inputs; we also compensate the annotators at level substantially above US federal minimum wage, and in-line with living wage estimates. Like most NLP research, our work has the potential to contribute dual-use use. However, we believe that overall, making models robust to ambiguity will contribute to safer and more reliable technology, and has limited potential for negative applications.
REPRODUCIBILITY STATEMENT
In order to further reproducibility, we release our dataset and our code. This includes the code for AMP, which can be extended to generate new ambiguities, well as the code for running all experiments. We do make use of a closed-source model (ChatGPT), which can hinder reproducibility; hedge against this, most of our results are based on open-source models that widely available. We also include our prompts in the appendix.
ACKNOWLEDGEMENTS
This work has been supported in part by the U.S. National Science Foundation under grant No. 2204926. Elias Stengel-Eskin was supported by an National Science Foundation Graduate Research Fellowship.
REFERENCES
Scott AnderBois, Adrian Brasoveanu, and Robert Henderson. The pragmatics of quantifier scope: corpus study. In Proceedings of Sinn und Bedeutung, volume 16, pp. 15-28, 2012.
Yoav Artzi and Luke Zettlemoyer. Weakly supervised learning of semantic parsers for mapping instructions to actions. Transactions of the Association for Computational Linguistics, 1:49-62, 2013.
Yoav Artzi, Kenton Lee, and Luke Zettlemoyer. Broad-coverage cog semantic parsing with amr. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pp. 1699-1710, 2015.
Jonathan Berant, Andrew Chou, Roy Frostig, and Percy Liang. Semantic parsing Freebase from question-answar pairs. In Proceedings of the 2013 conference on empirical methods natural language processing, pp. 1533-1544, 2013. 2013.
Yevgeni Berzak, Andrei Barbu, Daniel Harari, Boris Katz, and Shimon Ullman. Do you see what mean? Visual resolution of linguistic ambiguities. In Proceedings of the 2015 Conference on Empirical Methods Natural Language Processing, pp. 1477-1487, 2015.
Nilavra Bhattacharya, Qing Li, and Danna Gurari. Why does a visual question have different answers? In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 427 4271-4280, 2019.
Johan Bos. Computational semantics in discourse: Underspecification, resolution, and inference. Journal of Logic, Language and Information, 13:139-157, 2004.
Glenn W Brier et al. Verification of forecasts expressed in terms of probability. Monthly weather review, 78(1):1-3, 1950.
Alfonso Caramazza, Ellen Grober, Catherine Garvey, and Jack Yates. Comprehension of anaphoric pronouns. Journal of verbal learning and verbal behavior. 16(5):601-609, 1977.
Tongfei Chen, Zhengping Jiang, Adam Poliak, Keisuke Sakaguchi, and Benjamin Van Durme. Un- certain natural language inference. In Proceedings of the 58th Annual Meeting of the Association
10

---

Published as a conference paper at ICLR 2024
for Computational Linguistics, pp. -7799, Online, July 2020. Association for Computa- tional Linguistics. doi: 10.18653//1/2020..s maa.:.0..7.mmmmmm777........ URL URLLittps://aacaattallog wwcpl/oooo.ooo...goooo..o 2020
acl 1all-main.777
Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng. Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph E. Gonzalez, lon Stoica, and Eric P. Xing. Vicuna: An open-source chatbot impressing gpt-4 with 90%* * chatgpt quality, March 2023. uttpt
///msys.org/blog/ 2aalll///22200000 2023-03---2200 1000--00030-viuu-- 30-vicuna/
Ann Copestake, Dan Flickinger, Carl Pollard, and Ivan A Sag. Minimal recursion semantics: An introduction. Research on language and computation, 3:281-332, 2005.
Marco Damonte, Rahul Goel, and Tagyoung Chung. Practical semantic parsing for spoken language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Industry
Papers), pp. 16-23, 2019.
Li Dong and Mirella Lapata. Language to logical form with neural attention. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 33-43, Berlin, Germany, 2016. Association for Computational Linguistics. doi: 10.18653/v1/P16 1004.
Veena Dwivedi. Interpreting quantifier scope ambiguity: Evidence of heuristic first, algorithmic second processing. PloS one, 8(11):e81461, 2013.
Ruth Filik, Kevin B Paterson, and Simon P Liversedge. Processing doubly quantified sentences: Evidence from eye movements. Psychonomic Bulletin Review, 11(5):953 959, 2004.
Jerry A Fodor and Zenon W Pylyshyn. Connectionism and cognitive architecture: critical analysis. Cognition, 28(1-2)33-71, 1988.
Matthieu Futeral, Cordelia Schmid, Ivan Laptev. Benoît Sagot, and Rachel Bawden. Tackling ambiguity with images: Improved multimodal machine translation and contrastive evaluation. arXiv preprint 12.1::22/2.101444. 2022.
Rohit J Kate, Yuk Wah Wong, Raymond J Mooney, et al. Learning to transform natural to formal languages. In AAAI, volume 5, pp. 1062-1068. 2005.
Najoung Kim and Tal Linzen. COGS: compositional generalization challenge based on semantic interpretation. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 9087-9105, Online, 2020. Association for Computational Linguistics. doi: 10. 12..1665511122222mmmmmmmm......
URL httpp:://aclantholoogy wwwp:///.oooo.../2000./0// wtg//020.gemaaeeee.oo emnle- main
731
Howard S Kurtzman and Maryellen C MacDonald. Resolution of quantifier scope ambiguities.
Cognition, 48(3):243-279, 1993.
James R Lackner and Merrill F Garrett. Resolving ambiguity: Effects of biasing context in the unattended ear. Cognition, 1(4):359332,, 1972.
Brenden M. Lake and Marco Baroni. Generalization without systematicity: On the compositional recurrent networks. In Jennifer G. Dy and Andreas Krause (eds.),
skills of sequence-to-sequenee Proceedings of the 35th International Conference on Machine Learning, ICMI. 2018, Stock- holmsmässan, Stockholm, Sweden, July 10-15, 2018, volume 80 of Proceedings of Machine Learning Research, pp. 2879-2888, PMLR, 2018. URL http::/pproceeeiinggg mlr www.ppees..//ee........... press/v80/
lake18a. htmm
Alisa Liu, Zhaofeng Wu, Julian Michael, Alane Suhr, Peter West, Alexander Koller, Swabha Swayamdipta. Noah A Smith, and Yejin Choi. We're afraid language models aren't modeling ambiguity. arXiv preprint arXiv:2304. /4399, 2023.
Ninarch Mehrabi, Palash Goyal, Apurv Verma, Jwala Dhamala, Varun Kumar, Qian Hu, Kai-Wei Chang. Richard Zemel, Aram Galstyan, and Rahul Gupta. the elephant flying? resolving ambiguities in text-to-image generative models. arXiv preprint arXiv:2211 12503, 2022.
11

---

Published as a conference paper at ICLR 2024
Grégoire Mialon, Roberto Dessi, Maria Lomeli, Christoforos Nalmpantis, Ram Pasumuru, Roberta Raileanu, Baptiste Rozière, Timo Schick, Jane Dwivedi- Yu, Asli Celikyilmaz, et al. Augmented language models: a survey, arXiv preprint 233.v:2302.078842, 2023.
Sewon Min, Julian Michael, Hannaneh Hajishirzi, and Luke Zettlemoyer. AmbigQA: Answering ambiguous open-domain questions. In Proceedings of the 2020 Conference Empirical Methods in Natural Language Processing (EMNLP), pp- 5783-5797, 2020.
Richard Montague. English formal language. Linguaggi nella Societa e nella Tecnica, 1970.
Yixin Nie, Xiang Zhou, and Mohit Bansal. What can we learn from collective human opinions on natural language inference data? In Proceedings the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP). Association for Computational Linguisties, 2020.
Erik Nijkamp, Bo Pang, Hiroaki Hayashi, Lifu Tu, Huan Wang, Yingbo Zhou, Silvio Savarese, and Caiming Xiong. Codegen: An open large language model for code with multi-tum program synthesis. arXiv preprint arXiv:2203. 3374,, 2022.
Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang. Sandhini Agarwal, Katarina Slama, Alex Ray, al. Training language models to follow instructions with human feedback. Advances in Neural Information Processing Systems, 35: 27730-27744, 2022.
Aaron Parisi, Yao Zhao, and Noah Fiedel. Talm: Tool augmented language models. arXiv preprint arXiv:2205.12255 2022.
Terence Parsons. Events in the Semantics of English: A Study in Subatomic Semantics. MIT Press, 1990.
Ellie Pavlick and Tom Kwiatkowski. Inherent disagreements in human textual inferences. Transac- tions of the Association for Computational Linguistics, 2647-694, 2019.
Steven T Piantadosi, Harry Tily, and Edward Gibson. The communicative function of ambiguity in language. Cognition, 122(3):280-29) 291, 2012.
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. Language models are unsupervised multitask learners, 2019.
Nathan Ellis Rasmussen and William Schuler. A corpus of encyclopedia articles with logical forms. In Proceedings of the 12th Conference on Language Resources and Evaluation (LREC 2020), 2020.
Keith Rayner, Marcia Carlson, and Lyn Frazier. The interaction of syntax and semantics during sentence processing: Eye movements in the analysis of semantically biased sentences. Journal of verbal learning and verbal behavior, 22(3):358-374, 1983.
Subhro Roy, Sam Thomson, Tongfei Chen, Richard Shin, Adam Pauls, Jason Eisner, and Benjamin Van Durme, BenchCLAMP: A benchmark for evaluating language models on semantic parsing. arXiv preprint arXiv:22066 10668, 2022.
Rachel Rudinger and Benjamin Van Durme. Is the stanford dependency representation semantic? In Proceedings of the Second Workshop EVENTS: Definition, Detection, Coreference, and Representation, pp. 54-58, 2014.
Keisuke Sakaguchi and Benjamin Van Durme. Efficient online scalar annotation with bounded sup- port. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 208 218. Melbourne, Australia. July 2018. Association for Compu- tational Linguistics. doi: 10. 100.18653/v1// 1/t33//1/P11/// 1020. URL
Timo Schick, Jane Dwivedi- Yu. Roberto Dessi, Roberta Raileanu, Maria Lomeli, Luke Zettlemoyer, Nicola Cancedda, and Thomas Scialom. Toolformer: Language models can teach themselves to use tools. arXiv preprint arXiv:2302.0476// 2023.
12

---

Published as a conference paper at ICLR 2024
Hinrich Schutze. Ambiguity in language learning: Computational and cognitive models. Stanford University, 1995.
Semantic Machines, Jacob Andreas, John Bufe, David Burkett, Charles Chen, Josh Clausman, Jean Crawford, Kate Crim, Jordan DeLoach, Leah Dorner, Jason Eisner, Hao Fang. Alan Guo, David Hall, Kristin Hayes, Kellie Hill, Diana Ho, Wendy Iwaszuk, Smriti Jha, Dan Klein, Jayant Krishnamurthy, Theo Lanman, Percy Liang, Christopher H. Lin, Ilya Lintsbakh, Andy McGovern Aleksandr Nisnevich, Adam Pauls, Dmitrij Petters, Brent Read, Dan Roth, Subhro Roy, Jesse Rusak, Beth Short, Div Slomin, Ben Snyder, Stephon Striplin, Yu Su, Zachary Tellman, Sam Thomson, Andrei Vorobev, Izabela Witoszko, Jason Wolfe, Abby Wray, Yuchen Zhang, and Alexander Zotov. Task-oriented dialogue as dataflow synthesis. Transactions of the Association for Computational Linguisties, 8:556 571, September 2020. URL https: org/ 1112///ccle_0000000
Richard Shin and Benjamin Van Durme. Few-shot semantic parsing with language models trained on code. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pp. 5417-5425, 2022.
Richard Shin, Christopher Lin, Sam Thomson, Charles Chen Jr. Subhro Roy, Emmanouil Antonios Platanios, Adam Pauls, Dan Klein, Jason Eisner, and Benjamin Van Durme. Constrained language models yield few-shot semantic parsers. Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pp. 7699-7715, 2021.
Mark Steedman. Taking scope:
The natural semantics of quantifiers. Mit Press, 2011.
Elias Stengel-Eskin and Benjamin Van Durme. Calibrated interpretation: Confidence estimation in semantic parsing. arXiv preprint arXiv:221 2022.
Elias Stengel-Eskin and Benjamin Van Durme. Did you mean...? Confidence-based trade-offss in semantic parsing. arXiv preprint arXiv:2303. 00..303..66.... 16857, 2023.
Elias Stengel-Eskin, Jimena Guallar-BB Blascoo Yi Zhou, and Benjamin Van Durme. Why did the chicken cross the road? Rephrasing and analyzing ambiguous questions in vqa. Proceedings of the 61sl Annual Meeting of the Association for Computational Linguistics, 2023.
Stefanie Tellex, Thomas Kollar, Steven Dickerson, Matthew Walter, Ashis Banerjee, Seth Teller, and Nicholas Roy. Understanding natural language commands for robotic navigation and mobile manipulation. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 25, pp. 1507-1514, 2011.
Stefanie Tellex, Nakul Gopalan, Hadas Kress-Gazit, and Cynthia Matuszek. Robots that use language. Annual Review of Control, Robotics, and Autonomous Systems, 3:25-55, 2020.
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.1397.1 2023.
Siddharth Vashishtha, Benjamin Van Durme, and Aaron Steven White. Fine-grained temporal relation extraction. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pp. 2906-2911 Florence. Italy, 2019. Association for Computational Linguistics. doi: 10.18653/v1/P19. 1280. URLLAttee://ccaaattooogg..u wwwp:ooy...or//----1111-9
Jason Wei, Maarten Bosma, Vincent Zhao Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M Dai, and Quoe V Le. Finetuned language models are zero-shot learners. In International Conference on Learning Representations, 2022.
Terry Winograd. Understanding natural language. Cognitive psychology, 3(1):1-191, 1972.
Ludwig Wittgenstein. Tractalus
Annalen der Naturphilosophie, 1921.
Zhengxuan Wu, Christopher D Manning, and Christopher Potts. ReCOGS: How incidental details of logical form overshadow an evaluation of semantic interpretation. arXiy preprint arXiv:2303.13716 2023.
13

---

Published as a conference paper at ICLR 2024
Tao Yu, Rui Zhang. Kai Yang. Michihiro Yasunaga, Dongxu Wang, Zifan Li, James Ma, Irene Li, Qingning Yao, Shanelle Roman, et al. Spider: A large-scale numan-labeles dataset for complex and cross-domain semantic parsing and text-to-sql task. In Proceedings of the 2018 Conference Empirical Methods in Natural Lanuagge Processing, pp- 3911-3921, 2018.
Yuewei Yuan, Chaitanya Malaviya, and Mark Yatskar. AmbiCoref: Evaluating human and model sensitivity to ambiguous coreference. In Findings of the Association for Computational Linguistics: EACL 2023, pp. 993-1900 2023.
John M Zelle and Raymond J Mooney. Learning to parse database queries using inductive logic programming. In Proceedings of the national conference artificial intelligence, pp. 1050-1055,
1996.
Sheng Zhang, Xutai Ma, Kevin Duh, and Benjamin Van Durme. Broad cooer-c semantic parsing as transduction. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP- IJCNLP), pp. 3786-3798, Hong Kong, China, 2019. Association for Computational Linguistics. doi: 10.18653/v 1/D19-1392. URL
George Kingsley Zipf. Human Behaviour and the Principle of Least Effort. Addison- Addison-Wesley Press, 1949.
A APPENDIX
A.1 DATASET CONSTRUCTION
We take ao-Daavdsonian event semantics approach (Parsons 1990) to our logical forms (LFs), expressing our logical forms in quantified first-order logic (FOL). Events are represented as variables, with event-type predicates applied to them. For example, the statement a woman walks would be represented as 3 Fe..e..orr (o..ee.woma((2)) A walk(() A agenl(e, x), allowing for arbitrary number of semantic roles; the semantic roles covered our dataset are agent, patient, and instrument. Generic noun phrases like "a dog" existentially quantified: 3x.dog(z). Proper nouns are assumed have a single referrent, and are not quantified, c.g. Mary walks + Je.walks(e) agent(e, Mary).
A.1.1 LEXICAL ITEMS
For PP ambiguity, we pair visual verbs (e.g. see, observe, spot, etc.) with visual instruments (e.g. telescope, binoculars, etc.) and tactile verbs (e.g. grab, pick up, etc.) with things that can be worn/possessed and used for manipulation (e.g. gloves, ovenmills, tongs, etc.). For scope and reverse scope, we use common nouns and visual and tactile verbs. For pronoun coreference, the lexical items used here are gendered names (e.g Mary, John, etc.) and gendered nouns (c.g. woman, man, boy, girl). Conjunction examples use intransitive verbs.
A.1.2 EXISTENTIAL QUANTIFICATION
For uses of "the", we differ from Kim & Linzen 2020), who use the notation for definite articles to denote uniqueness clause. This is implemented as a existential quantifier at the widest scope, which can be ignored all cases except scope ambiguity, where we only have Similarly, Artzi et al. (2015) introduce Skolem terms (Steedman) 2011) for definite NPs, which are also globally scoped. Thus, we do not differentiate between definite and indefinite NPs AMP. This has no impact ambiguity.
ceefiiitely-qunntfitet NPs.
A.1.3 EXTENDING TO NEW TEMPLATES
The framework we release allows for the addition of new templates and lexical items. To add new template, the user specifies a surface-form template and an LF template, and provides the set of lexical items that can be used to fill slots in the templates. The framework enumerates all possible combinations of lexical items which respect the template constraints and produces paired inputs and L.Fs.
14

---

Published as a conference paper at ICLR 2024
A.2 ZERO-SHOT PROMPTS
PP Attachment For PP attachment, we show an example the main verb being used transitively, the instrumental of "with", and the accompaniment of "with".
Let's translate what human user says into what computer might say.
Human: Galileo Mary
Computer: exists a saw(a) agent (a, Galileo) AND patient Mary)
Human: Mary with the camera
Computer: exists x exists camera(x) AND have (a) AND agent (a, Mary) AND patient (a. x)
Human: Galileo saw with the camera Computer: exists x exists a camera(x) AND saw(a) AND agent Galileo) AND instrument(e, x)
Human:
Galileo saw Mary with the camera
Computer:
Conjunctions For conjunction ambiguities, include example of double conjunction (e.g. and and) and double disjunction (e.g, or or). The bracketing can vary.
Let's translate what human user says into what computer might say.
Human: the bird left and walked and ate
Computer: exists x exista a exista exista i bird(x) AND ( (ef AND agent(a, x) AND walked(e) AND agent(e, x) AND ate(i) AND agent(i, x)
Human: the bird left or walked or ate
Computer: exists x exists a exists exists i bird(x) AND ( left(() AND aa, x) ) OR ( walked(e) AND agent(e, x) ) OR ate(i) AND (i, x)
Human: the bird left or walked and ate
Computer:
Bound pronouns For pronoun coreference, show the transitive verb and each possible subject with the embedded verb separately. We also include an example of a subject with two verbs so that the model sees how compose verbs in the same sentence.
Let's 's translate what human user says into what computer might say.
the Computer: exists x exists woman (x) AND saw (a) AND agent(a, AND patient Marie)
Human:
Marie
Marie amiled Computer: exists smiled(a) AND agent (a, Marie)
Human:
Human: the woman frowned and smiled
Computer: exists x exists exists woman(x) AND frowned(a) AND agent (a, x)
AND smiled(e) AND agent(e, x)
Human: the woman smiled
Computer: exista x exists a woman (x) AND smiled(a) AND agent (a, x)
Human:
the Marie and she smiled
Computer:
15

---

Published as a conference paper at ICLR 2024
Scope ambiguity prompts include an example of the verb being used transitively, as well an example universal quantification.
Scope
Let's translate what human user says into what computer might say.
Human: a bird held a sweater
Computer: exists x exists exists bird(x) AND sweater (y) AND held(a) AND (a, x) AND patient(a, y)
Human: each bird
Computer: forall x bird(x)
Human:
each bird held sweater Computer:
Inverse scope Inverse scope prompts include the same information scope ambiguities but with reversed arguments.
Let's translate what human user says into what computer might say.
Human: a dog spotted a hat Computer: exists x exists exists dxg(x) AND hat (y) AND (aotted AND agent(a, x) AND patient(a, y)
Human: each hat
Computer: forall x hat (x)
dog spotted each hat Computer:
Human:
A.3 CONSTRAINED DECODING
For locally-run models, 2022; Roy et al. 2022) to ensure that the model produces syntactically-core decoding,
grammar- constrrined decoding (Shin al. 2021 Shin & Van Durmel
formulae. During the BenchCLAMP framework (Roy al., 2022) restrict the model's output vocabulary according to context-free grammar, such that the model can only produce strings accepted by the gramman This allows to separate the model's semantic performance from its syntactic abilities. We decode with beam search, using beam of 5.
A.4 ZERO-SHOT PARSING: QUALITATIVE ANALYSIS
Section 3.1 shows that models typically perform poorly on zero-shot parsing. Given that we use constrained decoding on all open-source models, the errors they make cannot be syntactic in nature, i.c. their outputs are guaranteed to be well-formed FOL expressions. This raises the question of what kinds of errors models are making. Here, we qualitatively analyze model errors. For each ambiguity type, we sample 10 incorrect examples from the Codegen- Madegen--16 16B model and classify the errors the model makes.
PP attachment: There predicate, c.g. exists v0 exists v1 agent (v1, Adele) AND have (v1) instrument (v1, v0) AND svied(v1) AND telescope (v0) is missing patient (v1, Sherlock) predicate for the sentence "Adele spied Sherlock with telescope". The remaining example had incorrect variable usage.
classes of errors. 9/10 examples have missing
Scope: 9/10 incorrect examples had in place of the Y quantifier (i.e. the right number of quantifiers but no V quantifier). 1/10 was missing a quantifier.
Inverse Scope: 10/10 examples had a in place of the V quantifier.
We release FOL and Lisp grammars.
16

---

Published as a conference paper at ICLR 2024
Bound: 8/10 examples used the same variable for verbs. For example, in exists v0 agent (vD, Katherine) AND frowned(vO) AND observed (v0) AND patient (vŌ, Mary). the event variable vO is used for frowned and observed when there should be an additional event variable v1. 1/10 examples had other incorrect variable use, and another had different missing predicate.
Conjunctions: 8/10 examples had the wrong connectives, i.c. OR, OR, AND instead of OR, AND, AND. 2/10 had bad scoping, where the correct predicates and connectives were produced but were grouped incorrectly.
A.5 ANNOTATION TASK
To ensure the validity of our results, we conducted a pilot paraphrasing task, where we asked Mechanical Turk annotators to verify that they native English speakers and paraphrase short passage. In the task, annotators from a trusted list were first asked if they were native English speakers. To verify the results, annotators sentences. The annotation interface precluded copying, preventing annotators from
additionally asked paraphrase short fable (The North Wind
and the Sun) using external resources, Annotators were paid $0.50 for the pilot task, corresponding to hourly payment of ~ $14.00 Each annotator's summary was then manually checked to verify fluency and adequacy; all annotators passed the quality check.
In the main HIT, annotators were again paid ~ $14.00 per hour, and each annotator performed exactly 20 annotations in a sequence. Each sequence has tuples of the ambiguity types. Annotators were shown a sliding scale with 3 ticks: not confident, somewhat confident, very confident, and annotators can slide the indicator anywhere along the scale. The interface can seen Fig. 7
Choosing Stuations Based on Natural Language: Exampla 20
What situation do think the following statement most compatible with?
Statement:
every girl
There are multiple girls, and there is at least one cat (there may be more).
There are multiple girls, and there is exactly one cat.
How contident you your
not confident
somewhat confident
very confident
Figure 7: Annotation interface for human evaluation.
To transform the annotators' confidence scores into probabilities, we first min-max normalize raw confidence scores, following past work using sliding bars /Vashishtha al. 2019). This accounts for the fact that different annotators may the slider differently. We then take the lowest confidence value to correspond to p(LF.) = 0.5, where LF, is the LF corresponding to the chosen interpretation. Intuitively, if p p(F.)) were less than 0.5, the annotator would have chosen the other LF. The highest confidence value corresponds to p(LF2) = 1.0.
Qualitatively, we find that visual verbs and nouns (e.g. saw-telescope, observed-glassed are matched more to LF,, where the PP an instrument, while tactile verbs and nouns (c.g. held- gloves, picked up-mitlens) yield possessive interpretation.
17

---

Published as a conference paper at ICLR 2024
Limitations of the Annotation Task To gather human preferences, elicit choices between verbalized interpretations of each logical form. This different task from what the models are being tasked with, and is motivated by the fact that annotators unlikely to know first-order logic. Even they did, it is difficult to constrain annotators to produce exactly the kind of first-order logic statements that would match the reference. In this sense, most of the models test have advantage, as we use constrained decoding according to a grammar. Thus, the model cannot produce LFs that deviate from the syntax expected by AMP.
Our method for eliciting judgements differs from standard methods in psycholinguistics, which are typically based on reading times, eye tracking, or other more elaborate experimental methods (Lackner & Garrett 1972 Rayner et al. 1983; Filik et al., [2004] Dwivedi] 2013). It is more akin to the paraphrase verification method used by Rayner al. (1983) to elicit interpretations. Note that these methods test for different things. While the former set of methods typically test for incremental and subconscious processes, our method and paraphrase verification test for conscious, non-incremental judgements. In the context of comparison to transformer-base/ models which do not receive incremental input, the second paradigm is a more accurate fit. Nevertheless, the uncertainty consciously expressed by our human annotators may differ, for example, from the uncertainty would obtain via more direct measurements like reading times.
18


Reviews:
[{"review_id": "x78TtdicYz", "reviewer": "Reviewer_PRrS", "summary": "The paper explores the longstanding issue of ambiguity in natural language and its implications for semantic parsing in AI. It delves into the nature of language ambiguity, highlighting how it stems from the balance of communication efficiency and interpretive flexibility, and poses challenges for AI systems that lack human-like commonsense knowledge and context. To address these challenges, the authors propose a novel framework and the Ambiguous Parsing (AMP) dataset, which includes various types of ambiguities paired with dual logical forms (LFs). This resource is aimed at enhancing the performance of large language models (LLMs) in semantic parsing tasks, especially in handling ambiguity.\n\nThe study introduces two tasks to assess how well LLMs, utilizing in-context learning (ICL), can capture multiple interpretations of an ambiguous input. These tasks are designed to evaluate model performance in both zero-shot and few-shot settings, with a series of metrics developed to quantify their ability to predict and represent ambiguity. The paper also reports on models' performance, noting that while models can sometimes mirror human preference for certain interpretations, they generally fall short in predicting all possible parses. Additionally, it is observed that some models are quite adept at reflecting the distribution of interpretations in mixed-prompt scenarios, offering insight into in-context learning amidst conflicting evidence.", "questions": "1. How do you foresee the findings of this research being applied in practical AI systems, particularly in areas where ambiguity can have significant consequences, like in legal or healthcare settings?\n\n2. Is the AMP dataset extensible, and are there plans to include more complex or nuanced forms of ambiguities, such as cultural or idiomatic ones?\n\n3/ Could you elaborate on the selection process for the five types of natural language ambiguities included in your study? Were there other types of ambiguities considered but excluded?\n\n4. The use of synthetic data might not fully capture the complexity of natural language ambiguities encountered in real-world scenarios. How well do the findings translate to naturally occurring datasets?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "1. The AMP dataset is a significant contribution, providing a resource specifically designed for investigating ambiguity in semantic parsing, which is a relatively unexplored area.\n\n2. The paper takes a comprehensive approach by addressing the challenge from the perspective of both dataset creation and model evaluation.\n\n3. The introduction of zero-shot and few-shot tasks offers a rigorous evaluation framework for future research on ambiguity in semantic parsing.\n\n4. The development of new metrics to assess the models’ ability to handle ambiguity is a noteworthy contribution that can guide subsequent model development.\n\n5. The results contribute interesting insights into the capabilities and limitations of current LLMs in capturing ambiguity through zero-shot and in-context learning.", "weaknesses": "1. While the paper provides a strong foundation, it could benefit from a more detailed exploration of how ambiguity affects real-world applications of semantic parsing.\n\n2. The AMP dataset, while novel, might still be limited in scope and diversity, potentially affecting the robustness of the study’s conclusions.\n\n3. It is unclear how the proposed methods deal with the dynamic nature of conversational context, which can significantly affect ambiguity resolution.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "UkKbjonFa0", "reviewer": "Reviewer_Qhxa", "summary": "This paper presents a benchmark suite and set of corresponding metrics for measuring models' behavior in the face of ambiguity in a semantic parsing task. The task is set in an in-context learning setting where models are provided with examples of (natural language, logical form) pairs and asked to translate a new natural language sentence into a logical form (LF).\n\nThere are two test settings:\n* A \"zero-shot\" setting where the model is expected to infer the existence of an ambiguity in the input example that did not show up in its few-shot examples. For example, by Appendix A.2, the model might see the LF for `Mary with the camera` (which, side note, seems ungrammatical to me at least when interpreted in the relevant sense) and \"Galileo saw with the camera\" (which...also seems ungrammatical absent context), it's tested on whether it recovers that \"Galileo saw Mary with the camera\" is ambiguous, by checking if both possible interpretations get high enough probability (or ranking) in its output distribution.\n* A \"mixed prompting\" setting where the model is given ambiguous examples in the prompt which resolve one way or the other a given proportion of the time (e.g., attaching high or low) and it is tested on the degree to which it approximates this ambiguity resolution behavior in its outputs (either on a dataset level in what it ends up decoding on lots of examples, or on an instance level in how it allocates relative probability).", "questions": "Section 3.1:\n* I think Figure 2 would be better as a grouped bar chart? It doesn't really make sense to me to have multiple models along a line here.\n* \"we rarely observe any models predicting both LFs correctly; one exception to this is conjunction ambiguity\" — seems also true for `bound`?\n\nSection 6:\n* \"statements be ambiguous\" (typo)\n* \"Making this assumption can reduce semantic parsing to syntactic parsing\" — it's very unclear to me what this means\n* The argument in the second to last paragraph that datasets need to have multiple judgments seems wrong to me. If there is indeed annotator disagreement, this disagreement will show up implicitly spread across multiple examples. Learning to maximize the likelihood of the data should teach the model the right kind of uncertainty. Multiple annotations aren't actually necessary for learning this — though they might be necessary for _evaluating_ it.\n\n---\n\n**EDIT:** Thanks to the authors for their response. I think all of my concerns were basically addressed except for the point about the choice to ask that the model represent ambiguity in its top-k probabilities, which I think is a problematic assumption (outlined why in my comment). I think this undermines the significance of the paper as it takes an outdated and limiting perspective on how to represent ambiguity, but I still think it constitutes a useful contribution to the literature, hence I'm sticking with my recommendation of a weak accept.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "* The paper aims at an important problem (handling ambiguity in semantic parsing).\n* The setup is clever and allows for some interesting analyses. I think the looking at the token-level confidences to see how model uncertainty is reflected in ambiguity-resolution–dependent choice points, as done in Figure 5, is a useful idea.\n* The comparison to human behavior (Section 3.2) is interesting and I imagine could seed future experiments.", "weaknesses": "I'm worried about how we assign meaning to the various results, and I'm not sure how this result would feed into future work that helps parsers handle ambiguity better.\n\n1. Human experiments: humans were given both interpretations and asked to assign confidences to them. This seems a bit different from what the models were asked to do in the zero-shot experiments, which is implicitly pick out the ambiguity on their own. I understand it'd be hard to elicit this kind of behavior from humans — ideally you would ask them a question whose answer hinges on the ambiguity and get some measure of the uncertainty in their answer, but it's unclear what that measure would be. The measured degree of uncertainty from humans in this setting may quite exceed the actual uncertainty when reading and interpreting language in practice, as humans may simply snap to one reading or the other. So it may not be fair to compare this result to model results — though of course, there is no clear analogy between what humans and models are doing anyway, so I'm not sure what the exact criteria for a fair comparison should be. It might seem like a safer bet to compare humans to models on a similar task, where the model is given both interpretations and is asked to give a confidence score, or repeat the one that it prefers — for example, I think it'd be a strict improvement on the current paper to do the human comparison to the FIM relative confidence results instead. But of course, the confidences reported through that process may not be ecologically valid with respect to the paper's broader goal of characterizing LM behavior on ambiguous semantic parsing examples.\n2. More broadly, the paper relies on a mostly implicit assumption that the _appropriate_ behavior for an LM is to represent multiple possible readings in its top-k outputs, or for the probabilities of each possible reading to reflect the proportions of related readings in their prompts / ICL data. It isn't obvious to me that this is what we want or that it is the right way of approaching the question of whether LMs can handle ambiguity in semantic parsing appropriately. When prompted for a single output, why _should_ the model distribute its probabilities in any particular way? There's nothing that says that we have to use the top-k candidates in order to represent a set of semantically distinct alternatives, and indeed it is suboptimal for this purpose, as the paper notes that the top-k results are usually variations of the same interpretation. I can see how this may make sense if we were to, for example, use the LM probabilities as a prior to guide some kind of search through the space of parses; we then want all plausible interpretations to be discoverable. But it's not clear that's the best strategy in practice if we're going to be using LMs anyway: why not just ask it, for example, to decode all possible interpretations in sequence? There's a big space of possible things to do here if we want models to handle ambiguity, and unfortunately the results of these experiments only bear on a small subset whose promise is unclear. Not that the experiments done are bad or invalid, but I think the paper needs to make a clear argument about the nature of the construct being tested and what these results are supposed to inform.\n3. On that note, it's unclear to me from the arguments in the paper how these results can actively guide future efforts to build semantic parsing systems that handle ambiguity better. I think the burden does lie on this paper to make that case, at least in principle (I'm not saying it needs more ML experiments).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "dKogOi61DJ", "reviewer": "Reviewer_49nJ", "summary": "The paper proposed a task to convert an ambiguous sentence to its logical form to evaluate LLMs with zero-shot and few-shot evaluation. The paper studies five types of ambiguity and evaluates LLMs with a decent number of examples. They also proposed multiple metrics to interpret the model results. The dataset and experiment all together provide insights into how LLMs understand ambiguity and its difference from human ambiguity understanding.", "questions": "1. Why not report ZM 100 in Figure 4?\n2. In section 3.1, the zero-shot experiment, how do you get multiple predictions? Do you sample k times? Your prompt doesn’t encourage the model to predict two LFs, so the model only predicts one by demonstration. The zero-shot results could get better if you encourage two predictions in your prompt.\n3. in figure 2, any intuition of why cg-2b is outperforming cg-6b on LF1?\n4. It's more like a suggestion: Do you plan to extend the ambiguity to more than two interpretations?\n\nNotes:\n\n- Section 2.1 Metrics. The last sentence is confusing: “The higher, the better. “ If you are talking about k values, shouldn't it be the lower, the better? If it refers to the metric, then it should be explicit.\n- Equation 1 is very confusing. Do i  and k mean the same thing here?\n- Section 3, paragraph 2, “Here”⇒ “In this setting”\n- Figure 4 is a Table, not a Figure.\n- Figure 5 is a bit hard to read when it comes to different types of semantic ambiguity. You can choose to put a different text color for important tokens inside the blue circle to increase the readability.\n- In Section 4, paragraph 2, what does “1 sentence at a time” mean when the percentage of the LF0 sentences is from 0 to 100 in increments of 10?", "limitations": "", "rating": 6, "confidence": 2, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. The motivation and writing are very clear. The paper is generally easy to follow.\n2. I like the human probability vs. model probability experiments personally, and seeing that humans have certain preferences on one interpretation than the other is interesting, and model prediction somehow matches it as well is very interesting too.", "weaknesses": "1. The generation task is hard, especially generating logical forms. Why not formulate this as a multi-choice problem?  Letting the model choose two from 10 possible combinations?\n2. Is there any quantitative analysis? What kind of errors does the model usually make?\n3. The evaluation metric can be improved. I have several questions about this. Why not use the same zero-shot and few-shot metric since the output format is the same? Why not use language interpretation instead of LF generations? Language interpretation is much more intuitive for the model than LF generations.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "SUHlBStOUa", "reviewer": "Reviewer_dZKu", "summary": "This paper addresses an important class of tasks: evaluating semantic parsing outside of the realm of 1:1 mappings from form to meaning, potentially including an actual distribution over meanings.   They develop a dataset of templatically generated text-LF pairs where each sentence maps to two possible interpretations, use smart annotation of scalar rankings to get distributions over those ambiguities, and evaluate models on their ability to generate both possible meanings given some prior in-context-learning prompts, and ability to match the distribution of human judgements and to match inbalances of interpretion in the ICL inputs.  They find that models (most of the codegen variety) do quite bad as \"zero shot\" prediction of these LFs, somewhat well at producing a best LF in certain conditions but also quite poorly at generating well-calibrated distribution over the two choices.", "questions": "- In more real-world semantic parsing tasks there can be dramatically more than just two possible semantic parses. Would this approach (particularly use of EASL in collecting human judgements) still work in contexts with far more acceptable parses per sentence?\n-  Insofar as all models were equally poor at a number of the zero-shot tasks at ZM_5, did the authors check the effect of raising that k parameter? \n- I was curious about the focus on capturing the training data biases rather than the human annotation judgements, for the few-shot setting.", "limitations": "", "rating": 8, "confidence": 3, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "- It is a well-written paper with both clear reasoning and clear explanation, enough detail to be replicable and with the promised of publication of data and code. \n- It served to target specific questions about how models handle these ambiguities (e.g. the mapping token-level confidence to LF outputs), and is properly planned out to answer those questions. \n- Both the human judgments annotation and the constrained decoding seem very rigorously done.", "weaknesses": "- While just a quibble, I feel like the combination of ICL and very constrained templates makes their zero-shot setting very hard, and it's hard to draw conclusions from their findings there.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)