Reasoning about Ambiguous Definite Descriptions
Authors: Stefan Frederik Schouten
Peter Bloem
Ilia Markov
Piek Vossen
Venue: EMNLP 2023 Findings (2023)
Keywords: Natural language reasoning
definite descriptions
ambiguity
large language models

Abstract: Natural language reasoning plays an increasingly important role in improving language models' ability to solve complex language understanding tasks. An interesting use case for reasoning is the resolution of context-dependent ambiguity. But no resources exist to evaluate how well Large Language Models can use explicit reasoning to resolve ambiguity in language. We propose to use ambiguous definite descriptions for this purpose and create and publish the first benchmark dataset consisting of such phrases. Our method includes all information required to resolve the ambiguity in the prompt, which means a model does not require anything but reasoning to do well.
We find this to be a challenging task for recent LLMs. Code and data available at: https://github.com/sfschouten/exploiting-ambiguity

Reasoning about Ambiguous Definite Descriptions
Stefan F. Schouten and Peter Bloem and Ila Markov and Piek Vossen Vrije Universiteit Amsterdam
(s.f. schouten, www.sshhccc...bleeeeee... pleemm, i markov, p. t. jm.. vossen)evu.n!
Abstract
2010, the pope native speaker of German.
Natural language reasoning plays an increas- ingly important role improving language models' ability to solve complex language un- derstanding tasks. An interesting use case for reasoning is the resolution of context- dependent ambiguity. But no resources exist to evaluate how well Large Language Models can explicit reasoning to resolve ambiguity in language. We propose to use ambiguous defi- nite descriptions for this purpose create and publish the first benchmark dataset consisting such phrases. Our method includes all infor- mation required resolve the ambiguity in the prompt, which means model does not require anything but reasoning to do well. We find this he challenging task for recent LLMs. Code and data available at: https: //gi thub. com/ sechouten/explott Laiinggmmigee imbiguity
Time
in 2010
Denotee
Benedict
Francis
Native
German
Spanish
Language
The pope 2010 (Benedict) nalive speaker of German.
The current pope (Francis) was, as of 2010, native speaker German.
De
Example ambiguous definite description. Since person's native language does change time, we know that the de dicto interpretation is correct.
Figure 1:
1 Introduction
Natural language understanding and reasoning interdependent skills: reasoning with natural lan- guage presupposes level of understanding; but full understanding may require the resolution of ambiguities through reasoning. Complex ambigu- ity in particular could benefit from explicit 'out loud' reasoning, such as the reasoning that is pro- duced with chain-of-thoug!! prompts.
explicit reasoning, such as factual recall. Thus, improvements on these tasks cannot be easily tributed to improvements in reasoning.
In this paper we create new benchmark dataset which requires models to resolve ambiguous defi- nite descriptions. Definite descriptions are phrases that denote entities by describing the properties or roles that are unique to them within the relevant context (e.g. "the pope", "john's mother", "our king"). We use ambiguous descriptions which de- note one of two entities, and include information both entities in context. Specifically, intro- duce de dicto / de re ambiguity by including temporal operator (sce Figure 1 for an example). By asserting something that is true only one of these entities, one of the two interpretations can be excluded by reasoning.
Existing resources used evaluate reasoning are not well suited to investigate the capability of resolving ambiguity by explicit reasoning. Some existing benchmarks require the resolution am- biguity, but focus only on ambiguity that humans can resolve intuitively (e.g. Winograd schemas, Levesque et al. 2012). The ability to reason with natural language is often evaluated with tasks con- sidered complex enough require it. These may may not include ambiguities of various types, making them poorly suited to evaluate when mod- cls are able to resolve ambiguity and which types. Such tasks may also benefit from abilities besides
We demonstrate the value of this approach by creating a new benchmark dataset generated from Wikidata. We explicitly include the knowledge re- quired for disambiguation the prompt of each

---

instance. Doing so ensures that when a system answers incorrectly, its failure stemmed from its inability to adequately reflect on the given infor- mation. We perform experiments showing the per- formance of recent large language models on our benchmark. We find that: (1) GPT4 and chain-of- thought prompting perform best (2) the LLaMA- based OpenAssistant model beats GPT3.5 in aver- age accuracy, although the latter is more consistent; and (3) de re instances are harder than de dicto instances for all models.
is used represent knowledge and LLMs are used
to reason over that knowledge. One challenge this
paradigm could have to overcome is the potential for ambiguity inherent in natural language. Our work intentionally introduces ambiguity, and eval- uates how well LLMs can resolve it by reasoning. Previously, Winograd schemas (Levesque et al., 2012) have been used in various benchmarks test for commonsense reasoning. These schemas con- sist of sentences with ambiguous pronouns which must be resolved in one of two ways. However, existing schemas focus on ambiguities that humans resolve intuitively (without the need for explicit reasoning). Creating new schemas that do require explicit reasoning does not appear straightforward. especially since creating them has proved difficult in general (Kocijan et al., 2023). Our method also involves ambiguous noun phrases, but requires ex- plicit reasoning about definite descriptions rather than implicit reasoning about pronouns.
In summary, our contributions are: (a) an outline of ambiguous definite descriptions, an under-ex- plored class of problems useful for the evaluation of systems that reason with and about natural lan- guage (section 3); (b) a new benchmark dataset con- sisting of ambiguous definite descriptions, show- casing the value of the problem class (section 4), and finally (c) experimental results showing the performance of recent large language models on this benchmark (section 5)
Ambiguous Definite Descriptions
2 Related Work
Definite descriptions are noun phrases that denote entities by describing the properties or roles that are unique to them within the relevant context. Exam- ples of such phrases in English include among oth- ers: "the pope", "john's 's mother", and "our king".
Reasoning is a broad term and the tasks related to it are varied. We focus on the type of reasoning that is explicit and is done with natural language, which has recently been surveyed extensively (Qiao et al., 2023; Huang and Chang, 2023; Yu et al., 2023). This paradigm became possible with the advent of large language models (LLMs) such as Chat- GPT and GPT4 (OpenAI, 2023). These models can be prompted to perform "chain-of-though reason- ing (Wei et al., 2022); even in zero-shot setting (Suzgun al., 2022). This has significantly im- proved performance for many tasks (Suzgun et al.. 2022). Using new dataset of ambiguous definite descriptions, our experiments evaluate to what ex- tent LLMs can use (chain-or (chain-of-thowwe oof-thoughtt reasoning to resolve ambiguity in language.
De dicto vs. de
The kind of ambiguities we use are known as de dicto de re ambiguities. They involve statement that is either true of what is said (de dicto) true of the thing (de re). Take for example the sentence depicted in Figure 1, where is unclear if the de- scription "the pope" denotes the current pope (Fran- cis) or the previous pope (Benedict). The source of the ambiguity is the phrase "In 2010", which can be read as primarily relating to the description ("the pope"), meaning the property is ascribed to the pope from 2010 (Benedict). Or, it can be read as relating primarily the property ascription itself ("was a native speaker of German"), in which case the property is being ascribed to the current pope but is qualified as being true in 2010.
Recent work ambiguity includes the construc- tion of both curated (Liu al., 2023; Min al., 2020) and synthetic datasets (Yuan et al., 2023). Such datasets investigate ambiguity in a variety of tasks such as: natural language inference, open- domain question answering, etc. Our dataset is synthetically generated (but factual), and focuses specifically on de dicto / de re ambiguities, forming a binary classification task.
This type of ambiguity is the result of descrip- tions being combined with non-extensional oper- ator of which the temporal 'In PERIOD' is just one example. Other non-extensional operators could be used, such as modal operators, or propositional attitudes like 'PERSON believes that' or PPESON hopes that'.
The term 'logical reasoning over natural lan- guage' (LRNL) was coined by Yang et al. (2023) to talk about a new trend where natural language

---

Component
Explanation
Example
entity -relations
number relations of the entity and the context in which it is true.
Lars Lervik was part of the Telemark Battalion from Jan- wary 2010 to January 2013.
Lars Lervik was part the Brigade Nord from January 2018 January 2020.
Premises
relation pproperties
The value the property each relation.
Brigade Nord is military unit of size class brigade. Telemark Battalion is a military unit of size class battalion.
regularity
The relevant rule or regularity that is required to resolve the ambiguity.
Militory units do not change size class.
context
Establishes the current context.
the current date is June 2019,
sentence
An amhiguous sentence with a tem- poral operator and property being ascribed to an entily denoted by a definite description.
what is the most likely interpretation of the following sentence: The military unit of Lars Lervik was of size class brigade in March 2010.
Question
interpretations
Two interpretations for the ambiguous sentence, one of which contradicts the regularity.
1. Lars Lervik's unit March 2010 (Telemark Battalion)
X
was of size class brigade
2. Lars Lervik's 's current unit (Brigade Nord) was of size class brigade in March 2010
Table I: An example from the RADD-Wikidata-S-FF dataset (property pair P7779_ (1779_021506400) The parts of the example that are bold are the same for each instance. For this example instance the de re interpretation is correct.
4
out property-pairs where the results were domi- nated by single entity, for example, had the data for the mersoonmilitary_ue wwwit-sizeeeeee " pair con- sisted of only a handful of people changing units many different times, we would not have included it. See Table 2 for details on which property pairs we include and what they represent.
To demonstrate the value of Reasoning about Ambiguous Definite Descriptions we create a (semi-)automatically generated dataset based on Wikidata. This dataset is based on 5 Wikidata property-paiee and is in English. We focus creating ambiguous sentences with temporal op- erator. Wikidata contains many triples qualified with the 'start time' and 'end time' properties to indicate the period in which they were true.
We include all facts relevant the main entity in each prompt whether they are necessary to resolve the ambiguity or not, this allows us to measure if models can work around distractor facts.
In Table 1, one can see a complete example in- stance from this dataset. For given main entity (e.g., Lars Lervik) find the relations correspond- ing that entity (Lars Lervik's military units and the beginning and end of his term with them). This information is combined with a property of the re- lations that does not change over time (the size class of the military units). Pairs of these relations are sampled such that the property (the size class) has different values for the two relations (in this case, one brigade and one battalion). Note that, by changing the property we ascribe the denotee, we can flip which interpretation correct (battalion instead of brigade) so we can generate two mirror instances for each entity and relation-pia pair..on-pair.
4.1 Template design considerations
We use templates generate the instances in the dataset, and while experimenting we identified a few things avoid when phrasing them.
Avoiding disambiguation by verb tense. The ambiguous description must not contain a verb whose tense favours one interpretation over the other. For example with:
"In MONTHYEAR, the club for which X is/was head coach was from Y."
If we use 'X is head coach' it implies that "In MONTHYEAR' does not scope over the definite de- scription, whereas with 'X was head coach' the implication is that it does scope over the definite description (assuming MONTHYEAR is in the past). Thus, we avoid using such phrases, for example. by formulating the example as:
We prioritized finding small diverse set of property-paing Besides requiring property whose value changes over time and property whose value does not, the combination of properties also must yield sufficient results on wikidata (we required at least 50 to create the 100 samples). We also filtered
"In MONTHYEAR, the club with as its head coach was from Y."

---

property-pair
description
#instances
#premises prompt
P6 P19
head of government birth place
7299
41.14
head coach country
1607
17.50
P159 P206
headquarter location in
body water
147
12.03
P7779 Q21506450
military unit size class
90
8.21
P8047 P30
ship country of registry continent
74
6.37
Table 2:
Property-pairs with their marks in Figure 2 and descriptions, the number of instances (each yielding de dicto and de re prompt), and average number of premises per prompt.
Avoiding event-referencing properties. We also avoid referencing events in the property ascrip- tion, which can cause 'in MONTHYEAR' to be read as referring to the time of the event, for example:
.8
Direet CoT
Accuracy
P286_P17
0 P6.
"In MONTHYEAR, the club with X as its head coach was founded by Y." Sometimes
FC PSI47_P30
8 P159. P206
avoid this by rephrasing: "In MONTHYEAR, the club with X as its head coach had Y as its founder."
.4
* P7779
021500450
Koula- 13B
OAsst-33B
GPT3.5
GPT4
One property pair was discarded because could find good way phrase the sentence, this was about the location of formation (P740) of dis- tributors creative works (P750).
Figure 2: Accuracies obtained by each model on the WADD-Wikidata-5-4 benchmark. Bars show average accuracy, marks overlaid on top indicate accuracy on each of the five property-pairs.
5 Experiments
We evaluate three callogue-oriented large language models (LLMs) on our benchmark in zero-shot setup. The models we include are:
"Option 1" or "Option 2"? The models were prone repeating the premises in their answer, thus warning them do this was necessary avoid them from running out of condext-window before providing the final answer.
(a) foala-13B' (Geng et al., 2023)!
(b) LLaMA-33Bt (Touvron et al., 2023) fine- tuned version of OpenAssistant Conver- sations (Köpf et al., 2023)²
(c) GPT-3.5(-turbo-0001) (OpenAI, 2022)
5.1 Quantitative results
(d) GPT-4(-0613) (OpenAI, 2022)
Figure shows the accuracies obtained by cach model for each property-pain and on average. The precision, recall and Fl scores for each model, class, and prompt style can be seen in Ta- ble 3. The highest average accuracy is obtained by GPT4 (83.2%) followed by OAsst-33B (67.7%) and GPT3.5's (64.8%). The mean accuracy of Koala-133 roughly equal to random guessing (52.2%). Looking at the performance for the var- ious property-paiss we can see that GPT3.5, al- though less accurate overall, is more consistent than OAsst-33B. The performance for OAsst-33B is particularly low on the P6_P19 pair. This can be explained by the average number of premises (41.14) which for the other property pairs (6.37 17.5 on average. never fewer than 5). The long prompts exceed the number of tokens included the context window during training for OAsst-33B (and Koala- 13B).
Models marked with were used with 4-bit OPTQ quantization (Frantar et al., 2023).
The evaluation is performed on 500 instances (100 per property-pair). To prompt the language models, we start with instances such as the one from the 'example' column in Table and number the premises P1 through P.V. Next, we shuffle the order of the interpretations, and then append one of the following two instructions: direct=" "Answer only with "Option 1" or "Option 2", explain your decision after." chain-of-thun thought= cfain-of-thoughtt swering, let's consider both options, and think step by step. DO NOT repeat the premises, only refer to their number." When using chain-of- thought (CoT), we follow up with a second mes- sage: "Based on this, what is your final answer,
twice the amount included
'aeloohheell 13B CPTQ 4bi 128g
oo www.eheeeeeeeeeeeneeeeeeeeoe SFT lamaa 308 GPTQ

---

de dicto
re
models have a strong preference for predicting de dicto (this can also be seen from the confusion ma- trix in Table 4). This explains why this behaviour mostly produces errors on the de re class.
Pr
Re
F1
Pr
Re
F1
Koala- 13B
0,46
0.42
0,44
0.47
0.47
0.47
Direct
OAsst-33B
0.59
0.79
0,67
0.61
0.36
0.45
G
0.53
1.00
0.69
0.97
0.10
0.18
About a third of all de re Mhain-of-thoughee an- swers conclude that neither option is correct. Al- 0.33
GPT4
0.56
1.00
0.72
1.00
0.20
Koala-135
0.56
0,51
0.53
0,56
0.54
0.54
though despite of this GPT4 almost always still pre- 0.61
COT
OAsst-33B
0.65
0.84
0.73
0.80
0.51
GPT3.5
0.64
0.82
0.72
0.73
0.48
0.56
dicts one of the two options. This seems to indicate that the models completely fail to consider the de re option. Take for example the following ambigu- ous sentence "The birth place of Prague 6's head of state was Třebič in November 2001" for which the correct interpretation is Option 2: 'Prague 6's current head of state (Marie Kousalíková) was, of November 2001, born in Trebfe'. GPT4 rea- sons as follows: "Option 2 can be eliminated based on premises (P6) and (P9), as Marie Kousalíková was born in Třebič, but she was not the head of state in November 2001, according to (PI) and (P2). Hence, this interpretation would be incor- rect." But, this reasoning is clearly incorrect, since her being head of state in November 2001 is not required for Option 2 to be correct. This type of error seems be a large part of why the models
GPT4
0.76
0.99
0.86
0.99
0.67
0.79
Table 3: Precision, recall and F1 score by class (columms), for each prompt styles and model (rows). Bold and italic show best scores overall and per prompt- style respectively.
Direct
CoT
de dieto
neither
de dieto
neither
de re
de re
Knala-13B de dicto
21.2
27.2
1.6
25.4
20.6
4.0
de
25.0 23.6
1.4
19.8
26.8
3.4
OAsst-33B de dicto
39.6
6.2
4.2
42,2
6.0
1.8
de re
27.6
18.0
4.4
22.8
25.4
1.8
GPT3.5 de dieto
49.8
0.2
0.0
41.0
8.8
0.2
de re 45.0
5.0
0.0
23.8
23.8
2.4
GPT4
de dicto
50.0
0.0
0.0
49.6
0.4
0.0
perform worse for de re instances.
de re
39.8
10.2
0.0
16.2
33.6
0.2
Together these two types of errors appear to make up the vast majority of GPT4's mistakes.
Table 4: Average confusion. The columns show prompt styles and predicted classes, the rows show models and label classes. Correct predictions are in gray.
6 Conclusion
We have introduced Reasoning about Ambiguous Definite Descriptions way to evaluate how well systems can natural language reasoning resolve ambiguous language and have created the first benchmark dataset to demonstrate the value of this task. Our findings show that recent LLMs are not yet capable of solving this task reliably, and that they particularly struggle with de re instances. Our error analysis suggests that beselss not always 'thinking step-by-step' as instructed- do not properly consider the de re options, instead hallucinating extra conditions for their correctness.
Note also that each model benefits from the use of ofain-of-though prompting.
In Table 4, we can see the confusion table for each model and prompt style. We can see that there are (up to 4%) cases where the models do not produce a parseable result. Finally, we observe that the de re instances are the more difficult class; only GPT4 with chain-of-thee of-in-of- is able to perform better than random guessing on these instances.
5.2 Error Analysis
We perform small analysis of the answers given by the best-performing model GPT4 to give insight into what areas need the most improvement.
In future work, we will expand our work by: (1) testing how changing the provided information (e.g. removing regularities) affects the performance; (2) embedding this problem in other tasks such that the ability to solve it depends the resolution of the ambiguity; (3) constructing multi-lingual ver- sions evaluate if the ability to resolve ambiguity through reasoning independent of the language; and (4) extending our method to other extensional operators, and other types of ambiguities.
About half of the mistakes made by GPT4 in- volve the model not properly following the chain- of-thought instruction, meaning the model already made a prediction in the first one or two sentences of its response. Although, ignoring this instruction seems to be common among both the de dicto and de re classes. It seems that absent any reasoning the

---

Limitations
Alisa Liu, Zhaofeng Wu, Julian Michael, Alane Suhr, Peter West, Alexander Koller, Swabha Swayamdipta. Noah A. Smith, and Yejin Choi. 2023. We're Afraid Language Models Aren't 't Modeling Ambigu- ity. ArXiv:2304. 14399 [cs].
The de dicto de re ambiguity we use in our dataset is one of many possible kinds of ambiguities. Com- pletely excluding the possibility that our results rely on the particulars of this ambiguity will require a diverse set of ambiguities.
Sewon Min, Julian Michael, Hannanch Hajishirzi, and Luke Zettlemyer. 2020. AmbigQA: Answering Am- biguous Open-domain Questions. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNP)), pages 5783- 5797, Online. Association for Computational Lin- guistics.
So far have only included English prompts in our benchmark. We leave the creation of prompt templates in other languages for future work.
We have performed a 'best-effort' tuning of the prompts. It is plausible that with a more exten- sive tuning better prompts can be found and that overall performance could be improved to some degree. An extensive tuning may also reveal that each model benefits from different prompts, which could also change their relative performance.
OpenAL. 2022. Introducing ChatGPT.
OpenAL. 2023. ArXiv:2303.08774 [es].
GPT-4 Technical
Report.
Shuofei Qiao, Yixin Ou, Ningyu Zhang, Xiang Chen, Yunzhi Yao, Shumin Deng, Chuanqi Tan, Fei Huang. and Huajun Chen. 2023. Reasoning with Language Model Prompting: A Survey. A1X15:2212.09597 [cs].
Acknowledgements
This research was supported by Huawei Finland through the DreamsLab project. All content repre- sented the opinions of the authors, which were not necessarily shared or endorsed by their respective employers and/ or sponsors.
Mirac Suzgun, Nathan Scales, Nathanael Schärli, Se- bastian Gehrmann, Yi Tay, Hyung Won Chung. Aakanksha Chowdhery, Quoc V. Le, Ed H. Chi, Denny Zhou, and Jason Wei. 2022. Challenging BIG-Bench Tasks and Whether Chain-or of Thouhghh Can Solve Them. ArXiv:2210.092: [es].
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. 2023. LLaMA: Open and Efficient Foundation Language Models. 2331 [cs].
References
Elias Frantar, Saleh Ashkboos. Torsten Hoefler, and Dan Alistarh. 2023. OPTQ: Accurate Quantization for Generative Pre-trained Transformers.
Xinyang Geng, Arnav Gudibande, Hao Liu, Eric Wal- lace, Pieter Abbeel, Sergey Levine, and Dawn Song. 2023. Koala: A dialogue model for academic re- search. Blog post.
Jason Wei, Xuezhi Wang. Dale Schuurmans, Maarten Bosma, Ed Chi, Quoe Le, and Denny Zhou, 2022. Chain of Thought Prompting Elicits Reasoning in Large Language Models. arXiv:2201.11900 [cs]. ArXiv: 2191.
Jie Huang and Kevin Chen-Chuan Chang. 2023, To- wards Reasoning in Large Language Models: A Sur- vey. ArXiv:2212. 10403 [cs].
Zonglin Yang, Xinya Du, Rui Mao, Jinjie Ni, and Erik Cambria. 2023. Logical Reasoning over Natural Language as Knowledge Representation: A Survey. ArXiv:2303.1220 [cs].
Vid Kocijan, Ernest Davis, Thomas Lukasiewiez. Gary Marcus, and Leora Morgenstern. 2023. The Defeat of the Winograd Schema Challenge. ArXiv:2201.022 .02387
[cs].
Fei Yu, Hongbo Zhang, Prayag Tiwari, and Benyou Wang. 2023. Natural Language Reasoning, A Survey. ArXiv:2303.1473 [cs].
Andreas Köpf, Yannic Kilcher, Dimitri von Rütte, Sotiris Anagnostidis, Zhi-Rui Tam, Keith Stevens, Abdullah Barhoum, Nguyen Minh Duc, Oliver Stanley, Richárd Nagyfi, Shahul ES, Sameer Suri, David Glushkov, Arnay Dantuluri, Andrew Maguire, Christoph Schuhmann, Huu Nguyen, and Alexan- der Mattick. 2023. OpenAssistant Conversations Democratizing Large Language Model Alignment. ArXiv.2304.07327 [cs].
Yuewei Yuan, Chaitanya Malaviya, and Mark Yatskar, 2023. AmbiCoref: Evaluating Human and Model Sensitivity to Ambiguous Coreference. In Findings of the Association for Computational Linguistics: EACL 2023, pages 1023-1030, Dubrovnik, Croatia. Association for Computational Linguisties.
Hector Levesque, Ernest Davis, and Leora Morgenstemn. 2012. The Winograd Schema Challenge. In Thir- leenth International Conference on the Principles of Knowledge Representation and Reasoning.


Reviews:
[{"review_id": "Rax4zY2IbA", "reviewer": "Reviewer_sY4f", "summary": "", "questions": "Question A: The authors state, regarding avoiding event-referencing properties, “If we cannot find a good way to phrase it, we discard the property all together” (lines 221-222). How many properties were discarded for this reason?\n\nQuestion B: The chain-of-thought prompt included the instruction “DO NOT repeat the premises, only refer to their number” (lines 244-245). What was the rationale for this?", "limitations": "", "rating": 4, "confidence": 3, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the review!\n\nWe recognize that more property-pairs is better, but rather than constructing as many pairs as possible (with many being similar), we prioritized finding a few representative property-pairs that are quite different. Of course, we specifically required a property whose value changes over time, and a property whose value does not. Furthermore, the combination of properties also must yield sufficient results (we required at least 50 to create the 100 samples). Finally, we also do not want those 50 to be dominated by a single entity, for example, had the data for the ‘person-military_unit-size_class’ pair consisted of only a handful of people changing units many different times, we would not have included it. We will expand the description of this process in the paper, so that larger versions of this benchmark may be easily created in the future.\n\n\n**Question A**\nThe only template we couldn’t phrase in a way we were happy with was about the location of formation (`P740`) of distributors of creative works (`P750`).\n\n**Question B**\nWe noticed that the LLMs were prone to repeating the information before actually answering the question. This made the answers of the LLM considerably longer, and increased the number of times that the responses exceeded the context-window. Since repeating the answers does not really add any additional information, we tried to discourage this behavior by adding this instruction. We will clarify this in the paper.\n\n\nWe agree that F1-score for each class is a good addition to better show the difference between classes, we will add this to our results section.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "This paper describes an interesting twist on the idea of testing whether language models can resolve ambiguity in language—instead of focusing on implicit reasoning based on world knowledge, it studies explicit logical reasoning about definite descriptions. The methods are mostly sound, and the paper is well-written overall. I appreciate that the code and data are publicly available, which should help others reproduce the results.", "reasons_to_reject": "The dataset contains, and the evaluation is performed on, only five property-pairs, which may introduce variability in the results. While this is probably sufficient to demonstrate the authors’ main claim, that recent large language models struggle with this task, other points, such as the relative performance of OAsst-33B and ChatGPT, may have been different had a different set of property-pairs been chosen.", "questions_to_authors": "Question A: The authors state, regarding avoiding event-referencing properties, “If we cannot find a good way to phrase it, we discard the property all together” (lines 221-222). How many properties were discarded for this reason?\n\nQuestion B: The chain-of-thought prompt included the instruction “DO NOT repeat the premises, only refer to their number” (lines 244-245). What was the rationale for this?", "paper_topic_and_main_contributions": "This paper investigates whether large language models can distinguish between de dicto and de re interpretations of definite descriptions in a temporal context. The authors generate a dataset, RADD-Wikidata-5-EN, inserting data from 5 English Wikidata property-pairs into context and ambiguous sentence templates, such that the property being ascribed to the denotee determines the correct interpretation of the ambiguous sentence. They test three large language models using both direct and chain-of-thought prompts, finding that the models generally struggle with this task, especially with de re interpretations, although chain-of-thought prompting helps somewhat.", "excitement": 4, "reproducibility": 5, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Typos_Grammar_Style_and_Presentation_Improvements": "The authors may want to consider presenting (in addition to accuracy and the confusion table) the F1 score for each class—this can succinctly show the difference in model performance classifying de re and de dicto instances."}}, {"review_id": "wh6rr15O7N", "reviewer": "Reviewer_VEf9", "summary": "", "questions": "Question A: Were there other types of ambiguities you considered using?\n\nQuestion B: Have you done any deeper analysis into the types of reasoning failures the models exhibit? Are there certain logical connectives or steps that are more problematic?\n\nQuestion C: You show *de re* problems are harder across models. Do you have any hypotheses for why this is the case?", "limitations": "", "rating": 4, "confidence": 3, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the review!\n\nWe agree that there are additional opportunities for insight. We will add a qualitative analysis of the type of reasoning failures that occur. \nAnd we do indeed have more ideas for future work that were previously cut to meet the page limit. We will add these back.\n\n\n**Question A**\nWhile the original idea for this research focused only on de dicto / de re we have thought about ways to expand it to other types of ambiguity.\nIn the context of analyzing reasoning about toxic language we have considered examples with other types of ambiguity, for example:\n```\nFrom 2016 to 2019 the prime minister of the UK was Theresa May.\nFrom 2019 to 2022 the prime minister of the UK was Boris Johnson.\nTheresa May is a woman.\nBoris Johnson is a man.\nIf the year is [2017/2020] and we are in the UK, does the following sentence express a stereotype?\n“Why is the prime minister always so hysterical?”\n```\nwhich relies on the lexical ambiguity of ‘hysterical’.\n\n**Question B**\nA deeper analysis of the failure modes of LLM reasoning is very interesting to us. A quantitative analysis of this will be the subject of future work.\n\n**Question C**\nOne possibility is that when these ambiguities occur, in the training data or in English in general, de dicto is more likely to be the intended interpretation. If this is the case it is possible that LLMs pick up on it. We are not aware of research that speaks to the prior probability of either interpretation.\n\n\nWe agree with the grammar / style improvements and will make the suggested changes.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- Addresses an interesting, underexplored problem - ambiguity resolution through reasoning.\n- New semi-automatically generated benchmark dataset based on real-world knowledge.\n- Careful prompt design that isolates reasoning ability\n- Solid experiments across major recent dialogue models showing poor performance.\n- The paper is clear and well written", "reasons_to_reject": "- There are opportunities to strengthen analysis and gain more insights:\n\t- what types of reasoning failures occur?\n\t- Finer-grained breakdown (per instance or per property)\n- Discussion could be expanded with more concrete future work proposals based on insights gained.\n\nHowever, I understand that, due to page limitations, these remarks can be addressed in the next iteration of the present work.", "questions_to_authors": "Question A: Were there other types of ambiguities you considered using?\n\nQuestion B: Have you done any deeper analysis into the types of reasoning failures the models exhibit? Are there certain logical connectives or steps that are more problematic?\n\nQuestion C: You show *de re* problems are harder across models. Do you have any hypotheses for why this is the case?", "paper_topic_and_main_contributions": "This paper introduces a new dataset and benchmark for evaluating the ability of models to resolve ambiguity in language through reasoning. The key idea is using ambiguous definite descriptions, where a phrase like \"the pope\" could refer to two different entities based on a temporal qualifier. By providing contextual facts, one interpretation can be excluded through reasoning. \nThe authors designed the prompts to supply all necessary information to resolve the ambiguity, requiring reasoning without additional knowledge. Experiments with recent LLMs like ChatGPT showed low performance, indicating this task is challenging.", "excitement": 4, "reproducibility": 5, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "- line 73: add a comma before \"we\"\n- line 181-182: add commas before and after \"by changing property we ascribe to the denotee\""}, "extra_text": {}}, {"review_id": "6WHOJHUKzP", "reviewer": "Reviewer_xFyH", "summary": "", "questions": "", "limitations": "", "rating": 2, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the review!\n\nWe agree that a human baseline would be valuable, but we think that measuring the relative performance of LLMs is valuable independently of human performance. Therefore, we believe that our current results form a sufficient contribution. \n\nBoth prompt engineering and fine-tuning seek to find small changes to the conditions of evaluation that work in a model’s favor. The reason we did not do these experiments is because we are primarily interested in evaluating the *emergent* reasoning capabilities of LLMs. The prompt-tuning we did was aimed at trying to find a clear formulation of the problem that permits each LLM to do reasonably well. Extensive prompt engineering would not benefit our evaluation, because a model that does well at this task should do so regardless of the way the prompt is phrased as long as it is semantically equivalent.  \nWith regards to fine-tuning, we wanted to avoid the model relying on some unknown biases in our data, rather than its emergent reasoning abilities. While we could have attempted to eliminate as many of these biases as possible, to know for sure if we were actually learning to reason we would have had to evaluate on other reasoning tasks as well. Such experiments would have required more research and a considerably longer paper. \n\nIn summary, in our work we focus on creating an *evaluative* benchmark, and measure how well emergent reasoning generalizes to our new task. We will further emphasize this goal in our paper. We leave a human baseline and the creation of training resources for future work.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The question about large language models’ ability to solve complex language understanding tasks, is an important one. The paper examines though one particular task.\nThe dataset is released publicly.", "reasons_to_reject": "The paper only addresses one particular task of NLU, particularly challenging because it involves temporal reasoning, on which LLMs are known to struggle.\nThe task is quite hard also for humans, hence a human performance baseline should be included in the study.\nThe conclusion from the experiments are hard to generalise.\nThe experiments do not consider fine-tuning the models on the task: since the dataset is semi-automatically constructed, it would be interesting to see how a model trained on the dataset might pick-up the knowledge necessary to tackle the task.", "questions_to_authors": "", "paper_topic_and_main_contributions": "The paper introduces a way to evaluate how well systems can use natural language reasoning to resolve ambiguous determinate descriptions.\nA benchmark dataset has been created to experiment on this task, based primarily on descriptions that have different referents through time. Examples are produced by means of templates and therefore they might be fairly repetitive. A fine-tuned system might become good at solving the challenge.\nExperimental on the benchmark are performed in a zero-shot setting, with no attempt to optimise the prompt to each model, which is known in other tasks may have significant effect on performance.\nThe best performance of the three tested models is an accuracy of 67.7%, but it can be hardly considered conclusive.", "excitement": 2, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper introduces a dataset which requires models to resolve ambiguous definite descriptions (phrases that denote entities by describing the properties or roles that are unique to them within the relevant context, e.g. "the pope", "john’s mother", "our king"). The dataset has been constructed semi-automatically using 5 property-value pairs from from Wikidata. Their experiments indicate that LLMs still struggle to resolve this kind of ambiguity. 

The paper tackles an interesting problem and makes a valuable contribution in terms of the dataset. It does have drawbacks -- there's no human baseline, the dataset has been constructed from only 5 property value pairs, and could have used stronger baselines (even if as upper bounds). However, I think the paper meets the criteria for a short paper, i.e., a short focused contribution which the community can build upon.
Decision: Accept-Findings