AmbigDocs: Reasoning across Documents on Different Entities under the Same Name
Authors: Yoonsang Lee
Xi Ye
Eunsol Choi
Venue: COLM (2024)
Keywords: multi-document reasoning, entity disambiguation, ambiguous QA
TL;DR: We study the reasoning capabilities of language models across multiple documents featuring confusing entities.

Abstract: Different entities with the same name can be difficult to distinguish. Handling confusing entity mentions is a crucial skill for language models (LMs). For example, given the question “Where was Michael Jordan educated?” and a set of documents discussing different people named Michael Jordan, can LMs distinguish entity mentions to generate a cohesive answer to the question? To test this ability, we introduce a new benchmark, AmbigDocs. By leveraging Wikipedia’s disambiguation pages, we identify a set of documents, belonging to different entities who share an ambiguous name. From these documents, we generate questions containing an ambiguous name
and their corresponding sets of answers. Our analysis reveals that current state-of-the-art models often yield ambiguous answers or incorrectly merge information belonging to different entities. We establish an ontology categorizing four types of incomplete answers and automatic evaluation metrics to identify such categories. We lay the foundation for future work on reasoning across multiple documents with ambiguous entities.

Published as a conference paper at COLM 2024
Ambig Docs: Reasoning across Documents on Different Enti- ties under the Same Name
Yoonsang Lee 0, Y Eunsol Choi
The University of Texas at Austin Seoul National University Lysianthus@snu.ac kr. xiyeêcs. utexas. eunsol@utexas.ed edu
Abstract
Different entities with the same name can be difficult to distinguish. Han- dling confusing entity mentions is a crucial skill for language models (LMs). For example, given the question "Where was Michael Jordan educated?" and a set of documents discussing different people named Michael Jordan, can LMs distinguish entity mentions to generate a cohesive answer to the question? test this ability, we introduce new benchmark, AmbigDocs. By leveraging Wikipedia's ss disambiguation pages, we identify set of docu- ments, belonging to different entities who share an ambiguous name. From these documents, generate questions containing an ambiguous name and their corresponding sets of answers. Our analysis reveals that cur- rent state-of-the-at models often yield ambiguous answers incorrectly merge information belonging to different entities. We establish an ontology categorizing four types of incomplete answers and automatic evaluation metrics to identify such categories. We lay the foundation for future work on reasoning across multiple documents with ambiguous entities."
1 Introduction
Different entities can be referred by the same name, and clustering such confusing entity mentions requires rich reasoning involving background knowledge. While it rare for different entities who share the same name to occur together within single document, such co-occurrences are prevalent in multi-document setting. For example, searching the web with the question "When is ACL 2024?" returns webpage on Austin City Limits Music Festival and another page on Association for Computational Linguistics conference. Given such two webpages, confusing entities, such as AAsstin City Limits Music Festival will be held on Oct 4-6 and 11-13, while the ACL conference will happen on August 11-16."?
LMs generate a complete answer correctly distinguishing
Despite rich study in ambiguous question answering (QA) (Min et al., 2020; Stelmakh et al., 2022), which covers such ambiguous entity resolution, few work studied how LMs reason when provided with a confusing document set. The major obstacle has been the lack of annotated confusing document sets.? 2 To enable research in this area, we construct synthetic dataset, AmbigDocs, whose single instance consists question asking about an ambiguous entity and list of gold document-answer pairs for each disambiguated entity (see Table 8 for examples). We leverage Wikipedia disambiguation pages, which provide multiple entities that can be referred to by the same surface name. From each disambiguation page, we generate question and multiple valid answers, where each answer is grounded document and corresponds to different entity in the same ambiguation page. We generate a total of 36K examples, covering 102K unique entities over all domains.
Work was done at UT Austin.
1 Our dataset released at https: http:///bbmgoos..../// gitb..ioo ii/.
WWhil newer datasets contain evidence documents for few examples, we identify only 111 instances for ASQA (Stelmakh et al., 2022) dataset and 106 instances for QuoteSum (Schuster et al., 2023) dataset that cover entity disambiguation.

---

Published as a conference paper at COLM 2024
where Judge Day waas homm?
Question: What is the state
Long form Answer
Per-Doc Doc Answer Eval
Answer Category
Dee
Charles Bemand
Dothan, Alabamo
Judge Day
X X
Ambigous
Dee jeoward Wilam Day)
LLM
Cranston, Rhode Ieland....
Judge Day was born in Alabama, Rhode Island, and Onio.
Merged
Dec
Cantor, ONO....
William Loubs
Born
The context does not grovide enough information..
X X
X
No Answer
Document Answer Evaluation:
William Louis was born Ohio, and Charles Bernard Day bom Alabama.
Answer with eniiyy disambiguation
Answer without entlyy disentiguatiin No answer
Charles Bernard Day born Alabamaa Edward William born in Rhood Island, William Louis Day was bomn in Ohio.
Complete
X
Figure 1: Given question containing ambiguous entity mention (e.g,, Judge Day) and a set of documents containing valid answer to different disambiguated entities, LLM should generate complete answer, pairing each disambiguated entity with its answer. Left bottom box: we first evaluate y with respect to a single input document, checking if y contains the answer with its disambiguated entity name (e.g., for the Doe 1, it should mention the answer " "Alabama" and disambiguated entity name "Charles Bernard Day"). Based on per-document scores, we assign one of the answer category labels (partial, no, ambiguous, merged, complete).
Equipped with new dataset that contains ambiguous question and set of documents with valid disambiguated entity-answer pairs, we evaluate how LMs answer questions containing ambiguous entity names when provided with set of documents suggesting multiple distinct answers. We first evaluate LMs with metrics used in prior ambiguous QA tasks (Adlakha et al., 2023; Stelmakh et al., 2022). Then, we develop types of answers (complete, partial, ambiguous, merged, and no answer) that captures the unique challenge in Ambi gDoc. We develop an automatic metric that can classify answers according to this ontology, with very high agreement of manual classification. We find that both open-source LLMs (Touvron et al., 2023; Jiang et al., 2023) and OpenAl's GPT models (OpenAI, 2023) suffer at identifying different entities, failing to generate complete answers. Yet, we find prompting with in-domain in-context examples can improve model performance significantly, GPT-4/Mistral- questions, 20% 10%
ontology of five
generating a complete answer for 58% /43% of 100% gain from the zero-shot setting. We provide a rich analysis showing the strengths and weaknesses of existing LMs under this challenging scenario.
To summarize, our contributions are as follows: (1) We present mmbigDocs, a large-scale dataset that pairs question and set of documents that suggests multiple answers based on question entity disambiguation.
(2) We establish an ontology categorizing five types of generated answers and an automatic classification heuristic, enabling in-depth analysis model behaviours. (3) We benchmark how current LMs perform the question answering task when provided ambiguous questions and document set that requires challenging entity disambiguation.
2
Overview
Definition We first define key terminology used in the paper.
Surface Name (sn): An ambiguous entity name that can be interpreted as any of disam- biguated entities, depending on the context. In Figure 1, su is "Judge Day".
A list of disambiguated entities ([DE, DE ..., DE D) A set of n distinct entities that share the same surface name sn. In Figure 1, this will be [Charles Benard Day, Edward William Day, William Louis Day].
Question (qau): A question that contains the surface name sn. The question is ambiguous as sn can refer to multiple entities, each with different answers. In Figure 1, it is "What is the state where Judge Day was born?".
A list of answers (ja,, a2,....,n)) Given the question qan, a; is the answer corresponding
to DE, In Figure 1, it is [ |"Alabama,, "Ohio", "Rhode Island").
A set of documents relevant to the question d2,... dm ): A set of m documents that are relevant to question qan that will be provided to the LMs.
2

---

Published as a conference paper at COLM 2024
Wikiodii Disamiiuutton Page Hughesville
Seed Document
Dataset Generation
3. Answer Set Expansion
Heghes le, Maypand
Higharals Milser
What is the populiation of Hughesville?
Hlugheer Pornaplanta Pareggiun
Higherrlis, Nen Jonny
Courty Maryari
Cuearyy Masial.
Highemila ls Woman
Answer
2,00
Hachevile Lie of scrend Jicns
1. 2,197
QA Model O
Detaset Genarator
O
2,220 4
183
2,220
No answer
Figure 2: Overview of our dataset generation. We identify surface name and a list of disambiguated entities from Wikipedia's 's disambiguation pages. We select two documents for generating a question and their corresponding answers. Subsequently, we gather additional answers from the remaining documents.
Task We study retrieval-augmeeted generation (Lewis et al., 2020; Izacard & Grave, 2020; Sachan et al., 2021), which allows LMs to incorporate external documents at the inference time. Given a question and a set of relevant documents, {9sm, (dmm,. }, LMs are tasked with generating a long-form answer, y, from which we can infer set of disambiguated entities [DE,, DF,, ... DEn] and their corresponding answers, (2,, an) The key dif- ference from prior work on ambiguous QA (Min et al., 2020; Stelmakh et al., 2022) is that we provide set of relevant documents, which contains multiple disambiguated entities and their corresponding answers. This allows an isolated evaluation of LMs on their ability to reason with a confusing document set, without conflating the retriever performance. 4
Evaluation We propose a reference-based evaluation. Given the reference set of disam- biguated entities [DE DE DE and their corresponding answers, [4t az..... (mj and predicted answer y, we categorize the answer y into one of the five categories that we define:
Complete Answer:
LM correctly generates all valid answers in the provided document set. Each answer is paired with the disambiguated entity name.
Partial Answer: LM generates one or more (but not all) of the valid answers in the provided document set. Each answer is paired with the disambiguated entity name.
No Answer: LM abstains from providng any answer.
Ambiguous Answer: LM generates one of the valid answers without providing the disambiguated entity name
Merged Answer: LM generates multiple answers without providing the disambiguated entity name (merging facts about multiple disambiguated entities).
Figure 1 provides an example answer corresponding to each output category. Ideally, LM should generate complete answer. Merged answer can be considered the most problematic, as it may provide confusing or misleading information. The preference among partial answer, no answer, and ambiguous answer can depend on the types of questions and the prominence among disambiguated entities. If one of the disambiguated entities is substantially more likely to be referred by surface name, providing an ambiguous answer corresponding to the likely entity can be more useful than abstaining. In Section 4.2, we will introduce an automatic metric that assigns model outputs to one of the five answer categories. Now we describe how we construct AmbigDocs to study this problem.
Creating AmbigDocs
We build an automated data creation pipeline that can generate large-scale data without manual annotations, once provided with Wikipedia disambiguation pages and access to high-quality LLMs. To ensure the quality of the synthetic data, our pipeline features carefully crafted prompts for sampling initial instances and various checks for filtering low
'Each answer consists of roughly 20 60 words (answer length statistics Table 2). "We additionally provide experimental results with retrieved document sets, but the main analysis performed with the gold document set.
3

---

Published as a conference paper at COLM 2024
quality instances. This follows recent work generating synthetic datasets for complex tasks with LMs (Xie et al., 2023; Yehudai et al., 2024; Oh et al., 2024; Zhao et al., 2024).
Data Instance Each instance in our benchmark consists of {a surface name sn, a question containing the surface name qmn and set of gold documents d' dd)) Each gold document d; contains a disambiguated entity DE; and its corresponding answer d..
Overview We generate data by first identifying two documents, each belonging to disambiguated entity under the same surface name. Next, we generate question and two answers; each answer belongs to a disambiguated entity. This diverges from prior work on this domain that applies heuristics to annotate gold documents for existing ques- tions (Stelmakh et al., 2022; Schuster et al., 2023), which leads to incomplete document annotation. Figure 2 summarizes our process. The first step is to identify (di dfj,d), two documents belonging to different entities which can be referred to by the same surface name and discuss similar topics (Sec. 3.1). The second step is to generate a question qan and answers corresponding to respective entities {ar az (Sec 3.2). As the second step only considers two disambiguated entities per surface name, we expand the answer set to gather remaining plausible answers (Sec 3.3).
3.1 Generating Seed Document Pair
For each surface name sn, we have a set of documents D (10.6 documents on average) gathered from Wikipedia disambiguation page, each belonging to one of the disambiguated entities. We use the Wikipedia snapshot from December 20th, 2018, where each article are split into 100 words (which we consider as documents) (Karpukhin al., 2020). For each disambiguated entity of the snapshot, we collect the documents where the disambiguated entity serves the title. Our goal is to find two documents discussing similar topic, belonging to different disambiguated entities, each containing distinct answer to some ambiguous question 9sn.
For each document pair, we measure their longest n-gram overlap as n. We define relevancy score between pair of documents based on n. The relevancy between a pair is set to 0 if и < 3 or > 10; we set the relevance as π otherwise. Here we set relevancy to be when n 2 10, as this typically indicates pair shares almost identical content, whereas our goal is to find pairs featuring similar topic but different answers. We compute this score for all pairs of documents in D, excluding pairs where both documents are from the same disambiguated entity. If all document pairs receive score of O, we disregard sn, otherwise, we select one document pair with the highest relevancy score, breaking a tie randomly. This process yields total of 67 294 document pairs, one per surface name.
3.2 Generating Ambiguous Question and Answer Pairs
Having identified two documents with confusing entities that cover a similar topic, we proceed to generate the question 9.n that can be answered differently by each document. We use an LLM to generate this: prompting it with (sn, di, d)) along with two-shot exemplars to generate (9sm. a;]] We initially use GPT-4 (T 2023), and then distill its ability to smaller model, Llama2-13b (Touvron et al., 2023), by finetuning with dataset generated from GPT-4 (West et al., 2021). We obtain a total of 43,839 questions after filtering low-quality examples. Details of the generation and filtering process can be found in Appendix B.2.
3.3 Answer Set Expansion
The generated data contains only two answers belonging to two disambiguated entities, while there can be more than two entities sharing the surface name sn. In this final stage, we expand the answer set. As we already have a question Jsn, we find additional answers using an LLM-based QA module. We consider all documents in D, excluding documents that belong to DE,, DEJ. For each document, we use a Llama2-13b model with 4-shot exemplars to generate an answer given the question (the prompt is in Figure 11 in the appendix). As
4

---

Published as a conference paper at COLM 2024
# ans
% Ex.
What is river crossed by the Bow Bridge?
sport Tlamtoen knownn
Who the
Who dimsted
sbout
Cenkury
Luchegg?
2
3
26.6
49.5
What is population density of Lake, Wisconsi??
What is the location the Griewold House?
Viert millway Sharhwel
onvertie
porent of
count
Who somtioaeed
4 13.3
postion did (sener)) Flayes hold iitrr???
when
where
5
6.4
What is the profession of Shahan?
When dld Jarmo
flox (Jedium
over 5
4.3
What are the school colors of Marietta High School?
what DO
mebarre sing?
ran New
de
What does Untversity
Nork
Riako 506
Table 1: Distribution
What was the destination of Flight 387?
100
of the number of answers per question (#ans) in AobigDocs. The maximum num- ber was 10.
Figure Treemap illustrating the distribution of "Wh" ques- tions in AmbigDocs. Each box's size corresponds to its fre- quency, Questions starting with "What" are the most prevalent, reflecting the dataset's focus entities.
the QA module generate incorrect answer, we keep the newly generated answer only when (1) it passes NLI entailment check (Chen et al., 2021b) and (2) length-normalized log perplexity of the answer is below threshold value of 0.2. For the NLI entailment check, we consider the document as the premise. Additionally, we convert 9DE; and a into declarative form (Chen et al., 2021b) to use as the hypothesis, where ADE represents disambiguated question created by replacing sn in the Asn with DEJ. multiple answers are found for the same disambiguated entity, we choose the answer with the lowest perplexity.
3.4 Dataset Analysis
We collected a total of 36,098 examples, where they are randomly split into train/ /den /test splits with ratios of 60% /10% /1%%/3%% respectively. We ensure that there is document overlap between distinct splits. On average, each question has 2.92 answers (distribution in Table 1), covering total of 102,624 distinct entities. In this work, we do not training or validation split, only using the test split for evaluation, but will make all data available.
Figure 3 visualizes questions in AmbigDocs. As our generation is synthetic, we (the authors) manually examine 100 instances to assess its quality. We observe two types of noise: when single document discusses multiple disambiguated entities (2%), and when an LLM generates incorrect answers (4%). Overall, we find question-answer pairs to be mostly valid and answers can be supported by the gold documents. Please refer to Appendix B.3 for further analysis and examples.
4 Evaluation for AmbigDocs
We discuss how to evaluate monel-generated long-form answer y for question 9sn with respect to the reference answer list (21,4,,... paired with its disambiguated entity list [DE DE,,...,
4.1 Standard QA Metrics
We first report automatic metrics, token overlap metric and model-based metric, that has been used in prior work (Adlakha et al., 2023; Stelmakh et al., 2022).
Token Recall We map the answer y to a set of tokens by following the answer processing script of SQuADv2 (Rajpurkar et al., 2018) (which lower cases and splits the string) and remove the stopwords from the nltk package. We perform the same preprocessing for individual answer string a; and disambiguated entity name DE to get token sets. For disambiguated entity name token set, we disregard tokens that appear in the surface name sn to exclude common information and capture the disambiguated entity. For instance, for disambiguated entity name 'Hughesvill,, New Jersey' in Figure/ 2, we will only consider (new, jersey), excluding 'hgghevvllle" which appears in the surface name. Then, define:

---

Published as a conference paper at COLM 2024
The average of recall scores of each gold answer aj, i.. 1. EE y), where R(a;, y) represents the token-level recall of a to y.
Answer Recall:
Entity Recall: The average of recall scores for each disambiguated entities DE i.e.
R(DE, V). where R(DE; .y) represents the token-level recall of DE; to y. Entity-Answer Recall (EAR): We introduce a new metric, which averages the product of answer recall and entity recall: in (R(DE, y) y); By multiplying two scores, we measure how many answers are generated with its disambiguated entity name." 5
Disambig-F1 (DF1) Stelmakh et al. (2022) introduces model-based metric that use RoBERTa-based (Liu et al., 2019) QA model trained on SQuADv2 to evaluate long-form answer. For each a;, the model ta kes disambiguated question and the generated answer, and extracts a short answer d; The Fl-score between extracted answer and gold answer is then averaged across all a; ie. (y J Σ (1 ). To simulate a disambiguated question without explicit human annotation, we replace sn inside 9Nn with DE; for each gold answer. This metric serves the same purpose the EAR metric introduce above.
4.2 Answer Categorization
In the overview (Sec. 2), we have defined five types of answers: complete answer, partial answer, ambiguous answer, merged answer, and no answer. We first present manual categorization and then introduce a heuristic to automatically classify answers.
Manual Annotation We generate long-form answers for 100 randomly sampled questions in the test set with five LMs (upcoming Sec. 5.1 provides the details of the answer generation process). Given the question Asn, the model-generated long-form answer y, and gold answers aij,a, mm paired with DDE categories. We provide the instruction for human evaluation in Appendix C.2. One of the authors annotated the initial data, and another person (who has background in NLP) annotated subset of the initial data (125 items, 5 model outputs for 25 questions). The Cohen's ss Kappa statistic for inter-annotation agreement is 0.85, showing high agreement and the validity of our ontology. We will discuss the distribution of model answer types in the results section (manual annotation distribution can be found in Figure in the appendix).
I the annotator classifies y into one of the five
Automatic Categorization We develop heuristics to automatically classify the generated answer y into one of the categories proposed in Section 2. Let T₃ be the set of tokens mapped from string s. Then, we check whether each reference answer pair (a;, DE,) is:
answered with disambiguation: both ot and DE are mentioned y, i.e. R(at,y) 2 TT
and R(DE,
1
answered without disambiguation: at is mentioned in y but DE is not, ie. R(a 2 2 and R RDE;; Mi
We introduce moving threshold for recall based on the length of the target string, such as |T and |T as longer tokens are less likely appear verbatim. Then we count the number of answers with disambiguation, denoted Cp, and the number of answers without disambiguation, denoted as Cup, for long-form answer y. Then, we can classify into one of five mutually exclusive categories (ı is the number of reference answers):
Complete Answer: Cp = m
Partial Answer: 1 S Cp < m, Cup =
Ambiguous Answer: Cp O, Cup
No Answer: Cp Cxp = 0
Merged Answer: otherwise
SThere may be cases where an answer exploits the metric design, such as DDF is az and DF; is at" However, we have not encountered such cases during the manual inspection 600 model outputs. We increase the numerator to reduce false cases. If Tap S 2, we set R(a y) 2 0.5.

---

Published as a conference paper at COLM 2024
We provide the pseudocode for classifying answers in Figure 7 in the appendix. Next, measure the agreement between automatic metrics and manual annotation.
Verifying Automatic Categorization We compare the answer category labeled by a human (one of the authors) and the category assigned by heuristic, on 500 example outputs. The Cohen's Kappa statistic between the two label sets 0.83, showing strong agreement, though slightly lower than the human agreement of 0.85. We provide the confusion matrix in Figure 5 in the appendix. Some Ambiguous answers are misclassified as Partial answers due to the limitations of lexical-based heuristics, such as plural forms (locomotive vs. locomotives) and spelling variations (organisation vs, organization).
Results
5.1 Experimental Setup
Models We evaluate open-sourced LMs, Llama2 (7b, 13b) (Touvron et al., 2023), Mistral (7b) (Jiang et al., 2023), and two commercial APIs (GPT 3.5 and GPT-4) (OpenAI, 2023).
Data We will evaluate the LMs on the test portion of our data, which contains 7,220 questions. For GPT-4 model, we evaluate on 500 random subsets, owning to its expensive API cost. We pair each question with three types of document sets.
Gold Only (n 5): The gold documents gathered during data generation process (Kand- pal et al., 2022). We randomly sample five documents if there are more than five. Retrieved Only (n 5): We retrieve five documents with question 4sn the query. We use GTR (Ni et al., 2021) as our retriever and Wikipedia snapshot from December 20th, 2018 retrieval corpus. On average, this contains 1.2 gold documents.
Gold + Retrieved (n = 5): We take all gold documents, and add top retrieved documents until we have five documents if there are fewer than five gold documents.
Prompt & Inference The documents are randomly shuffled before being fed into LMs. We present the prompt used for inference in Figure 11. The answers are greedy decoded.
5.2 Standard QA Metric Results
Table 2 presents the model performance on traditional QA metrics. When provided only the gold documents (GOLD ONLY), Mistral and GPT-4 exhibit the strongest performance, followed by Llama2-7b model. Surprisingly, we find some larger models (Llama2-13b, GPT- 3.5) significantly perform worse than smaller models (Llama2-7b, Mistral-7b). One potential hypothesis is that larger models with richer parametric knowledge have bias towards particular contexts or answers, resulting in incomplete answers. Even when provided only gold documents, none of the models achieve an EAR or DF1 score exceeding 0.5, indicating that existing models struggle to distinguish entities that share a surface name. For comparison, we test LMs on the same sets of questions, but providing only one gold document to the model. As opposed to multi-document setting, L.MSs can find the single answer, all achieving 0.70-0.80 EAR and above 0.60 DF1 scores (the full results in Appendix D.1).
When retrieved documents are presented with gold documents (GOLD+RETRIEVED)) per- formance declines, suggesting that non-gold but relevant documents distract the model's reasoning process (Cuconasu et al., 2024). Unsurprisingly, solely providing the retrieved cor- pus (RETRIEVED ONLY) results in significant performance drop across all models. We find that Mistral model shows the best performance in entity recall in all settings, contributing to its strong overall performance.
Difference between Entity-Answer Recall and DF1 metrics Both EAR and DF1 metrics exhibit similar trends, but open-source LLMs show higher EAR scores compared to GPT family, which shows higher DF1 scores. This disparity may be attributed to differences in the
7

---

Published as a conference paper at COLM 2024
Llama2-7b
Mistral-7b
Llama2-13b
GPT-3.5
GPT-
GOLD ONLY
Answer Length ((yl)
53.9
55.1
32.7
17.9
39.7
Answer Recall
0.536
0.561
0.441
0.393
0.632
Entity Recall
0.527
0.643
0.370
0.350
0.596
Entity-Answer Recall
0.372
0.465
0.244
0.234
0.473
Disambig-F1
0.219
0.248
0.178
0,180
0.289
RETREVET
Answer Length (yl)
67.0
59.9
36.0
18.4
40.6
Answer Recall
0.495
0.483
0.383
0.337
0.509
Entity Recall Entity-Answer Recall
0.520
0.571
0.351
0.319
0.482
0.356
0.392
0.219
0.198
0.358
Disambig-F1
0.189
0.199
0.144
0.151
0.222
RETRIEVED ONLY
Answer Length (lyl)
61.4
58.8
33.2
18.2
37.7
Answer Recall
0.370
0.384
0.305
0.294
0.410
Entity Recall Entity-Answe Recall
0.390
0.454
0,289
0.281
0.385
0.242
0.294
0.166
0.167
0.265
Disambig-F1
0.140
0.159
0.122
0.131
0.181
Table 2: Experimental results on the test split. Answer length is measured by the number of words split by whitespace. We bold the highest score and underline the second-highes score for each metric in each row. Mistral-7b and GPT-4 models exhibit strong performance compared to other models.
Complete
Ambiguous
Merged
Llama2-b-
26.9
32.1
4.7
15.2
21.0
Mistral-7b
32.8
41.9
2.0 4.4
18.9
Llama2-13b-
12.9
33.7
7.7
22.7
23.0
GPT-3.5
7.0
47.4
6.5
18.9
20.2
GPT-4
38.9
25.0
3.1
12.0
21.0
Distribution of answer categories (% in each box) under Gold Only setting. Different LMs show different failure modes.
Figure 4:
length of the generated answers. GPT models tend to produce shorter answers compared to open-source LLMs. While lengthy answers may lead to higher recall scores, the QA model may struggle extract gold answers, resulting in lower DF1 score.
5.3 Answer Categorization Results
We categorize model outputs into five output categories with the automatic metric we have built in Section 4.2. Figure 4 depicts the output distribution under the GOLD ONLY setting for five different models. GPT-4 exhibits strong performance, yielding the highest percentage of complete answer. Mistral generates more Partial answers but fewer Merged answers, leading to overall performance comparable to GPT-4. Larger models such as Llama2-13b and GPT-3.5 tend to generate more Ambiguous and Merged answers rather than Complete answers, resulting in poor overall performance. A large percentage of GPT-3.5 outputs are labeled as Partial. We find GPT-3.5 often generates only one correct answer, reflected in short length of generated answers. Table 3, we present an example of generated answers from five LMs.
8

---

Published as a conference paper at COLM 2024
Example
Asn When did Senator Grundy serve in the United States Senate?
(DEj af): (Felix Grundy, 1829 to 1838)
(DE2, a2): (Joseph R. Grundy, December 11, 1929, to December 1, 1930)
Predictions
Llama2-7b: (Merged) Based on the provided context, Senator Grundy served in the
United States Senate from December 11, 1929, to December 1, 1930, and again from October 19, 1829, to July 4, 1838.
Mistral-7b: (Partial) Felix Grundy served in the United States Senate from October 19, 1829, to July 4, 1838.
Llama2-13b: (Ambiguous) Based on the provided context, Senator Grundy served in the United States Senate from October 19, 1829, to July 4, 1838.
GPT-3.5: (Ambiguous)) Senator Grundy served in the United States Senate from October 19, 1829, to July 4, 1838.
GPT-4: (Complete) Joseph R. Grundy served in the United States Senate from December 11, 1929, to December 1, 1930. Felix Grundy served in the United States Senate from October 19, 1829 to July 4, 1838.
Table 3: Generated answers from five different LMs. The surface name is coloured in purple, where the information relevant to DE; and DE2 is coloured in orange and blue, respectively. Classification of each answer is presented next to the model name. LMs exhibit varying behaviours despite given the same prompt.
Llama2-7b
Mistral-7b
Llama2-13b
GPT--3.5
GPT-4
Answer Recall
0.455 (-001)
0.596 (+.035)
0.466 (+.025)
0.408 (+.025)
0.726 (+.094)
Entity Recall
0.444 (-.083)
0.723 (+.080)
0.442 (+.072)
0.397 (+.047)
0.796 (+.200)
Entity-Answer Recall
0.291 (-.081)
0.504 (+.039)
0.288 (+.044)
0.273 (+.039)
0.647 (+.174)
Disambig-F1
0.212 (-:007)
0.297 (+.049)
0.215 (+.037)
0.202 (+.022)
0.382 (+.093)
Table 4: Few-shot results under Gold Only setting. We provide the difference from the zero-shot result in the parenthesis.
5.4 In-Context Few-shot Learning Results
Prior work shows that in-context examples can help LMs' reasoning process (Liu et al., 2021; Lee et al., 2024). To help LMs, we craft two in-context examples with complete answer. Table presents the results under GOLD ONLY setting with two-shot demonstrations (prompt in Figure 12 in appendix). We observe performance gains across all models, except for Llama2-7b, particularly in entity recall. GPT-4 exhibits the most substantial gain among all models, nearly three times more than others. In the appendix (Table 10), we also present the answer category distribution. All models, except for Llama2-7b, show a significant increase in the number of Complete answers. We find that larger models generate less Ambiguous and Merged answers than before, likely due to their learning via few-shot examples. These findings encourage future works on teaching models to distinguish confusing entities and provide Complete answers.
5.5 Evaluating Answer Precision
So far, we measured the recall of the reference answer set. Here, we report K-Precision (Adlakha et al., 2023), the percentage of tokens in y that are from input documents during inference, to evaluate its faithfulness to input documents. We observe the K-precision scores are over 0.84 for all models and evaluation settings, indicating the model generates fairly extractive answers. The full result is reported in Table 11 in the appendix.

---

Published as a conference paper at COLM 2024
6 Related Works
Coreference Resolution Our task is closely related to multi-document coreference resolu- tion task (Yang et al., 2015; Cattan et al., 2021) in that it requires entity mention resolution across multiple documents. Yet, our task contains a purposefully ambiguous entity mention in the question. Thus, the entity mention in question can be mapped to multiple entities, making it hard to derive and evaluate against gold entity clusters.
Complex Reasoning with Retrieval Augmentation While language models are equipped with rich parametric knowledge (Roberts et al., 2020), providing relevant documents in context often results in improved performances for various end tasks (Lewis et al., 2020; Asai et al., 2024). Wan et al. (2024); Tan al. (2024) provides document set containing conflicting answers to contentious questions, while other work (Xie et al., 2023; Chen et al., 2022) simulates a situation where the document set and parametric knowledge of LM collide. WikiHowQA QA Dolotova-BBarnnove et al., 2023) examines multi-document reasoning of LLMs, without engaging conflicting documents. We study a setting where we provide a confusing set of documents which requires careful entity disambiguation. Concurrent to us, Chiang & Lee (2024) studies generating biographies from a set of documents that contains multiple people with the same name. While this work also studies similar entity confusion errors of LMs, its goal is to improve a factuality metric (Min et al., 2023). We cover a much broader set of entities and focus on multi-document reasoning for question answering. Similar to our answer categorization ontology, Mishra et al. (2024) establishes different types of factuality generation, while our study focuses on entity disambiguation. Compared to ontology from Chiang & Lee (2024), our ontology is more fine-grained, distinguishing ambiguous and partial outputs.
error. Their ontology aims evaluation of general aitrieval-augmente
Ambiguity in Question Answering Prior work studied ambiguous question answering in simple factoid QA (Min et al., 2020; Zhang & Choi, 2021) and long-form QA (Stelmakh et al., 2022) setting. While ambiguity can arise for multiple reasons (Amplayo et al., 2023), we focus on entity disanbiguation. Chen et al. (2021a) studied entity linking for the ambiguous entity in a question (e.g., linking ambiguous surface name "Apple" in the question such as "What is the record label of Apple?" to its correct Wikipedia entity). Our task setting, which aims to generate a long-form answer covering multiple answers given question and a set of relevant documents, is equivalent to recent QuoteSum (Schuster et al., 2023) dataset. However, their focus is on generating and evaluating extractive answer and does not model whether LMs disambiguate entity mentions. We explicitly simulate confusing document sets and provide metrics to identify whether LMs generate a coherent and complete answer with entity disambiguation.
LLMs can handle ambiguous questions differently than what is explored in this work, which is to present all possible answers with disambiguation (Amplayo et al., 2023; Kim et al., 2023; Sun et al., 2023). These include asking clarifying questions to allow users to specify the intended interpretation of the question (Zhang & Choi, 2023; Lee et al., 2023; Kuhn et al., 2022) or evaluating scores for each candidate answer (Cole et al., 2023).
7 Conclusion
This work studies the reasoning capabilities of language models across documents featuring distinct entities that share a surface name. We introduce AmbigDocs, where we identify such documents, paired with an ambiguous question and the answers from each document. Our empirical findings disclose that language models often fail at generating complete answers, exhibiting varying types of behaviours. We hope our findings encourage future investigations on LLMs when faced with entities that are hard to disambiguate.
While we did not explore in this work, the dataset can be used for other tasks such as fact verification (Min et al., 2023) and multi-document summarization (Chiang & Lee, 2024). Future work can also explore fine-grained analysis on the various factors that contribute to determining incomplete answers.
10

---

Published as a conference paper at COLM 2024
Acknowledgments
We thank the members of UT Austin NLP community for valuable feedback, especially to Hung-Ting Chen, Michael J.Q. Zhang, and Fangyuan Xu. We also thank Hyunji Lee for helpful discussions. The study is partially funded by NSF grant IIS-2312248. This work was supported by Korea Institute for Advancement of Technology (KIAT) grant funded by the Korea Government (Ministry of Education) (20256511002222200022200-1144440 momniconductor-Speiiliiie University).
References
Vaibhav Adlakha, Parishad BehnamGhader Xing Han Lu, Nicholas Meade, and Siva Reddy, Evaluating correctness and faithfulness of instruction-folloiing models for question answering. arXiv:2307.16877 2023.
Reinald Kim Amplayo, Kellie Webster, Michael Collins, Dipanjan Das, and Shashi Narayan. Query refinement prompts for closed-bood long-form qa. In Anmual Meeting of the Associa- tion for Computational Linguistics, 2023.
Akari Asai, Zexuan Zhong, Danqi Chen, Pang Wei Koh, Luke Zettlemoyer, Hannaneh Hajishirzi, and Wen-tau Yih. Reliable, adaptable, and attributable language models with retrieval. Xiv preprint arXio:2403.03187, 2024.
Valeriia Dolotova-Baranova, Vladislav Blinov, Sofya Filippova, Falk Scholer, and Mark Sanderson. Wikihowqa: comprehensive benchmark for multi-document non-factoid question answering. In Annull Meeting of the Association for Computational Linguistics, 2023.
Samuel R. Bowman, Gabor Angeli, Christopher Potts, and Christopher D. Manning. A
large annotated corpus for learning natural language inference. In Conference Empirical Methods in Natural Language Processing, 2015.
Arie Cattan, Alon Eirew, Gabriel Stanovsky, Mandar Joshi, and Ido Dagan. Cross-document coreference resolution over predicted mentions. ArXiv, abs/ 0t//2200.01220 2021.
Anthony Chen, Pallavi Gudipati, S. Longpre, Xiao Ling, and Sameer Singh. Evaluating entity disambiguation and the role of popularity in retrieval-based nlp. In Annual Meeting of the Association for Computational Linguistics, 2021a.
Hung-Tine Ting Chen, Michael J.Q. Zhang, and Eunsol Choi. Rich knowledge sources bring complex knowledge conflicts: Recalibrating models to reflect conflicting evidence. In Conference on Empirical Methods in Natural Language Processing, 2022.
Jifan Chen, Eunsol Choi, and Greg Durrett. Can nli models verify qa systems' predictions?
In Conference on Empirical Methods in Natural Language Processing, 2021b.
Cheng-Han Chiang and Hung-Yi Lee. Merging facts, crafting fallacies: Evaluating the contradictory nature of aggregated factual claims in long-form generations. arXiv preprint arXiv:2402.05662, 2024.
Jeremy R. Cole, Michael J.Q. Zhang, Daniel Gillick, Julian Martin Eisenschlos, Bhuwan Dhingra, and Jacob Eisenstein. Selectively answering ambiguous questions. ArXiv, 0bss/2305.1661 2023.
Florin Cuconasu, Giovanni Trappolini, Siciliano, Simone Filice, Cesare Campagnano, Yoelle Maarek, Nicola Tonellotto, and Fabrizio Silvestri. The power of noise: Redefining retrieval for rag systems. ArXio, abs/2401.14887. 2024.
Tim Dettmers, Artidoro Pagnoni, Ari Holtzman, and Luke Zettlemoyer. Qlora: Efficient finetuning of quantized lims. ArXiv, abs/ 2005//335.4431,, 2023.
Or Honovich, Roee Aharoni, Jonathan Herzig, Hagai Taitelbaum, Doron Kukliansy, Vered Cohen, Thomas Scialom, Idan Szpektor, Avinatan Hassidim, and Yossi Matias. True: Re-evaluating factual consistency evaluation. In Workshop on Document groundded Dialogue and Conversational Question Answering, 2022.
11

---

Published as a conference paper at COLM 2024
Gautier Izacard and Edouard Grave. Leveraging passage retrieval with generative models for open domain question answering. ArXiv, abs/ abs/ 2007001202 2020.
Albert Qiaochu Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Deven- dra Singh Chaplot, Diego de Las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, L'elio Renard Lavaud, Marie-Anne Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas Wang, Timothée Lacroix, and William El Sayed. Mistral 7b. ArXia, abs/ 2310.0622, 2023.
Nikhil Kandpal, H. Deng, Adam Roberts, Eric Wallace, and Colin Raffel. Large language models struggle to learn long-tail knowledge. In International Conference on Machine Learning, 2022.
Vladimir Karpukhin, Barlas Oguz, Sewon Min, Patrick Lewis, Ledell Yu Wu, Sergey Edunov, Danqi Chen, and Wen tau Yih. Dense passage retrieval for open-domain question answer- ing. ArXiv, 00bs/2004.44006, 2020.
Gangwoo Kim, Sungdong Kim, Byeongguk Jeon, Joonsuk Park, and Jaewoo Kang. Tree of clarifications: Answering ambiguous questions with retrieeal-augmented large language models. In Conference on Empirical Methods in Natural Language Processing, 2023.
Lorenz Kuhn, Yarin Gal, and Sebastian Farquhar. Clam: Selective clarification for ambiguous questions with generative language models. arv preprint arXiv:2212.07766 2022.
Dongryeol Lee, Segwang Kim, Minwoo Lee, Hwanhee Lee, Joonsuk Park, Sang-Woo Lee, and Kyomin Jung. Asking clarification questions to handle ambiguity in open-domain qa. ArXiv, abs/ https/2305.18000 13808, 2023.
Yoonsang Lee, Pranav Atreya, Xi Ye, and Eunsol Choi. Crafting in-context examples according to Ims' parametric knowledge. In Findings of the Association for Computational Linguistics: NAACL 2024, pp. 2069-20888 2085, 2024.
Patrick Lewis, Ethan Perez, Aleksandara Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Kuttler, Mike Lewis, Wen tau Yih, Tim Rocktäschel, Sebastian Riedel, and Douwe Kiela. eetreeallaugmmeneed generation for knowledge-intensiee nlp tasks. ArXiv, 0tt/20005.114411 2020.
Jiachang Liu, Dinghan Shen, Yizhe Zhang, Bill Dolan, Lawrence Carin, and Weizhu Chen. What makes good in-context examples for gpt-3? In Workshop on Knowlledge Extraction and Integration for Deep Learning Architectures; Deep Learning Inside Out, 2021.
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. Roberta: robustly optimized bert pretraining approach. ArXiv, abs/ htss/1907.10 1969,11692, 2019.
Sewon Min, Julian Michael, Hannaneh Hajishirzi, and Luke Zettlemoyer. Ambigqa: An- swering ambiguous open-domain questions. Conference on Empirical Methods in Natural Language Processing, 2020.
Sewon Min, Kalpesh Krishna, Xinxi Lyu, Mike Lewis, Wen tau Yih, Pang Wei Koh, Mohit lyyer, Luke Zettlemoyer, and Hannaneh Hajishirzi. Factscore: Fine-grained atomic evaluation of factual precision in long form text generation. ArXiv, abs/2305.14255 2023.
Abhika Mishra, Akari Asai, Vidhisha Balachandran, Yizhong Wang, Graham Neubig, Yulia Tsvetkov, and Hannaneh Hajishirzi. Fine-grained hallucination detection and editing for language models. ArXiv, abs/2401 2ass/2400.6885, 2024.
Jianmo Ni, Chen Qu, Jing Lu, Zhuyun Dai, Gustavo Hernandez Abrego, Ji Ma, Vincent Zhao, Yi Luan, Keith B. Hall, Ming-Wo Wei Chang, and Yinfei Yang. Large dual encoders are generalizable retrievers. ArXiv, abs/ 221 12.07889, 2021.
Hanseok Oh, Hyunji Lee, Seonghyeon Ye, Haebin Shin, Hansol Jang, Changwook Jun, and Minjoon Seo. Instructir: A benchmark for instruction following of information retrieval models. Xiv preprint arXim:2402.14334 2024.
12

---

Published as a conference paper at COLM 2024
OpenAl. Gpt-4 technical report. 2023.
Pranav Rajpurkar, Robin Jia, and Percy Liang. Know what you don't know: Unanswerable 2018.
questions for squad. ArXiv,
Adam Roberts, Colin Raffel, and Noam M. Shazeer. How much knowledge can you pack into the parameters of a language model? In Conference on Empirical Methods in Natural Language Processing, 2020.
Devendra Singh Sachan, Siva Reddy, William Hamilton, Chris Dyer, and Dani Yogatama. End-to-end training of multi-document reader and retriever for open-domain question answering. ArXiv, abs/ 22166005366 2021.
Tal Schuster, Ádám Dániel Lelkes, Haitian Sun, Jai Gupta, Jonathan Berant, William W. Cohen, and Donald Metzler. Semqa: Semi-extractive multi-source question answering. ArXiv, abs/a311.000 22////1310.8866 2023.
Ivan Stelmakh, Yi Luan, Bhuwan Dhingra, and Ming-Wei Chang. Asqa: Factoid questions meet long-form answers. ArXiv, abs/ 22044.6029, 2022.
Weiwei Sun, Hengyi Cai, Hongshen Chen, Pengjie Ren, Zhumin Chen, Maarten de Rijke, and Zhaochun Ren. Answering ambiguous questions via iterative prompting. In Annual Meeting of the Association for Compulational Linguistics, 2023.
Hexiang Tan, Fei Sun, Wanli Yang, Yuanzhuo Wang, Qi Cao, and Xueqi Cheng. Blinded by generated contexts: How language models merge generated and retrieved contexts for open-domain qa? ArXiv, abs/ 0bs/2401.11911, 2024.
Hugo Touvron, Louis Martin, Kevin R. Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei,, Nikolay Bashlykov, Soumya Batra, Prajiwal Bhargava, Shruti Bhosale, Daniel M. Bikel, Lukas Blecher, Cristian Cantón Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, Brian Fuller, Cynthia Gao, Vedanuj Goswami, Naman Goyal, Anthony S. Hartshorn, Saghar Hosseini, Rui Hou, Hakan Inan, Marcin Kardas, Viktor Kerkez, Madian Khabsa, Isabel M. Kloumann, A. V. Korenev, Punit Singh Koura, Marie-Anne Lachaux, Thibaut Lavril, Jenya Lee, Diana Liskovich, Yinghai Lu, Yuning Mao, Xavier Martinet, Todor Mihaylov, Pushkar Mishra, Igor Molybog, Yixin Nie, Andrew Poulton, Jeremy Reizenstein, Rashi Rungta, Kalyan Saladi, Alan Schelten, Ruan Silva, Eric Michael Smith, R. Subramanian, Xia Tan, Binh Tang, Ross Taylor, Adina Williams, Jian Xiang Kuan, Puxin Xu, Zhengxu Yan, lliyan Zarov, Yuchen Zhang, Angela Fan, Melanie Kambadur, Sharan Narang, Aurelien Rodriguez, Robert Stojnic, Sergey Edunov, and Thomas Scialom. Llama 2: Open foundation and fine-tuned chat models. ArXio, abs/ 0099880, 2023.
Alexander Wan, Eric Wallace, and Dan Klein. What evidence do language models find convincing? Xiti preprint ar arXtt:2402.11782, 2024.
Peter West, Chandrasekhar Bhagavatula, Jack Hessel, Jena D. Hwang, Liwei Jiang, Ronan Le Bras, Ximing Lu, Sean Welleck, and Yejin Choi. Symbolic knowledge distillation: from general language models to commonsense models. In North American Chapter of the Association for Computational Linguistics, 2021.
Jian Xie, Kai Zhang, Jiangjie Chen, Renze Lou, and Yu Su. Adaptive chameleon or stubborn sloth: Unraveling the behavior of large language models in knowledge clashes. ArXiv, 2bs/0305.33300, 2023.
Bishan Yang, Claire Cardie, and Peter Frazier. A hierarchical distance-dependent Bayesian model for event coreference resolution. Transactions of the Association for Computational Linguistics, 3:517-528, 2015. doi: 10.1162/tacla.0000
Asaf Yehudai, Boaz Carmeli, Yosi Mass, Ofir Arviv, Nathaniel Mills, Assaf Toledo, Eyal Shnarch, and Leshem Choshen. Genie: Achieving human parity in content-grounded datasets generation. ArXiv, /a/s//201.1434 2024.
13

---

Published as a conference paper at COLM 2024
Ori Yoran, Tomer Wolfson, Ori Ram, and Jonathan Berant. Making retrieval-augmented language models robust to irrelevant context. ArXio, 20100//3101 2023.
Michael J.Q. Zhang and Eunsol Choi. Situatedqa: Incorporating extra-linguistic contexts into qa. ArXio, abs (bbs/21090611,, 2021.
Michael J.Q. Zhang and Eunsol Choi. Clarify when necessary: Resolving ambiguity through interaction with lms. ArXiv, abs/ 0ts/2331.094450 2023.
Bowen Zhao, Zander Brumbaugh, Yizhong Wang, Hannaneh Hajishirzi, and Noah A Smith. Set the clock: Temporal alignment of pretrained language models. arviv preprint arXi::2402.16777, 2024.
14

---

Published as a conference paper at COLM 2024
A Models
Open-source models in our paper are from huggingface. We provide the model names here:
Llama-2-7b-chat-ff
t5 5_xx1_truee true nli mixture
2-13b----b-
gtr-t5-xx1
Mistral-7B-Insttrut--0-.2
question converter-3b
B Dataset Generation Details
B.1 Wikipedia Disambiguation Pages
We use the Wikipedia API identify disambiguation pages and gather ambiguous entities along with their disambiguations from these pages. We only take that su have 10 or fewer resolutions, and DE, that have 20 or fewer documents.
B.2 Model Distillation
Due the computational cost associated with GPT-4, we initially generate 1500 questions. Following prior works in automatic dataset construction (Xie et al., 2023; Yehudai et al., 2024), we conduct post hoc filtering to ensure data quality. We divide the filtration into three steps and present the proportion of removed questions during each step in Table 5.
Not Formatted: We remove the questions if (1) the generated output does not follow the format in Figure 9, or (2) each answer contains too much sentences (answer that contains more than 4 periods).
Not Ambiguous: We remove the questions if (1) Asn does not contain sn, (2) 9mn con- tains both DET and DES, (3) 9sn contains prompt-specific phrases such as 'passage' and 'context', (4) one answer contains another, or (5) two answers are too similar (IoU(a;, 45) > 0.75).
Not Entailed: We assess the faithfulness of each answer by framing as a Natural Language Inference (NLI) problem (Bowman et al., 2015). We substitute su in the question with DE,, creating a disambiguated question 9DE We convert 4DE; and a; into declarative form (Chen et al., 2021b) to use as the hypothesis, while the gold document serves as the premise. We employ T5-11B NLI model (Honovich et al., 2022) to evaluate entailment between hypothesis and premise, retaining only instances where the premise is entailed by hypothesis for both answers.
Not formatted
Not ambiguous
Not entail
Success
GPTT4
0.0%
8.0%
26.5%
65.5%
Llama2-13b
1.7%
8.4%
23.3%
66.6%
Table 5; Percentage of questions removed during three filtration steps. Success indicates the questions that were not removed, and therefore collected as our dataset.
This process results in 983 questions, accounting for 65.5% of the original 1500 questions. We distill the generation capabilities of GPT-4 to the lama2-13b-cha model by finetuning Llama2-13b-cha model with QLoRA (Dettmers al., 2023) for 5 epochs, with learning rate of 2e-4. The training script is taken from Yoran et al. (2023). After repeating the same generation and filtration steps to the remaining seed pairs, we obtain total of 43,839 questions, achieving success rate of 66.6%, comparable to that of GPT-4.
15

---

Published as a conference paper at COLM 2024
Train
Dev
Test
Total
Number of questions
25,268
3,610
7,220
36,098
Number of covered disambiguated entities
71,641
10,430
20,733
102,624
Average number of answers per question
2.93
2.89
2.87
2.92
Average answer length
8.48
8.58
8.38
8.47
Table 6: Statistics of our dataset. Answer length is measured by the number of words split by whitespace.
known 11.2%
profession 7.5%
refer 5.7%
primary 5.5%
population 5.4% located 5.0%
main 4.8%
location 4.1%
Table 7: Frequency of the most common words in the question after being converted to lowercase, removing stopwords, and split by whitespace. The questions fall into two categories: those seeking specific information (profession, population, location, primary, main) and those requesting clarification ambiguous entities (known, refer).
B.3 Dataset Analysis
Table 6 presents detailed statistics of dataset and Table 7 presents the eight most common words appearing in the questions. Additionally, we present three examples of our dataset in Table 8.
Furthermore, we evaluated the quality of 100 instances of our dataset and identified two types of noise. The first occurs when document discusses multiple entities. For exam- ple, the document about the person 'Francisco Prestes Maia' also mentions 'Prestes Maia (building)', which another disambiguated entity: In such cases, the generated question asks about the building. The second type of noise arises when an answer is incorrect, which typically occurs during the answer expansion process. We leave employing a better QA model and developing better filtration methods for future work.
C Evaluation Details
C.1 Preprocessing
1.0
Some surface names contain "(disambigua- tion)' at the end, which we remove before using anywhere. In addition, some dis- ambiguated entities contain phrases enclosed in parenthesis, e.g. 'Michael Jordan (mycolo- gist)'. We remove the parenthesis only when mapping the string into tokens. During map- ping, removing token overlaps of su may re- sult in an empty set of tokens. For such cases, we compute the exact match of the disam- biguated entity, ie. DE iny.
No AROWOS Merqed Aeriquuuus Fenual camplec
0.83
0.01
Numan Anahyate
0.03
0.15
0.73
0.02
0.05
0.4
0.02
0.59
Completo
Palttal
Ambiguous
Menged
No Arcwer
Quantilier
C.2 Instruction for Human Annotation
Figure 5: Confusion matrix comparing quan- tifier metrics with our manual analysis (n=500). Our automatic classification shows high agreement with manual classification, although Ambiguous answers are some- times misclassified as Partial answers.
The instructions provided to human annota- tors include the overview illustrated in Figure the content outlined in Section 2, and a sam- ple annotation interface (Figure 6). For each y, the annotator is provided with the follow-
16

---

Published as a conference paper at COLM 2024
qsn Who is the artist of the "Cherry Bomb" project?
DEj: Cherry Bomb (EP)
di: Cherry Bomb (EP) Cherry Bomb is the third extended play of NCT 127, the Seoul- based sub-unit the South Korean boy group NCT. It was released by SM
df: NCT 127
DE3: Cherry Bomb (album)
d3: Cherry Bomb (album) Cherry Bomb is the third studio album by American rapper Tyler, the Creator. was released on April 13, 2015,
a: Tyler, the Creator
qn When was Nengren Temple first established?
DET: Nengren Temple (Guangzhou)
4: Nengren Temple (Guangzhou) Nengren Temple () is a Buddhist temple located was first established by Yinjian () in 1824, during the reign of Daoguang Emperor (1821-1850) of the Qing dynasty (1644-1911)....
4:: 1824
DE3: Nengren Temple (Biujiang)
d3: Nengren Temple (Jiujiang) Nengren Temple () is a Buddhist temple located was first built in the Northern and Southern dynasties (420-589), and went through many changes and repairs through the following dynasties....
45 Northern and Southern dynasties (420-589)
qsn What is the primary use of Lake Whitney?
DET: Lake Whitney (Connecticut)
di: Lake Whitney (Connecticut)... ... Now with new water treatment facility rated for up to 15 million gallons per day, Lake Whitney has been reconnected as a reserve water source for the South Central Connecticut Regional Water Authority. This act has ...
af: Water source
DE3: Lake Whitney (Texas)
d;: Lake Whitney (Texas) Lake Whitney is a flood control reservoir on the main stem of the Brazos River in Texas. It is located on River Mile Marker 442
4; Flood control reservoir on the main stem of the Brazos River
DE5: Lake Whitney Ranch
dž: were presented in various town hall meetings where ... Lake Whitney Ranch is property near Whitney, Texas that is under development as summer youth camp, Pathfinder camp, church camp and conference center for az: Summer youth camp, Pathfinder camp, church camp and conference center
Table 8: Three examples from our dataset. The question is presented at the first line, with the bold text indicating the surface name sn. Below the question, we present the gold documents, each paired with its disambiguated entity and the answer. Bold text in each document highlights the evidence for the answer.
ing: (sn, [DE DEE (4 Subsequently, the annotator classifies into one of the five categories by selecting from a dropdown menu. The distribution of human-annotated answers presented in Figure 8.
C.3 Automatic Categorization
Figure 7 describes the pseudocode for classifying a generated answer into one of five categories. We present the confusion matrix comparing automatic metrics and our manual analysis in Figure 5.
17

---

Published as a conference paper at COLM 2024
Disamiiguaeed Erdty Answer
Ambiguous Enity
Disambiguteed Entty
Ckstar
Model's Generaled Arsewe
(This is an example (please feel free fide)
Beraad Alsbama
Edward Willaam
What where Judge born?
ambiguous
Judga Day was bom Oiiio.
merged
Judge Day
partal
Willaa Louis Day was bomn In Ohic, and Chares Bemard Day was born in Nabamo.
complete
The Bernant Day was
Willia Day was Rhode
fall
provide information.
Figure 6: Interface for the human annotation process. For each model generated answer, the annoatator is provided the surface name, the question, and the list of reference (disam- biguated entity, gold answer) pairs. The dropdown menu contains five categories, where the annotator can easily select one.
We do not provide which model y was generated from.
Input: question 9su, reference [DF,, DES,..,, D, model-generated long-form answer Output: Category cls of among (Complete answer, Partial answer, No answer, Ambiguous answer, Merged answer)
list -n. disambiguated entity list
1: Cp Cup 0 to m do 3: Toz - GetTokens(a))
TDE; GefTokens(DE.)
5: if ifa 2 [Tal and R(DE, 2 Fogy
1 then
6:
Cp - Cp +1
7: else if R(a,,y) 2 [Ta
2 and R(DE7,y) = 0 then
8: Cup + Cup +1 1
9: end if
10:
end for
11:
12: if Cp = 0 and Cup = then
13: cls No answer
14: else if c,
then
15: cls Complete answer
16: else if cp = 0 and Cup = then
17: cls - Ambiguous answer
18: else if 2 and Cap 0 then
19: cls Partial answer
20: else
21: cls + Merged answer
22: end if
23:
24: return cls
Figure 7: Algorithm for automatically classifying the generated answer y into one of five categories.
D Experimental Details
D.1 Single Document Study
We provide only one gold document to the LM and examine if the model has identified the correct answer. Since the model is not given multiple entities, the Ambiguous answer is resolvable. Therefore, we compute entity recall twice, once with DE and once with sn, and select the higher value. We do the same for the DF1 metric, both with and without
18

---

Published as a conference paper at COLM 2024
Complete
Partial
No Answer
Ambiguous
Merged
Llma2-7b
23
33
Mistral-7b
Llama2-13b
GPT-3.5
50
44
GPT-4
39
Figure 8: Distribution of human annotated answer categories (% in each box) under Gold
Only setting.
Llama2-7b
Mistral-7b
Llama2-13b
GPT-3.5
GPT-4
Answer Recall
0.784
0.852
0.768
0.804
0.855
Entity Recall
0.985
0.988
0.990
0.943
0.956
Entity-Answer Recall
0.770
0,841
0.762
0.764
0,816
Disambig-Fl
0.644
0.613
0.656
0.625
0.648
Table 9: Results for the single document study. Results indicate that when provided with only one gold document, the model finds the answer correctly.
replacing sn with DE. Table 9 presents the results. Unsurprisingly, all models exhibit the highest answer recall scores throughout our study, well as other metrics.
D.2 In-Context Few-shot Learning
Table 10 shows the answer category distribution for few-shot experiment in Section 5.4. All models, except for Llama2-7b, demonstrate improvements in generating Complete answers. Moreover, larger models generate fewer Ambiguous and Merged answers, suggesting that they may have learned how to generate Complete answers via few-shot examples.
D.3 Answer Precision
Table 11 presents K-Precision scores of the models across experimental settings. We observe that the generated answers are well factually grounded to the reference documents.
E Prompts
We present the prompts used throughout our paper in Figure 9-12.
19

---

Published as a conference paper at COLM 2024
Llama2-7b
Mistral-7b
Llama2-13b
GPT-3.5
GPT-4
Complete
0.203 (-066))
0.435 (+.107)
0.215 (+.086)
0.101 (+.031)
0.586 (+.197)
Partial
0.331 (+.010)
0.326 (-0933
0.303 (-.034)
0.486 (+.012)
0.240 (-011)
No Answer
0.072 (+.025)
0.017 (-003)
0.061 (-0166)
0.076 (+-011)
0.012 (-019)
Ambiguous
0.178 (+.026)
0.033 (-001)
0.193 (-.034)
0.150 (-.039)
0.034 (-006)
Merged
0.217 (+.007)
0.189 (+,000)
0.227 (-0003)
0.186 (-016)
0.128 (-0882
Table 10: Automatic quantifier metrics for few-shot experiments under Gold Only setting. We provide the difference from zero-shot result the parenthesis.
Llama2-7b
Mistral-7b
Llama2-13b
GPT-3.5
GPT-4
GOLD ONLY
0.841
0.843
0.852
0.933
0.898
GOLD+RETRIEVED
0.841
0,886
0,868
0.945
0.919
RETRIEVED ONLY
0.850
0.898
0.875
0.949
0.934
Table K-precision scores on the test split.
20

---

Published as a conference paper at COLM 2024
You are given two passages about an ambiguous entity with different interpretations. Create question about the entity, which each passage must answer differently. Find the shortest span of each passage as an answer to each passage.
Entity: Young Conservatives
Passage 1: Young Conservatives (Czech Republic) The Young Conservatives is a political youth organisation in the Czech Republic. is the youth wing of the Civic Democratic Party (ODS), centre-right political party, and shares that party's conservative and economically liberal ideology. Young people within the age from 15 to 35 apply for membership in the MK. Several significant politicians from the ODS party started as members of Young Conservatives, including Jan Zahradil, Ji015900ed Posp00ed016lil, Petr Sokol, Martin Baxa, Petr Gandalovi010d, Ivan Langer, Martin NovotnOOfd, and Pavel Drobil. Former Chairman of Young Conservatives Petr Mach went on to found a Passage 2: Young Conservatives (UK) The Young Conservatives (YC) is the youth wing of the Conservative Party in the United Kingdom for members aged 25 and under. The organisation shares the same values and policies as its parent political party with branches being an integrated part of local associations, with the exception of college and university branches which are run independently. YC is both social and political, aiming to bring together young conservatives and encouraging young people to get involved in campaigning. The Junior Imperial and Constitutional Leaguewas formed in 1906 with objectives to encourage practical political work and organisation among
Question: What the age range for membership in the Young Conservatives?
Answer to Article 1: 15 to 35
Answer to Article 25 and under
Entity: {sm)
Passage 1: {DE [di)
Passage 2: (DE; (d))
Question:
Figure Prompt template for generating question and two corresponding answers with fol- lowing placeholders: sn for the shared surface name, di and d for selected seed documents, and DE and DEŽ for its corresponding entities. Due the limited space, we provide only one-shot example. We expect the model to generate {9sm. [a],a]]] a,]), adhering to the format in the template.
Given a passage related to the question, answer to the question. Find the shortest span of the passage as an answer.
Passage: Proletarian Unity Party (France) Proletarian Unity Party (France) The Party of Proletarian Unity (, PUP) was a French socialist political party. It was formed December 21, 1930 by leftists expelled from the French Communist Party (PCF), together with some who had previously belonged to the left-wing of the Section franO0e7aise de l'Internationale ouvri00eSre (SFIO). Its members were known France as pupistes, and one of its notable leaders was Alexandre Bachelet. Owing to proportional representation, it at one time had ten seats in the Chamber of Deputies of the Third Republic. The PUP affiliated to the London Bureau of left-socialist parties. On January Question: When was the Proletarian Unity Party formed?
Answer: December 21, 1930
Passage: {DE,) {d.}
21

---

Published as a conference paper at COLM 2024
Question: {9sH}
Answer:
Figure 10: Prompt template for answer set expansion with following placeholders: 9sn for the question and d; for the document related to DE,. Due to the limited space, we provide only one-shot example.
Answer for the given question using only the provided context.
Question: qsn
Context: (DE) (ay)
Context:
(m)) {d= ((m)
Answer:
Figure 11: Prompt template for inference. We random shuffle the documents and map d to 700
Answer for the given question using only the provided context.
Question: What the location of the Griswold House?
Context: Griswold House (Guilford, Connecticut) - a limited number of days each week. The Griswold House is located east of Guilford Center, on the south side of Boston Street (Connecticut Route 146) at its junction with Lovers Lane. It 2-1/2 story wood frame structure, with a gabled roof, large central chimney, and clapboarded exterior. A leanto section to the rear gives the house classic New England saltbox appearance. The main facade three bays wide, with sash windows arranged symmetrically around the entrance. The entrance is flanked by pilasters and topped by a fully pedimented gable. The house was probably built around 1764
Context: John N. A. Griswold House - John N. A. Griswold House The John N. A. Griswold House a National Historic Landmark 76 Bellevue Avenue in Newport, Rhode Island. It is home to the Newport Art Museum and houses an art gallery. The home was built in 1864 by Richard Morris Hunt for John Griswold, an Old China Trade merchant and member of the Griswold Family. The house is one of the earliest American Stick2013sth 3stylee buildings and one of Hunt's first works in Newport. The house is an official project of Save America2019; Treasures. The Griswold House is 2-1/2 story wood frame structure, set Answer: There are two Griswold Houses mentioned in the passages. John N. A. Griswold House located at 76 Bellevue Avenue in Newport, Rhode Island and Griswold House (Guilford, Connecticut) is located east of Guilford Center, on the south side of Boston Street (Connecticut Route 146) at its junction with Lovers Lane.
Question: 9s
Context: (DE (d))
Context: (DE (d (m)
Answer:
Figure 12: Prompt template for few-shot learning. We random shuffle the documents and map de to d') Due to the limited space, we provide only one-shot example.
22


Reviews:
[{"review_id": "HGEWfixGVJ", "reviewer": "Reviewer_JT3K", "summary": "This paper proposes AmbigDocs, a new benchmark for studying the behavior of LLMs on ambiguous document-based QA, and a methodological pipeline that can be used to create similar benchmarks from Wikipedia disambiguation pages using LLMs. The questions in the benchmark are ambiguous in the sense that they contain a surface form of a named entity, which may refer to multiple actual entities in the answer documents.\n\nThe paper also gives a new spin to the definition of an answer in the context of LLMs and document-based QA and provides a taxonomy of 5 answer types, against which several SOTA closed- and open-source LLMs (LLaMa 7b and 13b, Mistral 7b, GPT-3.5 and GPT-4) are evaluated. The benchmark and the answer taxonomy are the main contributions.\n\nThe key findings are twofold: 1) generating complete answers for ambiguous questions in AmbigDocs is a challenging task for the open- and close-source LLMs 2) prompting with in-domain in-context examples significantly improves the performance of LLMs.\n\nThe paper is generally well-written and the benchmark creation pipeline is clearly presented, although the analysis of results is quite short and shallow, which is the main limitation of this paper. Another weakness is the artificial nature of ambiguity in benchmark questions and its unclear practical applications.", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your review and we are happy to see that you found our paper to investigate behaviors of LLMs that have not been studied before. We address your concerns below:\n\n[R1] Detailed analysis of results\n\nWe will strengthen our analysis in the final version of our paper. Examining the performance across different types of questions (e.g. Figure 3, Table 6) could enable a more fine-grained understanding of the results. Manually coding model prediction comparison can provide deeper understanding of model performance as well (e.g. example in appendix Table 8).\n\n[R2] Artificial ambiguity of questions\n\nThis is a valid concern as our dataset is synthetically generated. Upon manual inspection, however, we find that our questions are mostly reasonable. Please see Section 3.4 and Appendix B.3 for our random samples of examples. \nWe also emphasize that entity ambiguity – “noun phrase in the question which may be resolved in multiple ways” [2], the problem we study, occurs naturally in human interaction data [1]. \n\n[Q1] Intuition behind counter-intuitive findings\n\nGreat question! As these models are released without precise description on how they are trained, it is a bit hard to understand why. One hypothesis is that larger models, with richer parametric knowledge, have a bias towards particular contexts or answers, resulting in incomplete answers. \n\n[Q2] Applications besides ambiguous QA\n\nOur benchmark offers challenging document sets for tasks involving multi-document reasoning, which can be used to evaluate multi-document summarization and fact verification. I.e., Given the confusing document set, see whether LLM can generate a coherent summary. We will discuss alternative use cases of our datasets in the revision.\n\n[1] Min, Sewon, et al. AmbigQA: Answering Ambiguous Open-domain Questions. EMNLP 2020\n\n[2] Amplayo, Reinald Kim, et al. Query Refinement Prompts for Closed-Book Long-Form QA. ACL 2023", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The behavior of LLMs on ambiguous document-based QA has not been studied before\n2. Taxonomy of answer types with heuristics for answer classification", "reasons_to_reject": "1. Detailed analysis of results is lacking\n2. The ambiguity in questions seems quite artificial, unclear whether this benchmark has any practical applications", "questions_to_authors": "1. Some of your findings are quite counter-intuitive (e.g. LLaMa 2 with 13b parameters performing worse than the one with 7b parameters, Mistral demonstrating the best overall performance). Do you have any intuition why?\n2. What are some practical applications of your benchmark besides examining the performance of LLMs on the ambiguous QA task?", "extra_scores": {}}, {"review_id": "nD4jKP3sMD", "reviewer": "Reviewer_mQPG", "summary": "The paper defines a methodology for analysing entity disambiguation performance of LLMs. A part of the methodology is the creation of a large scale data set that leverages Wikipedia for entity disambiguation.\n\nI have concerns about this paper because it misses a baseline. There are many Wikipedia based tools that allow entity disambiguation, e.g.\nhttps://www.dbpedia-spotlight.org/\n\nIn addition also since several years, in industry NLP libraries there are tools to create entity linking based on custom data, e.g.\n\nhttps://github.com/explosion/projects/blob/master/nel-emerson/scripts/notebook_video.ipynb\n\nWithout a baseline using existing tools or an existing language model data set (e.g. based on Spacy), the usefulness of the new benchmark is hard to evalute.", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your review and we are glad to see that you found our benchmark to be novel. We address your concerns below:\n\n[R1] Comparing with baselines using existing Entity Linking tools\n\nThis work evaluates LM’s long-form answer generation ability, while distinguishing confusing entities. Entity linking, on the other hand, links existing entity mention in the input text to disambiguated Wikipedia pages. Because of this major difference, it is not clear how to use entity disambiguation tools for this task. We will add discussions on this in the final draft.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "A novel benchmark about entity disambiguation for LLMs is introduced.", "reasons_to_reject": "The benchmark misses a baseline against which the LLMs are compared. I am withdrawing this concern, the authors explained in the rebuttal that they have something else in mind.", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "BqQgpKJA65", "reviewer": "Reviewer_dFmw", "summary": "Distinguishing between different entities that share the same name poses a significant challenge for language models (LMs). The paper addresses this challenge by introducing a novel benchmark, AmbigDocs, which leverages Wikipedia's disambiguation pages to identify documents related to entities sharing an ambiguous name. The method involves generating questions containing ambiguous names and their corresponding sets of answers from these documents. The goal of this benchmark is to evaluate the ability of LMs to reason across multiple documents with ambiguous entities and provide cohesive answers despite the confusion caused by entities with the same name.", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your review and we are delighted to see that you found our paper to offer insightful analysis. We address your concerns below:\n\n[R1] Distinguishing between genuine ambiguity and other factors\n\nWe assume that LMs should ideally generate complete answers as all necessary information is provided in-context, thereby not considering external factors such as dataset bias during evaluation. Future work could perform fine-grained analysis on the various factors that contribute to determining incomplete answers.\n\n[Q1] Collection of documents\n\nWe use the Wikipedia snapshot from December 20th, 2018, where each article is split into 100 words (which we consider as documents) [1]. For each disambiguated entity of the snapshot, we collect the documents where the disambiguated entity serves as the title.\n\n[Q2] Threshold for calculating the relevancy score\n\nYes, we agree we might be missing some potential document pairs. We focused more on precision of identified document pairs, to manage the computational load of data generation in Section 3.2. We ran some pilot studies to balance the exploration of all possible document pairs against limited computational budget and manually selected threshold value. We found that documents with a maximum 3-gram overlap were mostly independent, whereas those with high n-gram overlap often contained the exact same sentences. \n\n[Q3] Filtering low-quality examples\n\nWe removed examples that were not formatted, were not ambiguous, and failed to pass the NLI check. For details, please refer to Appendix B.2. \n\n[Q4] Retaining newly generated answers\n\nWe follow prior works to evaluate the QA model’s correctness by converting it into a NLI problem [2, 3]. When multiple answers were found, we computed the perplexity of each answer to select the most adequate one, where we also observed that answers with perplexity above 0.2 were often wrong. Following [3], we manually evaluated 100 instances, where 4% of questions included a single incorrect answer in their set of answers (Appendix B.3). We will make this clearer in the revision. \n\n[1] Karpukhin, Vladimir, et al. Dense Passage Retrieval for Open-Domain Question Answering. EMNLP 2020\n\n[2] Chen, Jifan, et al. Can NLI Models Verify QA Systems’ Predictions? EMNLP 2021 Findings\n\n[3] Xie, Jian, et al. Adaptive chameleon or stubborn sloth: Revealing the behavior of large language models in knowledge conflicts. ICLR 2024", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The paper introduces a new benchmark, AmbigDocs, designed specifically to tackle the task of reasoning across documents with entities that share the same name. This benchmark provides a solid resource for evaluating the performance of LMs in handling such ambiguity.\n\n2. Empirical findings presented in the paper shed light on the performance of current LLMs in generating answers for questions involving ambiguous entity mentions. By categorizing incomplete answers and establishing an ontology for evaluation, the paper offers insightful analysis into the challenges faced by LMs when dealing with ambiguity.", "reasons_to_reject": "1. The paper proposes an ontology categorizing incomplete answers and automatic evaluation metrics to identify them. However, how accurate are these metrics in distinguishing between incomplete answers caused by genuine ambiguity and those due to other factors such as model limitations or dataset biases?\n\n2. In the section of \"3. Creating AmbigDocs\", some details still require further clarification. Please see my questions in \"Questions To Authors\".", "questions_to_authors": "1. Regarding seed document pair generation (Section 3.1), could the authors elaborate on how the set of documents D is collected from Wikipedia's disambiguation pages? Are there any specific criteria used to select these documents? How to ensure the quality and relevance of the documents obtained from the disambiguation pages?\n\n2. Also in Section 3.1, the method for calculating the relevancy score between document pairs based on n-gram overlap is interesting. However, could the authors explain why a threshold of 3 and 10 is chosen for determining relevancy? How was this threshold decided? The paper mentioned setting relevancy to 0 when n ≥ 10 to indicate almost identical content. However, could this threshold potentially exclude relevant document pairs where the content is similar but not identical?\n\n3. In section 3.2, the paper mentioned obtaining a total of 43,839 questions after filtering low-quality examples. What criteria were used to determine the quality of the generated questions? Could the authors provide some examples of what constitutes a low-quality question in this context?\n\n4. In section 3.3, the paper mentioned two criteria for retaining newly generated answers: passing an NLI entailment check and having a length-normalized log perplexity below a threshold value of 0.2. Could the authors explain the rationale behind these criteria? How effective are these criteria in filtering out incorrect or irrelevant answers?", "extra_scores": {}}, {"review_id": "vQSQa6WNUt", "reviewer": "Reviewer_v8yv", "summary": "The paper focuses on evaluation of LLMs in factoid question answering where the salient entities in the question are ambiguous and the answer to the question could change depending on the ambiguity resolution of the salient entities. The authors create AmbigDocs dataset to evaluate multiple LLMs and conclude that LLMs are not currently created with proper strength considering such questions.", "questions": "", "limitations": "", "rating": 9, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your review and we are excited to see that you found our paper to present a realistic idea. Thank you for the detailed suggestions, we will make edits incorporating your suggestions.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "* The paper studies the knowledgabilty of LLMs and the factuality of their generated answers in a realistic setting.\n* The idea is quite well thought and the paper asks the right questions and properly answers those questions.\n* Multiple LLMs are studied so the results are more trustable.", "reasons_to_reject": "* The write up has a few missing citations and could be improved.\n \nPlease refer to the suggested improvements in the next section.", "questions_to_authors": "Suggested write-up improvements:\n\n* Page 1 / Section 1: I'd suggest a retouch on this sentence: \" Then, we generate a question and answers from the set of documents, ensuring full annotation of gold document for each answer.\"\n\n* Page 2/ Section 1: For the sentence: \"We find that both open-source LLMs and GPT models suffer at identifying different entities, failing to generate complete answers.\" I would replace GPT with \"OpenAI's GPT\" for clarification and I would add a citation for it as well (You have citations for LLaMA and Mixtral earlier but (OpenAI, 2023) is missing and is cited in Section 3.2).\n\n* Page 2/ Section 2 / Task definition: I believe this definition could be improved by citing and properly explaning the task of Retrieval-augmented generation (RAG; Lewis et al., 2020b; Izacard and Grave, 2021; Singh et al., 2021) which is briefly mentioned in the related works section.\n\n* Page 7/ Section 5.1 and 5.2/ Gold only setting: authors may find the work of Kandpal et al. (2022; https://arxiv.org/pdf/2211.08411) related to oracle or gold retrieval settings (for completion of related works coverage).\n\n* Page 8/ Section 5.3: \"We categorize model outputs into five output categories with the automatic metric we have\nbuilt in Section 4.2\" a period is missing at the end of this sentence.", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper studies the problem of ambiguous entity names, e.g., where two well-known people share the same name and there are a variety of documents that refer to each of the different entities. The paper creates a new dataset exemplifying this challenge, including questions that require disambiguating ambiguous entity names (e.g., a name two people share) to a specific individual entity. Authors perform analysis on the performance of SOTA models and categorize their failures to disambiguate entities in to several error categories. This problem is particularly important for applications of such SOTA systems, e.g., for RAG tasks.

One concern that reviewers hold is that questions are synthetically generated and may not reflect actual naturalistic scenarios where entity disambiguation is required. However, reviewers are generally positive about the creation of this benchmark as a way to directly investigate LLM performance on this important problem.

If accepted, the authors should ensure that they address reviewer concerns and suggestions in the final version of the paper.
Decision: Accept