AmbigDocs: Reasoning across Documents on Different Entities under the Same Name

Authors: yoonsang  lee
xi  ye
eunsol  choi
Venue: colmweb_org_COLM_2024_Conference
Published: 2024-03-23 07:05:05

Abstract: Different entities with the same name can be difficult to distinguish. Handling confusing entity mentions is a crucial skill for language models (LMs). For example, given the question “Where was Michael Jordan educated?” and a set of documents discussing different people named Michael Jordan, can LMs distinguish entity mentions to generate a cohesive answer to the question? To test this ability, we introduce a new benchmark, AmbigDocs. By leveraging Wikipedia’s disambiguation pages, we identify a set of documents, belonging to different entities who share an ambiguous name. From these documents, we generate questions containing an ambiguous name
and their corresponding sets of answers. Our analysis reveals that current state-of-the-art models often yield ambiguous answers or incorrectly merge information belonging to different entities. We establish an ontology categorizing four types of incomplete answers and automatic evaluation metrics to identify such categories. We lay the foundation for future work on reasoning across multiple documents with ambiguous entities.

Main paper: 
 introduction: 
 Different entities can be referred by the same name, and clustering such confusing entity mentions requires rich reasoning involving background knowledge. While it is rare for different entities who share the same name to occur together within a single document, such co-occurrences are prevalent in multi-document setting. For example, searching the web with the question "When is ACL 2024?" returns a webpage on Austin City Limits Music Festival and another page on Association for Computational Linguistics conference. Given such two webpages, can LMs generate a complete answer correctly distinguishing confusing entities, such as "Austin City Limits Music Festival will be held on Oct 4-6 and 11-13, while the ACL conference will happen on August 11-16."? Despite rich study in ambiguous question answering (QA) (Min et al., 2020; Stelmakh et al., 2022) , which covers such ambiguous entity resolution, few work studied how LMs reason when provided with a confusing document set. The major obstacle has been the lack of annotated confusing document sets. 2 To enable research in this area, we construct a synthetic dataset, AmbigDocs, whose single instance consists of a question asking about an ambiguous entity and a list of gold document-answer pairs for each disambiguated entity (see Table 8 for examples). We leverage Wikipedia disambiguation pages, which provide multiple entities that can be referred to by the same surface name. From each disambiguation page, we generate a question and multiple valid answers, where each answer is grounded in a document and corresponds to a different entity in the same ambiguation page. We generate a total of 36K examples, covering 102K unique entities over all domains.Question: What is the state where Judge Day was born?Figure 1 : Given a question containing an ambiguous entity mention (e.g., Judge Day) and a set of documents containing a valid answer to different disambiguated entities, LLM should generate a complete answer, pairing each disambiguated entity with its answer. Left bottom box: we first evaluate y with respect to a single input document, checking if y contains the answer with its disambiguated entity name (e.g., for the Doc 1, it should mention the answer "Alabama" and disambiguated entity name "Charles Bernard Day"). Based on per-document scores, we can assign one of the answer category labels (partial, no, ambiguous, merged, complete) .Equipped with a new dataset that contains an ambiguous question and a set of documents with valid disambiguated entity-answer pairs, we evaluate how LMs answer questions containing ambiguous entity names when provided with a set of documents suggesting multiple distinct answers. We first evaluate LMs with metrics used in prior ambiguous QA tasks (Adlakha et al., 2023; Stelmakh et al., 2022) . Then, we develop an ontology of five types of answers (complete, partial, ambiguous, merged , and no answer) that captures the unique challenge in AmbigDocs. We develop an automatic metric that can classify answers according to this ontology, with very high agreement of manual classification. We find that both open-source LLMs (Touvron et al., 2023; Jiang et al., 2023 ) and OpenAI's GPT models (OpenAI, 2023) suffer at identifying different entities, failing to generate complete answers. Yet, we find prompting with in-domain in-context examples can improve model performance significantly, GPT-4/Mistral-7B generating a complete answer for 58%/43% of questions, 20%/10% gain from the zero-shot setting. We provide a rich analysis showing the strengths and weaknesses of existing LMs under this challenging scenario.To summarize, our contributions are as follows: (1) We present AmbigDocs, a large-scale dataset that pairs a question and a set of documents that suggests multiple answers based on question entity disambiguation. (2) We establish an ontology categorizing five types of generated answers and an automatic classification heuristic, enabling in-depth analysis on model behaviours. (3) We benchmark how current LMs perform the question answering task when provided ambiguous questions and a document set that requires challenging entity disambiguation. 
overview: 
 Definition We first define key terminology used in the paper.• Surface Name (sn): An ambiguous entity name that can be interpreted as any of disambiguated entities, depending on the context. In Figure 1 , sn is "Judge Day". • Question (q sn ): A question that contains the surface name sn. The question is ambiguous as sn can refer to multiple entities, each with different answers. In Figure 1 , it is "What is the state where Judge Day was born?". • A list of answers ([a 1 , a 2 , . . . , a n ]): Given the question q sn , a i is the answer corresponding to DE i . In Figure 1 , it is ["Alabama", "Ohio", "Rhode Island"]. • A set of documents relevant to the question ({d 1 , d 2 , . . . d m }): A set of m documents that are relevant to question q sn that will be provided to the LMs.Figure 2 : Overview of our dataset generation. We identify a surface name and a list of disambiguated entities from Wikipedia's disambiguation pages. We select two documents for generating a question and their corresponding answers. Subsequently, we gather additional answers from the remaining documents.Task We study retrieval-augmented generation (Lewis et al., 2020; Izacard & Grave, 2020; Sachan et al., 2021) , which allows LMs to incorporate external documents at the inference time. Given a question and a set of relevant documents, {q sn , {d 1 , . (Min et al., 2020; Stelmakh et al., 2022 ) is that we provide a set of relevant documents, which contains multiple disambiguated entities and their corresponding answers. This allows an isolated evaluation of LMs on their ability to reason with a confusing document set, without conflating the retriever performance. ] and predicted answer y, we categorize the answer y into one of the five categories that we define:• Complete Answer: LM correctly generates all valid answers in the provided document set. Each answer is paired with the disambiguated entity name. • Partial Answer: LM generates one or more (but not all) of the valid answers in the provided document set. Each answer is paired with the disambiguated entity name. • No Answer: LM abstains from providng any answer.• Ambiguous Answer: LM generates one of the valid answers without providing the disambiguated entity name. • Merged Answer: LM generates multiple answers without providing the disambiguated entity name (merging facts about multiple disambiguated entities).Figure 1 provides an example answer corresponding to each output category. Ideally, LM should generate a complete answer. Merged answer can be considered the most problematic, as it may provide confusing or misleading information. The preference among partial answer, no answer, and ambiguous answer can depend on the types of questions and the prominence among disambiguated entities. If one of the disambiguated entities is substantially more likely to be referred by surface name, providing an ambiguous answer corresponding to the likely entity can be more useful than abstaining. In Section 4.2, we will introduce an automatic metric that assigns model outputs to one of the five answer categories. Now we describe how we construct AmbigDocs to study this problem. 
creating ambigdocs: 
 We build an automated data creation pipeline that can generate large-scale data without manual annotations, once provided with Wikipedia disambiguation pages and access to high-quality LLMs. To ensure the quality of the synthetic data, our pipeline features carefully crafted prompts for sampling initial instances and various checks for filtering low quality instances. This follows recent work on generating synthetic datasets for complex tasks with LMs (Xie et al., 2023; Yehudai et al., 2024; Oh et al., 2024; Zhao et al., 2024) .Data Instance Each instance in our benchmark consists of {a surface name sn, a question containing the surface name q sn , and a set of gold documents ({d Overview We generate data by first identifying two documents, each belonging to a disambiguated entity under the same surface name. Next, we generate a question and two answers; each answer belongs to a disambiguated entity. This diverges from prior work on this domain that applies heuristics to annotate gold documents for existing questions (Stelmakh et al., 2022; Schuster et al., 2023) , which leads to incomplete document annotation. Figure 2 summarizes our process. The first step is to identify (d * 1 , d * 2 ), two documents belonging to different entities which can be referred to by the same surface name and discuss similar topics (Sec. 3.1). The second step is to generate a question q sn and answers corresponding to respective entities {a * 1 , a * 2 } (Sec 3.2). As the second step only considers two disambiguated entities per surface name, we expand the answer set to gather remaining plausible answers (Sec 3.3).For each surface name sn, we have a set of documents D (10.6 documents on average) gathered from Wikipedia disambiguation page, each belonging to one of the disambiguated entities. We use the Wikipedia snapshot from December 20th, 2018, where each article are split into 100 words (which we consider as documents) (Karpukhin et al., 2020) . For each disambiguated entity of the snapshot, we collect the documents where the disambiguated entity serves as the title. Our goal is to find two documents discussing a similar topic, belonging to different disambiguated entities, each containing a distinct answer to some ambiguous question q sn . For each document pair, we measure their longest n-gram overlap as n. We define a relevancy score between a pair of documents based on n. The relevancy between a pair is set to 0 if n ≤ 3 or n ≥ 10; we set the relevance as n otherwise. Here we set relevancy to be 0 when n ≥ 10, as this typically indicates a pair shares almost identical content, whereas our goal is to find pairs featuring similar topic but different answers. We compute this score for all pairs of documents in D, excluding pairs where both documents are from the same disambiguated entity. If all document pairs receive a score of 0, we disregard sn, otherwise, we select one document pair with the highest relevancy score, breaking a tie randomly. This process yields a total of 67,294 document pairs, one per surface name.Having identified two documents with confusing entities that cover a similar topic, we proceed to generate the question q sn that can be answered differently by each document. We use an LLM to generate this: prompting it with (sn, d * 1 , d * 2 ) along with two-shot exemplars to generate {q sn , [a * 1 , a * 2 ]}. We initially use GPT-4 (OpenAI, 2023) , and then distill its ability to a smaller model, Llama2-13b (Touvron et al., 2023) , by finetuning with dataset generated from GPT-4 (West et al., 2021) . We obtain a total of 43,839 questions after filtering low-quality examples. Details of the generation and filtering process can be found in Appendix B.2.The generated data contains only two answers belonging to two disambiguated entities, while there can be more than two entities sharing the surface name sn. In this final stage, we expand the answer set. As we already have a question q sn , we find additional answers using an LLM-based QA module. We consider all documents in D, excluding documents that belong to DE * 1 , DE * 2 . For each document, we use a Llama2-13b model with 4-shot exemplars to generate an answer given the question (the prompt is in Figure 11 the QA module can generate an incorrect answer, we keep the newly generated answer only when (1) it passes NLI entailment check (Chen et al., 2021b) and ( 2) length-normalized log perplexity of the answer is below a threshold value of 0.2. For the NLI entailment check, we consider the document as the premise. Additionally, we convert q DE * i and a * i into a declarative form (Chen et al., 2021b) to use as the hypothesis, where q DE * i represents a disambiguated question created by replacing sn in the q sn with DE * i . If multiple answers are found for the same disambiguated entity, we choose the answer with the lowest perplexity.We collected a total of 36,098 examples, where they are randomly split into train/dev/test splits with ratios of 60%/10%/30%, respectively. We ensure that there is no document overlap between distinct splits. On average, each question has 2.92 answers (distribution in Table 1 ), covering a total of 102,624 distinct entities. In this work, we do not use training or validation split, only using the test split for evaluation, but will make all data available. Figure 3 visualizes questions in AmbigDocs. As our generation is synthetic, we (the authors) manually examine 100 instances to assess its quality. We observe two types of noise: when a single document discusses multiple disambiguated entities (2%), and when an LLM generates incorrect answers (4%). Overall, we find question-answer pairs to be mostly valid and answers can be supported by the gold documents. Please refer to Appendix B.3 for further analysis and examples. 
evaluation for ambigdocs: 
 We discuss how to evaluate model-generated long-form answer y for question q sn with respect to the reference answer list[a * 1 , a * 2 , ...a * m ] paired with its disambiguated entity list [DE * 1 , DE * 2 , . . . , DE * m ].We first report automatic metrics, a token overlap metric and a model-based metric, that has been used in prior work (Adlakha et al., 2023; Stelmakh et al., 2022) .We map the answer y to a set of tokens by following the answer processing script of SQuADv2 (Rajpurkar et al., 2018) (which lower cases and splits the string) and remove the stopwords from the nltk package. We perform the same preprocessing for individual answer string a * i and disambiguated entity name DE * i to get token sets. For disambiguated entity name token set, we disregard tokens that appear in the surface name sn to exclude common information and capture the disambiguated entity. For instance, for disambiguated entity name 'Hughesville, New Jersey' in Figure 2 , we will only consider {new, jersey}, excluding 'hughesville' which appears in the surface name. Then, we define: ∑ R(DE * i , y) • R(a i , y). By multiplying two scores, we measure how many answers are generated with its disambiguated entity name.5 Disambig-F1 (DF1) Stelmakh et al. (2022) introduces a model-based metric that use a RoBERTa-based (Liu et al., 2019) QA model trained on SQuADv2 to evaluate long-form answer. For each a * i , the model takes a disambiguated question and the generated answer, and extracts a short answer a i . The F1-score between extracted answer and gold answer is then averaged across all a * i , i.e. DF1(y) = 1 m ∑ F1(a i , a * i ). To simulate a disambiguated question without explicit human annotation, we replace sn inside q sn with DE * i for each gold answer. This metric serves the same purpose as the EAR metric we introduce above.In the overview (Sec. 2), we have defined five types of answers: complete answer, partial answer, ambiguous answer, merged answer, and no answer. We first present manual categorization and then introduce a heuristic to automatically classify answers.We generate long-form answers for 100 randomly sampled questions in the test set with five LMs (upcoming Sec. 5.1 provides the details of the answer generation process). Given the question q sn , the model-generated long-form answer y, and gold answers[a * 1 , a * 2 , ...a * m ] paired with [DE * 1 , DE * 2 , . . . , DE * m ], the annotator classifies y into one of the five categories. We provide the instruction for human evaluation in Appendix C.2. One of the authors annotated the initial data, and another person (who has a background in NLP) annotated a subset of the initial data (125 items, 5 model outputs for 25 questions). The Cohen's Kappa statistic for inter-annotation agreement is 0.85, showing high agreement and the validity of our ontology. We will discuss the distribution of model answer types in the results section (manual annotation distribution can be found in Figure 8 in the appendix).We develop heuristics to automatically classify the generated answer y into one of the categories proposed in Section 2. Let T s be the set of tokens mapped from string s. Then, we check whether each reference answer pair (a * i , DE * i ) is: • answered with disambiguation: both a * i andDE * i are mentioned in y, i.e. R(a * i , y) ≥ 1 |T a * i | and R(DE * i , y) ≥ 1 |T DE * i | . • answered without disambiguation: a * i is mentioned in y but DE * i is not, i.e. R(a * i , y) ≥ 2 |T a * i | 6 and R(DE * i , y) = 0.We introduce a moving threshold for recall based on the length of the target string, such as 1|T a * i | and 1 |T DE * i |, as longer tokens are less likely to appear verbatim. Then we count the number of answers with disambiguation, denoted as c p , and the number of answers without disambiguation, denoted as c np , for long-form answer y. Then, we can classify y into one of five mutually exclusive categories (m is the number of reference answers):• Complete Answer: c p = m • Ambiguous Answer:c p = 0, c np = 1 • Merged Answer: otherwise • Partial Answer: 1 ≤ c p < m, c np = 0 • No Answer: c p = c np = 0We provide the pseudocode for classifying answers in Figure 7 in the appendix. Next, we measure the agreement between automatic metrics and manual annotation.We compare the answer category labeled by a human (one of the authors) and the category assigned by our heuristic, on 500 example outputs. The Cohen's Kappa statistic between the two label sets is 0.83, showing strong agreement, though slightly lower than the human agreement of 0.85. We provide the confusion matrix in Figure 5 in the appendix. Some Ambiguous answers are misclassified as Partial answers due to the limitations of lexical-based heuristics, such as plural forms (locomotive vs. locomotives) and spelling variations (organisation vs. organization). 
experimental setup: 
 Models We evaluate open-sourced LMs, Llama2 (7b, 13b) (Touvron et al., 2023) , Mistral (7b) (Jiang et al., 2023) , and two commercial APIs (GPT 3.5 and GPT-4) (OpenAI, 2023) .We will evaluate the LMs on the test portion of our data, which contains 7,220 questions. For GPT-4 model, we evaluate on 500 random subsets, owning to its expensive API cost. We pair each question with three types of document sets.• Gold Only (n ≤ 5): The gold documents gathered during data generation process (Kandpal et al., 2022) . We randomly sample five documents if there are more than five. • Retrieved Only (n = 5): We retrieve five documents with question q sn as the query. We use GTR (Ni et al., 2021) as our retriever and Wikipedia snapshot from December 20th, 2018 as a retrieval corpus. On average, this contains 1.2 gold documents. • Gold + Retrieved (n = 5): We take all gold documents, and add top retrieved documents until we have five documents if there are fewer than five gold documents.The documents are randomly shuffled before being fed into LMs. We present the prompt used for inference in Figure 11 . The answers are greedy decoded.Table 2 presents the model performance on traditional QA metrics. When provided only the gold documents (GOLD ONLY), Mistral and GPT-4 exhibit the strongest performance, followed by Llama2-7b model. Surprisingly, we find some larger models (Llama2-13b, GPT-3.5) significantly perform worse than smaller models (Llama2-7b, Mistral-7b). One potential hypothesis is that larger models with richer parametric knowledge have a bias towards particular contexts or answers, resulting in incomplete answers. Even when provided only gold documents, none of the models achieve an EAR or DF1 score exceeding 0.5, indicating that existing models struggle to distinguish entities that share a surface name.For comparison, we test LMs on the same sets of questions, but providing only one gold document to the model. As opposed to multi-document setting, LLMs can find the single answer, all achieving 0.70-0.80 EAR and above 0.60 DF1 scores (the full results in Appendix D.1).When retrieved documents are presented with gold documents (GOLD+RETRIEVED), performance declines, suggesting that non-gold but relevant documents distract the model's reasoning process (Cuconasu et al., 2024) . Unsurprisingly, solely providing the retrieved corpus (RETRIEVED ONLY) results in a significant performance drop across all models. We find that Mistral model shows the best performance in entity recall in all settings, contributing to its strong overall performance. We categorize model outputs into five output categories with the automatic metric we have built in Section 4.2. Figure 4 depicts the output distribution under the GOLD ONLY setting for five different models. GPT-4 exhibits strong performance, yielding the highest percentage of complete answer. Mistral generates more Partial answers but fewer Merged answers, leading to overall performance comparable to GPT-4. Larger models such as Llama2-13b and GPT-3.5 tend to generate more Ambiguous and Merged answers rather than Complete answers, resulting in poor overall performance. A large percentage of GPT-3.5 outputs are labeled as Partial. We find GPT-3.5 often generates only one correct answer, as reflected in short length of generated answers. In Table 3, we present an example of generated answers from five LMs. 
related works: 
 Coreference Resolution Our task is closely related to multi-document coreference resolution task (Yang et al., 2015; Cattan et al., 2021) in that it requires entity mention resolution across multiple documents. Yet, our task contains a purposefully ambiguous entity mention in the question. Thus, the entity mention in question can be mapped to multiple entities, making it hard to derive and evaluate against gold entity clusters.Complex Reasoning with Retrieval Augmentation While language models are equipped with rich parametric knowledge (Roberts et al., 2020) , providing relevant documents in context often results in improved performances for various end tasks (Lewis et al., 2020; Asai et al., 2024) . Wan et al. (2024) ; Tan et al. (2024) provides document set containing conflicting answers to contentious questions, while other work (Xie et al., 2023; Chen et al., 2022) simulates a situation where the document set and parametric knowledge of LM collide.WikiHowQA (Bolotova-Baranova et al., 2023) examines multi-document reasoning of LLMs, without engaging conflicting documents. We study a setting where we provide a confusing set of documents which requires careful entity disambiguation. Concurrent to us, Chiang & Lee (2024) studies generating biographies from a set of documents that contains multiple people with the same name. While this work also studies similar entity confusion errors of LMs, its goal is to improve a factuality metric (Min et al., 2023) . We cover a much broader set of entities and focus on multi-document reasoning for question answering. Similar to our answer categorization ontology, Mishra et al. (2024) establishes different types of factuality error. Their ontology aims evaluation of general retrieval-augmented generation, while our study focuses on entity disambiguation. Compared to ontology from Chiang & Lee (2024) , our ontology is more fine-grained, distinguishing ambiguous and partial outputs.Prior work studied ambiguous question answering in simple factoid QA (Min et al., 2020; Zhang & Choi, 2021) and long-form QA (Stelmakh et al., 2022) setting. While ambiguity can arise for multiple reasons (Amplayo et al., 2023) , we focus on entity disambiguation. Chen et al. (2021a) studied entity linking for the ambiguous entity in a question (e.g., linking ambiguous surface name "Apple" in the question such as "What is the record label of Apple?" to its correct Wikipedia entity). Our task setting, which aims to generate a long-form answer covering multiple answers given a question and a set of relevant documents, is equivalent to recent QuoteSum (Schuster et al., 2023) dataset. However, their focus is on generating and evaluating extractive answer and does not model whether LMs disambiguate entity mentions. We explicitly simulate confusing document sets and provide metrics to identify whether LMs generate a coherent and complete answer with entity disambiguation.LLMs can handle ambiguous questions differently than what is explored in this work, which is to present all possible answers with disambiguation (Amplayo et al., 2023; Kim et al., 2023; Sun et al., 2023) . These include asking clarifying questions to allow users to specify the intended interpretation of the question (Zhang & Choi, 2023; Lee et al., 2023; Kuhn et al., 2022) or evaluating scores for each candidate answer (Cole et al., 2023) . 
conclusion: 
 This work studies the reasoning capabilities of language models across documents featuring distinct entities that share a surface name. We introduce AmbigDocs, where we identify such documents, paired with an ambiguous question and the answers from each document.Our empirical findings disclose that language models often fail at generating complete answers, exhibiting varying types of behaviours. We hope our findings encourage future investigations on LLMs when faced with entities that are hard to disambiguate.While we did not explore in this work, the dataset can be used for other tasks such as fact verification (Min et al., 2023) and multi-document summarization (Chiang & Lee, 2024) . Future work can also explore fine-grained analysis on the various factors that contribute to determining incomplete answers. 
 
 Appendix: Open-source models in our paper are from huggingface. We provide the model names here:• Llama-2-7b-chat-hf • Llama-2-13b-chat-hf • Mistral-7B-Instruct-v0.2 • question converter-3b• t5 xxl true nli mixture• gtr-t5-xxl• roberta-base-squad2We use the Wikipedia API to identify disambiguation pages and gather ambiguous entities along with their disambiguations from these pages. We only take that sn have 10 or fewer resolutions, and DE i that have 20 or fewer documents.Due to the computational cost associated with GPT-4, we initially generate 1500 questions. Following prior works in automatic dataset construction (Xie et al., 2023; Yehudai et al., 2024) , we conduct post hoc filtering to ensure data quality. We divide the filtration into three steps and present the proportion of removed questions during each step in Table 5 .• Not Formatted: We remove the questions if (1) the generated output does not follow the format in Figure 9 , or (2) each answer contains too much sentences (answer that contains more than 4 periods).• Not Ambiguous: We remove the questions if (1) q sn does not contain sn, (2) q sn contains both DE * 1 and DE * 2 , (3) q sn contains prompt-specific phrases such as 'passage' and 'context', (4) one answer contains another, or (5) two answers are too similar (IoU(a * 1 , a * 2 ) > 0.75). • Not Entailed: We assess the faithfulness of each answer by framing it as a Natural Language Inference (NLI) problem (Bowman et al., 2015) . We substitute sn in the question with DE * i , creating a disambiguated question q DE * i . We convert q DE * i and a * i into a declarative form (Chen et al., 2021b) to use as the hypothesis, while the gold document serves as the premise. We employ T5-11B NLI model (Honovich et al., 2022) to evaluate entailment between hypothesis and premise, retaining only instances where the premise is entailed by hypothesis for both answers.Not formatted Not ambiguous Not entail Success GPT-4 0.0% 8.0% 26.5% 65.5% Llama2-13b 1.7% 8.4% 23.3% 66.6%Table 5 : Percentage of questions removed during three filtration steps. Success indicates the questions that were not removed, and therefore collected as our dataset.This process results in 983 questions, accounting for 65.5% of the original 1500 questions.We distill the generation capabilities of GPT-4 to the Llama2-13b-chat model by finetuning Llama2-13b-chat model with QLoRA (Dettmers et al., 2023) for 5 epochs, with a learning rate of 2e-4. The training script is taken from Yoran et al. (2023) . After repeating the same generation and filtration steps to the remaining seed pairs, we obtain a total of 43,839 questions, achieving a success rate of 66.6%, comparable to that of GPT-4. known 11.2% profession 7.5% refer 5.7% primary 5.5% population 5.4% located 5.0% main 4.8% location 4.1%Table 7 : Frequency of the most common words in the question after being converted to lowercase, removing stopwords, and split by whitespace. The questions fall into two categories: those seeking specific information (profession, population, location, primary, main) and those requesting clarification on ambiguous entities (known, refer).Table 6 presents detailed statistics of our dataset and Table 7 presents the eight most common words appearing in the questions. Additionally, we present three examples of our dataset in Table 8 .Furthermore, we evaluated the quality of 100 instances of our dataset and identified two types of noise. The first occurs when a document discusses multiple entities. For example, the document about the person 'Francisco Prestes Maia' also mentions 'Prestes Maia (building)', which is another disambiguated entity. In such cases, the generated question asks about the building. The second type of noise arises when an answer is incorrect, which typically occurs during the answer expansion process. We leave employing a better QA model and developing better filtration methods for future work.C.1 Preprocessing Figure 5 : Confusion matrix comparing quantifier metrics with our manual analysis (n=500). Our automatic classification shows high agreement with manual classification, although Ambiguous answers are sometimes misclassified as Partial answers.Some surface names contain '(disambiguation)' at the end, which we remove before using sn anywhere. In addition, some disambiguated entities contain phrases enclosed in parenthesis, e.g. 'Michael Jordan (mycologist) '. We remove the parenthesis only when mapping the string into tokens. During mapping, removing token overlaps of sn may result in an empty set of tokens. For such cases, we compute the exact match of the disambiguated entity, i.e. DE * i is in y.The instructions provided to human annotators include the overview illustrated in Figure 1 , the content outlined in Section 2, and a sample annotation interface (Figure 6 ). For each y, the annotator is provided with the follow-q sn : Who is the artist of the "Cherry Bomb" project? ). Subsequently, the annotator classifies y into one of the five categories by selecting from a dropdown menu. The distribution of human-annotated answers is presented in Figure 8 .Figure 7 describes the pseudocode for classifying a generated answer into one of five categories. We present the confusion matrix comparing automatic metrics and our manual analysis in Figure 5 .Figure 6 : Interface for the human annotation process. For each model generated answer, the annoatator is provided the surface name, the question, and the list of reference (disambiguated entity, gold answer) pairs. The dropdown menu contains five categories, where the annotator can easily select one. We do not provide which model y was generated from.question q sn , reference answer list [a : c p ← 0, c np ← 0 2: for i ← 1 to m do 3: T a * i ← GetTokens(a * i ) 4: T DE * i ← GetTokens(DE * i ) 5: if R(a * i , y) ≥ 1 |T a * i | and R(DE * i , y) ≥ 1 |T DE * i | then 6: c p ← c p + 1 7: else if R(a * i , y) ≥ 2 |T a * i | andWe provide only one gold document to the LM and examine if the model has identified the correct answer. Since the model is not given multiple entities, the Ambiguous answer is resolvable. Therefore, we compute entity recall twice, once with DE and once with sn, and select the higher value. We do the same for the DF1 metric, both with and without Table 11 presents K-Precision scores of the models across experimental settings. We observe that the generated answers are well factually grounded to the reference documents.We present the prompts used throughout our paper in Figure 9 -12. Our dataset is released at https://ambigdocs.github.io/.While newer datasets contain evidence documents for a few examples, we identify only 111 instances for ASQA(Stelmakh et al., 2022) dataset and 106 instances for QuoteSum(Schuster et al., 2023) dataset that cover entity disambiguation.Each answer consists of roughly 20 to 60 words (answer length statistics in Table2).We additionally provide experimental results with retrieved document sets, but the main analysis is performed with the gold document set.There may be cases where an answer exploits the metric design, such as 'DE * 1 is a * 2 and DE * 2 is a * 1 '. However, we have not encountered such cases during the manual inspection ofmodel outputs.6 We increase the numerator to reduce false cases. If |T a * i | ≤ 2, we set R(a * i , y) ≥ 0.5. 


Decision: This paper studies the problem of ambiguous entity names, e.g., where two well-known people share the same name and there are a variety of documents that refer to each of the different entities. The paper creates a new dataset exemplifying this challenge, including questions that require disambiguating ambiguous entity names (e.g., a name two people share) to a specific individual entity. Authors perform analysis on the performance of SOTA models and categorize their failures to disambiguate entities in to several error categories. This problem is particularly important for applications of such SOTA systems, e.g., for RAG tasks.

One concern that reviewers hold is that questions are synthetically generated and may not reflect actual naturalistic scenarios where entity disambiguation is required. However, reviewers are generally positive about the creation of this benchmark as a way to directly investigate LLM performance on this important problem.

If accepted, the authors should ensure that they address reviewer concerns and suggestions in the final version of the paper.

Reviews: review: paper_summary: paper_summary: This paper proposes AmbigDocs, a new benchmark for studying the behavior of LLMs on ambiguous document-based QA, and a methodological pipeline that can be used to create similar benchmarks from Wikipedia disambiguation pages using LLMs. The questions in the benchmark are ambiguous in the sense that they contain a surface form of a named entity, which may refer to multiple actual entities in the answer documents.

The paper also gives a new spin to the definition of an answer in the context of LLMs and document-based QA and provides a taxonomy of 5 answer types, against which several SOTA closed- and open-source LLMs (LLaMa 7b and 13b, Mistral 7b, GPT-3.5 and GPT-4) are evaluated. The benchmark and the answer taxonomy are the main contributions.

The key findings are twofold: 1) generating complete answers for ambiguous questions in AmbigDocs is a challenging task for the open- and close-source LLMs 2) prompting with in-domain in-context examples significantly improves the performance of LLMs.

The paper is generally well-written and the benchmark creation pipeline is clearly presented, although the analysis of results is quite short and shallow, which is the main limitation of this paper. Another weakness is the artificial nature of ambiguity in benchmark questions and its unclear practical applications.
strength_weakness: strength_weakness: 1. Detailed analysis of results is lacking
2. The ambiguity in questions seems quite artificial, unclear whether this benchmark has any practical applications
questions: questions: 1. Some of your findings are quite counter-intuitive (e.g. LLaMa 2 with 13b parameters performing worse than the one with 7b parameters, Mistral demonstrating the best overall performance). Do you have any intuition why?
2. What are some practical applications of your benchmark besides examining the performance of LLMs on the ambiguous QA task?
paper_summary: paper_summary: The paper defines a methodology for analysing entity disambiguation performance of LLMs. A part of the methodology is the creation of a large scale data set that leverages Wikipedia for entity disambiguation.

I have concerns about this paper because it misses a baseline. There are many Wikipedia based tools that allow entity disambiguation, e.g.
https://www.dbpedia-spotlight.org/

In addition also since several years, in industry NLP libraries there are tools to create entity linking based on custom data, e.g.

https://github.com/explosion/projects/blob/master/nel-emerson/scripts/notebook_video.ipynb

Without a baseline using existing tools or an existing language model data set (e.g. based on Spacy), the usefulness of the new benchmark is hard to evalute.
strength_weakness: strength_weakness: The benchmark misses a baseline against which the LLMs are compared. I am withdrawing this concern, the authors explained in the rebuttal that they have something else in mind.
paper_summary: paper_summary: Distinguishing between different entities that share the same name poses a significant challenge for language models (LMs). The paper addresses this challenge by introducing a novel benchmark, AmbigDocs, which leverages Wikipedia's disambiguation pages to identify documents related to entities sharing an ambiguous name. The method involves generating questions containing ambiguous names and their corresponding sets of answers from these documents. The goal of this benchmark is to evaluate the ability of LMs to reason across multiple documents with ambiguous entities and provide cohesive answers despite the confusion caused by entities with the same name.
strength_weakness: strength_weakness: 1. The paper proposes an ontology categorizing incomplete answers and automatic evaluation metrics to identify them. However, how accurate are these metrics in distinguishing between incomplete answers caused by genuine ambiguity and those due to other factors such as model limitations or dataset biases?

2. In the section of "3. Creating AmbigDocs", some details still require further clarification. Please see my questions in "Questions To Authors".
questions: questions: 1. Regarding seed document pair generation (Section 3.1), could the authors elaborate on how the set of documents D is collected from Wikipedia's disambiguation pages? Are there any specific criteria used to select these documents? How to ensure the quality and relevance of the documents obtained from the disambiguation pages?

2. Also in Section 3.1, the method for calculating the relevancy score between document pairs based on n-gram overlap is interesting. However, could the authors explain why a threshold of 3 and 10 is chosen for determining relevancy? How was this threshold decided? The paper mentioned setting relevancy to 0 when n ≥ 10 to indicate almost identical content. However, could this threshold potentially exclude relevant document pairs where the content is similar but not identical?

3. In section 3.2, the paper mentioned obtaining a total of 43,839 questions after filtering low-quality examples. What criteria were used to determine the quality of the generated questions? Could the authors provide some examples of what constitutes a low-quality question in this context?

4. In section 3.3, the paper mentioned two criteria for retaining newly generated answers: passing an NLI entailment check and having a length-normalized log perplexity below a threshold value of 0.2. Could the authors explain the rationale behind these criteria? How effective are these criteria in filtering out incorrect or irrelevant answers?
paper_summary: paper_summary: The paper focuses on evaluation of LLMs in factoid question answering where the salient entities in the question are ambiguous and the answer to the question could change depending on the ambiguity resolution of the salient entities. The authors create AmbigDocs dataset to evaluate multiple LLMs and conclude that LLMs are not currently created with proper strength considering such questions.
strength_weakness: strength_weakness: * The write up has a few missing citations and could be improved.
 
Please refer to the suggested improvements in the next section.
questions: questions: Suggested write-up improvements:

* Page 1 / Section 1: I'd suggest a retouch on this sentence: " Then, we generate a question and answers from the set of documents, ensuring full annotation of gold document for each answer."

* Page 2/ Section 1: For the sentence: "We find that both open-source LLMs and GPT models suffer at identifying different entities, failing to generate complete answers." I would replace GPT with "OpenAI's GPT" for clarification and I would add a citation for it as well (You have citations for LLaMA and Mixtral earlier but (OpenAI, 2023) is missing and is cited in Section 3.2).

* Page 2/ Section 2 / Task definition: I believe this definition could be improved by citing and properly explaning the task of Retrieval-augmented generation (RAG; Lewis et al., 2020b; Izacard and Grave, 2021; Singh et al., 2021) which is briefly mentioned in the related works section.

* Page 7/ Section 5.1 and 5.2/ Gold only setting: authors may find the work of Kandpal et al. (2022; https://arxiv.org/pdf/2211.08411) related to oracle or gold retrieval settings (for completion of related works coverage).

* Page 8/ Section 5.3: "We categorize model outputs into five output categories with the automatic metric we have
built in Section 4.2" a period is missing at the end of this sentence.

Comments: title: Rebuttal
Rebuttal
Re: Rebuttal by Authors
Discussion period now open
comment: Hi AC and all,

After reviewing all the comments and rebuttals, I still retain my score (according to ACL/EMNLP standards). I share reviewer JT3K's concerns regarding the analysis of experimental results, and I also found some details in methodology unclear (although the author's rebuttal has addressed my questions). However, I believe these are minor issues; differing views and debates indicate the community's interest in this paper. Consider that this paper is the first to study the performance of LLMs on ambiguous entities (the same problem we've also observed affecting retrievers in RAG-QA, despite mitigation strategies in GPT-4 and Claude 3), introducing this perspective is significant. If this paper is accepted, I suggest that AC remind the authors in the meta-review to address the reviewers' concerns as much as possible in the camera-ready version. 

Many thanks. --dFmw
Rebuttal acknowledged.
Rebuttal acknowledged.
Rebuttal acknowledged.
I read the authors' responses and chose to keep my original evaluation.
Hi reviewers, please take a look at the author's rebuttals and the other reviews for this paper!

If the rebuttals addressed your concerns, please let the authors know about this and update your review. If not, please continue to engage with the authors and the other reviewers in the discussion forum.

It seems there is significant disagreement among the reviewers about this paper. 

mQPG, you are the most negative about the paper, and your concern is that the paper doesn't compare with existing benchmarks for the task. Does the authors' response clarify this concern, or do you still hold this concern?

dFmw and v8yv, you are both very positive about the paper. Would you like to argue for its acceptance?