Venue: NAACL 2025

# IdentifyMe: A Challenging Long-Context Mention Resolution Benchmark

for LLMs

Kawshik Manikantan\({}^{1}\), Makarand Tapaswi\({}^{1}\), Vineet Gandhi\({}^{1}\), Shubham Toshniwal\({}^{2}\)

\({}^{1}\)CVIT, IIIT Hyderabad \({}^{2}\)NVIDIA

###### Abstract

Recent evaluations of LLMs on coreference resolution have revealed that traditional output formats and evaluation metrics do not fully capture the models' referential understanding. To address this, we introduce IdentifyMe, a new benchmark for mention resolution presented in a multiple-choice question (MCQ) format, commonly used for evaluating LLMs. IdentifyingMe features long narratives and employs heuristics to exclude easily identifiable mentions, creating a more challenging task. The benchmark also consists of a curated mixture of different mention types and corresponding entities, allowing for a fine-grained model performance analysis. We evaluate both closed- and open-source LLMs on IdentityMe and observe a significant performance gap (20-30%) between the state-of-the-art sub-10B open models _vs._ closed ones. We observe that pronominal mentions, which have limited surface information, are typically harder for models to resolve than nominal mentions. Additionally, we find that LLMs often confuse entities when their mentions overlap in nested structures. The highest-scoring model, GPT-4o, achieves 81.9% accuracy, highlighting the strong referential capabilities of state-of-the-art LLMs while also indicating room for further improvement. 1

Footnote 1: Code for the paper is available at: [https://github.com/KawshikitManikantan/IdentifyMe](https://github.com/KawshikitManikantan/IdentifyMe)

+
Footnote †: _Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 2: Short Papers)_, pages 768–777

+
Footnote †: _Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 2: Short Papers)_, pages 768–777

## 1 Introduction

Coreference Resolution (CR) consists of identifying the entity mentions and clustering them based on the entity identity. It is a fundamental task for text comprehension and can therefore be used to assess a model's textual understanding. While LLMs have made tremendous strides on a wide array of NLP tasks Brown et al. (2020); OpenAI (2024a); Gemini Team et al. (2024), their performance on CR has been relatively underwhelming. It remains uncertain to what extent this is due to the LLMs' weak referential abilities, as traditional coreference setups--both datasets and metrics--require LLMs to adhere to varying definitions of mentions, boundaries, and entities across datasets.

For instance, Le and Ritter (2023) report that on document-level coreference annotation, LLMs perform well at mention linking but struggle with mention detection, particularly due to varying definitions of what constitutes an entity and how mention boundaries are defined. While Manikantan et al. (2024) mitigate the variability of entity definition by assuming major entities as inputs, their

Figure 1: Sample instance from the validation set of IdentifyMe. The mention of interest is highlighted in the text. The answer options include frequently occurring entities in the text, and None of the Above.



evaluation remains limited by dataset-specific span boundaries. Recent work by Gan et al. (2024) demonstrates through manual analysis that LLMs perform markedly better when evaluated in an unrestricted output mode. This suggests that traditional evaluations may underestimate LLMs' coreference capabilities, highlighting the need to adapt traditional CR datasets and metrics to better assess LLMs.

Along these lines, we introduce the IdentifyMe benchmark for mention resolution in a multiple-choice question (MCQ) format. The MCQ format is commonly used in large language model (LLM) evaluations Hendrycks et al. (2021) and offers two key advantages. First, its widespread presence in pretraining datasets enables LLMs to answer questions in this format effectively. Second, it eliminates the need for exact antecedent span identification during mention resolution evaluation, thus mitigating errors caused by dataset-specific annotation choices.

To construct the benchmark, we use annotations from two long-text coreference benchmarks, namely LitBank Bamman et al. (2020) and FantasyCoref Han et al. (2021). To make the benchmark challenging, we restrict it to pronominal and nominal mentions and apply heuristics for each mention type to filter out easily resolvable cases (Section 2.1). Each MCQ instance consists of text marked with the mention of interest and choices comprising frequently occurring entities in the text and the _None of the Above_ (NoA) option. Fig. 1 shows an example in IdentifyMe, derived from LitBank.

We evaluate both closed- and open-source LLMs with the following key findings:

* Among the mention types, LLMs perform worse on pronominal mentions (which have limited surface information) than on nominal mentions.
* The instances where _None of the Above_ is the correct answer prove particularly challenging for all the models, with open-source models experiencing a performance drop of more than 50%.
* With nested mentions, LLMs frequently confuse entities with overlapping mentions (\(e.g.\), [\(\overline{\text{his}}\)] mother).
* The highest-scoring model GPT-4o scores 81.9% on IdentifyMe, highlighting the strong performance of _frontier_ LLMs while indicating scope for further improvement in referential capabilities.

## 2IdentifyMe Benchmark

IdentifyMe is an MCQ-based benchmark where, given a text document with a marked mention, the task is to identify the entity the mention refers to. We derive these mentions from two coreference datasets focused on literary texts: LitBank and FantasyCoref. These datasets provide long contexts (1700 words on average for FantasyCoref and 2000 words for LitBank) and feature multiple entities with rich inter-dependencies (\(e.g.\), _Mr. and Mrs. Pett_) that make resolving mentions more challenging. While LitBank offers diverse writing styles and linguistic structures, FantasyCoref includes entities that often take on different forms (\(e.g.\), disguises and transformations), or undergo title change (\(e.g.\), _Prince Rudolph_ is called _The Emperor_ after his coronation), which further complicates entity mapping.

Coreference annotations cluster mentions that refer to the same entity, but creating an MCQ requires a representative phrase for each entity cluster. We use GPT-4o-mini (see Table 9) to generate these phrases based on the mentions and their frequencies. The generated annotations undergo manual review to ensure each entity has a distinct representative phrase.

To prevent confusion, we discard and avoid labeling clusters that: (i) contain annotation errors (\(e.g.\), due to cluster merging or splitting Kummerfeld and Klein (2013)); (ii) are too small (\(<3\) mentions) or difficult or ambiguous to label (\(e.g.\), entities like _some money_); (iii) are plural, as they often lack explicit surface forms that can be derived from mentions.

An MCQ is created from a document using mentions from labeled clusters, with all labeled entities provided as options. To ensure benchmark quality, we exclude short context documents (\(<1000\) words) or those where the discarded entities represent more than 50% of the mentions.

### Selecting Mentions for IdentifyingMe

Based on previous works which utilize rule-based linguistic patterns to perform Zhou and Su (2004); Lee et al. (2013) or analyze Haghighi and Klein (2009); Otmazgin et al. (2023) coreference resolution, we propose a two-step heuristic to identify 

challenging mentions.

Step 1: Discard easy mentions.We apply two criteria to filter out mentions that can be easily resolved due to syntactic similarity:

_Nominal fuzzy score_: We calculate the fuzzy similarity2 between a nominal mention and its entity's representative phrase, allowing for variations in word order and subsets. We discard mentions with similarity scores above 75%, as these cases typically provide obvious surface-form clues for identification.

Footnote 2: [https://github.com/seatgeek/thefuzz](https://github.com/seatgeek/thefuzz)

_Net distractor score_: We categorize pronominal mentions based on attributes like gender, number, and animacy (LingMess Otmazgin et al., 2023)). For a candidate marked pronominal mention, nearby pronouns of the same category that refer to the same entity can provide disambiguating context. However, pronouns that either share the category but refer to different entities, or refer to the same entity but have different categories, can increase ambiguity. We define the Net-Distractor-Score as the difference between the count of ambiguity-increasing and disambiguating neighboring pronouns. We discard mentions with non-positive scores (\(\leq 0\)).

Step 2: Ranking mentions by difficulty.Filtered mentions are ranked from most to least difficult: for nominals, a low Nominal-Fuzzy-Score is preferred; and for pronouns, a high Net-Distractor-Score is preferred. Additionally, the distance between the marked mention and other mentions of the same entity also indicate difficulty. We consider distances to the nearest mention, the nearest nominal mention, and the nearest mention resembling the representative phrase as further criteria for ranking. All the individual criteria are combined using Copeland's method Copeland (1951), evaluating pairwise wins and losses to determine the final ranking.

### Dataset Statistics

IdentifyMe comprises the 1800 most challenging questions based on our ranking method, drawn from 159 documents (64 from LitBank, 95 from FantasyCoref). We randomly select 600 of these questions as a validation set for prompt tuning and ablation experiments. Each question includes a _None of the Above (NoA)_ option to encourage more confident entity selection. To test NoA detection, we remove the correct entity from 10% of the questions, making NoA the correct choice. Both validation and test splits maintain balance across source datasets and mention types (pronominals and nominals).

### Does IdentifyMe have Hard Mentions?

We conduct an ablation experiment to assess the effectiveness of our mention selection process. As a baseline, we randomly sample mentions and evaluate model performance on their identification. The performance drops of 9.5% for Mistral-7B and 7.2% for the more robust GPT-4o-mini demonstrate that IdentifyMe captures more challenging mentions compared to random sampling (see Table 1).

### Human Evaluation on IdentifyMe Subset

We perform human evaluation on a randomly selected subset of 10 FantasyCoref documents from the test split of IdentifyMe. A set of 50 mention resolution questions are extracted from these documents, comprising 25 nominals and 25 pronominal mentions. As seen in Table 2, there is a significant performance gap of \(\sim\)23% between humans and the best performing LLM, GPT-4o. This confirms that there is substantial scope for improvement and IdentifyMe poses a challenge to current LLMs.

## 3 Experiments

Models.Among closed-source models, we evaluate GPT-4o OpenAI (2024a),

\begin{table}
\begin{tabular}{l c c} \hline \hline Model & Random (10 runs) & IdentifieryMe Val.) \\ \hline Mistral-7B & 64.8 \(\pm\) 2.1 & 55.3 \\ GPT-4o-mini & 70.5 \(\pm\) 1.9 & 63.3 \\ GPT-4o* & 83.8 & 80.7 \\ \hline \hline \end{tabular}
\end{table}
Table 1: Performance of models on the IdentifyMe validation set vs. comparable-sized evaluation set consisting of randomly chosen mentions (repeated 10 times).

\begin{table}
\begin{tabular}{l l} \hline \hline Model/Approach & Accuracy \\ \hline Mistral-7B & 46.0 \\ Llama-3.1-8B & 50.0 \\ \hline GPT-4o-mini & 62.0 \\ Gemini-1.5-Flash & 66.0 \\ GPT-4o & 70.0 \\ \hline Human-1 & 92.0 \\ Human-2 & 94.0 \\ \hline \hline \end{tabular}
\end{table}
Table 2: Performance of various models and human annotators on a subset of 50 questions from IdentifyMe.

 

GPT-4o-mini [15], and Gemini-1.5-Flash3[6]. Due to computational constraints, we limit the evaluation of open-source models to sub-10B variants: Llama-3.1-8B [27] and Mistral-7B [9].

Footnote 3: Due to safety filters, evaluated on 1197 questions

MCQ setup.The selected mention is highlighted in the original text by enclosing it with special tokens (_e.g_."\(\dots\) eject _a fourteen-year old boy_ from \(\dots\)" \(\rightarrow\) "... eject {{_a fourteen-year old boy_}} (#This is the marked span) from \(\ldots\)". A zero-shot prompt instructs the model to retrieve and resolve the mention and identify who or what it refers to from a given set of entities and NoA (detailed prompt in Appendix A.3).

Inference details.For open-source models, we use regex-based constrained decoding with the outlines library [19] to limit answers to specific entity representative phrases. We also experiment with a chain-of-thought (CoT) approach [20], instructing the model to explain its reasoning before answering the question. As seen in Table 3, using CoT improves the model performance (_e.g_., +9.5% for Llama-3.1-8B, +3.7% for GPT-4o-mini). Based on these results, we use the CoT decoding for evaluation over the test set. For details on prompts used and decoding regular expressions, see Appendix A.3.

### Results

Table 4 presents the overall LLM performance on the IdentifyMe test set, along with a breakdown by nominal and pronominal mention types. The Random baseline, where answers are uniformly randomly chosen, achieves 8% on our benchmark. Although all LLMs outperform the Random baseline, open-source models show considerable room for improvement, with Llama-3.1-8B reaching only 53.3% accuracy. GPT-4o is the top-performing model with an accuracy of 81.9%. Meanwhile, GPT-4o-mini, an affordable closed-source option, surpasses smaller open-source models but lags behind top performers like GPT-4o and Gemini-1.5-Flash. Across mention types, all closed-source models perform significantly better at resolving nominal mentions than pronominal ones.

Table 5 presents the performance split across mention types and source datasets. For nominal mentions, the FantasyCoref (FC) instances are, on average, considerably more challenging than those from LitBank (LB). This could be because of the higher surface similarity across FantasyCoref entities (_e.g_. _The eldest princess_, _The youngest princess_). In contrast, LitBank's pronominal mentions are harder to resolve than FantasyCoref's, possibly due to its complex linguistic structure.

### Error Analysis

Comparing entities NoA.Table 6 provides the accuracy distribution when the correct option is an entity (Ent) NoA. Furthermore, we classify errors into three categories: (a) ground truth is an entity and the model chooses another entity (EntEnt), (b) ground truth is an entity, but the model predicts NoA (Ent-NoA), and (c) ground truth is NoA, but the model chooses an entity (NoA-Ent). Open-source models perform extremely poorly on the NoA subset (120 MCQs), leading to high NoA-Ent errors. Among closed-source models,

\begin{table}
\begin{tabular}{l c c c} \hline \hline Model & \multicolumn{2}{c}{Total} & Nominal & Pronominal \\  & (1200) & (600) & (600) \\ \hline Random & 8.0 & 7.6 & 8.5 \\ \hline Mistral-7B & 51.5 & 52.5 & 50.5 \\ Llama-3.1-8B & 53.3 & 53.2 & 53.5 \\ \hline GPT-4o-mini & 63.3 & 67.7 & 59.0 \\ Gemini-1.5-Flash & 73.9 & 77.7 & 70.0 \\ GPT-4o & **81.9** & **85.2** & **78.7** \\ \hline \hline \end{tabular}
\end{table}
Table 4: Performance of various models on the IdentifyMe test set.

\begin{table}
\begin{tabular}{l c c c} \hline \hline Model & w/o CoT & w/ CoT \\ \hline Mistral-7B & **55.3** & 53.8 \\ Llama-3.1-8B & 50.2 & **59.7** \\ \hline GPT-4o-mini & 63.3 & **67.0** \\ \hline \hline \end{tabular}
\end{table}
Table 3: Validation accuracy of LLMs w/ and w/o CoT.

\begin{table}
\begin{tabular}{l c c c c} \hline \hline \multirow{2}{*}{Model} & \multicolumn{2}{c}{Nominal} & \multicolumn{2}{c}{Pronominal} \\  & FC & LB & FC & LB \\  & (300) & (300) & (300) & (300) \\ \hline Mistral-7B & 39.0 & 66.0 & 51.7 & 49.3 \\ Llama-3.1-8B & 42.3 & 64.0 & 55.0 & 52.0 \\ \hline GPT-4o-mini & 60.7 & 74.7 & 63.3 & 54.7 \\ Gemini-1.5-Flash & 72.1 & 83.3 & 73.7 & 66.3 \\ GPT-4o & **79.3** & **91.0** & **81.3** & **76.0** \\ \hline \hline \end{tabular}
\end{table}
Table 5: Performance split by mention type and dataset source. FC: FantasyCoref, LB: LitBank.

 

Gemini-1.5-Flash achieves sub-par performance on NoA MCQs (\(\downarrow\) 48.3%) and prefers to select an entity when the answer is NoA (83/120). Interestingly, GPT-4o and GPT-4o-mini are much more resilient on NoA questions, with drops of only \(\downarrow\) 9.6\(\%\) and \(\downarrow\) 0.9%, respectively.

Nested mentions.The dataset contains 352 instances of nested mentions, where the span of one mention overlaps with another. Table 7 shows that the accuracy of nested mentions is comparable to the overall accuracy. However, when models err in resolving these mentions, about 40% of these errors are because the predicted entity corresponds to an overlapping mention. Figure 2 illustrates a sample nested mention error made by GPT-4o.

## 4 Conclusion

We present IdentifyMe, a challenging MCQ benchmark designed for the evaluation of the referential capabilities of LLMs. Our analysis reveals several key challenges for LLMs, including: (i) pronominal resolution which has limited surface form information, (ii) questions where "None of the Above" is the correct answer, and (iii) nested mentions that require distinguishing between overlapping spans. GPT-4o scores 81.9% on IdentifyMe, highlighting the strong referential capabilities of _frontier_ LLMs while still leaving ample room for improvement. We believe the IdentifyMe benchmark, with its curated mix of diverse and challenging mentions, will serve as an effective tool for fine-grained assessment of state-of-the-art LLMs' referential capabilities.

## 5 Limitations

The IdentifyMe has several limitations: it covers only the literary domain, includes only nominal and pronominal mentions, and excludes plural entities. The source datasets we used are publicly available, and our preliminary investigations suggest limited contamination risk, as none of our evaluated LLMs could accurately reproduce the original CoNLL annotations for complete stories. While we significantly transformed the original coreference annotations to construct our benchmark, we acknowledge the potential possibility of data contamination.

## References

* B. Bamman, O. Lewke, and A. Mansoor (2020)An Annotated Dataset of Coreference in English Literature. In LREC, Cited by: SS1.
* T. B. Brown, B. Mann, N. Ryder, M. Subbiah, J. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell, S. Agarwal, A. Herbert-Voss, G. Krueger, T. Henighan, R. Child, A. Ramesh, D. M. Ziegler, J. Wu, C. Winter, C. Hesse, M. Chen, E. Sigler, M. Litwin, S. Gray, B. Chess, J. Clark, C. Berner, S. McCandlish, A. Radford, I. Sutskever, and D. Amodei (2020)Language Models are Few-Shot Learners. In NeurIPS, Cited by: SS1.

\begin{table}
\begin{tabular}{l l l l r r} \hline \hline  & \multicolumn{2}{c}{Accuracy} & \multicolumn{2}{c}{\#Misclassifications} \\ Model & Ent & NoA & Ent-Ent & Ent-NoA & NoA-Ent \\ \hline \hline \multirow{2}{*}{Mistral-7B} & 57.0 & 1.7 & 453 & 11 & 118 \\  & 59.2 & 0.8 & 438 & 3 & 119 \\ \hline GPT-4o-mini & 63.4 & 62.5 & 221 & 174 & 45 \\ Genini-1.5-Flash & 78.6 & 30.3 & 192 & 38 & 83 \\ GPT-4o & **82.9** & **73.3** & **135** & **50** & **32** \\ \hline \hline \end{tabular}
\end{table}
Table 6: Left: Model accuracy for MCQs with correct answer as an entity (Ent, 1080 samples) NoA (120 samples) Right: Number of misclassifications within entities (Ent-Ent) or with NoA (Ent-NoA, NoA-Ent).

\begin{table}
\begin{tabular}{l c c c} \hline \hline Model & \multicolumn{2}{c}{Accuracy} & Span Error \\  & Non-nested & Nested & \\ \hline \hline \multirow{2}{*}{Mistral-7B} & 50.1 & 54.8 & 40.3 \\  & Llama-3.1–8B & 53.2 & 53.7 & 42.9 \\ \hline \hline GPT-4o-mini & 60.8 & 69.3 & 34.3 \\ \hline \multirow{2}{*}{Gemini-1.5-Flash} & 73.3 & 75.1 & 36.8 \\  & 82.1 & 81.5 & 47.7 \\ \hline \hline \end{tabular}
\end{table}
Table 7: ILM performance on nested mentions (352 of 1200) versus non-nested mentions. The Span Error column indicates the error for nested mentions where the predicted entity corresponds to an overlapping mention.

Figure 2: An error by GPT-4o in resolving a nested mention where the model incorrectly resolves _his artless victim_ to the entity referred to by _his_ i.e. _M. Capoul_.

 

* Copeland (1951) A. Copeland. 1951. A Reasonable Social Welfare Function. In _Seminar on Applications of Mathematics to Social Sciences_.
* Gan et al. (2024) Yujian Gan, Massimo Poesio, and Juntao Yu. 2024. Assessing the Capabilities of Large Language Models in Coreference: An Evaluation. In _LREC-COLING_.
* Team et al. (2024) Gemini Team, Petko Georgiev, and Ying Ian Lei. 2024. Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context.
* Haghighi and Klein (2009) Aria Haghighi and Dan Klein. 2009. Simple Coreference Resolution with Rich Syntactic and Semantic Features. In _EMNLP_.
* Han et al. (2021) Sooyoun Han, Sumin Seo, Minji Kang, Jongin Kim, Nayoung Choi, Min Song, and Jinho D. Choi. 2021. FantasyCoref: Coreference Resolution on Fantasy Literature Through Omniscient Writer's Point of View. In _Fourth Workshop on Computational Models of Reference, Anaphora and Coreference_.
* Hendrycks et al. (2021) Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. 2021. Measuring Massive Multitask Language Understanding. In _ICLR_.
* Jiang et al. (2023) Albert Q Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, et al. 2023. Mistral 7B. _arXiv preprint arXiv:2310.06825_.
* Kummerfeld and Klein (2013) Jonathan K. Kummerfeld and Dan Klein. 2013. Error-Driven Analysis of Challenges in Coreference Resolution. In _EMNLP_.
* Le and Ritter (2023) Nghia T. Le and Alan Ritter. 2023. Are Large Language Models Robust Coreference Resolvers? _arXiv preprint arXiv:22305.14489_.
* Lee et al. (2013) Heeyoung Lee, Angel Chang, Yves Peirsman, Nathanael Chambers, Mihai Surdeanu, and Dan Jurafsky. 2013. Deterministic Coreference Resolution Based on Entity-Centric, Precision-Ranked Rules. _Computational Linguistics_.
* Manikantan et al. (2024) Kawshik Manikantan, Shubham Toshniwal, Makarand Tapaswi, and Vineet Gandhi. 2024. Major Entity Identification: A Generalizable Alternative to Coreference Resolution. In _EMNLP_.
* Meta-AI (2024)Meta-AI. 2024. The Llama 3 Herd of Models.
* OpenAI (2024a) OpenAI. 2024a. GPT-4 Technical Report.
* OpenAI (2024b) OpenAI. 2024b. GPT-4o-mini: Advancing Cost-Efficient Intelligence.
* Otmazgin et al. (2023) Shon Otmazgin, Arie Cattan, and Yoav Goldberg. 2023. LingMess: Linguistically Informed Multi Expert Scorers for Coreference Resolution. In _EACL_.
* Wei et al. (2023) Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed Chi, Quoc Le, and Denny Zhou. 2023. Chain-of-Thought Prompting Elicits Reasoning in Large Language Models. In _NeurIPS_.
* Willard and Louf (2023) Brandon T Willard and Remi Louf. 2023. Efficient Guided Generation for LLMs. _arXiv preprint arXiv:2307.09702_.
* Zhou and Su (2004) GuoDong Zhou and Jian Su. 2004. A High-Performance Coreference Resolution System using a Constraint-based Multi-Agent Strategy. In _COLING_.

 



## Appendix A Appendix

### Model Details

Table 8 presents the precise model identifies used in this work.

### Sample Model Outputs

Figure 3 presents another example error where the model is confused due to complicated first and second person references within dialog.

Figure 4 and 5 present instances which both GPT-4o and Llama3.1 get right. Their explanation makes sense as well.

### Prompts

Table 9 presents the prompt passed to GPT-4o-mini to generate representative phrases for each cluster/entity. Table 10 and Table 11 present the prompts and regular expressions used for inference with and without CoT respectively.



* [150] M. C. Dickson, J. R. Friedman, and A. K. S. Rao, "A new approach to the problem of computing a class of graphs," in _Proceedings of the 24th ACM SIGKDD international conference on Knowledge discovery and data mining_, pages 37-68, 2009.
* [151] P. F. G. Johnson, "The complexity of graph theory," _Journal of Graph Theory_, vol. 1, no. 1, pp. 135-140, 1996.

[MISSING_PAGE_POST]

 

* [10] M. R. Bennet, A. K. S. and J. C. H. Lee, "The role of the word-level information in a word-level context: The word-level information in a word-level context: The word-level information in a word-level context: The word-level information in a word-level context: The word-level information in a word-level context: The word-level information in a word-level context: The word-level information in a word-level context: The word-level information in a word-level context: The word-level information in a word-level context: The word-level information in a word-level context: The word-level information in a word-level context: The word-level information in a word-level context: The word-level information in a word-level context: The word-level information in a word-level context: The word-level information in a word-level context: The word-level information in a word-level context: The word-level information in a word-level context: The word-level information in a word-level context: The word-level information in a word-level context: The word-level information in a word-level context: The word-level information in a word-level context: The word-level information in a word-level context: The word-level information in a word-level context: The word-level information in a word-level context: The word-level information in a word-level context: The word-level information in a word-level context: The word-level information in a word-level context: The word-level information in a word-level context: The word-level information in a word-level context: The word-level information in a word-level context: The word-level information in a word-level context: The word-level information in a word-level context: The word-level information in a word-level context: The word-level information in a word-level context: The word-level information in a word-level context: The word-level information in a word-level context: The word-level information in a word-level context: The word-level information in a word-level context: The word-level information in a word-level context: The word-level information in a word-level context: The word-level information in a word-level context: The word-level information in a word-level context: The word-level information in a word-level context: The word-level information in a word-level context: The word-level information in a word-level context: The word-level information in a word-level context: The word-level information in a word-level context: The word-level information in a word-level context: The word-level information in a word-level context: The word-level information in a word-level context: The word-level information in a word-level context: The word-level information in a word-level context: The word-level information in a word-level context: The word-level information in a word-level context: The word-level information in a word-level context: The word-level information in a word-level context: The word-level information in a word-level context: The word-level information in a word-level context: The word-level information in a word-level context: The word-level information in a word-level context: The word-level information in a word-level context: The word-level information in a word-level context: The word-level information in a word-level context: The word-level information in a word-level context: The word-level information in a word-level context: The word-level information in a word-level context: The word-level information in a word-level context: The word-level information in a word-level context: The word-level information in a word-level context: The word-level information in a word-level context: The word-level information in a word-level context: The word-level information in a word-level context: The word-level information in a word-level context: The word-level information in a word-level context: The word-level information in a word-level context: The word-level information in a word-level context: The word-level information in a word-level context: The word-level information in a word-level context: The word-level information in a word-level context: The word-level information in a word-level context: The word-level information in a word-level context: The word-level information in a word-level context: The word-level information in a word-level context: The word-level information in a word-level context: The word-level information in a word-level context: The word-level information in a word-level context: The word-level information in a word-level context: The word-level information in a word-level context: The word-level information in a word-level context: The word-level information in a word-level context: The word-level information in a word-level context: The word-level information in a word-level context: The word-level information in a word-level context: The word-level information in a word-level context: The word-level information in a word-level context: The word-level information in a word-level context: The word-level information in a word-level context: The word-level information in a word-level context: The word-level information in a word-level context: The word-level information in a word-level context: The word-level information in a word-level context: The word-level information in a word-level context: The word-level information in a word-level context: The word-level information in a word-level context: The word-level information in a word-level context: The word-level information in a word-level context: The word-level information in a word-level context: The word-level information in a word-level context: The word-level information in a word-level context: The word-level information in a word-level context: The word-level information in a word-level context: The word-level information in a word-level context: The word-level information in a word-level context: The word-level information in a word-level context: The word-level information in a word-level context: The word-level information in a word-level context: The word-level information in a word-level context: The word-level information in a word-level context: The word-level information in a word-level context: The word-level information in a word-level context: The word-level information in a word-level context: The word-level information in a word-level context: The word-level information in a word-level context: The word-level information in a word-level context: The word-level information in a word-level context: The word-level information in a word-level context: The word-level information in a word-level context: The word-level information in a word-level context: The word-level information in a word-level context: The word-level information in a word-level context: The word-level information in a word-level context: The word-level information in a word-level context: The word-level information in a word-level context: The word-level information in a word-level context: The word-level information in a word-level context: The word-level information in a word-level context: The word-level information in a word-level context: The word-level information in a word-level context: The word-level information in a word-level context: The word-level information in a word-level context: The word-level information in a word-level context: The word-level information in a word-level context: The word-level information in a word-level context: The word-level information in a word-level context: The word-level information in a word-level context: The word-level information in a word-level context: The word-level information in a word-level context: The word-level information in a word-level context: The word-level information in a word-level context: The word-level information in a word-level context: The word-level information in a word-level context: The word-level information in a word-level context: The word-level information in a word-level context: The word-level information in a word-level context: The word-level information in a word-level context: The word-level information in a word-level context: The word-level information in a word-level context: The word-level information in a word-level context: The word-level information in a word-level context: The word-level information in a word-level context: The word-level information in a word-level context: The word-level information in a word-level context: The word-level information in a word-level context: The word-level information in a word-level context: The word-level information in a word-level context: The word-level information in a word-level context: The word-level information in a word-level context: The word-level information in a word-level context: The word-level information in a word-level context: The word-level information in a word-level context: The word-level information in a word-level context: The word-level information in a word-level context: The word-level information in a word-level context: The word-level information in a word-level context: The word-level information in a word-level context: The word-level information in a word-level context: The word-level information in a word-level context: The word-level information in a word-level context: The word-level information in a word-level context: The word-level information in a word-level context: The word-level information in a word-level context: The word-level information in a word-level context: The word-level information in a word-level context: The word-level information in a word-level context: The word-level information in a word-level context: The word-level information in a word-level context: The word-level information in a word-level context: The word-level information in a word-level context: The word-level information in a word-level context: The word-level information in a word-level context: The word-level information in a word-level context: The word-level information in a word-level context: The word-level information in a word-level context: The word-level information in a word-level context: The word-level information in a word-level context: The word-level information in a word-level context: The word-level information in a word-level context: The word-level information in a word-level context: The word-level information in a word-level context: The word-level information in a word-level context: The word-level information in a word-level context: The word-level information in a word-level context: The word-level information in a word-level context: The word-level information in a word-level context: The word-level information in a word-level context: The word-level information in a word-level context: The word-level information in a word-level context: The word-level information in a word-level context: The word-level information in a word-level context: The word-level information in a word-level context: The word-level information in a word-level context: The word-level information in a word-level context: The word-level information in a word-level context: The word-level information in a word-level context: The word-level information in a word-level context: The word-level information in a word-level context: The word-level information in a word-level context: The word-level information in a word-level context: The word-level information in a word-level context: The word-level information in a word-level context: The word-level information in a word-level context: The word-level information in a word-level context: The word-level information in a word-level context: The word-level information in a word-level context: The word-level information in a word-level context: The word-level information in a word-level context: The word-level information in a word-level context: The word-level information in a word-level context: The word-level information in a word-level context: The word-level information in a word-level context: The word-level information in a word-level context: The word-level information in a word-level context: The word-level information in a word-level context: The word-level information in a word-level context: The word-level information in a word-level context: The word-level information in a word-level context: The word-level information in a word-level context: The word-level information in a word-level context: The word-level information in a word-level context: The word-level information in a word-level context: The word-level information in a word-level context: The word-level information in a word-level context: The word-level information in a word-level context: The word-level information in a word-level context: The word-level information in a word-level context: The word-level information in a word-level context: The word-level information in a word-level context: The word-level information in a word-level context: The word-level information in a word-level context: The word-level information in a word-level context: The word-level information in a word-level context: The word-level information in a word-level context: The word-level information in a word-level context: The word-level information in a word-level context: The word-level information in a word-level context: The word-level information in a word-level context: The word-level information in a word-level context: The word-level information in a word-level context: The word-level information in a word-level context: The word-level information in a word-level context: The word-level information in a word-level context: The word-level information in a word-level context: The word-level information in a word-level context: The word-level information in a word-level context: The word-level information in a word-level context: The word-level information in a word-level context: The word-level information in a word-level context: The word-level information in a word-level context: The word-level information in a word-level context: The word-level information in a word-level context: The word-level information in a word-level context: The word-level information in a word-level context: The word-level information in a word-level context: The word-level information in a word-level context: The word-level information in a word-level context: The word-level information in a word- 



**Instruction**

Read the text given below. The text has an entity mention marked within ""({mention}) (#This is the marked mention) """. Extract the mention and find who/what the mention refers to in the text.

**Example Input: Text: Chapter 1 It is a truth universally acknowledged, that a single man in possession of a good fortune, must be in want of a wife. However little known the feelings or views of such a man may be on his first entering a neighbourhood, this truth is so well fixed in the minds of the surrounding families, that he is considered the rightful property of some one or other of their daughters. "My dear Mr. Bennet," said his lady to him one day, ...

Chapter 2 Mr. Bennet was among the earliest of those who waited on Mr. Bingley. He had always intended to visit him, though to the last always assuring ({his wife}) (#This is the marked mention) that he should not go; and till the evening after the visit was paid she had no knowledge of it. It was then disclosed in the following manner. Observing his second daughter employed in trimming a hat, he suddenly addressed her with: "I hope Mr. Bingley will like it, Lizzy." "We are not in a way to know what Mr. Bingley likes," said her mother resentfully, "since we are not to visit" ....

 I do not know how you will ever make him amends for his kindness; or me, either, for that matter. At our time of life it is not so pleasant, I can tell you, to be making new acquaintances every day; but for your sakes, we would do anything. Lydia, my love, though you are the youngest, I dare say Mr. Bingley will dance with you at the next ball.

- The mention refers to: Decoding Regex (Constrained Decoding):
- Mention: \(\langle\{[A-Za-z ,!'..]_{[1,125]}\}\rangle\rangle\)
- The mention refers to: (Mary|Kitty|Mrs. Bennet|Mrs. Long|Elizabeth|Mr. Bingley| Mr. Bennet|Lydia|Netherfield Park|None of the Above) Example Output:
- Mention: his wife
- The mention refers to: Mrs. Bennet

## Appendix C

\begin{table}
\begin{tabular}{l} \hline
**Instruction** \\ Read the text given below. The text has an entity mention marked within ""({mention}) (\#This is the marked mention) """. Extract the mention and find who/what the mention refers to in the text. \\ \hline
**Example Input:** \\ \hline
**Text:** \\ Chapter 1 It is a truth universally acknowledged, that a single man in possession of a good fortune, must be in want of a wife. However little known the feelings or views of such a man may be on his first entering a neighbourhood, this truth is so well fixed in the minds of the surrounding families, that he is considered the rightful property of some one or other of their daughters. “My dear Mr. Bennet," said his lady to him one day, ...

Chapter 2 Mr. Bennet was among the earliest of those who waited on Mr. Bingley. He had always intended to visit him, though to the last always assuring ({his wife}) (#This is the marked mention) that he should not go; and till the evening after the visit was paid she had no knowledge of it. It was then disclosed in the following manner. Observing his second daughter employed in trimming a hat, he suddenly addressed her with: ”I hope Mr. Bingley will like it, Lizzy.” "We are not in a way to know what Mr. Bingley likes," said her mother resentfully, "since we are not to visit” ....

 I do not know how you will ever make him amends for his kindness; or me, either, for that matter. At our time of life it is not so pleasant, I can tell you, to be making new acquaintances every day; but for your skes, we would do anything. Lydia, my love, though you are the youngest, I dare say Mr. Bingley will dance with you at the next ball.

- The mention refers to: \\ \hline
* Mention: \(\langle\{[A-Za-z ,!’..]_{[1,125]}\}\rangle\rangle\)
- The mention refers to: (Mary|Kitty|Mrs. Bennet|Mrs. Long|Elizabeth|Mr. Bingley| Mr. Bennet|Lydia|Netherfield Park|None of the Above) Example Output: \\ \hline
- Mention: his wife
- The mention refers to: Mrs. Bennet \\ \hline \end{tabular}
\end{table}
Table 11: QA prompt without CoT.