Bridging Context Gaps: Leveraging Coreference Resolution for Long Contextual Understanding
Authors: Yanming Liu
Xinyue Peng
Jiannan Cao
Shi Bo
Yanxin Shen
Tianyu Du
Sheng Cheng
Xun Wang
Jianwei Yin
Xuhong Zhang
Venue: ICLR 2025 Poster (2025)
Keywords: Coreference Resolution
Long Contextual Understanding
Information Extraction
Primary area: unsupervised, self-supervised, semi-supervised, and supervised representation learning

Abstract: Large language models (LLMs) have shown remarkable capabilities in natural language processing; however, they still face difficulties when tasked with understanding lengthy contexts and executing effective question answering. These challenges often arise due to the complexity and ambiguity present in longer texts. To enhance the performance of LLMs in such scenarios, we introduce the Long Question Coreference Adaptation (LQCA) method. This innovative framework focuses on coreference resolution tailored to long contexts, allowing the model to identify and manage references effectively. The LQCA method encompasses four key steps: resolving coreferences within sub-documents, computing the distances between mentions, defining a representative mention for coreference, and answering questions through mention replacement. By processing information systematically, the framework provides easier-to-handle partitions for LLMs, promoting better understanding. Experimental evaluations on a range of LLMs and datasets have yielded positive results, with a notable improvements on OpenAI-o1-mini and GPT-4o models, highlighting the effectiveness of leveraging coreference resolution to bridge context gaps in question answering. Our code is public at https://github.com/OceannTwT/LQCA.

Published as a conference paper at ICLR 2025
BRIDGING CONTEXT GAPS: LEVERAGING COREFER- ENCE RESOLUTION FOR LONG CONTEXTUAL UNDER- STANDING
Yanming Liu', Xinyue P Jiannan Cao', Shi Bo Yanxin Shen', Tianyu Du
Sheng Cheng', Xun Wang", Jiannel Yin', Xuhong Zhang!* 'Zhejiang University, SSutheeat University. Booton University *Massachusetts Institute of Technology
(oceann24, ssyysyx, zhangxuhong, zjuyjw, zjradty/ezju. edu.cn j jiannanemit. kinyuepengêseu.eduu edu. cn, shibo@bu. edu
ABSTRACT
Large language models (LLMs) have shown remarkable capabilities in natural language processing; however, they still face difficulties when tasked with un- derstanding lengthy contexts and executing effective question answering. These challenges often arise due to the complexity and ambiguity present in longer texts. To enhance the performance LLMs in such scenarios, we introduce the Long Question Coreference Adaptation (LQCA) method. This innovative framework focuses on coreference resolution tailored to long contexts, allowing the model identify and manage references effectively. The LQQA method encompasses four key steps: resolving coreferences within sub-documents, computing the dis- tances between mentions, defining representative mention for coreference, and answering questions through mention replacement. By processing information systematically, the framework provides easier-to-hande to-handle partitions for LLMs, pro- moting better understanding. Experimental evaluations on a range of LLMs and datasets have yielded positive results, with notable improvements on OpenAI-ol- mini and GPT-4o models, highlighting the effectiveness of leveraging coreference resolution to bridge context gaps in question answering. Our code is public at https: thub. com wwwp///ceeannww/////
INTRODUCTION
Large language models (LLMs) (Brown 2020
Chowdhery et al. 2023 Ouyang et al.) 2002) have
"Hensy Bue" by Englis tarned
fectbor auy tutitian
demonstrated exceptional competitiveness across
ectress Bille Sres her debut studi alkam.
various tasks, including question answering, sum-
(1000 woeds)
marization, and generation (Wang al. (2023). Re-
-The single also uxpassed "Unteriia" und becazse
longest running single
lustic 1356 ctireigg Ai?
cently, many new versions of LLMs have begun extend their capability handle longer contexts (Yang et al.) [0244] Dubey et al. 2024). Both open- source and proprietary models show some profi- ciency in understanding long texts and support ex-
singer's
top
(1000)
frightering,
playing with lighesing con-an" Band awenber Tem Furker said the song. "Mi not the seand bise welve aever dooe before. Lts dee feet perlode to dhe alben, and great song get you in the moed for party""
In auy teferexeg
tended reading (Zhang et al. 2024a). However, LLMs still struggle to accurately identify key pas- sages from the middle of long contexts (Liu et al. 2024h and generate effective responses expected content based on these passages when overwhelmed by excessive information (Shi al.) 2023). This suggests that despite having larger context windows, many LLMs still face challenges in handling long- text tasks (Li et al. 2024b).
(0000 words)
das for relaue the tib of Seytender, 2003. Studio albams Extenced plays Stagln Fautared single Other charted sorge Mixic rdeess Fooraates
crttatt Courtneee
Figure I: The complexity arises from multiple enti- ties and coreference relations in long contexts, mak- ing it difficult for large models to effectively lear contextual information.
Corrreponniigg author.

---

Published as a conference paper at ICLR 2025
Currently, range of studies are being conducted long-text reasoning, covering areas such as foundational models for long texts, long-text datasets (Li et al. 2024a An et al. 2024), task types, and multi-turn dialogues. To more efficiently retrieve contextual information, techniques like retrieval- augmented methods (Phan et al. 2024 Bai et al. 2024 Zhu et al. 2024a) for slicing long contexts or compressing (Huang et al. 2024) and reinterpreting articles are rapidly advancing.
In the domain of information extraction, LLMs still fall short in tasks such as entity extraction, coreference resolution, and event detection, often described as "large but not precise" [Isik et al. 20244 Kumar 2024). Recent research has attempted coreference resolution through prompt engineering,
while some studies based on the DeBERTa (He al. (02T) model series have shown improvements in coreference resolution tasks (Martinelli et al. 2024). However, these models remain limited by
their smaller parameter sizes and constrained context windows (Oimazgin al. 2023
Although LLMs exhibit some capabilities in handling long-text tasks, their performance remains highly dependent on the quality of the text. In lengthy texts, the use of numerous similar referring expressions, modifiers, and varying noun phrases can pose significant comprehension challenges for the models. This is especially critical when dealing with ambiguous sentences, where LLMs are prone to confusion (Shi et al. 2023). Therefore, improving the quality of reasoning for long-text tasks and focusing on the quality of long-text content have become critical research topics.
To address the challenges of LLMs in understanding long contexts and question answering, propose the Long Question Coreference Adaptation (LQCA) method, an efficient framework for long-context coreference resolution and question answering. The LQCA framework is capable of resolving all mentions in long contexts and selecting the best answer mention to replace the original text. This framework consists of four steps: coreference resolution on sub-document, mentions distances computation, defining coreference representative mention, and question answering with mention replacement. These four steps systematically process information within long contexts, ultimately replacing it with partitions that are easier for large models to understand. We conducted experiments on five large models with varying parameter sizes and nine datasets focused on long- context questions. The results indicate that long contexts processed through coreference resolution perform best during inference with language models, achieving a 3.61% % improvement on GPT-4. Our method demonstrates the advantages and capabilities of information extraction and coreference resolution in understanding long contexts while emphasizing the importance of text quality for model inference.
Our Contributions. Our main contributions can be summarized as follows:
We propose the LQCA framework, a coreference resolution method designed to enhance the LLM capability of long context responses. Our approach partition the context and merges coreference information to achieve comprehensive coreference resolution results for the entire document, replacing segments of the text to improve its quality, thereby helping the model better understand the context.
We demonstrate that text quality plays a crucial role in answering questions related to long text comprehension. By optimizing text quality and eliminating ambiguity, enable the model to learn and reason more effectively for the target tasks.
Extensive experiments show that our method performs exceptionally well across various tasks, including long text question answering and long text classification, providing a new paradigm for addressing long text issues.
2 PRELIMINARIES
Coreference Resolution and Mentions. In information extraction and dialogue systems, traditional methods involve dealing with entities, relationships between entities, and semantic slot filling when processing context. However, in certain contexts and dialogue scenarios, such as speaker recognition and long context reading, the context often contains a large number of pronouns and modifiers, which significantly affects the model's understanding of the original information. To address this situation, the context can be optimized through two steps. The first step is mentions extraction: Given the input text æ, extract all pronouns, nouns, noun phrases, and modifiers in the context to obtain the mention set M = {mi, .y..... m2). For a specific mention m, its reference information is represented R(m:) my In the input text, the reference information is likely have large number of
2

---

Published as a conference paper at ICLR 2025
Context Travels Euroge Neihher er Here nor Thene: Travels in Europe a 1991 humorous trave cque American writer cJ wyy impressions of the life, cullure and smeni les oo lack there off each locality
Sub documents (rice volund)
Contewt: As Broughout chilhood Iiko lead hers wore unareddo by Biil actisito)) fact Bryoón rocouite
Parttffioing CR
Purt-ol-Speech
Sub documents
realy wast.
Documment
Contentt Je throughout
last, Brysom recounti
Conteett in "ac Bigson recounts how getting
chistoodd actieterl txtt
barzzre noen th not even giling school very othert (obaro)
gerling
odhool very aften
boforr thus goinng
Bryson recounts how the IC3 woe unantoredted gett actore even going oftens.
6d]
Montions:
1452p, 1435p this actlaitin 1142h, 14270-
adhpol Desprie this unicua beteenzz Bryson in jais stony, through the eyss of diidi..)
Conetrranoe EGaideen (FE,) 1035 [Go [Calsssdoeea menatoomminn 14840L
Coreferance: paryoon, 11482 14320 the [1435, 14350-] activities [143) 1427|L-
I can this
Context: Travels Neither Here There: Traves wy mmpeesion the ilfe culture and amerilies lock thereof) ir Australla
195" travelogue
Quastion: Bil Bryson in city that the seat of which county?
Figure 2: Our Long Question Corefcrence Adaptation(1.00 (CAQ method addresses the coreference resolution problem in long texts through four steps, leveraging indirect coreference and mention relations, and providing the resolved context to the LLM for question reasoning.
reduudant references within the mention set, referred to co-reference c. For two meaning- redundant mentions my and mj. it holds that {R(m,), R{m,)) S c. Since mentions inherently refer to themselves, this can also be expressed as (m;, my] S c. By obtaining all coreference relationships and replacing mentions the original text, the clarity of the text can be effectively enhanced.
METHODOLOGY
We employ four-step Long Question Coreference Adaptation (LQCA) method. Each step involves the integration and extraction of information from the text. By partitioning long documents and resolving references within each sub-document, we use clustering techniques to merge the results from different sub-documents into the same cluster, thereby achieving overall reference resolution for the entire document. Finally, we replace the references and provide the cleaned text to the large model for contextual understanding and response.
3.1 COREFERENCE RESOLUTION ON SUB-DOCUMENT
The sate-of-the-an reference resolution model, Maverick (Martinelli et al., 2024) is based on the DeBERTa-v33 (He et al., 2021) architecture, which has specific requirements regarding the length of the input tokens. We cannot directly perform reference resolution on the entire long input. Given an long input context X, initiate the process by partitioning the context into sub-documents {S J.V each constrained to a maximum length of L. is the total number of partitioning documents. Our partitioning is based sliding window approach, where each partitioning starts from the beginning of sentence and extends to the position of the last sentence that does not exceed the length of L. If the last sentence exceeds this limit, truncation is applied for the whole sentence.
For each sub-document S;, we utilize the maver www.ccc---s-oooooooeeeeeeo traction and coreference resolution. .M- {m1, m2, m& represent the set of detected mentions in the input sequence, where each mention m; is characterized by its position Pt. In the coreference resolution of sub-documents, e/ represents the j-th coreference set in the i-th sub-document and C. cf .f...., c/) } is the total coreferences in single uob-document. The mentions under this coreference set belong to this reference. After providing the sub-documents to the Maverick model through batch inference, ing to the long text. The results help us in the subsequent processing of recognizing and merging the same coreferences and mentions, allowing us to combine the information from these sub-documents.
model to perform mentions ex-
obtained all the coreferences C [C.) w and mention M correspond-
3

---

Published as a conference paper at ICLR 2025
3.2 MENTIONS DISTANCES COMPUTATION
To address the issue of co co-referent mentions being distributed across multiple partitions, we construct a sparse matrix representing mention relationships. For any two mentions my and my occurring in the same sub-document, we define their co-reference score under particular sub-document and specific reference as follows:
{
if ma; S C.. otherwise
$ (mas mb)
Similarly, the score for non-corte con-coreference i; is defined as:
{1
if ma. my & c/
0 (may ms) = {0 otherwise
Thus, the co-reference score for these mentions within the sub-document is s;(ma, mm) EE sy and the non-co-reeerence score is (m, ma) E, t/. Since two mentions can only belong to the same reference in single sub-document, the scores 8; (Ma,, ms) and 4 ms) satisfy s,mma; my) @ 4,(m (Ma, m,) = 1, indicating whether the two mentions are the same reference.
Through this process, can determine the co-reference relationship of any two mentions within sub-document. Only when both mentions are present in the sub-document can they receive mention score. For cross-ubbdoccumen mention relationships, we integrate information across multiple sub-documents based on the distance between the two mentions.
For the entire long context, define d(ma, m2) as the distance between two mentions. Since mentions that appear in the same sub-document have already had their co-reference relationship evaluated by the Maverick model, can assess their distance determine whether they refer the same entity in the context of the long document. The score between mentions is computed as:
EN s: (:maa mb)
d(ma, mp)
EN 8; (Ma, mb) + ΣN ty (m my)
For mentions not in the same sub-doeument, since they lack coreference and coreference scores, merging their relationships requires evaluation based on distance information. For any mention m.s the distance between mentions m, and m. is d(m,, mc), , and the distance between mentions m& and m. is d(m,, mc), we apply the multiplication principle:
d(m.; my) = max mmax(dmm, x d(ms, mc)
cEM
Using this distance evaluation, the distance information between any two mentions in the mention cluster can calculated. This form can be easily computed using variations of the shortest path problem to obtain the corresponding distance information. The algorithm could be refered in Appendix G
After computing the longest dot product path, compare the longest path score to predetermined threshold k and based on this information, we construct mention relationship graph G(M, E):
(ma, E F,, if d(m,; ms) >
In the mention relationship graph, each strongly connected component C, = {u E M
Ve E
C₁ ++ f} represents set of coreferent mentions.
This classification helps bridge the gaps in context, ensuring that relevant mentions are identified as referring to the same entity, thereby improving the model's understanding of the document content.
4

---

Published as a conference paper at ICLR 2025
3.3 DEFINING COREFERENCE REPRESENTIVE MENTIONS
For the strongly connected components we obtained, each strong connection serves as coreference, and when inputting into the LLM later, all mentions under the same coreference need be replaced. especially those mentions that are pronouns, which should be transformed into specific, meaningful content.
We first use the lightweight spaCy model htt_core_web... to perform part-of-speech tagging on all words the article. the tagging results, if token the mention span belongs to PRON, it is marked p(m4)= PRON. Additionally, for coreference, we want the transformed text to select the mention that contains the most meaningful equivalent. Let me, be the representative mention of reference C₁ and [m,, e2) be the number of times mention my appears under coreference C₁. The process could formulated as:
mag argmax (s(m, ci) x (p(m:) o PRON]]
there are multiple mentions that satisfy this condition, we select the carliest mention based on the principle of first selection. This representative mention serves the normalized text for the entire coreference. By effectively replacing vague mentions with their n the text, enabling LLMs to maintain coherent understanding of the context.
3.4 QUESTION ANSWERING WITH MENTIONS REPLACEMENT
need to modify the original text into the target text by replacing mentions in the text based on each coreferent representative mention. When handling overlapping cases, we prioritize replacing the mention with the largest overlapping range. no suitable replacement is available (for example, only pronouns). we retain the original text.
To execute the Question Answering (QA) task, we leverage the reasoning capabilities of the LLM, formalized in the following process:
i = arg min (- log P(R|C', R
where P(R(C", Q) represents the probability of generating a response A given the modified context O' and the question Q. By implementing these steps, approach effectively addresses the coreference issues long texts, enabling subsequent tasks (such as QA) to interact more accurately with the content and enhance contextual awareness. This integrated method only tackles the challenges posed by long texts but also effectively bridges contextual gaps, improving LLM performance in understanding long contexts.
4 EXPERIMENTAL SETUP
4.1 DATASETS
To evaluate the performance of LQCA in long text contexts, we primarily conduct assessments on question-answering data across three datasets, targeting different task categories including sum- marization tasks, auuettin-answerring these three datasets are LooGLE (Li et al.) 2024a). L-Eval (An et al.) (2024), and LongBench (Bai et al., 2024). For the summarization task, we select the arXiv paper abstract category from the LooGLE dataset for evaluation. For the question-answent in L-Eval, emphasizing multiple-choice and true/false question-answering tasks. For the dataset dataset requiring information extraction from texts, we utilize the multi-document question-answerte under LongBench, primarily selecting three English-based datasets: HotpotQA (Yang et al. 2018 8), 2WikiMultihoQAA (Ho et al., 2020). and MuSiQue (Trivedi et al., 2022).. This diverse selection enables to better understand the performance of our approach across different tasks and datasets, providing a comprehensive perspective on its strengths and weaknesses in multi-tate task scenarios. For more details the datasets, please refer Appendix B
tasks, and multiple-choice classification tasks. Specifically,
task, we focus on closed-ended tasks
5

---

Published as a conference paper at ICLR 2025
4.2 BASELINES
Currently, methods for addressing long-context issues mainly involve guiding models through chain- of-thought techniques for specific step reasoning. One method the compression strategy, which condenses key information from long texts into concise paragraphs, allowing questions to be answered and processed based on this. Another commonly used method involves slicing text segments for retrieval. Our comparison benchmarks include the following methods:
Vanilla LM: Guides the model to generate responses through simple text input, testing its basic ability to handle long contexts.
Zero-shot Chain-of-Thought (Kojima al. 2022): Utilizes prompt templates, designing specific prompts guide the model's reasoning, leveraging the model's inference capabilities without the need for additional training data.
RAG (Lwiis et al., 2020 Ram et al.) 2023 with Self-Corpus: Unlike traditional RAG, do not introduce additional knowledge but instead use slices of long text segments as the corpus. When questioning relevant issues, we retrieve from the corpus and combine it with context for reasoning, aiming to improve the model's understanding and response quality to long texts.
RECOMP (Xu al., 2024) with Self-Corpus: Builds upon RAG with added compression strategy, using the same long text segment slice corpus. This method improves processing efficiency by compressing key information from long texts, aiming optimize the model's 's reasoning process without losing important information.
4.3 MODELS
We utilize five models, including three from OpenAl's GPT series (Brown 2020): 01-mini-2024-99-11 2oot-4o-2024-8--2 and got-400miini-2224-000
as well as two open- source large models from Llama-3 (Dubey et al., (202) and Qwen-2 (Yang et al.] (002):: 1lama3-gradient-8) and qwen2-7b. The five tested large models all have a 128k context win- dow, which meets our requirements for evaluating and testing long context datasets, reducing the information loss caused by additional text segmentation. Comparing different models helps us understand whether their core reference performance in long contexts is similar.
4.4 IMPLEMENTATION
Our experiments were primarily evaluated in zero-shot setting. For inference predictions of general models, we used default settings and adopted greedy search as the inference benchmark. For zero-shot reasoning chains, referred to the prompt templates the default chain-of of-thought scheme. The evaluated models have long context window, allowing us to retain almost all contextual information for the questions without truncation.
Nonetheless, we still establish a safety mechanism for ultra-long texts. When the input length L exceeds the model's maximum context length (indicated by the name suffix), we truncate from the middle of the input sequence S to avoid losing the beginning and end portions, which may contain key information. During the generation process, we employed greedy decoding ensure the reproducibility of results. For LQCA, our default experimental setting is coreference score threshold of 0.9, meaning the distance between two mentions exceeds 0.9, we consider them within the same coreference. For the spaCy model, we used en core web sm model for tokenization and part-of-speech tagging.
In terms of benchmark comparison, we use retrieval method, with Contriever-msmarco (Zacard ct al. 2022) as the retrieval model responsible for slicing articles in a sliding, non-overlapping manner, with each segment limited to 512 tokens. Additionally, in the RECOMP experiments, we choose the abstract compression tool provided by RECOMP as the compression tool to help the model effectively extract important information from text segments. The comparison of LQCA with other baselines is based on the differences in the comparison datasets, evaluated using Rouge-L, accuracy and F1 scores.

---

Published as a conference paper at ICLR 2025
Table Evaluations of the performance of different models and methods across three datasets. Assess the long context ability in question answering, summarization, and knowledge extraction.
Modill
LooGLE
L-Eval
Longlienhh
Aug.
Aralv paper abstract (Ronge-L)
Long dependency QA (Rouge-L.)
TOEEL
QUALITY (Arc.)
Courseru (Acc.)
SFciiin (Aec.)
HetgotQA (F1)
ZWIldMateehoOQA (F1)
MusiQue (F7)
(Aec.)
OpenAl al mini
35.62
33,98
55.44
89.94
66.33
66.60
30.47
27.46
90.55
64.12
25.55
60.15
27.95
25,61
73.99
80.20
62.57
33.97
57.75
Liama -bb
6.25
9:13
46.22
00.94
64.56
32.55
23.51
38.00
Qwen2-76
7.56
6.64
57.35
48.92
50.79
62.33
46.19
54.11
20.69
37.17
Zero sker CoT
OpenAl al mini
34.89
32.56
89-42
88.55
K5:97
87.66
71.55
50.24
37.85
64.30
40 GPT-4ommiii
31.25
28.75
88.91
87.12
$2.15
81.46
66.94
69.36
34.95
61.21
Llamm-3-4b
5.65
Owes2-76
6.95
61.74
OpznAl mini
30.95
30.12
70.26
36.73
61.85
GPT-4o
27.65
22.35
85.11
80.12
50.65
78. 12
66.35
47.01
32.11
57.72
GPT-4o-min
25.71
23.14
83.23
82.23
73
79.19
62.94
42.55
35.52
56.26
Lhaan-3-88 3-5b
5.84
5.56
63.15
45.66
48.77
52.94
40.72
51.78
22.03
35.60
Qwen Te
8,33
5.23
55.79
45.12
60.21
58.99
61,80
$4.09
19.42
35.10
RECOMP with anl Carpan
Open -mii-
21.55
76:25
Qhristion Congfoennee AdtonNtooNL.CAA
48.65
37.36
03.71
91.66
K7.25
24
75.92
56.85
18.55
67.5K
GPT
$4.53
30.17
91.03
89.62
85.12
81.92
71.66
53,62
36.25
63.76
GPT do mini
29.59
28.45
88.72
88.65
K3.99
79.89
70.33
46.16
34.82
61.17
Liam- -b
5.91
9.23
69:55
24 08.07
09.15
28.50
66.23
54.21
25.03
39.37
Qwes2-76 76
7:51
5.70
61.49
50.22
53.76
61.02
47.11
37.16
21.95
38.44
EXPERIMENTS
5.1
MAIN RESULTS
We evaluate the performances of five LLMs on multiple datasets with LQCA and baselines, and the results are shown in the Table [
Context with coreference resolution. LQCA has demonstrated highly competitive performance among other long context processing methods. It achieves the best results on almost all datasets when applied to LLMs with a higher number of parameters and strong contextual understanding, consistently outperforming other baseline performances on GPT-4o. Meanwhile, the other two 7-8 out of 9 baselines methods, respectively. Compared
models also achieve the best performance to directly providing the question to the LLM, LQCA achieves average improvement of +3.61 % on GPT-4o. On the latest o1-mini model, the coreference resolution method also significantly improves by average of +3.18 % across various metrics compared to other methods. At the same time, LQCA has also achieved good performance on open-source large models with relatively small parameter counts, showing improvements in metrics compared direct prompting and zero-shot chain-of-thought methods. However, due to differences in context understanding capabilities, retrieval methods can sometimes find corresponding answers more accurately for these types of large models. Overall, texts that have undergone coreference resolution can help models understand contextual dependencies to varying degrees, thereby assisting models solving various problems.
LQCA in Long dependency question. When addressing tasks involving long-dependency issues, such summarization.long dependency QA, and strategy-relatee question answering (HotpotQA. 2WikiMHQA) tasks, coreference resolution methods demonstrate a clear advantage over other approaches, especially when the model possesses stronger contextual understanding capabilities. For example, in summarization tasks, the O1-mini model improved performance by +5.05 % compared to other methods. Even more notably, in datasets like HotpotQA, which contain extensive text passages and dependencies, the impact of coreference resolution is even more pronounced. For instance, this dataset, GPT4o showed a +7.59% improvement in results after incorporating coreference resolution, outperforming other methods by about +6.5%. The most challenging aspect of long-de oong-dependency tasks is determining various coreference relations and the specific entities being referred to. When a question is directly presented to the model, it may confuse these coreferences, leading incorrect
guestion-answering
7

---

Published as a conference paper at ICLR 2025
X021
Figure 3: The difference between various coreference resolution variants and the LQCA method on GPT-4o(left) and Llama-3-Sb(righ))
answers. Thus, effective coreference resolution enhances contextual coherence, helping the model provide more accurate responses.
LQCA in Knowledge-based question. For knowledge-basee based question answering tasks. the im- provement brought by coreference resolution methods is not significant compared to handling oongdeependency issues. However, when dealing with long-text knowledge-brad basee question answering. clarifying the references within the text helps enhance the generalization ability of the model. In addressing scientific questions, certain inherent knowledge may have specific representations and specialized terminology, where the model's comprehension ability becomes more important than coreference resolution. Additionally, effective information compression and retrieval strategies contribute to providing accurate answers in commonsense question answering. In commonsense QA, based on four L-Eval datasets, the Llama3-a model demonstrates an improvement of +8.44 % compared with RECOMP method, indicating its ability to enhance the contextual understanding of smaller models.
5.2
IMPACT OF DIFFERENT COREFFRENCE REALIZATIONS
To compare the performance of different coreference resolution designs and the LQCA framework in long document comprehension, as well as to explore the specific roles of each component, We adopt three variations LQCA follows:
LQCA-LLM: We use LLM to perform coreference resolution on each document slice, while other steps remain unchanged. The prompts used for the L.LM are provided Appendix C
LQCA-w/o overlap: This slicing method segments the document into non-overlapping slices, each longer than 512 tokens. The coreference resolution results for each slice are used to replace information in the original text, following the same replacement methods as in Defining Representation and Mention Replacement.
LQCA-w/ RAG: After replacing the text, we introduce a Contriever-msmarco retriever during the question-answerte answering phase. The retriever uses the corpus provided by the document slices, similar to the setup of the bascline method. Compared to the original LQCA method, this approach reduces the model's s dependency on extended context length.
As shown in Figure we evaluate different methods. The coreference resolution method using the LLM shows significant shortcomings in answering questions, likely because the model is currently unable to effectively handle specific downstream tasks. In complex contextual environments with extensive coreference information, expert-level models are often required to handle and annotate mentions accurately. The non-overlapping slicing replacement method performs similarly to LQCA in most tests, but the inability to merge coreference information between different sub-documents slightly limits the question-ant -nnswrring effectiveness for long documents.
Additionally, the LOCA-w/ RAG method helps smaller models extract relevant information and even outperforms the LQCA framework certain metrics. This suggests that retrieval augmentation is more suitable for models with limited capabilities and computational resources in long -ocgmmecnment QA tasks. By gathering relevant information related to the question, it helps the model provide
8

---

Published as a conference paper at ICLR 2025
better answers. The experimental results of these variants further validate the effectiveness of LQCA framework.
5.3 SETTING IN SUB-DOCUMENTS
In LQCA, two critical parameters have significant impact on the coreference resolution performance of the framework. The first is the length of the text segmentation. We ensure that the segmented text length stays within the token limit of the model's 's input. The second is the preset threshold parameter K, which largely determines whether two mentions refer to the same entity. Due to variations between different texts, the number of sub-documents generated by segmentation may differ, which influences the model's inference performance. address this, we conduct grid search on these two variables to analyze their impact on performance, with the experimental results shown in the Figure
Longer sub-documents help capture contextual information, improving coreference resolution performance. For example, in the figure, when the sub- document length approaches the upper limit of 512, the model shows good per- formance with most Fl scores above 70 when the k-value is less than 0.8. Since we resolve coreferences across multiple documents and establish connections be- tween mentions, longer contexts provide more precise information.
34.10
22.31
33.00
31.99
20.30
34.33
20.12
48.00
40
-79
$4.28
55.12
56.20
57.00
56.31
64.02
01:31
63.07
09:50
07.22
69:10
68.26
15
68.6 32
69 35
96
20.33
33.54
Coreference scores threhold is length affected. The impact old between mentions across the entire document on the framework's perfor- mance is strongly correlated with the subdocument length. When the seg- shorter,
score thresh-
66:15
71.6
-55
M6.31
05:12
06.30
07.75
GB 22
03:47
08.02
73
0O 03
69
325.6
272.5
mented sub-documents lower ie value helps maintain consistency between related mentions. However, if the threshold is set too high, some pro- nouns may not find the correct referent. On the other hand, an excessively high threshold may degrade coreference resolution performance, as incomplete information in certain sub-documents may cause the model to misclassify mentions or assign them to separate categories. Therefore, moderate value helps the framework achieve optimal performance.
Figure 4: Performance of the LQCA method on the LongBench- HotpotQA dataset using the GPT-40 model under different val- ues of and We use score as evaluation metric.
5.4 EVALUATION OF KEY INFORMATION POSITIONS
Since current large language models tend to fo- cus more on the beginning and the end of document in long-text scenarios, they often over- look the middle sections where answers may be located. To better assess the effectiveness of our framework, we conducted inference evalu- ations using the GPT-4o model the Coursera dataset, marking the percentage position of an- swers within the text. We divided the positions of the answers into five intervals, using the per- centage value at the end of each interval marker.
Figure 5: The effect of the position of relevant informa- tion on Coursera QA dataset performance.
Middle position performace is highly im- proved by LQCA. As shown in the Figure the model with coreference resolution significantly alleviates the issue of ignoring mid-document

---

Published as a conference paper at ICLR 2025
information. It can be observed that after applying coreference resolution, the accuracy of key information in the [0.2, 0.4], [0.4, 0.6] and [0.6, 0.8] intervals improve significantly. Specifically, the accuracy of key information in the [0.4, 0.6] interval increased by +6.1% compared to directly providing the context. This indicates that providing accurate coreference relations enhances the quality of context, reducing information loss and contextual ambiguity, thereby enabling the language model to perform more effectively and accurately when answering questions.
RELATED WORK
6.1 LONG CONTEXT UNDERSTANDING IN LLM
Recent advancements LLMs highlight the significance of long-contexs understanding across diverse applications (Xiong et al. 2024 Pan al. 2024). Zhu et al. (2024a) explores strategies to extend the context window of embedding models to 32k tokens without additional training, enabling their application in tasks with long inputs such as legal contracts. LongBench, L-eval and LooGLE serve as multitask benchmarks, providing a comprehensive evaluation of long-context understanding (Bai et al., 2024| An et al., 2024 Li et al. 2024a). new context understanding benchmark has been proposed, validating the need for improved generative models in understanding context amidst varying training conditions (Zhu al., 2024b). The study temporal complex events through LLMs sheds light on the ability to analyze event chains effectively, particularly when utilizing suitable retrieval mechanisms (Zhang et al. 2024b). However, challenges persist; current LLMs struggle with contextual relevance based on information positioning, as indicated by performance degradation in identifying important details (Liu et al. 2024). Our LQCA method improves text quality through coreference resolution, thereby enhancing the long context understanding of tasks by LLMs.
6.2 COREFERENCE RESOLUTION
Coreference resolution is an important task in the field of information extraction (Lee al. 2017 Dobrovolskil 2021). Techniques such as finetuning pretrained seq2seq transformers have proven effective, where document inputs are mapped to gereference-tagger sequences, emphasizing the importance of model size and supervision levels (Zhang et al.] (023). Furthermore, novel approach focusing on event coreference emphasizes learning from events rather than entities, integrating multiple representations for improved resolution (Yao et al., 2023a b). The complexity of coreference evaluation is addressed by highlighting the need for standardized measurement methodologies across different datasets (Porada al.) 2023). Meanwhile, prompt-based methods like Core/Prompt allow for the modeling of events and coreference simultaneously through a masked language model setup (Xu et al., 2023). Linguistic insights contribute to performance enhancements by categorizing mention-pairss into distinct decision types (Otmazgin et al. 2023). We combine coreference resolution with long context, improving text quality by replacing referents in long context texts, which aids in enhancing downstream tasks.
CONCLUSIONS
This paper presents the LQCA method, a framework aims at enhancing long-context understanding in LLMs by leveraging coreference resolution. The framework operates through four systematic steps: resolving coreferences within uub-documents, calculating mention distances, defining a repre- sentative mention for coreferences, and performing question answering with mention replacement. By processing long contexts in this manner, the method simplifies the information, making it more comprehensible for the language models. Experiments conducted on five large models and nine long-context question datasets show a notable improvement during inference, with a recorded 3.61 % enhancement on GPT-4o. The findings illustrate the effectiveness of integrating coreference resolution with information extraction to improve comprehension of lengthy texts. Text quality remains critical factor influencing model inference performance. We believe this framework will contribute to the long-term development of the long context understanding of L.LMs.
10

---

Published as a conference paper at ICLR 2025
8 ACKNOWLEDGEMENTS
This work was partly supported by the NSFC under No. 62402418 and No. 62102360. This work was also partly supported by the Key R&D Program of Ningbo under No. 2024Z115.
REFERENCES
Chenxin An, Shansan Gong. Ming Zhong, Xingjian Zhao, Mukai Li, Jun Zhang, Lingpeng Kong. and Xipeng Qiu. L-eval: Instituting standardized evaluation for long context language models. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar (eds.), Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 14388-14411, Bangkok, Thailand, August 2024, Association for Computational Linguistics. doi: 10.18653/v1/ 2024.acl-long, long.776.
776
Yushi Bai, Xin Lv, Jiajie Zhang, Hongchang Lyu, Jiankai Tang. Zhidian Huang. Zhengxiao Du, Xiao Liu, Aohan Zeng, Lei Hou, Yuxiao Dong, Jie Tang, and Juanzi Li. LongBench: A bilin- gual, multitask benchmark for long context understanding. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar (eds.), Proceedings of the 62nd Annual Meeting of the Association for Com- putational Linguistics (Volume 1: Long Papers), pp. 3119-3137, Bangkok, Thailand, August 2024. Association for Computational Linguistics. doi: 10.18653/1/2024.aa 172. URL
Tom B Brown. Language models are few-shot learners. arXiv preprint arXiv:2005. 2020.
Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung. Charles Sutton, Sebastian Gehrmann, et al. Palm: Scaling language modeling with pathways. Journal of Machine Learning Research, 141243):1-113, 2023.
Pradeep Dasigi. Kyle Lo, Iz Beltagy, Arman Cohan, Noah A Smith, and Matt Gardner. A dataset of information-seekle seeking questions and answers anchored in research papers. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguisties: Human Language Technologies, pp. 4599 4610, 2021.
Vladimir Dobrovolskii. Word-level coreference resolution. In Huang, Lucia Specia, and Scott Wen-tau Yih (eds.), Proceedings of the 202/ Conference Empirical Methods in Natural Language Processing, pp. 7670-7675, Online and Punta Cana, Dominican Republic, November 2021. Association for Computational Linguistics. doi: 10.18653. emnlp-main. wrgp:/looooooo/2222// .em.://22m.mmmmmmaa...... 605
Moens, Xuanjing
17200llemmll-mmai..cr URL
Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang. Angela Fan, et al. The Ilama herd of models. arXiv preprint 2xXiv:2407.21788 2024.
Pengcheng He, Xiaodong Liu. Jianfeng Gao, and Weizhu Chen. (DEBERTA): (DECODING)- {enhanced) [bert} [with]: [disentangled) {attention). In International Conference on Learning Representations, 2021. URL httptstt/ppeenreeeiii wttp://wwwwwwe.eeeefeoooreaaaaoo
Wei Hc, Kai Liu, Jing Liu, Yajuan Lyu, Shiqi Zhao, Xinyan Xiao, Yuan Liu, Yizhong Wang, Hua Wu, Qiaoqiao She, Xuan Liu, Tian Wu, and Haifeng Wang. DuReader; Chinese machine reading comprehension dataset from real-world applications. In Eunsol Choi, Minjoon Seo, Danqi Chen, Robin Jia, and Jonathan Berant (eds.). Proceedings of the Workshop Machine Reading for Question Answering, pp. 37-46, Melbourne, Australia, July 2018, Association for Computational Linguisties. doi: 10.18653/v1/W M0W./1--200 18-2605. URL https:://acaathhheeee www.olloyyoooggw/o..ooo wwwpoor/g////////
Xanh Ho, Anh-Khoa Duong Nguyen, Saku Sugawara, and Akiko Aizawa. Constructing a multi- hop QA dataset for comprehensive evaluation of reasoning steps. In Donia Scott, Nuria Bel, and Chengqing Zong (eds.), Proceedings of the 28th International Conference on Computa- Barcelona. Spain (Online), December 2020. International
tional Linguistics, pp. Committee on Computational Linguistics. doi: 10.18653/v 186653/v12022.cooingg 580. URL https://aclaantooogg..rrg/////// wow.:oooa..innnn.........

---

Published as a conference paper at ICLR 2025
Chensen Huang, Guibo Zhu, Xuepeng Wang, Yifei Luo, Guojing Ge, Haoran Chen, Dong Yi, and Jinqiao Wang. Recurrent context compression: Efficiently expanding the context window of Ilm. arXiv preprint arXiv:2406.00 0614400666611100 10, 2024.
Berivan Isik, Natalia Ponomareva, Hussein Hazimeh, Dimitris Paparas, Sergei Vassilvitskii, and Sanmi Koyejo. Scaling laws for downstream task performance of large language models. In ICLR 2024 Workshop on Muthematical and Empirical Understanding of Foundation Models, 2024.
Gautier Izacard, Mathilde Caron, Lucas Hosseini, Sebastian Riedel, Piotr Bojanowski, Armand Joulin, and Edouard Grave. Unsupervised dense information retrieval with contrastive learning. Transactions on Machine Learning Research, 2022. ISSN 1335-8856... URL
net/forumiid=jkte pX1760
Tomáš Kočiský, Jonathan Schwarz, Phil Blunsom, Chris Dyer, Karl Moritz Hermann, Gábor Melis, and Edward Grefenstette. The narrativeqa reading comprehension challenge. Transactions of the Association for Computational Linguistics, 6:317-328, 2018.
Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa. Large language models are zero-shot reasoners. Advances in neural information processing systems, 35: 22199-22213 2022.
Pranjal Kumar. Large language models (Ilms): survey, technical frameworks, and future challenges. Artificial Intelligence Review, 57(9):260, 2024.
Kenton Lee, Luheng He, Mike Lewis, and Luke Zettlemoyer. End-to-end neural coreference resolu- tion. In Martha Palmer, Rebecca Hwa, and Sebastian Riedel (eds.), Proceedings of the 2017 Confer- ence on Empirical Methods in Natural Language Processing, pp. 188-197, Copenhagen, Denmark, September 2017. Association for Computational Linguistics. doi: 10. 10118833/1/DD1-10088 URL otttss//aalaantooogyy..r///1/111
Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, al. Retrieval- augmented genera- tion for nowwledge-intessive nlp tasks. Advances in Neural Information Processing Systems, 33: 9459-9474, 2020.
Jiaqi Li, Mengmeng Wang, Zilong Zheng, and Muhan Zhang. LooGLE: Can long-context language models understand long contexts? In Lun-Wei Ku, Andre Martins, and Vivek Srikumar (eds.), Proceedings of the 62nd Annual Meeting of the Association for Computa- tional Linguistics (Volume 1: Long Papers), pp. 16304-16333, Bangkok, Thailand, August 2024a. Association for Computational Linguistics. doi: 10.18655/1122044.ddlloong 859, URL https://aclanttoolgy..orrg wrwp:/oggy.or//222220 acl-llng.......... 8899
Tianle Li, Ge Zhang, Quy Due Do, Xiang Yue, and Wenhu Chen. Long-context lims struggle with long in-context learning. arXiv preprint arXiv:2404.02061, 2024b.
Nelson F. Liu, Kevin Lin, John Hewitt, Ashwin Paranjape, Michele Bevilacqua, Fabio Petroni, and Percy Liang. Lost in the middle: How language models use long contexts. Transactions of the Association for Computational Linguistics, 12:157-173, 2024. doi: 00.1162/aae.a.000333 URL
https://aclanntho..og hwwpthhooogg..oo/222222 tac1-1.9
Giuliano Martinelli, Edoardo Barba, and Roberto Navigli. Maverick: Efficient and accurate coreference resolution defying recent trends. In Lun-Wet Ku, Andre Martins, and Vivek Srikumar (eds.), Proceedings of the 62nd Annual Meeting of the Association for Computa- tional Linguistics (Volume I: Long Papers), pp. 13380-13333 Bangkok, Thailand, August 2024. Association for Computational Linguisties. doi: 10.18653/ ht https://aclantholl htw.aaaaaanntoooooooo//////0 2024/ acl acl-long. 722
722. URL
Shon Otmazgin, Arie Cattan, and Yoav Goldberg. LingMess: Linguistically informed multi expert scorers for coreference resolution. In Andreas Vlachos and Isabelle Augenstein (eds.), Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics, pp. 2752-2760, Dubrovnik, Croatia, May 2023. Association for Computational Linguistics. doi: 10. 1 1/2/2//////222....//// eacl- main.202. htt.::ts::
www.gy..oggg0000 www.orgg/.2....... facl-mate 202
12

---

Published as a conference paper at ICLR 2025
Long Ouyang, Jeffrey Wu, Xu Jiang. Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang. Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. Advances in neural information processing systems, 35:27730- 27744, 2022.
Yichen Pan, Dehan Kong, Sida Zhou, Cheng Cui, Yifei Leng, Bing Jiang, Hangyu Liu, Yanyi Shang, Shuyan Zhou, Tongshuang Wu, al. Webcanvas: Benchmarking web agents in online environments. arXiv preprint arXiv:2406. 12373, 2024.
Hung Phan, Anurag Acharya, Sarthak Chaturvedi, Shivam Sharma, Mike Parker, Dan Nally, Ali Jannesari, Karl Pazdernik, Mahantesh Halappanavar, Sai Munikoti, et al. Rag vs. long context: Examining frontier large language models for environmental review document comprehension. arXiv preprint wxXhv2247.07322/1 2024.
Ian Porada, Alexandra Olteanu, Kaheer Suleman, Adam Trischler, and J. Cheung. Challenges to evaluating the generalization of coreference resolution models: A measurement modeling perspective. pp. 15380 15395, 2023.
Ori Ram. Yoav Levine, Itay Dalmedigos, Dor Muhlgay, Amnon Shashua, Kevin Leyton- and Yoav Shoham. In-context retrieval-augmeente language models. Transactions of the Association for Computational Linguistics, 1 11:1316-1331, 2023.
Freda Shi, Xinyun Chen, Kanishka Misra, Nathan Scales, David Dohan, Ed H. Chi, Nathanael Schärli, and Denny Zhou. Large language models can be easily distracted by irrelevant context. In Andreas Krause, Emma Brunskill, Kyunghyun Cho, Barbara Engelhardt, Sivan Sabato, and Jonathan Scarlett (eds.), Proceedings of the 40th International Conference Machine Learning, volume 202 of Proceedings of Machine Learning Research, pp. 31210-31227. PMLR, 23-29 Jul 2023. hnttps://proceedinngg .ml.mmm...... wwt.rrereee/e22/sh22.......
Harsh Trivedi, Niranjan Balasubramaniann, Tushar Khot, and Ashish Sabharwal. MuSiQue: Multihop questions via single-hop question composition. Transactions of the Association for Computational Linguistics, 10:539-5544 2022. doi: 10.1 10.1162/tacl.a a 00475.. URL httlhhtts///aaaathhot .rg// 2022. 1.31
Yidong Wang, Zhuohao Yu, Wenjin Yao, Zhengran Zeng, Linyi Yang, Cunxiang Wang, Hao Chen, Chaoya Jiang, Rui Xie, Jindong Wang, al. Pandalm: An automatic evaluation benchmark for Il instruction tuning optimization. In The Twelfth International Conference on Learning Representations, 2023.
Wenhan Xiong, Jingyu Liu, Igor Molybog, Hejia Zhang, Prajjwal Bhargava, Rui Hou, Louis Mar- tin, Rashi Rungta. Karthik Abinav Sankararaman, Barlas Oguz, Madian Khabsa, Han Fang, Yashar Mehdad, Sharan Narang, Kshitiz Malik, Angela Fan, Shruti Bhosale, Sergey Edunov, Mike Lewis, Sinong Wang, and Hao Ma. Effective long-context scaling of foundation models. In Kevin Duh, Helena Gomez, and Steven Bethard (eds.), Proceedings of the 2024 Confer- ence of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), pp. 4643-4663, Mexico City, Mexico, June 2024. Association for Computational Linguistics. doi: 10. 1118653//1/2022.aaacllleee
URL
https:://alanttologg. org 2024. www../22222...cc11ooroo... 266
Fangyuan Xu, Weijia Shi, and Eunsol Choi. RECOMP: Improving retrieval-augmente LMs with context compression and selective augmentation. In The Twelfth International Conference Learning Representations, 2024. URL
Sheng Xu, Peifeng Li, and Qiaoming Zhu. Corefprompt: Prompt-based event coreference resolution by measuring event type and argument compatibilities. ArXiv, abs/2310. 14512, 2023.
An Yang, Baosong Yang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Zhou, Chengpeng Li, Chengyuan Li, Dayiheng Liu, Fei Huang, et al. Qwen2 technical report. arXiv preprint MxXi:22407.166711. 2024.
Zhilin Yang, Peng Qi, Saizheng Zhang, Yoshua Bengio, William Cohen, Ruslan Salakhutdinov, and Christopher D. Manning. HotpotQA: dataset for diverse. explainable multi-hop question answer- ing. In Ellen Riloff, David Chiang, Julia Hockenmaier, and Jun'ichi ichi Tsujii (eds.), Proceedings
13

---

Published as a conference paper at ICLR 2025
of the 2018 Conference Empirical Methods in Natural Language Processing, pp. 2369-2388 Brussels, Belgium. October- Nooober-ovvebbereeeo 2018. Association for Computational Linguistics. doi: 10.18653/v1/D13 1259. URL https:://alaantholloe http://ooooooogoooggog// 1259
Yao Yao, Z. Li, and Hai Zhao. Learning event-aware measures for event coreference resolution. pp. 13542-13556, 2023a.
Yao Yao, Zuchao Li, and Hai Zhao. Learning event-aware measures for event coreference resolution. In Anna Rogers, Jordan Boyd-Graber, and Naoaki Okazaki (eds.), Findings of the Association for Computational Linguistics: ACL 2023, pp. 13542-13556, Toronto, Canada, July 2023b. URL https
Association for Computational Linguistics. doi: 10. Faaclanthology.orgg/22222 wtnd://.orr/.2.....in-.n--e--ee findings-acl. 855
Wenzheng Zhang, Sam Wiseman, and K. Stratos. Seq2seq is all you need for coreference resolution. ArXiv, wwwp///1..13774... 2023.
Xinrong Zhang, Yingfa Chen, Shengding Hu, Zihang Xu, Junhao Chen, Moo Hao, Xu Han, Zhen Thai, Shuo Wang, Zhiyuan Liu, and Maosong Sun. oBBenhh Extending long context evaluation beyond 100K tokens. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar (eds.), Proceedings the 62nd Annual Meeting of the Association for Computational Linguisticy (Volume I: Long Papers), pp. 15262-15277, Bangkok, Thailand, August 2024a, Association for Computational Linguistics. doi: 10.18653/w1/0024.coo 1ttp33//112222.ooo
URL htttps://aclanthooooo hwwpt/hhhhogoooooggggeoo wrgg/2224..0 ac1-long 814
Zhihan Zhang, Yixin Cao, Chenchen Ye, Yunshan Ma, Lizi Liao, and Tat-Seng Chua. Analyzing temporal complex events with large language models? benchmark towards temporal, long context understanding. In Lun-Wa Ku, Andre Martins, and Vivek Srikumar (eds.), Proceedings of the 62nd Anmual Meeting of the Association for Computational Linguistics (Volume I: Long Papers), pp. 1588-1608 1606, Bangkok, Thailand, August 2024b. Association for Computational Linguistics. doi: 10. 10.18655/112022.allooo
URL https: http:s////cclanhhloogooo org. 2024 acl- 87
Dawei Zhu, Liang Wang, Nan Yang, Yifan Song, Wenhao Wu, Furu Wei, and Sujian Li. Longembed: Extending embedding models for long context retrieval. arXiv preprint arXiv:2404. /2096, 2024a.
Yilun Zhu, Joel Ruben Antony Moniz, Shruti Bhargava, Jiarui Lu, Dhivya Piraviperumal, Site Li, Yuan Zhang, Hong Yu, and Bo-Hsiang Tseng. Can large language models understand context? In Yvette Graham and Matthew Purver (eds.), Findings of the Association for Computational Linguistics: EACL 2024, pp. 2004-2018, St. Julian's, s. Malta, March 2024b. Association for Computational Linguistics. httttttt///cccanttaa hwwpo/lloyyoggggo/00/ wwg.//24. findings-eacl. 135
A BROADER IMPACT AND LIMITATIONS
Broader Impact. Using corefcrence resolution methods to optimize long contexts inference is another significant improvement in information extraction within dialogue systems powered by large language models. We believe that by introducing more techniques, such as semantic recognition and entity extraction, large models can achieve better performance when handling long texts. Additionally, we can further investigate the integration of current compression and aurreval-auumented methods with coreference resolution techniques, exploring more potential solutions to enhance text quality. Higher-quality question texts help us better address various downstream tasks and provide appropriate solutions.
Limitation. LQCA presents innovative advancements in addressing long-context understanding and question answering. However, certain limitations warrant consideration. Firstly, the coreference resolution might struggle in dealing with ambiguous references or contexts with high complexity, pos- sibly leading to inaccurate mention replacements. Secondly, while the method improves performance on long-context questions, its effectiveness may diminish when applied to shorter contexts, where the overhead of processing could outweigh the benefits. Furthermore, the reliance on pre- pre-defined mention distances may limit adaptability to varied linguistic structures and usages across different domains. Future endeavors may focus on enhancing coreference resolution techniques and exploring adaptive approaches better manage diverse contexts and improve robustness in various settings.
14

---

Published as a conference paper at ICLR 2025
B DATASET DETAILS
B. LOOGLE
LooGLE et al. 2024a) a benchmark specifically designed to evaluate large language models (LLMs) in long-context understanding tasks. This benchmark emphasizes tasks that rely on both short- term and long-term dependencies in text inputs, such as question answering (QA), summarization, and cloze tasks. LooGLE supports automated evaluation metrics, such as BLEU, ROUGE, METEOR, and BERTScore, to assess model performance. Notable baseline models include GPT-4-32K, GPT- 3.5-16K, and ChatGLM2-66B3322 which have been evaluated on short-term dependency tasks like cloze tests and long-term dependency tasks like document summarization.
The benchmark provides detailed configurations on how optimize large language models for long-context tasks, with particular focus on retrieval-based tasks and long-form generation tasks.
B.2 L-EVAL
L-Eval An et al. (2024) is part of growing suite of long-text understanding evaluation tools. It is designed for multilingual evaluation, testing the performance of large language models on diverse tasks across language families. These tasks include document-level question answering, summarization, and cloze tasks, with particular focus the models' ability to handle large amounts of contextual information. The datasets are derived from real-world domains, such as scientific papers, narratives, and technical reports, making L-Eval highly valuable for tasks that require extensive context retention. L-Eval is an ideal testing platform for evaluating models in scenarios like multi-document retrieval, cross-document summarization, and multi-hop question answering, where retaining long and diverse information is crucial.
B.3 LONGBENCH
LongBench (Bai et al. 2024 is bilingual, multi-task benchmark designed to test large language models' ability to handle long contexts, covering both English and Chinese. It evaluates model performance tasks such as narrative understanding, multi-domain question answering, and summa- rization, where the datasets require handling large amounts of complex input, such as legal documents, scientific reports, and news articles.
This benchmark includes datasets like NarrativeQA (Kočiský et al. 2018), Qasper (Dasigi et al. 2021). HotpotQA (Yang et al. 2018), and DuReader(He et al. 2018), offering diverse application scenarios from muuti--ocummntt retrieval to entity tracking in long narratives. The models are evaluated not only on single- document tasks but also multi-document tasks and zero-shot performance tests. LongBench also incorporates Chinese-specific datasets, further extending its applicability in multilingual scenarios.
C PROMPT FOR VARIATION OF LCQA
I.QA--LLM
Please analyze the following context, do coreference resoulution. identify the mentions, and replace them with their corresponding golden mentions which have their actual reference and meaning. Mentions could be Pronouns, Nouns, Noun Phrases or Modifiers. Ensure that the revised text maintains the original meaning and reads naturally. Please only output result only contains revised context, don't I output any other information. Here is the text to be processed: Context: [Sentence S] Result:
15

---

Published as a conference paper at ICLR 2025
Table Experiments LQCA, RAG, and Variation in perturbations of coreference in LQCA.
Model
Methods
Lontit--omo--ppeee peprr-aberaat L-
L-teal-Courern
Longliench-Hippagg
Longlendb-WWuuMiig
Op
with Corpee
ruplassmeni
71.65
LQCA
95.71
51.25
GPT
KAG with seff Corpes
27,65
15.13
80.05
66.33
47.01
L0CAweth SR munion replacmmni
28.54
82.51
81.35
68.77
47.25
LOCA
34.53
91.03
85.12
71.65
53.62
with Celi-Capia
LQCA mentinss replacement
69.55
46.23
D PERTURBATIONS ON COREPERENCE
To assess the effectiveness of our algorithm on the accuracy of coreference, our method demonstrates an effective filtering mechanism specifically designed to handle highly ambiguous references. Specif- ically, in our algorithm, when the similarity between two mentions exceeds set threshold, we ensure that at least out of the 10 subdocuments containing both mentions classify them into the same cluster before considering them as co-referential. This establishes a higher threshold for merging similar mention words.
Therefore, if two mention words are highly ambiguous and have a lower similarity score (e.g. both have score of around 0.45), they will be treated as separate co-reference clusters due the high threshold (0.8-0.9). This ensures that our method can robustly handle ambiguous context issues. Furthermore, for ambiguous mentions, we treat them as independent co-reference cluster. This means that such mentions will not be replaced in subsequent processing, and their interpretation will rely on the background knowledge encoded in the original model.
We provide mechanism to simulate errors. Among all mentions, we randomly replace 5% of them, with replacement range of +10, evaluate the impact of ambiguous references the entire reasoning process. This allows us to assess how this situation affects the robustness and reliability of our method. Table shols shows experimental results about perturbation. It can be observed that after applying the disturbance, performance decreases, and the results differ from the previous baseline. This suggests the importance coreference resolution the reasoning process, as model performance can be impacted by incorrect coreference resolution.
E FUTURE SCOPE
Our LQCA implementation handles long coreference relations based on language model inference in short texts. In long texts, a potential optimization direction is using causal inference methods. The causal relationship measure formula C(ei, traditional entity relationship measures with causal inference. The traditional entity relationship measure (e,, e,) is based factors such syntactic distance, entity type matching, and co-occurrence, while the causal relationship measure (eisa(ei, eÿ) considers whether entities e, and ej are in causal chain or connected through causal events. The weight coefficients and control the influence of these two parts. The causal relationship measure Reausal(,, eÿ) reflects the causal dependencies between entities and can be calculated as follows:
Reonter (ei, ez) + B Reausal(i, Eş) combines
First, based on a chain of events with causal connections, if the events E; and Eş associated with entities e, and e, have a causal relationship, and if E, is the causal consequence of E,, then Reassa (e,, eÿ) will be large; for example, in the case of "The storm caused the flood," there is a strong causal relationship between the storm and the flood. Next, using causal inference models, such as Bayesian networks or causal graphs, the causal relationships between entities can be inferred from the model structure; for example, deduced from the network structure and used to calculate Reausal (e,, ej). Causal inference based on causal lexicons: causal connecting words in the text help to mark the causal relationships between events. When one entity triggers another, the causal relationship measure will be high.
graph neural network, the relationships between entities can be
To further enhance the reasoning of relationships between entities, causal relationship constraints are added to the entity relationship network. ensuring that connections based not only similarity but also on causal inference. This helps improve the accuracy of reasoning. Some entities may be at the cause effect positions in a causal chain. Changes in entities also be triggered by events,
16

---

Published as a conference paper at ICLR 2025
such a disease triggering change in health status. This causal relationship can be encoded through a causal model and assist in reasoning between entities. We expect that better coreference could assist the model to understand the context better and provide robost result on various cases.
F ETHICS STATEMENTS
Our work is based open-source datasets and code for experimentation. All data and information comply with relevant code standards and data regulations, ensuring that there no risk of privacy breaches information leaks. The use of large language models in paper is primarily applied to handling and solving long context understanding problem in most scenarios, as well as text processing during Chinese-to- Shinese-to-Englishh translation in token level. This fully complies with the conference's requirements and privacy security guidelines.
When interacting with large language models, we may utilize relevant information from instruction. It is important to note that ballucinations from large language models may lead to incorrect answers. Our approach can be further integrated into other frameworks.
G LONGEST DOT PRODUCT PATH ALGORITHM
We use this algorithm to compute the distance of mentions that calculates the distance across the entire graph. The corresponding algorithm updates paths in manner similar to Dijkstra's 's algorithm. By extracting the node with the smallest distance from the priority queue, the algorithm traverses its neighboring nodes, with the selection of these neighbors constrained by range parameter to ensure that only nodes within given distance are considered. For each neighbor, the algorithm determines whether update the optimal path from the current node to that neighbor by calculating the possible path lengths. If the newly calculated path length greater than the currently recorded optimal path, update is made accordingly.
Algorithm 1 All-Pairs Longest Path for Restricted Graph
Input: Graph: = (V, E); Weight function: w E - IR+; Range parameter:
Output: Longest dot product path distances: de, for all i, E V
1: for each node V do
// Initialize distances for node
2: for each node m E V do
3:
if then
4:
dn,m +
// Distance to itself is
5:
else
6:
d,.m + 0
& // Initial distance set to
7:
end
8:
end for
9:
Q <
|/ Priority queue to store unvisited nodes
10: while
Q is not empty do u + Extract-Max(() for E {m V Im
> // Dijkstra's algorithm b // Node with the smallest distance // Neighbors within range // Calculate potential distance
11:
12:
ul s L, (u, E} do all 5- dn. x w(u,
13:
if all > dn,v then
15:
d, + all
b // Update distance longer
end if 17: end for
end while
19:
end for
20:
return d, for all i E V
> Return all-pairs longest path distances
17


Reviews:
[{"review_id": "mFPduqSkp6", "reviewer": "Reviewer_ZAZy", "summary": "This work proposes to disambiguate mentions in a document by replacing pronouns with corresponding disambiguated entities identified by coreference resolution so that a large language model (LLM) can easily leverage a long context document for better contextual understanding. One of the crucial problem for an LLM is to understand a long context to respond to a query, but ambiguities in mentions in a document can easily impact the understanding to derive an answer. This work employs a BERT-based model to identify coreference resolution as a preprocessing step to modify a document-wise context to disambiguate mentions. Experiments are carried out on diverse tasks demanding long context understanding, e.g., LongBench, and show large gains when compared with other methods. e.g., RAGs.", "questions": "I'd like to know how the errors in coreference resolution might have an impact to the end performance. Probably it is possible to inject noises in coreference clusters to see how that will impact to the end performance.", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- A simple method to resolve ambiguity of mentions in a document so that an LLM can easily leverage the contextual information to generate a response. The use of a BERT-based model is a limitation in that it cannot handle longer contexts, but this work proposes a method to combine coreference resolution results for sub-documents using a simple statistics, i.e., whether a mention is within a cluster or not, to compute a distance metric.\n\n- Experiments show gains when compared with other methods considering long contexts, e.g., RAGs. Analysis is also interesting enough to quantify the gains of the proposed method.", "weaknesses": "- It is not clear how the errors in coreference resolution might have an impact to the end performance since coreference resolution itself is a rather challenging task.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "PJ5Tb8ugoB", "reviewer": "Reviewer_syvQ", "summary": "This paper investigates LLM performance in question answering and summarization tasks with very long\ninputs (128K). The hypothesis put forward is that preprocessing the input with coreference \nresolution allows the model to understand the long context better, in particular to identify \nand manage references effectively. The proposed method  encompasses\nfour key steps: resolving coreferences within sub-documents, computing the distances between mentions,\n defining a representative mention for coreference, and\nanswering questions through mention replacement. Experimental evaluations on a range of LLMs and\ndatasets yield overall improvement.", "questions": "- Please explain what I am supposed to be seeing in Figure 2. Your caption is not very informative either. \n\n- It would been useful to see  an actual example for the method in Section 3.2. \n\n- Please present the prompts you use for the various tasks (e.g., QA, summarization), in the appendix. \n\n- What is the accuracy of your coreference algorithm with the mention merging computation? How accurate is your approach? \n I understand that the input is very long, but perhaps you could perform some manual evaluation on shorter documents. It would be interesting to see what the degree of noise is, and how much better your results would be if coreference was perfect. \n\n- Please describe Maverick in more detail, ILCR readers may not be familiar with what it does. \n\n- I would have liked to see if results improve further with fine-tuning (e.g., using Lora). \n\n- What is the computational complexity of the coreference preprocessing step?\n- RAG and RECOMP are not incompatible with your method, could you not run coreference on the retrieved segments? \n\n- It would be good to analyze why you observe performance improvements. Is it because you identify named entities more accurately? \n\n- Existing work (e.g., https://aclanthology.org/2024.lrec-main.145.pdf) shows that LLMs are not very good at coreference resolution, however, it would have been more reasonable to give the LLM some examples (my understanding is you try to perform coreference in a zero-shot setting). \n\n-  Is your Avg column in table 1 averaging across Rouge, Accuracy, and F1? \n\n-", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- The idea of performing coreference resolution before attempting to do the task at hand, makes sense and is corroborated\nby empirical results showing improvements across models and datasets. \n\n-  The paper is well-written and the proposed method easy to follow. \n\n- The proposed coreference approach seems intuitive and could be also used during LLM pertaining or fine-tuning.", "weaknesses": "- The technical contribution is somewhat limited. Coreference merging is interesting, however, without empirical evaluation it is not possible to tell whether it actually works (only down-stream evaluation results are presented, see my questions below). \n\n- What is the main contribution of this paper, is it the algorithm in Section 3.2? \n\n-", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "34FEGDFhOq", "reviewer": "Reviewer_CakD", "summary": "This paper introduces the LQCA method, which aims to enhance the performance of LLMs in understanding lengthy contexts and executing effective question answering. The LQCA method focuses on coreference resolution within long contexts, systematically processing information to replace ambiguous or complex references with clearer mentions, thereby improving the model's comprehension. The paper claims that this approach leads to significant improvements in question answering performance on various LLMs and datasets, particularly on models like OpenAI-o1mini and GPT-4o.", "questions": "How does the LQCA method perform compared to other state-of-the-art approaches in terms of accuracy and efficiency, especially when dealing with extremely long or complex texts?\n\nHow does the LQCA method handle highly ambiguous contexts where multiple interpretations of coreferences are possible?\n\nWhat are the potential impacts of incorrect coreference resolution on the quality of question answering, and how does the LQCA method address these issues?\n\nCan the authors provide more insights into the computational overhead introduced by the LQCA method and its scalability for even longer contexts?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "The paper addresses an important issue in natural language processing, which is the difficulty LLMs face in dealing with long contexts.\n\nThe proposed LQCA method is a novel approach that combines coreference resolution with question answering in long contexts.", "weaknesses": "While the paper demonstrates improvements in performance, it does not provide a comprehensive comparison with alternative methods that also aim to improve long-context understanding.\n\nThe paper could benefit from a more detailed discussion on how the LQCA method handles cases with highly ambiguous references, which could be a common occurrence in real-world applications.\n\nThere is a lack of analysis on the computational overhead introduced by the LQCA method, which is crucial for understanding its scalability and practical applicability.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)