Bridging Context Gaps: Leveraging Coreference Resolution for Long Contextual Understanding

Authors: Yanming Liu, Xinyue Peng, Jiannan Cao, Shi Bo, Yanxin Shen, Tianyu Du, Sheng Cheng, Xun Wang, Jianwei Yin, Xuhong Zhang
Primary area: unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords: Coreference Resolution, Long Contextual Understanding, Information Extraction

Abstract:
Large language models (LLMs) have shown remarkable capabilities in natural language processing; however, they still face difficulties when tasked with understanding lengthy contexts and executing effective question answering. These challenges often arise due to the complexity and ambiguity present in longer texts. To enhance the performance of LLMs in such scenarios, we introduce the Long Question Coreference Adaptation (LQCA) method. This innovative framework focuses on coreference resolution tailored to long contexts, allowing the model to identify and manage references effectively. The LQCA method encompasses four key steps: resolving coreferences within sub-documents, computing the distances between mentions, defining a representative mention for coreference, and answering questions through mention replacement. By processing information systematically, the framework provides easier-to-handle partitions for LLMs, promoting better understanding. Experimental evaluations on a range of LLMs and datasets have yielded positive results, with a notable improvements on OpenAI-o1-mini and GPT-4o models, highlighting the effectiveness of leveraging coreference resolution to bridge context gaps in question answering. Our code is public at https://github.com/OceannTwT/LQCA.

Decision: accept (poster)
Reviewer scores: 8, 6, 6

Reviews and discussion:
<thread>
creation date:
1730350196840

thread_type:
official_review

<thread_object>
creator: reviewer_cakd

<content>
summary:
This paper introduces the LQCA method, which aims to enhance the performance of LLMs in understanding lengthy contexts and executing effective question answering. The LQCA method focuses on coreference resolution within long contexts, systematically processing information to replace ambiguous or complex references with clearer mentions, thereby improving the model's comprehension. The paper claims that this approach leads to significant improvements in question answering performance on various LLMs and datasets, particularly on models like OpenAI-o1mini and GPT-4o.

strengths:
The paper addresses an important issue in natural language processing, which is the difficulty LLMs face in dealing with long contexts.

The proposed LQCA method is a novel approach that combines coreference resolution with question answering in long contexts.

weaknesses:
While the paper demonstrates improvements in performance, it does not provide a comprehensive comparison with alternative methods that also aim to improve long-context understanding.

The paper could benefit from a more detailed discussion on how the LQCA method handles cases with highly ambiguous references, which could be a common occurrence in real-world applications.

There is a lack of analysis on the computational overhead introduced by the LQCA method, which is crucial for understanding its scalability and practical applicability.

questions:
How does the LQCA method perform compared to other state-of-the-art approaches in terms of accuracy and efficiency, especially when dealing with extremely long or complex texts?

How does the LQCA method handle highly ambiguous contexts where multiple interpretations of coreferences are possible?

What are the potential impacts of incorrect coreference resolution on the quality of question answering, and how does the LQCA method address these issues?

Can the authors provide more insights into the computational overhead introduced by the LQCA method and its scalability for even longer contexts?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730652851814

thread_type:
official_review

<thread_object>
creator: reviewer_syvq

<content>
summary:
This paper investigates LLM performance in question answering and summarization tasks with very long
inputs (128K). The hypothesis put forward is that preprocessing the input with coreference 
resolution allows the model to understand the long context better, in particular to identify 
and manage references effectively. The proposed method  encompasses
four key steps: resolving coreferences within sub-documents, computing the distances between mentions,
 defining a representative mention for coreference, and
answering questions through mention replacement. Experimental evaluations on a range of LLMs and
datasets yield overall improvement.

strengths:
- The idea of performing coreference resolution before attempting to do the task at hand, makes sense and is corroborated
by empirical results showing improvements across models and datasets. 

-  The paper is well-written and the proposed method easy to follow. 

- The proposed coreference approach seems intuitive and could be also used during LLM pertaining or fine-tuning.

weaknesses:
- The technical contribution is somewhat limited. Coreference merging is interesting, however, without empirical evaluation it is not possible to tell whether it actually works (only down-stream evaluation results are presented, see my questions below). 

- What is the main contribution of this paper, is it the algorithm in Section 3.2? 

-

questions:
- Please explain what I am supposed to be seeing in Figure 2. Your caption is not very informative either. 

- It would been useful to see  an actual example for the method in Section 3.2. 

- Please present the prompts you use for the various tasks (e.g., QA, summarization), in the appendix. 

- What is the accuracy of your coreference algorithm with the mention merging computation? How accurate is your approach? 
 I understand that the input is very long, but perhaps you could perform some manual evaluation on shorter documents. It would be interesting to see what the degree of noise is, and how much better your results would be if coreference was perfect. 

- Please describe Maverick in more detail, ILCR readers may not be familiar with what it does. 

- I would have liked to see if results improve further with fine-tuning (e.g., using Lora). 

- What is the computational complexity of the coreference preprocessing step?
- RAG and RECOMP are not incompatible with your method, could you not run coreference on the retrieved segments? 

- It would be good to analyze why you observe performance improvements. Is it because you identify named entities more accurately? 

- Existing work (e.g., https://aclanthology.org/2024.lrec-main.145.pdf) shows that LLMs are not very good at coreference resolution, however, it would have been more reasonable to give the LLM some examples (my understanding is you try to perform coreference in a zero-shot setting). 

-  Is your Avg column in table 1 averaging across Rouge, Accuracy, and F1? 

-
</content>
</thread_object>

</thread>
<thread>
creation date:
1730695382735

thread_type:
official_review

<thread_object>
creator: reviewer_zazy

<content>
summary:
This work proposes to disambiguate mentions in a document by replacing pronouns with corresponding disambiguated entities identified by coreference resolution so that a large language model (LLM) can easily leverage a long context document for better contextual understanding. One of the crucial problem for an LLM is to understand a long context to respond to a query, but ambiguities in mentions in a document can easily impact the understanding to derive an answer. This work employs a BERT-based model to identify coreference resolution as a preprocessing step to modify a document-wise context to disambiguate mentions. Experiments are carried out on diverse tasks demanding long context understanding, e.g., LongBench, and show large gains when compared with other methods. e.g., RAGs.

strengths:
- A simple method to resolve ambiguity of mentions in a document so that an LLM can easily leverage the contextual information to generate a response. The use of a BERT-based model is a limitation in that it cannot handle longer contexts, but this work proposes a method to combine coreference resolution results for sub-documents using a simple statistics, i.e., whether a mention is within a cluster or not, to compute a distance metric.

- Experiments show gains when compared with other methods considering long contexts, e.g., RAGs. Analysis is also interesting enough to quantify the gains of the proposed method.

weaknesses:
- It is not clear how the errors in coreference resolution might have an impact to the end performance since coreference resolution itself is a rather challenging task.

questions:
I'd like to know how the errors in coreference resolution might have an impact to the end performance. Probably it is possible to inject noises in coreference clusters to see how that will impact to the end performance.
</content>
</thread_object>

</thread>
<thread>
creation date:
1733193464748

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
General Response for Readers

comment:
We sincerely appreciate the thorough review and valuable suggestions provided by everyone on our paper. We are also deeply grateful to the reviewers for their insightful comments and recognition of our work. We believe these suggestions not only contribute to enhancing the quality of our paper but also have a profound impact on advancing the research on the proposed method. Here, we wish to reiterate the strengths and uniqueness of our study to assist readers in gaining a more comprehensive understanding of our approach while addressing common concerns more explicitly.

> 1: A New Perspective and Approach to Long Contextual Understanding

Our method offers a novel solution to long-text understanding. Previous approaches often relied on improving the base models or leveraging retrieval-augmented and tool-integrated methods to aid context understanding and access relevant information. While effective, these methods frequently overlook the quality of the text itself. In traditional scenarios, such as dialogue systems, many information extraction techniques effectively capture the core content of an article.

Our research demonstrates that coreference resolution not only enhances the performance of large language models (LLMs) in processing long texts and handling coreference but also improves the overall quality of the text. This highlights the auxiliary potential of information extraction techniques in future LLMs. 

> 2: An Innovative Coreference Resolution Approach for Long Texts

Traditional coreference resolution methods are predominantly designed for short texts, mainly due to the input length limitations of models like BERT. Consequently, many coreference resolution models can only function within the confines of short text ranges. Our research overcomes this limitation. Our method integrates coreference information across different segments of a long document and consolidates mentions belonging to the same coreference set. Even for mentions that are far apart in a long text, we propose a merging scheme based on distance information.

This innovation has demonstrated significant improvements across various downstream tasks, including long-text summarization and question answering. Our study marks an important advancement in the field of coreference resolution for long texts.

> 3: Comprehensive Experimental Analysis and Performance on SOTA LLMs

Through detailed ablation studies and fine-grained analysis of problem placements, we have showcased the potential of our method. Experimental results indicate that our coreference resolution approach effectively mitigates the "contextual confusion" issues that LLMs often encounter when processing medium-to-long texts. Compared to direct retrieval-augmented segmentation methods, our approach is not only more efficient and convenient but also significantly enhances task performance.

__Common Problems from Reviewers:__

> 1: how the errors in coreference resolution might have an impact to the end performance.

Our method demonstrates an effective filtering mechanism for handling highly ambiguous references. if two mentions are highly ambiguous and their similarity is relatively low (e.g., both scoring around 0.45), they would remain as separate coreference clusters due to the high threshold of 0.8–0.9. This ensures that our method robustly handles ambiguous contextual issues. Furthermore, in cases where a single mention appears ambiguously, it is treated as its own coreference cluster. This means that such mentions are not replaced in later stages, and their interpretation relies on the background knowledge encoded in the original model.

After applying perturbations, the performance declines, with results varying compared to the previous baseline. This indicates that coreference resolution is relatively important, as the model's performance may degrade due to incorrect coreference resolution

> 2: What is the computational complexity?

The computational overhead of our method mainly involves the complexity of two processes. The first is the complexity of coreference resolution for sub-documents, handled during the Maverick process. The second is the complexity of distance computation.

For the distance computation, the complexity is primarily influenced by the sliding window length. In merging long-distance coreferences, the complexity is $O(nlogn)$, where $n$ represents the number of mentions. During the process of finding strongly connected components, the overall resolution can be achieved using Tarjan's strongly connected component algorithm, which has a complexity of $O(n)$ due to its depth-first search approach. Thus, the overall complexity remains within a manageable range.

We have provided detailed responses to more specific questions and experiments in our replies to the reviewers. We sincerely appreciate everyone's feedback on our paper and for contributing to the improvement of our work.
</content>
</thread_object>

</thread>


Metareview:
metareview:
(a) Scientific Claims and Findings
The paper claims that leveraging coreference resolution improves LLM performance in long-context tasks by reducing ambiguity and enhancing comprehension. The LQCA method systematically resolves and integrates coreferences, significantly boosting performance on downstream tasks like question answering.

(b) Strengths

- Novel approach for applying coreference resolution to long texts.
- Demonstrated improvements across various datasets and tasks.
- Detailed analysis and ablation studies highlight the robustness and scalability of the method.

(c) Weaknesses

- Limited discussion of computational overhead and efficiency in real-world scenarios.
- Lack of direct comparison with some alternative long-context understanding methods.
- Coreference accuracy and its standalone effectiveness are only indirectly evaluated.

(d) Reasons for Decision
The paper has strong empirical results and proposes a novel approach. The method’s clear potential and the significant improvements on downstream tasks warrant its acceptance.