Revisiting Long-context Modeling from Context Denoising Perspective
Authors: name: Zecheng Tang, name: Baibei Ji, name: Juntao Li, name: Lijun Wu, name: Haijia Gui, name: Min Zhang
Organization: Soochow University

Long-context models (LCMs) have demonstrated great potential in processing
long sequences, facilitating many real-world applications. The success of LCMs
can be attributed to their ability to locate implicit critical information
within the context for further prediction. However, recent research reveals
that LCMs are often susceptible to contextual noise, i.e., irrelevant tokens,
that can mislead model attention. In this paper, we conduct a fine-grained
analysis of the context noise and propose an effective metric, the Integrated
Gradient (IG) score, to detect and quantify the noise information within the
context. Our findings reveal that even simple mitigation of detected context
noise can substantially boost the model's attention on critical tokens and
benefit subsequent predictions. Building on this insight, we propose Context
Denoising Training (CDT), a straightforward yet effective training strategy
that improves attention on critical tokens while reinforcing their influence on
model predictions. Extensive experiments across four tasks, under both context
window scaling and long-context alignment settings, demonstrate the superiority
of CDT. Notably, when trained with CDT, an open-source 8B model can achieve
performance (50.92) comparable to GPT-4o (51.00).