Graph of Records: Boosting Retrieval Augmented Generation for Long-context Summarization with Graphs

Authors: Haozhen Zhang, Tao Feng, Jiaxuan You
Primary area: learning on graphs and other geometries & topologies
Keywords: Retrieval-Augmented Generation, Long-context Summarization, Graph Neural Networks, Large Language Models

Abstract:
Retrieval-augmented generation (RAG) has revitalized Large Language Models (LLMs) by injecting non-parametric factual knowledge. 
Compared with long-context LLMs, RAG is considered an effective summarization tool in a more concise and lightweight manner, which can interact with LLMs multiple times using diverse queries to get comprehensive responses. However, the LLM-generated historical responses, which contain potentially insightful information, are largely neglected and discarded by existing approaches, leading to suboptimal results. In this paper, we propose \textit{graph of records} (\textbf{GoR}), which leverages historical responses generated by LLMs to enhance RAG for long-context global summarization. Inspired by the \textit{retrieve-then-generate} paradigm of RAG, we construct a graph by creating an edge between the retrieved text chunks and the corresponding LLM-generated response. To further uncover the sophisticated correlations between them, GoR further features a \textit{graph neural network} and an elaborately designed \textit{BERTScore}-based objective for self-supervised model training, enabling seamless supervision signal backpropagation between reference summaries and node embeddings. We comprehensively compare GoR with 12 baselines on four long-context summarization datasets, and the results indicate that our proposed method reaches the best performance. Extensive experiments further demonstrate the effectiveness of GoR.

Decision: reject
Reviewer scores: 6, 6, 5, 6

Reviews and discussion:
<thread>
creation date:
1730119630012

thread_type:
official_review

<thread_object>
creator: reviewer_mvtz

<content>
summary:
This paper presents GoR, a retrieval-augmented generation method aimed at enhancing the summarization of long documents. The authors propose a novel approach to distilling the knowledge of large language models (LLMs) to improve retrieval by incorporating both historical LLM responses and text chunks into a structured graph. Their second key contribution is a comprehensive framework for training a graph neural network (GNN)-based retriever on this graph, learning to rank chunk and response nodes using contrastive and ranking objectives. Experimental comparisons with traditional retrievers and long-context LLMs demonstrate the advantages of GoR in long-document summarization.

strengths:
1. Using LLM's parametric knowledge—referred to as historical responses in this paper—to enhance retrieval-augmented generation (RAG) is a valuable addition to the static corpus used in traditional RAG. The authors claim to be the first to apply this approach to long-document summarization.
2. The experimental results are promising, though some analyses may not fully address the research question.
3. The paper is well-written and easy to follow.

weaknesses:
1. The analyses lack sufficient convincing evidence. See the “Questions” section for more details.
2. The graph construction process is complex, which could potentially increase inference time—a critical factor in RAG systems.

questions:
1. The authors discuss the impact of different GNN architectures; however, a more fundamental research question remains: does the constructed structure itself actually improve performance? A follow-up question is: does the LLM's historical responses are really helpful, as claimed by the authors? I understand these are challenging to address, but the paper could benefit from empirical studies, such as ablation experiments on the constructed graph (e.g., ablating the response nodes).

2. A small issue for the presentation. In the analysis on L424, a more informative metric on the x-axis might be the number of tokens used for training. Increasing the number of simulated queries also scales the number of responses generated by the LLM, meaning the model could benefit from both the queries and the additional response tokens.

3. I would appreciate a comparison of inference time per query for the baselines in the main table. This information is valuable for readers interested in inference efficiency for deployment.

4. Regarding "Supervised Training on Global Summarization Queries," are you referring to a baseline where LLaMA-2-7B-chat is fine-tuned using supervised training on global summarization queries and reference summaries?

5. Would it make sense to add query nodes between the response and context nodes in the graph? My intuition is that query nodes could help the model better understand logical constraints between context and response (e.g., negation).
</content>
</thread_object>

</thread>
<thread>
creation date:
1730262012180

thread_type:
official_review

<thread_object>
creator: reviewer_roml

<content>
summary:
This paper proposes a novel method called GoR (Graph of Records) to enhance RAG in long-context global summarization tasks. GoR utilizes historical responses generated by LLMs to construct a graph structure that connects retrieved text chunks with their corresponding responses. This paper further employs a graph neural network and BERTScore-based self-supervised training to reveal correlations between them. Experimental results show that GoR outperforms 12 baselines across four long-context summarization datasets, demonstrating its effectiveness in improving summarization quality.

strengths:
Innovative Use of Historical Responses: The paper introduces a novel approach by leveraging LLM-generated historical responses for enhancing RAG, which is largely neglected by existing methods. This approach enriches the summarization process, potentially increasing the relevance and depth of generated summaries.

weaknesses:
__I.__ Computational Efficiency Evaluation: The paper lacks experimental validation of computational efficiency. The reliance on LLM-generated responses and retrieved chunk graphs, combined with the incorporation of graph neural networks and BERTScore-based objectives, could introduce substantial computational overhead.

__II.__ Dependency on Data Quality: The effectiveness of GoR may rely heavily on the quality and coherence of the historical responses generated by LLMs. Inconsistencies in these responses could impact the model's overall summarization performance.

__III.__ The quality of the presentation is below ICLR 2025 standards. For example:

  a. The format of the references should be consistent to ensure neatness and professionalism. For instance, the names of conferences should be uniformly presented either in abbreviations or full names, rather than a mixture of both.

  b. For the line 265, it should be written as “we only use “general queries” for evaluating”, and the symbol use is wrong.

questions:
Please see weakness
</content>
</thread_object>

</thread>
<thread>
creation date:
1730670075045

thread_type:
official_review

<thread_object>
creator: reviewer_xmhj

<content>
summary:
This paper presents a novel retrieval-augmented generation (RAG) approach for long-context document summarization. The method leverages a graph neural network (GNN) to retrieve relevant information from structured textual nodes. These nodes comprise two types of content: (1) original text chunks from the source document and (2) intermediate summaries previously generated by large language models (LLMs). To optimize the GNN retriever, the authors propose aligning the GNN's representations with semantic similarities from textual encoder, creating a more semantically-aware retrieval mechanism.

Key contributions include:

- A hierarchical text representation using GNNs that combines both source text and intermediate summaries
- A novel training objective that aligns GNN embeddings with semantic similarity metrics
- An end-to-end framework that integrates retrieval and generation for long-document summarization

strengths:
1. Introduces an innovative approach to integrate LLMs' intermediate summaries with original document chunks in a structured graph representation
2. Implements an efficient sparse connectivity strategy using top-K similar textual chunks, reducing computational complexity while maintaining information flow
3. This hierarchical structure effectively bridges local and global document understanding
4. Develops a well-motivated semantic alignment mechanism for the GNN by leveraging BERTScore-based similarities
5. Employs a dual-objective training strategy combining contrastive learning and pairwise ranking loss
6. This approach ensures the GNN's representations capture meaningful semantic relationships beyond surface-level textual similarity
7. Provides a thorough analysis of existing long-context summarization approaches, clearly identifying their limitations
8. Effectively positions the work within two key research streams: retrieval-augmented generation and graph-based document processing
9. Demonstrates clear technological advancement over previous methods through thoughtful experimental design

weaknesses:
1. Clarity and Presentation Issues: 
- **Problem Definition** : Lacks a dedicated subsection that formally defines long-context summarization; Missing explicit comparison between GoR and existing RAG approaches for long-context summarization in the introduction; Would benefit from a clear positioning diagram or framework overview

- **Technical Clarity** : Graph construction description (lines 147-194) lacks sufficient detail and clear visualization; Equation 3's retrieval mechanism is ambiguous about whether it includes LLM-generated responses or only original document chunks; The relationship between the proposed contrastive learning approach and BERTScore needs clearer explanation; The semantic alignment objective would benefit from step-by-step derivation


2. Limited Evaluation Metric: Over-reliance on ROUGE metrics for evaluation; Absence of crucial evaluation metrics: Human evaluation for factual consistency and coherence; BERTScore for semantic similarity assessment; Coverage metrics for long-document comprehension.

3. Insufficient Analysis of Critical Components: Lacks detailed analysis of edge creation strategy between document chunks and LLM responses; Insufficient investigation of the impact of LLM-generated responses; No comparative analysis showing the benefit of including historical LLM responses.

questions:
1. What are Ec and Eq context encoders in line 224-230? Are they also optimized by eq. 7? 
2. The paper mentioned BERTscore many times in this paper but how does BERTscore contribute to the GoR? 
3. In both contrastive learning and ranking loss, what is the motivation of  choosing the positive samples based on the semantic similarity from the encoder? 
4. Provide a case study of GoR and other baseline methods' summarization.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730687889276

thread_type:
official_review

<thread_object>
creator: reviewer_essc

<content>
summary:
The paper introduces the Graph of Records (GoR), a novel method designed to enhance retrieval-augmented generation (RAG) systems for long-context summarization using graph structures. RAG systems improve factuality and relevance in summarization by retrieving relevant content chunks and generating summaries based on those chunks. GoR innovates on traditional RAG by constructing a graph of linked nodes where historical user queries, LLM-generated responses, and retrieved text chunks are interconnected. This graph structure is then refined using a graph neural network (GNN), enabling the model to capture complex relationships between text chunks and responses. By employing a BERTScore-based self-supervised learning objective, GoR aligns node embeddings with semantic relevance to improve summarization accuracy. The model outperforms several baselines across four long-context summarization datasets (e.g., QMSum, AcademicEval), showcasing its effectiveness in creating more comprehensive summaries.

strengths:
1. The paper is well-structured, with clear explanations and logically organized sections, making it easy for readers to follow the methodology and findings.

2. The authors test GoR on four long-context summarization datasets, and the comprehensive evaluation against a variety of baselines (including both sparse and dense retrievers) demonstrates the robustness and generalizability of the method.

3. The proposed approach builds upon existing RAG techniques, making it relatively easy to implement and reproduce, which is valuable for the research community and practical applications.

weaknesses:
I didn’t find significant weaknesses in this paper. The entire architecture is built on existing modules, making the proposed framework both sound and replicable. However, this reliance on established methods might also be a limitation, as the framework feels less innovative or exciting despite being well-presented with informative experimental results.

questions:
I have no further questions about this paper. I am inclined to give it a weak positive score and will review the feedback from other reviewers before finalizing my assessment.
</content>
</thread_object>

</thread>
<thread>
creation date:
1731870083080

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Summary of the major revision

comment:
We thank the reviewers for the thorough and detailed reviews on our submission. We summarize major changes that we have made below. All changes are marked in blue in the updated submission.

- We fixed typos mentioned by reviewer RomL and mVtZ

- We provided a case study of GoR and other baseline methods' summarization in Appendix B (Page 17), mentioned by reviewer XmHJ

- We added additional experimental results on inference efficiency in Appendix A.5 and Table 5 (Page 16 and 17), mentioned by reviewer RomL and mVtZ

- We provided a further explanation of the training objective of GoR in Appendix A.3 (Page 16), mentioned by reviewer XmHJ

- We added a detailed baseline description in Appendix A.2 (Page 15)

- We added a "Broader Impacts" section in Appendix D (Page 19)


We respectfully request the reviewers consider these improvements based on the constructive feedback provided. We believe these enhancements strengthen the paper and provide valuable insights to the community, and we hope for consideration of increasing the scores of our submission. Thank you for your thoughtful consideration.
</content>
</thread_object>

</thread>


Metareview:
metareview:
**Summary:** 
The paper proposes Graph of Records (GoR), a method to enhance RAG for long-context summarization by leveraging LLM-generated historical responses, which are often neglected. GoR constructs a graph that links retrieved text chunks with their corresponding LLM responses, enabling the discovery of complex correlations through a GNN. A self-supervised training framework using a BERTScore-based objective ranks node embeddings by their relevance to global summaries, supported by contrastive and pairwise ranking losses. Evaluated on several long-context summarization datasets, the proposed method outperforms baselines, demonstrating its effectiveness in modeling and utilizing historical response data to improve summarization quality.

**Strength:** 
- The framework is sound, replicable, and built on established modules, ensuring reliability. The experimental results are informative and well-presented.
- Hierarchical structure seems to effectively bridge local and global document understanding

**Weakness:*
- The reliance on established methods might also be a limitation.
- Missing a formal definition of long-context summarization and a clear comparison with existing RAG approaches.
- Heavy reliance on ROUGE metrics, with no human evaluation or additional metrics like BERTScore or coverage for long-document comprehension.